Shipping

Mountain View, California

September 23

I work as a software developer at Coordination Engineering. I had joined the company when it was still fairly small, just 40 people in a sublet space in Mountain View, but we had grown by leaps and bounds over the past four years, and established ourselves as the number one provider in a fast growing category that had been termed by Gartner as "synchronization solutions."

Basically, our software solution tied different pieces of companies together with an easy to use interface. manufacturing, orders, returns, sales ,and product development were all brought together in an easy to use way. We tied into the big names in the different areas: SAP, PeopleSoft, Oracle, Microsoft, and filtered out the data and put it in a nice to use interface and made money by the boatloads.

Recently we had added an entire heuristic subsystem that I had designed. It was codenamed "Benny" internally (named after a slow, retiring gentleman in manufacturing that was two cards short of a full deck but nonetheless was always able to point out problems that incredibly intelligent Ph.D.'s had completely missed). What it would do it watch all the different facets of the company and highlight potential problems. For example: let's say that you're approaching the end of the quarter. Benny might spot that you aren't going to have enough manuals to cover the orders that you're probably going to book in the next three weeks, and it would tell you in time so that you could actually order them and get them by the time that you need them. Or: Benny might notice that the product development schedule doesn't allow enough time for manufacturing since manufacturing is going to be creating the boxes for another product that is shipping at the same time, so he'll tell you that you need to push off the product ship date for another three weeks (and, incidentally, giving the engineers a huge reprieve).

At any rate, even with the market slowdown the company was still doing reasonably well, and Benny had done well for both my company and for me. It was very easy to demonstrate, and we would let companies try out the module for three months for free, and in the vast majority of cases it came up with useful advice and proved itself worthy to the IT people and the VPs that had to make the final decision on whether or not to buy it.

Lesson to be learned here: it helps if you can target your product at the people that have to sign the checks.

I enjoyed my job, actually. There are certain facets of software development that are fairly unique, and it was hard for me to imagine another job where my skills and talents could be used as fully. It is very hard to describe my job to someone that isn't familiar with the technology, but let me explain:

My job was to build complex things out of very simple things (and this is what would lead to my downfall, but more on that later). A software program is written in a variety of different languages, but the fundamental operations -- the atoms of programs, if you will -- are very simple operations: get a value from memory. Add these two values. Store this value somewhere.

On top of this, amazing things are built. The text of a software program (called "code") looks like gibberish to the vast majority of the population:

public void processPendingDeliverables(PendingQueue q)

{

for (Iterator i = q.iterator(); i.hasMore(); i.next())

((PendingDeliverable) q.elementAt(i)).process();

}

... but this actually becomes quite readable with enough practice and experience. The tricky and insidious thing about software, however, is how big the programs get. The program above is, obviously, five lines. A program that is written in a typical college course usually ranges from 300-500 lines. As programs get larger, they get more complex than you might think from just looking at the size of the program; so, for example, a 2000 line program is more than twice as complex as a 1000 line program.

A typical software project can easily involve 30 people and over a million lines of code.

Making the situation significantly worse is the fact that every line of code can be critical. If you were writing a million lines of text for a novel, an ungrammatical statement or misspelled word is not going to affect the final product all that much. However, one badly stated "phrase" in a program can be the downfall of the entire system. The (true) story often told as a cautionary note is the fact that there was a space mission to Venus that had to be cancelled and destroyed due to one misplaced comma, while one of the Mars series of probes was lost because the units used between two different modules were not converted properly.

So, you're working with a large team, and you have to review and trust the other people, because the code that they write could very well affect the code that you write, and if either piece is substandard or there is some little thing that you forgot about when the pieces were woven together, then the entire program can be flawed, and it can mean that you ship a little later or that you have to go out to a customer and try to fix a problem that they're having or that they return the software or that your company goes under.

Now, there are a fair number of people that can program, and usually you can find enough people that are competent to write sections of that program. These people are important, but you need one more thing, which is unfortunately far rarer. One in a while, you run into an individual that is able to keep in his or her mind the workings of a million line program; that is not only able to edit an individual module, but sees the ebb and flow of that module with thirty others; that knows the right thing to do, as opposed to the quick thing to do; that comprehends large pieces of code just by the fact that it couldn't work any other way; that is able to recognize the trees and the forest and the moss on the trees and the root rot on tree #5136.

One can make a project successful.

Two can make a company successful.

I had come up with the basic idea for Benny about a year ago, and I had enough political chips and support from the other groups of the company that I was able to beg, borrow, and steal a team of 10 people, and recently we had been working on an upgrade to it. No matter how much you plan out a program and design it and try to anticipate all contingencies, in the end it will come down to long nights and hard work and effort as you rush to try and finish the program in time. It's incredibly difficult, but also fun, in a gruesome sort of way -- you've been working with people for the past year, spent more time with them than with your significant others, have fought and disagreed and forgiven and forgotten, and it all comes together in a hurried haze of stress, pressure, and not a little giddiness. It reminds me of all-night study sessions in college, somewhat, and when you're done with it all there's no better feeling in the entire world, and the bonds forged in the crucible of shipping can last for the rest of your life.

Errors within a program are called "bugs," and the dirty fact that no one likes to admit in the software industry is that every real product ships with thousands and thousands of these defects. Every company has some way of tracking bugs for their products, and if you were to reveal the bug list for the products in the industry, it would be a truly scary revelation for the customer. Given the realities of the market and depending on the product for revenue, it's not a question of trying to fix all the bugs; this isn't even a realistic possibility in the scheme of things. Instead, it's more a question of picking the right bugs to fix, and hoping that you haven't missed a critical one due to the fact that you were exhausted and bleary-eyed when you reviewed the bug list.

It was 2:00am in the morning, and the entire team was working late, finishing the product so that we could get it to manufacturing and start getting it shipped out in time to make the quarter. Given the success of the first version of Benny, we already had millions of dollars of revenue booked on the upgrade, and the company was depending on it to be able to make the numbers for the quarter. All in all, though, even given the crunch towards the end, I was fairly happy with the product; it had the right set of features, it was easy to use and very approachable for people, it did a useful task, and I thought that we had picked the right bugs and fixed the right things in order to make it a success. It's hard to tell these things until after the product ships, but I thought it would do well with the customers.

We made the final build of the software, and burnt it onto the CD, and everyone tested the CDs, and finally we were done. Anticlimactically enough, we slipped it under the door of the person that would pick it up and deliver it to manufacturing in the morning, and then we all congregated in the large meeting room. As per the tradition at the company, we opened up the Jack Daniels and wine and beer and toasted the birth of another product while we recited the humorous events that had occurred in the process of delivering the software.

It was finally 3:00am in the morning when we rolled out of there. I walked two of the women that I work with to their cars, and then went to my vehicle, got in, and managed to get home without mishap. I got home and got ready for bed and tried to sleep, but I was still wired from the excitement and caffeine of shipping the product, so I just lay there in the dark with my eyes open and thought about nice it was to have finally shipped, and all the things that I could finally do and take care of.

Without realizing it, my mental images became more and more vivid, and I realized that I was drifting off into sleep, but as I started to slumber I wondered what in my house was making the odd click, click, click noise.