Spence Green

After reading Steve McConnell’s Software Estimation: Demystifying the Black Art, I called a friend to discuss my newfound insight. Like a child who first learns to write his name, I circled around the central object for no less than 15 minutes. Software is hard in a “different” way! We need statistical methods and mountains of historical data to estimate it properly! Heed these commands or perish! Now my friend is an architect, and she was not moved by this euphoria.

“Are you telling me that your software is more complicated than the Burj Dubai? No one has ever built a building that tall. Moreover, the water table is one meter below the sand, so the whole structure is founded upon a massive concrete pad. Your software has greater complexity?”

Her response stymied me. Some pieces of software exceed that building in complexity by orders of magnitude–the Space Shuttle software, avionics controllers on the Airbus A380, Windows Vista–but how many of us work on those systems? Most engineers slave away on J2EE business platforms, or better yet, “In house” software [link] that solve mundane problems as inelegantly as possible. Not to be stopped, I posited a second argument: software is free from physical constraints, thereby enlarging the solution space. A bridge’s incline, for instance, is limited by the coefficient of friction between the road surface and a car tire: vertical bridges may be possible, but they are not useful.

“Innovation in building has never been more rapid or unbounded,” she countered, “Computer modeling makes unprecedented structures possible today. Although inconveniences like gravity do limit design to an extent, they are probably no more limiting than the constraints imposed upon you by APIs and frameworks. Just look at the Walt Disney Opera House in LA.”

The abstract nature of software is therefore not a reasonable excuse for 99.9% of late software projects. Most of us aren’t “in technology”: we use tools and methodologies that academics developed years ago. You’re not on the bleeding edge. Get over yourself.

I finally advanced an enervated argument based on estimation: it’s hard to finish software on time because software design is difficult to estimate. How long will it take you to finish your math homework? How long does it take to solve a Sudoku puzzle? How long does it take to catch three fish? We’re trying to predict an unpredictable task riddled with risk.

“How long does it take to design anything new? Architects deal with unreasonable requirements, unreasonable customers, and unreasonable deadlines. How long will it take me to design a building down to its moldings and handrails? I’ll tell you how long: a lot of sleepless nights. You can only estimate something if you’ve done it before.”

And this is precisely McConnell’s thesis. Unfortunately, expert opinion is not a sufficient resource. Historical data, on the other hand, has been used in study after study to achieve reliably accurate software estimates. Productivity is an organizational thing, so one organization’s data may not apply to another organization’s projects. Do you think that expert who has worked at 10 different companies can give you a useful estimate based on judgment?

So why are software projects late? I see three problems:

Bad requirements–Ask yourself these questions: does your organization employ a professionally-trained requirements engineer? Hold requirements inspections? Version control requirements at the line-item level? Link those line-items to code? Maintain requirements throughout the entire system lifecycle, including maintenance?

Brooks’s Law—Adding people to a late project makes it later. Graph theory holds the proof to this axiomatic observation. Adding more nodes to a connected graph makes the edge count increase exponentially, not linearly. This is why the scheduling equation to convert staff months to schedule months–the most “agreed-upon” equation in software–has a coefficient and an exponent.

The Iowa Theory–Software engineers are unwilling to do the book-keeping work necessary to make large projects succeed because their brains are trained to look for optimal solutions. Associating requirements with code is tedious, but necessary. Drawing algorithm diagrams is tedious, but necessary. Managing software change is tedious, but necessary. But as software engineers, we think these tasks should be easier. Tools exist to make them effortless–the Telelogic Lifecycle suite, for example–but most organizations don’t invest money in these solutions. So software engineers gripe about tools, and go back to hacking. When we will admit that we’re part of the problem?

Consider the USS Iowa. Her keel was laid down in June 1940 and completed in August 1942. She was 890 feet long, could shoot 1225kg shells over 40km, and could cruise at 30 knots. She was built when the country’s survival was at stake. Does your project have that kind of schedule pressure? She was designed by hand, using pencils, paper, and slide rules. Think about that. All that complexity was managed with filing cabinets, folders, and blueprints. Are software engineers willing to exert that kind of effort?