This is why software development at large, legacy corporations can be so stultifying. In his prescient IEEE Computer article, Barry Boehm labels the “That’s How We’ve Always Done It” (THWADI) attitude as a paralyzing disability in the rapidly changing software world:

Software teams that grasp this reality and its implications “look” and “feel” different from the moribund organizations that churn out the same old Micro-crap. The Guardian’s web team is a recent example of the former that comes to mind.

These criticisms are unremarkable. Jeff Atwood, among others, labeled the multicore hype an extension of the clock-speed race in the late 90’s. 900MHz is better than 700Mhz, so four cores must be better than two. Right? So say the marketing panjandrams. Most consumers lack a rudimentary understanding of computer architecture, so marketers need a comprehensible “hook.” Core count, like clock speed, seems analogous to horsepower, torque, and other “power” metrics. Consumers need such a gimick.

The criticism of Agile is even less significant. Agile introduces practices that good programmers intuitively follow. Knuth is a good programmer, and he does not need advice.

What the masses seem to have missed was this insight into Knuth’s working habits:

This passage reveals the character of genius, which is realized through these disciplines:

• Real progress comes not from independent ideas, but from the careful accumulation and synthesis of quality information.
• Understanding comes through action. Knuth does not jsut write about algorithms. He programs them, dissects them, and then disseminates them.
• Distractions must be minimized. Complicating access to the internet during the work day is one of the best ways to start.
• Real genius cannot be separated from art, which has much to do with taste. In the 70’s, Knuth grew tired of visually imprecise electronic typesetting systems. His frustration must not have been unique, but his ability to identify, decompose, and solve it was.

Edward Said once wrote that the American university remains the only refuge for those interested in reflection and the refinement of the intellect. What must be considered is that personal habits–not just talent and environment–have much to do with the expansion of human knowledge.

I am trying to fathom what you have against an additional library into the architecture. The AJAX framework provided by MS$ is an additional library we have to use, there are Oracle libraries we have to use…what is the roadblock you have with an IronRuby, IronPython, or Lua library? Limiting the number of libraries, selective in the process is essential, but if to restrictive, can ignore industry standard flexibility in our system.

My reply follows:
I’m not limiting the inclusion of other libraries. I prefer to think about the problem first, then select the best method of expression. Language is simply that: a method of expression. It is almost always better to program in the language–namely, through the use of native syntax–than to program through it. The latter mode is a common mistake: have you ever seen someone write Java as if it were C?

An ad hoc approach is the alternative. In this case, we choose technologies first and solve problems later. This technique is used often in OSS. Here you must recall the essential difference between commercial software development and experimentation. Read this post.

We don’t exercise all of those steps here because we don’t do proper software development. The lesson, however, is this: the decisions that you make as a coder have lifecycle costs associated with them. Think about it: you include library X. We have to learn that technology. I&T has to test it. CM has to integrate it into the nightly builds. Maintenance has to update it, changing custom code as necessary. A whole succession of maintenance programmers for the next 5-10 years must follow this process.

You must also consider the risk of the technology disappearing during the software’s lifecycle. This is always a risk in software, but it can be approached intelligently. Microsoft has made a significant capital investment in .NET, and they have included AJAX in .NET 3.5. Moreover, other companies have staked their viability on .NET (incidentally, this is the anti-trust argument against Windows in the enterprise space: companies cannot afford to move away from it). Can you say the same for other technologies? Perl was the “next” silver bullet in 1995, but we are still waiting on Perl 6. In the meantime, it has been superseded by faster bullets: Python, Ruby, and Scheme.

I recommend that you read the book Beyond Software Architecture by Hohmann. Technology decisions cannot be made in isolation because they impact the business. The converse is also true. We must always be thinking not only in terms of “wow, this is cool”, but also in light of the question: “Does this make good, long-term business sense?”

I’m not trying to limit your creative freedom. I am trying to show you that we must make considered decisions at this point in the “cone of uncertainty.”

Solve the Problem
Every good system solves a problem, which is often elusive. Think about any famous product and try to describe it with a single sentence or a single image. What does the iPod do? It allows you carry digital audio with you. Google? It lets you find relevant stuff on the Internet. Linux? The world needs a good, free operating system. This is not a pedantic exercise. It took me three months of reading and observation to discover the purpose of a recent project. Not a single person in our organization could articulate the system’s raison d’etre clearly, and I found that when I could, my design work took on a new level of coherence. What does the system do and why does it do it? Why is it useful? Answering these questions can go a long way toward unifying the design process. If the questions can’t be answered, then it might be prudent to get real and kill the project.

Write It Down
Once the system concept becomes clear, write a detailed spec. Specs serve several purposes:

• A reason to think: Writing is a specific activity whereas conversation and drawing are often unspecific. If you write, “The system should have a login page,” and then reflect on that statement, then you immediately realize the need for more detail. What does the page look like? What screen elements does it have? What text does it display? The answers to these questions are all requirements that go into the spec.
• A common design decision repository: The login page’s design details need a persistent home. Engineers, just like football players, need to operate from a common playbook. This is why email, PowerPoint, and the lauded whiteboard are not suitable design media: they are not permanent.
• A context: Our system has more than a login page. What is the next page in the interface hierarchy? How does authentication work? Where is user information stored? It is impossible to construct software without knowledge of context.

Software specs exist in myriad degrees of formality, breadth, and depth. But the most important thing is that they exist at the beginning:

Writing a spec is a great way to nail down all those irritating design decisions, large and small, that get covered up if you don’t have a spec. Even small decisions can get nailed down with a spec. For example, if you’re building a web site with membership, you might all agree that if the user forgets their password, you’ll mail it to them. Great. But that’s not enough to write the code. To write the code, you need to know the actual words in that email.

I don’t advocate Joel’s informal approach to specs. His method may be sufficient for designing web sites and business systems, but it cannot be used for Space Shuttle avionics or air defense systems. You wouldn’t use the instructions that came with your Coleman tent to build the Empire State building. The IEEE830-1998 standard is a better reference.

Draw It
Any software architect will quickly learn that it is difficult to model a system in its entirety. The software blueprint will probably never exist, a conclusion reached by the Agile crowd a decade ago. Metaformats suffer from a variety of issues, including:

• Inability to represent the whole system: The UML As Blueprint people have in mind the “exploded” view of systems that is so useful to mechanical engineers. Software is not tangible, though, meaning that it can behave strangely. Try to model a recursive function in UML and you’ll notice the problem.
• A certain level of “OO-ness”: The existence of “Class diagrams” in UML reveals its intellectual heritage. But system architects shouldn’t think in terms of classes, which are like cells in a body. We must first decide if the body should have two arms or three, not the number of cells in each arm.
• Time to develop: Drawing diagrams takes time that could be used to actually write the code. This is particularly true of interfaces. A good Interface Control Document (ICD) and an interface definition are infinitely more valuable than a diagram. Once coding begins, diagrams are often discarded. Why were they created at all?

A multi-faceted approach to design seems more prudent. I prefer a combination of tables (for data definitions), sequence diagrams (for modeling interactions between systems), flow charts (for designing processes), schemas (for databases and XML formats) and natural language requirements. If done properly, the latter can be remarkably effective. Like a good specification, an effective natural language requirement should be:

Correct;
Unambiguous;
Complete;
Consistent;
Verifable;
Modifable;
Traceable

Good specs and designs do not guarantee success. As in the entrepreneurial world, good plans do not make rich men. Execution matters.

Build the Organization
This is the most misunderstood diagram in software development:

Systems engineers design the systems that developers implement. Developers should not make judgments about how the user should behave. Likewise, systems engineers should not decide how to implement code. These competing interests need an arbitrator. In this diagram, I have called him an Integrated Product Team (IPT) lead after the Chrysler convention. Microsoft calls him a PM. Whatever his label, he knows enough about systems and software to mediate between the engineering factions. He also understands the business objectives, and can make the difficult distinction between too much schedule pressure, which harms analysis, and too much analysis, which leads to paralysis. He becomes the technical authority, the “System Solon”. Most importantly, he is at the top of the triangle. If software engineering dominates, then cross-cutting attributes such as performance may not be properly evaluated. If systems engineering sets the project tone, then code-level technical insights–”bottom-up” analysis–may be ignored. The Solon is the bulwark against both outcomes.

“Are you telling me that your software is more complicated than the Burj Dubai? No one has ever built a building that tall. Moreover, the water table is one meter below the sand, so the whole structure is founded upon a massive concrete pad. Your software has greater complexity?”

Her response stymied me. Some pieces of software exceed that building in complexity by orders of magnitude–the Space Shuttle software, avionics controllers on the Airbus A380, Windows Vista–but how many of us work on those systems? Most engineers slave away on J2EE business platforms, or better yet, “In house” software [link] that solve mundane problems as inelegantly as possible. Not to be stopped, I posited a second argument: software is free from physical constraints, thereby enlarging the solution space. A bridge’s incline, for instance, is limited by the coefficient of friction between the road surface and a car tire: vertical bridges may be possible, but they are not useful.

“Innovation in building has never been more rapid or unbounded,” she countered, “Computer modeling makes unprecedented structures possible today. Although inconveniences like gravity do limit design to an extent, they are probably no more limiting than the constraints imposed upon you by APIs and frameworks. Just look at the Walt Disney Opera House in LA.”

The abstract nature of software is therefore not a reasonable excuse for 99.9% of late software projects. Most of us aren’t “in technology”: we use tools and methodologies that academics developed years ago. You’re not on the bleeding edge. Get over yourself.

I finally advanced an enervated argument based on estimation: it’s hard to finish software on time because software design is difficult to estimate. How long will it take you to finish your math homework? How long does it take to solve a Sudoku puzzle? How long does it take to catch three fish? We’re trying to predict an unpredictable task riddled with risk.

“How long does it take to design anything new? Architects deal with unreasonable requirements, unreasonable customers, and unreasonable deadlines. How long will it take me to design a building down to its moldings and handrails? I’ll tell you how long: a lot of sleepless nights. You can only estimate something if you’ve done it before.”

And this is precisely McConnell’s thesis. Unfortunately, expert opinion is not a sufficient resource. Historical data, on the other hand, has been used in study after study to achieve reliably accurate software estimates. Productivity is an organizational thing, so one organization’s data may not apply to another organization’s projects. Do you think that expert who has worked at 10 different companies can give you a useful estimate based on judgment?

So why are software projects late? I see three problems:

Bad requirements–Ask yourself these questions: does your organization employ a professionally-trained requirements engineer? Hold requirements inspections? Version control requirements at the line-item level? Link those line-items to code? Maintain requirements throughout the entire system lifecycle, including maintenance?

Brooks’s Law–Adding people to a late project makes it later. Graph theory holds the proof to this axiomatic observation. Adding more nodes to a connected graph makes the edge count increase exponentially, not linearly. This is why the scheduling equation to convert staff months to schedule months–the most “agreed-upon” equation in software–has a coefficient and an exponent.

The Iowa Theory–Software engineers are unwilling to do the book-keeping work necessary to make large projects succeed because their brains are trained to look for optimal solutions. Associating requirements with code is tedious, but necessary. Drawing algorithm diagrams is tedious, but necessary. Managing software change is tedious, but necessary. But as software engineers, we think these tasks should be easier. Tools exist to make them effortless–the Telelogic Lifecycle suite, for example–but most organizations don’t invest money in these solutions. So software engineers gripe about tools, and go back to hacking. When we will admit that we’re part of the problem?

Consider the USS Iowa. Her keel was laid down in June 1940 and completed in August 1942. She was 890 feet long, could shoot 1225kg shells over 40km, and could cruise at 30 knots. She was built when the country’s survival was at stake. Does your project have that kind of schedule pressure? She was designed by hand, using pencils, paper, and slide rules. Think about that. All that complexity was managed with filing cabinets, folders, and blueprints. Are software engineers willing to exert that kind of effort?

Below, I describe the task model used in my team’s software process. An iterative software project has three task types:

• Non-technical: Any task not associated with code. Includes most work completed during the Preparation phase. Design schematics, system-level use cases, and whitepapers are all non-technical tasks.
• Implementation Request (IR): Directly associated with both a Level2 software requirement and the code that implements the requirement.
• Change Request (CR): Directly associated with a change to an existing requirement and the code that implements that change.

An effective tasking model has several key requirements:

• Unambiguously answers “What”: The task data object should include a title, detailed work instructions, and links to relevant documentation.
• Enables Easy Estimate Tracking: An engineer must be able to enter labor estimates AND evaluate his performance.
• Associates Tasks with Exit Criteria: Each task should be associated with a reviewable work product.

Finally, each engineer should have a personalized view of his work assignments that is not cluttered by unrelated tasks. He can immediately determine not only the current day’s objective, but also his future workload.

Organization
All tasks should appear in centralized container that is collectively owned, ie all programmers have write permissions to it. Most web-based project management tools (FogBugz, Basecamp) and enterprise portal platforms (OpenText Livelink, Microsoft Sharepoint) can satisfy this requirement. Tasks are organized in the “To do List” using the following abstractions:

• Tasks: Each of which is associated with a task group.
• Task Groups: A collection of tasks associated with a milestone.
• Milestones: Date upon which a specific business objective must be accomplished. A milestone frequently coincides with a Work Product Inspection (WPI).

If your project management tool is integrated with both your requirements management system and you change management (CM) tool, then you can use the same organizer for IRs and CRs. Otherwise, you must create a separate tasking model in your CM package.

What to Do Next
Programmers should receive an email notification after a task is assigned to them. They must then provide estimates:

• Set due dates for each milestones.
• Set due dates and durations for each of your tasks relative to the associated milestones.
• Add any missing tasks
• Provide work estimates.

IMPORTANT: Programmers should not change task due dates if they encounter a delay! The tasking activity has little benefit unless programmers improve their personal estimation skills. When an engineer changes a task status to ‘Completed’, the task manager will record the differential between the estimated ‘Due Date’ and the ‘Completed Date’. Programmers should challenge themselves to minimize this interval.

A Daily Regime…
Programmers start their days with the following process:

• Review current task assignments.
• Set a task’s status to ‘In Process’ when starting work; change the status to ‘Completed’ after finishing the task.
• Add new tasks as necessary.
• Arrive at the morning ‘Stand-up’ meeting aware of not only your current work assignments, but also the team’s current location in the project trajectory.

IMPORTANT: The task list has no purpose unless it actually reflects what the programmers are doing during the day. THEY are the source of project status, for management without data is nothing more than sorcery. Good estimates build credibility, which the team should gather rapaciously.

Parnas’s 1979 paper, “Designing Software for Ease of Extension and Contraction”  responded to four common software objectives:

• Deliver a subset of a system’s features
• Add a simple feature
• Remove an un-needed feature
• Tailor the system to meet changing user requirements.

The common complaint against software here was that such ostensibly jejune objectives became intractable. Parnas began with the surprising description of programs as “abstract mathematical objects.” But mathematicians state and prove theorems, seeking generality. If a mathematician becomes aware of “a set of closely related theorems, he responds by proving a more general theorem.” A good theorem is invariant across instances. We would prefer a commutative property that applied to both addition and multiplication, for instance, over one that applied to only one or the other.

Programmers, on the other hand, respond to a minimal requirements set. Generality therefore has necessary design and construction costs, for that principal is de facto “not minimal” (A performance cost often exists, too). Further, programmers often associate generality with the absence of local invariants such as magic numbers and tight coupling. Real re-use has system-level implications, though.

Parnas suggests dogmatic use of a “uses” relationship between “components” during system specification. The “uses” relationship is of course the intellectual ancestor of composition, a tenet of OO design. Jacobsen defined composition in 1992 as structuring an object–which is a number of operations and a state–by its parts. A family, for example, could be composed of a man, woman, and child (more liberal compositions are of course possible). The “uses” relationship still exists in UML and frequently appears in object diagrams. Furthermore, “real” object-oriented languages such as Ada and Smalltalk make expression of this concept possible. Why then does re-use remain a novelty?

Two principal problems exist. Within a company, the cost of finding a general component is often high. Basic information management issues such as the absence of a centralized repository and poor internal search capabilities are obvious causes. This argument is unspecific, however. The real problem is an instance of the free-rider problem (this is especially true in contracting organizations). Projects are often unwilling to accept the costs associated with generality even in light of the utility realized by both that project and other company groups. “Re-use” therefore becomes a topic of much comment and little consequence. It costs too much.

Re-use works if it is the chief end. Four scenarios come to mind:

• Extensions–This category includes frameworks (eg, .NET, J2EE) and APIs (eg, Ruby on Rails).
• Components–UNIX-style applications that exist as input/output functions. grep is the obvious example.
• Schemas/Ontologies–Here we find XML schemas and interface standards.
• Design patterns*–Popularized by the GoF, concepts such as the Singleton have become the software equivalents of Charles Sanders Pierce’s hobby horse. One paper has shown, however, that patterns are usually mutilated in the process of instantiation, making them hard to “recover.” In other words, patterns do not possess the same order of invariance as mathematical theorems.

The OSS community has been the principal beneficiary of re-use’s promise. An OSS component must exist as an autonomous element on the Internet to gain support, therefore making generality and interoperability key considerations. The absence of formal funding also eliminates the free-riding phenomenon on the developer side.

Re-use has failed as a “silver bullet” in the software industry, and it will continue to do so.

In my experience, most project teams scramble to meet the CDR deadline, often submitting work that is incomplete or unconsidered. This circumstance harms both the team and the customer. For example, our group tried to meet a program management quota of 30 slides per person. The “fluff” did not escape the customer’s notice, damaging the team’s credibility.

Instead, I suggest that CDR teams emphasize quality over quantity and use the CDR forum to educate the customer and capture requirements. Think of the CDR in as:

• A pre-validation meeting: Demonstrate analytically the design’s compatibility with the current requirements set. Eventually, the customer will validate the code against the requirements. Show him that you have integrated all of his needs into your current design.
• A training exercise: Have you ever listened to a fellow developer describe a complicated algorithm or component? That guy is working on your system, but you don’t easily understand his approach because you aren’t familiar with his area of the code. Now think of the customer. He must digest the entire design with NO experience in any part of the system. Use the CDR to train the educate the customer, not overwhelm him. Other members of the engineering team will benefit as well, for their awareness of the system will increase.
• A requirements capture meeting: Each presenter should identify a list of design issues and requirements questions prior to the presentation. During a CDR, you have the customer’s undivided attention. Instead of lecturing him, ask him questions and make him think.
• A technical design forum: The team’s engineers should attend every CDR brief. Require them to take notes and submit evaluations. A CDR exists within the same genre as the Fagan inspection and the requirements review. Treat it with the same formality.

Below is a list of guidelines for maximizing the CDR’s benefits. Both project managers and engineers can use these suggestions, for a well-integrated team blurs the distinction between these two roles.

Planning and Preparation

• Don’t use a boilerplate checklist: A CDR’s main objectives are gathering and disseminating information. Circumstances dictate the best methods for achieving those ends. This is not brain surgery. Develop a considered approach based on the project, avoiding “Big M” methodologies.
• Emphasize quality over quantity: Don’t use slide quotas. Treat slides like money and spend them wisely.
• Use the 4-6 rule: Each slide should have between 4-6 bullets, and each bullet should have between 4-6 words. If a slide requires elaboration, talk through the slide using a whiteboard or create a handout for the audience.
• Print the Presentation: Every participant should have a bound copy of the presentation. In this way, you may include complex drawings and figures that may be difficult to read on the projector.
• Plan an internal CDR (iCDR) or “dress rehearsal”: The team should go through the entire CDR sequence together at least a week prior to the real conference. The presenters should speak as if giving the real presentation and the audience should not interrupt the talk unnecessarily. The latter observation brings us to…
• Identify a moderator: CDRs inevitably stall under the weight of digressions. Identify a moderator keep the meeting moving.
• Identify a scribe: It is too difficult to compile a piebald stack of scribbled notes after the meeting. Identify a scribe, give him a laptop, and require him to enter observations formally into Telelogic DOORS or an equivalent tool. Important comments and decisions are often lost immediately after a discussion takes place. Instead, capture all of the comments (annotating them by speaker and slide number) and present them as a “Minutes of Meeting” document to the customer.
• Assign a Master Reviewer: Select a technical leader with broad system knowledge to review all of the slides prior to the iCDR. The reviewer should eliminate redundancies, identify major themes, and resolve formatting inconsistences. The body of slides should reflect a coherent picture of the system. A single mind can evaluate the team’s work with respect to this objective.
• Detail = Credibility: Everyone knows that block diagrams, state charts, and data schemas require time and effort to develop. Include them in the presentation. Together, they show that the team is operating at a satisfactory level of precision. 
• Think about Food: Don’t order the usual corporate garbage: donuts and coffee for breakfast, sandwiches and chips for lunch, Coke in the afternoon. Set the tone for the day with food. Start the day with fresh fruit, toast and French jam, fresh juices, and smoked salmon. Stimulate the body with healthy food, then stimulate the mind with a careful presentation. Impress the customer.

Execution

• Choose good communicators: No leaning on furniture. No staring at the projector screen. Use vocal inflection. Discard nervous habits. Dress well. A CDR is not a political rally, but it is also not Thursday night at the pub.
• Use amphitheater-style seating: Ensure that audience members can talk to each other. “Lecture room” seating prohibits discussion.
• Ban Blackberries: Place a box at the door and force people to dump their weapons into it. This is a serious discussion that requires undivided attention. Distractions caused by Joe Flowingut and his HIGH PRIORITY ISSUES!!! cannot be tolerated.
• Use two projectors: Display diagrams on one projector and talking points on the other.

Post-Op

• Debrief: Hold an engineering meeting immediately after the CDR. Sluggishness usually occurs after such an ordeal. The PM should develop a near-term list of action items and assign them after the last slide is presented. Don’t wait for the final analysis before you start moving forward.
• Requirements changes: Identify them, analyze them, and present them to the customer.
• Design changes: Get the engineers working on these immediately. Impact analysis should be the first step.
• Risk assessment: Develop a risk list and use it to re-evaluate the project schedule.
• Action items: Develop a list of action items for the customer. What do you need from him? What issues must he address? Set a date for follow-up and get your questions answered while the issues still occupy his mind.

A CDR comes from the “Waterfallish” world, but it works well at the beginning of an Agile project as well. Although design work may be performed during Agile/XP iterations, architecture must necessarily come first. A raw feature set must also be developed to allocate builds and iterations. These work artifacts are obvious candidates for a CDR.