spencegreen.com |
Stuff |
spencegreen.com |
Stuff |
The Toyota Production System (TPS) was the progenitor for a variety of change-oriented manufacturing techniques. Six-sigma, Lean, and other such constructs trace their heritage to TPS. Because Agile methodologies were influenced by “lean” thinking and an abhorrence of “Big M” processes, they too have eastern roots. For me, the allure of Agile methods, regardless of flavor, has always been the recognition of software as a human act: Programmers are not automata on an assembly-line tacking trunk lids to mechanical foetuses. Incidentally, the Japanese reached the same conclusion decades ago, as described by Teruyuki Minoura, a Toyota executive:
There can be no successful monozukuri (making thing) without hito-zukuri (making people). To keep coming up with revolutionary new production techniques, we need to develop unique ideas and knowledge by thinking about problems in terms of genchi genbutsu. This means it’s necessary to think about how we can develop people who can come up with these ideas. As our operations become increasingly global, there’s also a need to think how to implant the Toyota DNA in our overseas personnel.”
This is why software development at large, legacy corporations can be so stultifying. In his prescient IEEE Computer article, Barry Boehm labels the “That’s How We’ve Always Done It” (THWADI) attitude as a paralyzing disability in the rapidly changing software world:
Of course, some THWADI is good. We will need to separate obsolete practices from enduring principles that need to be conserved.
Some other implications for software engineers’ careers are that learning how to learn will be more important than learning things…”
Software teams that grasp this reality and its implications “look” and “feel” different from the moribund organizations that churn out the same old Micro-crap. The Guardian’s web team is a recent example of the former that comes to mind.
Much attention has been paid to Knuth’s recent interview on Informit. The Slashdot thread shows all the signs of a flame war, and the blogosphere has evidenced a vigorous response as well. The maelstrom has two focii: Knuth’s rejection of most eXtreme programming (XP) practices and his admission that he wouldn’t “be surprised at all if the whole multithreading idea turns out to be a flop.” To him, the emergence of multicore processors ”looks more or less like the hardware designers have run out of ideas, and that they’re trying to pass the blame for the future demise of Moore’s Law to the software writers by giving us machines that work faster only on a few key benchmarks!” Revisions of TAOCP will not contain parallized versions of his algorithms, nor will he devote significant research time to the subject.
These criticisms are unremarkable. Jeff Atwood, among others, labeled the multicore hype an extension of the clock-speed race in the late 90’s. 900MHz is better than 700Mhz, so four cores must be better than two. Right? So say the marketing panjandrams. Most consumers lack a rudimentary understanding of computer architecture, so marketers need a comprehensible “hook.” Core count, like clock speed, seems analogous to horsepower, torque, and other “power” metrics. Consumers need such a gimick.
The criticism of Agile is even less significant. Agile introduces practices that good programmers intuitively follow. Knuth is a good programmer, and he does not need advice.
What the masses seem to have missed was this insight into Knuth’s working habits:
I program every algorithm that’s discussed (so that I can thoroughly understand it) using CWEB, which works splendidly with the GDB debugger. I make the illustrations with MetaPost (or, in rare cases, on a Mac with Adobe Photoshop or Illustrator). I have some homemade tools, like my own spell-checker for TeX and CWEB within Emacs. I designed my own bitmap font for use with Emacs, because I hate the way the ASCII apostrophe and the left open quote have morphed into independent symbols that no longer match each other visually. I have special Emacs modes to help me classify all the tens of thousands of papers and notes in my files, and special Emacs keyboard shortcuts that make bookwriting a little bit like playing an organ. I prefer rxvt to xterm for terminal input. Since last December, I’ve been using a file backup system called backupfs, which meets my need beautifully to archive the daily state of every file.
I currently use Ubuntu Linux, on a standalone laptop—it has no Internet connection. I occasionally carry flash memory drives between this machine and the Macs that I use for network surfing and graphics; but I trust my family jewels only to Linux.
This passage reveals the character of genius, which is realized through these disciplines:
Edward Said once wrote that the American university remains the only refuge for those interested in reflection and the refinement of the intellect. What must be considered is that personal habits–not just talent and environment–have much to do with the expansion of human knowledge.
A colleague wrote the following note to me today:
I am trying to fathom what you have against an additional library into the architecture. The AJAX framework provided by MS$ is an additional library we have to use, there are Oracle libraries we have to use…what is the roadblock you have with an IronRuby, IronPython, or Lua library? Limiting the number of libraries, selective in the process is essential, but if to restrictive, can ignore industry standard flexibility in our system.
My reply follows:
I’m not limiting the inclusion of other libraries. I prefer to think about the problem first, then select the best method of expression. Language is simply that: a method of expression. It is almost always better to program in the language–namely, through the use of native syntax–than to program through it. The latter mode is a common mistake: have you ever seen someone write Java as if it were C?
An ad hoc approach is the alternative. In this case, we choose technologies first and solve problems later. This technique is used often in OSS. Here you must recall the essential difference between commercial software development and experimentation. Read this post.
We don’t exercise all of those steps here because we don’t do proper software development. The lesson, however, is this: the decisions that you make as a coder have lifecycle costs associated with them. Think about it: you include library X. We have to learn that technology. I&T has to test it. CM has to integrate it into the nightly builds. Maintenance has to update it, changing custom code as necessary. A whole succession of maintenance programmers for the next 5-10 years must follow this process.
You must also consider the risk of the technology disappearing during the software’s lifecycle. This is always a risk in software, but it can be approached intelligently. Microsoft has made a significant capital investment in .NET, and they have included AJAX in .NET 3.5. Moreover, other companies have staked their viability on .NET (incidentally, this is the anti-trust argument against Windows in the enterprise space: companies cannot afford to move away from it). Can you say the same for other technologies? Perl was the “next” silver bullet in 1995, but we are still waiting on Perl 6. In the meantime, it has been superseded by faster bullets: Python, Ruby, and Scheme.
I recommend that you read the book Beyond Software Architecture by Hohmann. Technology decisions cannot be made in isolation because they impact the business. The converse is also true. We must always be thinking not only in terms of “wow, this is cool”, but also in light of the question: “Does this make good, long-term business sense?”
I’m not trying to limit your creative freedom. I am trying to show you that we must make considered decisions at this point in the “cone of uncertainty.”
Software engineers do not often have the luxury of designing new systems from first principles. It is frequently the case that they must labor through some dreary chore, such as implementing version 49 of the SuperWhamo! application, or adhering to design constraints imposed not by reason, but by suits. When that rare opportunity to write new code does present itself, two paths are possible. The coder leaps into development, but the architect tries first to solve the problem. This essay contains my observations on the latter approach.
Solve the Problem
Every good system solves a problem, which is often elusive. Think about any famous product and try to describe it with a single sentence or a single image. What does the iPod do? It allows you carry digital audio with you. Google? It lets you find relevant stuff on the Internet. Linux? The world needs a good, free operating system. This is not a pedantic exercise. It took me three months of reading and observation to discover the purpose of a recent project. Not a single person in our organization could articulate the system’s raison d’etre clearly, and I found that when I could, my design work took on a new level of coherence. What does the system do and why does it do it? Why is it useful? Answering these questions can go a long way toward unifying the design process. If the questions can’t be answered, then it might be prudent to get real and kill the project.
Write It Down
Once the system concept becomes clear, write a detailed spec. Specs serve several purposes:
Software specs exist in myriad degrees of formality, breadth, and depth. But the most important thing is that they exist at the beginning:
Writing a spec is a great way to nail down all those irritating design decisions, large and small, that get covered up if you don’t have a spec. Even small decisions can get nailed down with a spec. For example, if you’re building a web site with membership, you might all agree that if the user forgets their password, you’ll mail it to them. Great. But that’s not enough to write the code. To write the code, you need to know the actual words in that email.
I don’t advocate Joel’s informal approach to specs. His method may be sufficient for designing web sites and business systems, but it cannot be used for Space Shuttle avionics or air defense systems. You wouldn’t use the instructions that came with your Coleman tent to build the Empire State building. The IEEE830-1998 standard is a better reference.
Draw It
Any software architect will quickly learn that it is difficult to model a system in its entirety. The software blueprint will probably never exist, a conclusion reached by the Agile crowd a decade ago. Metaformats suffer from a variety of issues, including:
A multi-faceted approach to design seems more prudent. I prefer a combination of tables (for data definitions), sequence diagrams (for modeling interactions between systems), flow charts (for designing processes), schemas (for databases and XML formats) and natural language requirements. If done properly, the latter can be remarkably effective. Like a good specification, an effective natural language requirement should be:
Correct;
Unambiguous;
Complete;
Consistent;
Verifable;
Modifable;
Traceable
Good specs and designs do not guarantee success. As in the entrepreneurial world, good plans do not make rich men. Execution matters.
Build the Organization
This is the most misunderstood diagram in software development:
Systems engineers design the systems that developers implement. Developers should not make judgments about how the user should behave. Likewise, systems engineers should not decide how to implement code. These competing interests need an arbitrator. In this diagram, I have called him an Integrated Product Team (IPT) lead after the Chrysler convention. Microsoft calls him a PM. Whatever his label, he knows enough about systems and software to mediate between the engineering factions. He also understands the business objectives, and can make the difficult distinction between too much schedule pressure, which harms analysis, and too much analysis, which leads to paralysis. He becomes the technical authority, the “System Solon”. Most importantly, he is at the top of the triangle. If software engineering dominates, then cross-cutting attributes such as performance may not be properly evaluated. If systems engineering sets the project tone, then code-level technical insights–”bottom-up” analysis–may be ignored. The Solon is the bulwark against both outcomes.
After reading Steve McConnell’s Software Estimation: Demystifying the Black Art, I called a friend to discuss my newfound insight. Like a child who first learns to write his name, I circled around the central object for no less than 15 minutes. Software is hard in a “different” way! We need statistical methods and mountains of historical data to estimate it properly! Heed these commands or perish! Now my friend is an architect, and she was not moved by this euphoria.
“Are you telling me that your software is more complicated than the Burj Dubai? No one has ever built a building that tall. Moreover, the water table is one meter below the sand, so the whole structure is founded upon a massive concrete pad. Your software has greater complexity?”
Her response stymied me. Some pieces of software exceed that building in complexity by orders of magnitude–the Space Shuttle software, avionics controllers on the Airbus A380, Windows Vista–but how many of us work on those systems? Most engineers slave away on J2EE business platforms, or better yet, “In house” software [link] that solve mundane problems as inelegantly as possible. Not to be stopped, I posited a second argument: software is free from physical constraints, thereby enlarging the solution space. A bridge’s incline, for instance, is limited by the coefficient of friction between the road surface and a car tire: vertical bridges may be possible, but they are not useful.
“Innovation in building has never been more rapid or unbounded,” she countered, “Computer modeling makes unprecedented structures possible today. Although inconveniences like gravity do limit design to an extent, they are probably no more limiting than the constraints imposed upon you by APIs and frameworks. Just look at the Walt Disney Opera House in LA.”
The abstract nature of software is therefore not a reasonable excuse for 99.9% of late software projects. Most of us aren’t “in technology”: we use tools and methodologies that academics developed years ago. You’re not on the bleeding edge. Get over yourself.
I finally advanced an enervated argument based on estimation: it’s hard to finish software on time because software design is difficult to estimate. How long will it take you to finish your math homework? How long does it take to solve a Sudoku puzzle? How long does it take to catch three fish? We’re trying to predict an unpredictable task riddled with risk.
“How long does it take to design anything new? Architects deal with unreasonable requirements, unreasonable customers, and unreasonable deadlines. How long will it take me to design a building down to its moldings and handrails? I’ll tell you how long: a lot of sleepless nights. You can only estimate something if you’ve done it before.”
And this is precisely McConnell’s thesis. Unfortunately, expert opinion is not a sufficient resource. Historical data, on the other hand, has been used in study after study to achieve reliably accurate software estimates. Productivity is an organizational thing, so one organization’s data may not apply to another organization’s projects. Do you think that expert who has worked at 10 different companies can give you a useful estimate based on judgment?
So why are software projects late? I see three problems:
Bad requirements–Ask yourself these questions: does your organization employ a professionally-trained requirements engineer? Hold requirements inspections? Version control requirements at the line-item level? Link those line-items to code? Maintain requirements throughout the entire system lifecycle, including maintenance?
Brooks’s Law–Adding people to a late project makes it later. Graph theory holds the proof to this axiomatic observation. Adding more nodes to a connected graph makes the edge count increase exponentially, not linearly. This is why the scheduling equation to convert staff months to schedule months–the most “agreed-upon” equation in software–has a coefficient and an exponent.
The Iowa Theory–Software engineers are unwilling to do the book-keeping work necessary to make large projects succeed because their brains are trained to look for optimal solutions. Associating requirements with code is tedious, but necessary. Drawing algorithm diagrams is tedious, but necessary. Managing software change is tedious, but necessary. But as software engineers, we think these tasks should be easier. Tools exist to make them effortless–the Telelogic Lifecycle suite, for example–but most organizations don’t invest money in these solutions. So software engineers gripe about tools, and go back to hacking. When we will admit that we’re part of the problem?
Consider the USS Iowa. Her keel was laid down in June 1940 and completed in August 1942. She was 890 feet long, could shoot 1225kg shells over 40km, and could cruise at 30 knots. She was built when the country’s survival was at stake. Does your project have that kind of schedule pressure? She was designed by hand, using pencils, paper, and slide rules. Think about that. All that complexity was managed with filing cabinets, folders, and blueprints. Are software engineers willing to exert that kind of effort?
The key to efficient programmer tasking involves telling programmers exactly what to do and then allowing them the space to do it. Practically, this means providing them with specific development tasks in a sequential order. If the project’s tasking model can achieve these mischievously difficult conditions, then programmers can enter the ‘Flow’, which is impossible with heavy context-switching.
Below, I describe the task model used in my team’s software process. An iterative software project has three task types:
An effective tasking model has several key requirements:
Finally, each engineer should have a personalized view of his work assignments that is not cluttered by unrelated tasks. He can immediately determine not only the current day’s objective, but also his future workload.
Organization
All tasks should appear in centralized container that is collectively owned, ie all programmers have write permissions to it. Most web-based project management tools (FogBugz, Basecamp) and enterprise portal platforms (OpenText Livelink, Microsoft Sharepoint) can satisfy this requirement. Tasks are organized in the “To do List” using the following abstractions:
If your project management tool is integrated with both your requirements management system and you change management (CM) tool, then you can use the same organizer for IRs and CRs. Otherwise, you must create a separate tasking model in your CM package.
What to Do Next
Programmers should receive an email notification after a task is assigned to them. They must then provide estimates:
IMPORTANT: Programmers should not change task due dates if they encounter a delay! The tasking activity has little benefit unless programmers improve their personal estimation skills. When an engineer changes a task status to ‘Completed’, the task manager will record the differential between the estimated ‘Due Date’ and the ‘Completed Date’. Programmers should challenge themselves to minimize this interval.
A Daily Regime…
Programmers start their days with the following process:
IMPORTANT: The task list has no purpose unless it actually reflects what the programmers are doing during the day. THEY are the source of project status, for management without data is nothing more than sorcery. Good estimates build credibility, which the team should gather rapaciously.
A recent software presentation that I saw started with an “architectural discussion.” One slide listed “Object/component-based design” as a feature, which was qualified by the usual platitudes including “Maximizes software re-use.” The slide then made the bold leap to “faster development” as the principal benefit of this “design decision.” I sighed. Two decades after “re-use” entered the technical nomenclature, the term is still misapplied. Re-use remains a misunderstood ideal with more promise than payoff.
Parnas’s 1979 paper, “Designing Software for Ease of Extension and Contraction” responded to four common software objectives:
The common complaint against software here was that such ostensibly jejune objectives became intractable. Parnas began with the surprising description of programs as “abstract mathematical objects.” But mathematicians state and prove theorems, seeking generality. If a mathematician becomes aware of “a set of closely related theorems, he responds by proving a more general theorem.” A good theorem is invariant across instances. We would prefer a commutative property that applied to both addition and multiplication, for instance, over one that applied to only one or the other.
Programmers, on the other hand, respond to a minimal requirements set. Generality therefore has necessary design and construction costs, for that principal is de facto “not minimal” (A performance cost often exists, too). Further, programmers often associate generality with the absence of local invariants such as magic numbers and tight coupling. Real re-use has system-level implications, though.
Parnas suggests dogmatic use of a “uses” relationship between “components” during system specification. The “uses” relationship is of course the intellectual ancestor of composition, a tenet of OO design. Jacobsen defined composition in 1992 as structuring an object–which is a number of operations and a state–by its parts. A family, for example, could be composed of a man, woman, and child (more liberal compositions are of course possible). The “uses” relationship still exists in UML and frequently appears in object diagrams. Furthermore, “real” object-oriented languages such as Ada and Smalltalk make expression of this concept possible. Why then does re-use remain a novelty?
Two principal problems exist. Within a company, the cost of finding a general component is often high. Basic information management issues such as the absence of a centralized repository and poor internal search capabilities are obvious causes. This argument is unspecific, however. The real problem is an instance of the free-rider problem (this is especially true in contracting organizations). Projects are often unwilling to accept the costs associated with generality even in light of the utility realized by both that project and other company groups. “Re-use” therefore becomes a topic of much comment and little consequence. It costs too much.
Re-use works if it is the chief end. Four scenarios come to mind:
The OSS community has been the principal beneficiary of re-use’s promise. An OSS component must exist as an autonomous element on the Internet to gain support, therefore making generality and interoperability key considerations. The absence of formal funding also eliminates the free-riding phenomenon on the developer side.
Re-use has failed as a “silver bullet” in the software industry, and it will continue to do so.
Over the last several days, my project has gone through the Critical Design Review (CDR) process. Although the format differed from the CDR that I had organized in February, the presentation problems, organizational needs, and technical outcomes were similar. CDRs have become a staple in most engineering organizations, and government contractors in particular must hold them.
In my experience, most project teams scramble to meet the CDR deadline, often submitting work that is incomplete or unconsidered. This circumstance harms both the team and the customer. For example, our group tried to meet a program management quota of 30 slides per person. The “fluff” did not escape the customer’s notice, damaging the team’s credibility.
Instead, I suggest that CDR teams emphasize quality over quantity and use the CDR forum to educate the customer and capture requirements. Think of the CDR in as:
Below is a list of guidelines for maximizing the CDR’s benefits. Both project managers and engineers can use these suggestions, for a well-integrated team blurs the distinction between these two roles.
Planning and Preparation
Execution
Post-Op
A CDR comes from the “Waterfallish” world, but it works well at the beginning of an Agile project as well. Although design work may be performed during Agile/XP iterations, architecture must necessarily come first. A raw feature set must also be developed to allocate builds and iterations. These work artifacts are obvious candidates for a CDR.
Powered by WordPress
Entries and comments feeds.
Valid XHTML and CSS.