spencegreen.com |
Stuff |
spencegreen.com |
Stuff |
Good writing elucidates truths that seem self-evident. Paul Graham’s most recent essay resonated deeply with me not because of its originality, but because he articulated an idea–however lost in his mellifluous prose–that unconsciously motivated my recent resignation. I live in Baltimore, but Baltimore does not inspire me. It does not ask me to be something different. It does not make me feel inadequate. Rather, it makes me bored and tired, and I am weary from these feelings.
For two years, I lived in the UAE. This is the country that has spawned Dubai, that sprawling pearl on the arid Gulf coast. Living in the UAE was like standing atop a power plant: you cannot ignore the sound of the whirring dynamo under your feet. The leaders of that tiny federation have an unfettered desire to make their country the exclusive haven of the well-heeled. These sheikhs are rich and they are happy to divulge that information to the world. The resulting schemes, however indecorous they appear by ‘Old World’ standards, nonetheless challenge the imagination. One folk tale has it that Sheikh Zayed, envious of the sheet-shaped hotel on his northern border, commissioned some European engineers to study various methods for moving the edifice. Specifically, he wanted them to excavate its foundation and float it down the coast to Abu Dhabi. A year and $10 million were needed to conclude that this plan would not sail.
In Abu Dhabi, I was never at ease. It seemed to say: you can run faster. In those two years, I ran myself to exhaustion. I learned to dive. I climbed mountains. I became fluent in a language. I mastered new skills at work. I traveled. Those years have left an indelible imprint on my life.
In October, I moved back to Baltimore. In the ensuing eight months, I can scarcely identify one meaningful change in my life. I am not a better engineer, my Arabic has fallen into disrepair, and I sleep too much. It took little meditation to speculate that this course could not hold. When the next big decision presented itself this spring, I thus thought hard about where I wanted to live. An unusual job offer could have kept me in Baltimore, but I feared stagnation. Chicago was a possibility, but I felt crushed by its girth (and its winter). California, Joan Didion’s ‘Golden Land’, made its case one hour after I departed from the airport.
Famished after four hours on a poorly-stocked plane, I parked my car in front of the first restaurant that I saw on University Ave in Palo Alto. Several patrons milled near the drink dispenser, waiting on others to vacate the five tables arranged at angles in the main dining area. One party vacated the table nearest the door and, undeterred by the pile of fetid dishes left on the table, I sat down at it. A prematurely white-haired man lingered in the aisle, and I invited him to sit with me. Presently, we ascertained that we were both in software and had entrepreneurial ambitions. The notable exception was that he was running a software company, and I was still dreaming about such a project. You should talk to so-and-so, he said, and he recovered his BlackBerry from his jacket pocket and made a call. He’s a Stanford professor, my new friend said. The professor said: come to Stanford. My new friend, a Wharton graduate, said: this is where it’s happening. Come to Stanford. Everyone I met said the same thing: join us.
How many places in the world exude that level of energy?
The Toyota Production System (TPS) was the progenitor for a variety of change-oriented manufacturing techniques. Six-sigma, Lean, and other such constructs trace their heritage to TPS. Because Agile methodologies were influenced by “lean” thinking and an abhorrence of “Big M” processes, they too have eastern roots. For me, the allure of Agile methods, regardless of flavor, has always been the recognition of software as a human act: Programmers are not automata on an assembly-line tacking trunk lids to mechanical foetuses. Incidentally, the Japanese reached the same conclusion decades ago, as described by Teruyuki Minoura, a Toyota executive:
There can be no successful monozukuri (making thing) without hito-zukuri (making people). To keep coming up with revolutionary new production techniques, we need to develop unique ideas and knowledge by thinking about problems in terms of genchi genbutsu. This means it’s necessary to think about how we can develop people who can come up with these ideas. As our operations become increasingly global, there’s also a need to think how to implant the Toyota DNA in our overseas personnel.”
This is why software development at large, legacy corporations can be so stultifying. In his prescient IEEE Computer article, Barry Boehm labels the “That’s How We’ve Always Done It” (THWADI) attitude as a paralyzing disability in the rapidly changing software world:
Of course, some THWADI is good. We will need to separate obsolete practices from enduring principles that need to be conserved.
Some other implications for software engineers’ careers are that learning how to learn will be more important than learning things…”
Software teams that grasp this reality and its implications “look” and “feel” different from the moribund organizations that churn out the same old Micro-crap. The Guardian’s web team is a recent example of the former that comes to mind.
Much attention has been paid to Knuth’s recent interview on Informit. The Slashdot thread shows all the signs of a flame war, and the blogosphere has evidenced a vigorous response as well. The maelstrom has two focii: Knuth’s rejection of most eXtreme programming (XP) practices and his admission that he wouldn’t “be surprised at all if the whole multithreading idea turns out to be a flop.” To him, the emergence of multicore processors ”looks more or less like the hardware designers have run out of ideas, and that they’re trying to pass the blame for the future demise of Moore’s Law to the software writers by giving us machines that work faster only on a few key benchmarks!” Revisions of TAOCP will not contain parallized versions of his algorithms, nor will he devote significant research time to the subject.
These criticisms are unremarkable. Jeff Atwood, among others, labeled the multicore hype an extension of the clock-speed race in the late 90’s. 900MHz is better than 700Mhz, so four cores must be better than two. Right? So say the marketing panjandrams. Most consumers lack a rudimentary understanding of computer architecture, so marketers need a comprehensible “hook.” Core count, like clock speed, seems analogous to horsepower, torque, and other “power” metrics. Consumers need such a gimick.
The criticism of Agile is even less significant. Agile introduces practices that good programmers intuitively follow. Knuth is a good programmer, and he does not need advice.
What the masses seem to have missed was this insight into Knuth’s working habits:
I program every algorithm that’s discussed (so that I can thoroughly understand it) using CWEB, which works splendidly with the GDB debugger. I make the illustrations with MetaPost (or, in rare cases, on a Mac with Adobe Photoshop or Illustrator). I have some homemade tools, like my own spell-checker for TeX and CWEB within Emacs. I designed my own bitmap font for use with Emacs, because I hate the way the ASCII apostrophe and the left open quote have morphed into independent symbols that no longer match each other visually. I have special Emacs modes to help me classify all the tens of thousands of papers and notes in my files, and special Emacs keyboard shortcuts that make bookwriting a little bit like playing an organ. I prefer rxvt to xterm for terminal input. Since last December, I’ve been using a file backup system called backupfs, which meets my need beautifully to archive the daily state of every file.
I currently use Ubuntu Linux, on a standalone laptop—it has no Internet connection. I occasionally carry flash memory drives between this machine and the Macs that I use for network surfing and graphics; but I trust my family jewels only to Linux.
This passage reveals the character of genius, which is realized through these disciplines:
Edward Said once wrote that the American university remains the only refuge for those interested in reflection and the refinement of the intellect. What must be considered is that personal habits–not just talent and environment–have much to do with the expansion of human knowledge.
A colleague wrote the following note to me today:
I am trying to fathom what you have against an additional library into the architecture. The AJAX framework provided by MS$ is an additional library we have to use, there are Oracle libraries we have to use…what is the roadblock you have with an IronRuby, IronPython, or Lua library? Limiting the number of libraries, selective in the process is essential, but if to restrictive, can ignore industry standard flexibility in our system.
My reply follows:
I’m not limiting the inclusion of other libraries. I prefer to think about the problem first, then select the best method of expression. Language is simply that: a method of expression. It is almost always better to program in the language–namely, through the use of native syntax–than to program through it. The latter mode is a common mistake: have you ever seen someone write Java as if it were C?
An ad hoc approach is the alternative. In this case, we choose technologies first and solve problems later. This technique is used often in OSS. Here you must recall the essential difference between commercial software development and experimentation. Read this post.
We don’t exercise all of those steps here because we don’t do proper software development. The lesson, however, is this: the decisions that you make as a coder have lifecycle costs associated with them. Think about it: you include library X. We have to learn that technology. I&T has to test it. CM has to integrate it into the nightly builds. Maintenance has to update it, changing custom code as necessary. A whole succession of maintenance programmers for the next 5-10 years must follow this process.
You must also consider the risk of the technology disappearing during the software’s lifecycle. This is always a risk in software, but it can be approached intelligently. Microsoft has made a significant capital investment in .NET, and they have included AJAX in .NET 3.5. Moreover, other companies have staked their viability on .NET (incidentally, this is the anti-trust argument against Windows in the enterprise space: companies cannot afford to move away from it). Can you say the same for other technologies? Perl was the “next” silver bullet in 1995, but we are still waiting on Perl 6. In the meantime, it has been superseded by faster bullets: Python, Ruby, and Scheme.
I recommend that you read the book Beyond Software Architecture by Hohmann. Technology decisions cannot be made in isolation because they impact the business. The converse is also true. We must always be thinking not only in terms of “wow, this is cool”, but also in light of the question: “Does this make good, long-term business sense?”
I’m not trying to limit your creative freedom. I am trying to show you that we must make considered decisions at this point in the “cone of uncertainty.”
In his eclectic book Let My People Go Surfing: The Education of a Reluctant Businessman, Yvon Chouinard traces the unusual development of Patagonia from a one-man smithing operation in California to the world’s leading producer of outdoor clothing. Chouinard’s self-deprecating style belies his preternatural understanding of the universal human craving for individual freedom. The same impulse that drove him to scale peaks using homemade tools manifests itself in the desire to skip work on Wednesdays or wear unusual clothing. People oppose systems that treat them as cogs. This is one reason for communism’s failure, and it also explains why the assembly line is at once man’s most efficient and least inspiring contrivances:
It is the closest thing to a perpetual motion machine, for its inertia alone seems sufficient to sustain it. In many ways, the modern engineering organization is no different than this assembly line. Whereas Ford has its conveyors and pneumatic arms, the large engineering company has its “Big M” methodologies. Use Python for a business system? Too risky. Compress the management hierarchy? Too controversial. Go on the offensive during requirements development? Too costly. Breaking free from this order takes a refractory personality. This is precisely Chouinard’s conclusion:
One of my favorite sayings about entrepreneurship is: If you want to understand the entrepreneur, study the juvenile delinquent. The delinquent is saying with his actions, “This sucks. I’m going to do my own thing.” Since I had never wanted to be a businessman, I needed a few good reasons to be one. One thing I did not want to change, even if we got serious: Work had to be enjoyable on a daily basis. We all had to come to work on the balls of our feet and go up the stairs two steps at a time. We needed to be surrounded by friends who could dress whatever way they wanted, even be barefoot. We all needed to have flextime to surf the waves when they were good, or ski the powder after a big snowstorm, or stay home and take care of a sick child. Breaking the rules and making my own system work are the creative part of management that is particularly satisfying to me.
Chouinard now has the luxury of reflecting on his ascent, which was fraught with challenges. At one point, he resorted to eating dog food when his money ran out. Such is the life of one who challenges convention, which by definition is a position arrived at by force. A terminal moraine, a huge stone pushed by a glacier, is a natural corollary:
Business, climbing, and even car detailing each have their customs that were developed through natural selection over extended periods. Objecting to an established position is no more palatable than exterminating a particular species of animal. This is why Machiavelli wrote in “The Prince”:
And it should be considered that nothing is more difficult to handle, more doubtful of success, nor more dangerous to manage, than to put oneself at the head of introducing new orders. For the introducer has all those who benefit from the old orders as enemies, and he has lukewarm defenders in all those who might benefit from the new orders.
We often forget that some of our greatest luminaries were not overnight successes, but lately recognized geniuses.
Software engineers do not often have the luxury of designing new systems from first principles. It is frequently the case that they must labor through some dreary chore, such as implementing version 49 of the SuperWhamo! application, or adhering to design constraints imposed not by reason, but by suits. When that rare opportunity to write new code does present itself, two paths are possible. The coder leaps into development, but the architect tries first to solve the problem. This essay contains my observations on the latter approach.
Solve the Problem
Every good system solves a problem, which is often elusive. Think about any famous product and try to describe it with a single sentence or a single image. What does the iPod do? It allows you carry digital audio with you. Google? It lets you find relevant stuff on the Internet. Linux? The world needs a good, free operating system. This is not a pedantic exercise. It took me three months of reading and observation to discover the purpose of a recent project. Not a single person in our organization could articulate the system’s raison d’etre clearly, and I found that when I could, my design work took on a new level of coherence. What does the system do and why does it do it? Why is it useful? Answering these questions can go a long way toward unifying the design process. If the questions can’t be answered, then it might be prudent to get real and kill the project.
Write It Down
Once the system concept becomes clear, write a detailed spec. Specs serve several purposes:
Software specs exist in myriad degrees of formality, breadth, and depth. But the most important thing is that they exist at the beginning:
Writing a spec is a great way to nail down all those irritating design decisions, large and small, that get covered up if you don’t have a spec. Even small decisions can get nailed down with a spec. For example, if you’re building a web site with membership, you might all agree that if the user forgets their password, you’ll mail it to them. Great. But that’s not enough to write the code. To write the code, you need to know the actual words in that email.
I don’t advocate Joel’s informal approach to specs. His method may be sufficient for designing web sites and business systems, but it cannot be used for Space Shuttle avionics or air defense systems. You wouldn’t use the instructions that came with your Coleman tent to build the Empire State building. The IEEE830-1998 standard is a better reference.
Draw It
Any software architect will quickly learn that it is difficult to model a system in its entirety. The software blueprint will probably never exist, a conclusion reached by the Agile crowd a decade ago. Metaformats suffer from a variety of issues, including:
A multi-faceted approach to design seems more prudent. I prefer a combination of tables (for data definitions), sequence diagrams (for modeling interactions between systems), flow charts (for designing processes), schemas (for databases and XML formats) and natural language requirements. If done properly, the latter can be remarkably effective. Like a good specification, an effective natural language requirement should be:
Correct;
Unambiguous;
Complete;
Consistent;
Verifable;
Modifable;
Traceable
Good specs and designs do not guarantee success. As in the entrepreneurial world, good plans do not make rich men. Execution matters.
Build the Organization
This is the most misunderstood diagram in software development:
Systems engineers design the systems that developers implement. Developers should not make judgments about how the user should behave. Likewise, systems engineers should not decide how to implement code. These competing interests need an arbitrator. In this diagram, I have called him an Integrated Product Team (IPT) lead after the Chrysler convention. Microsoft calls him a PM. Whatever his label, he knows enough about systems and software to mediate between the engineering factions. He also understands the business objectives, and can make the difficult distinction between too much schedule pressure, which harms analysis, and too much analysis, which leads to paralysis. He becomes the technical authority, the “System Solon”. Most importantly, he is at the top of the triangle. If software engineering dominates, then cross-cutting attributes such as performance may not be properly evaluated. If systems engineering sets the project tone, then code-level technical insights–”bottom-up” analysis–may be ignored. The Solon is the bulwark against both outcomes.
After reading Steve McConnell’s Software Estimation: Demystifying the Black Art, I called a friend to discuss my newfound insight. Like a child who first learns to write his name, I circled around the central object for no less than 15 minutes. Software is hard in a “different” way! We need statistical methods and mountains of historical data to estimate it properly! Heed these commands or perish! Now my friend is an architect, and she was not moved by this euphoria.
“Are you telling me that your software is more complicated than the Burj Dubai? No one has ever built a building that tall. Moreover, the water table is one meter below the sand, so the whole structure is founded upon a massive concrete pad. Your software has greater complexity?”
Her response stymied me. Some pieces of software exceed that building in complexity by orders of magnitude–the Space Shuttle software, avionics controllers on the Airbus A380, Windows Vista–but how many of us work on those systems? Most engineers slave away on J2EE business platforms, or better yet, “In house” software [link] that solve mundane problems as inelegantly as possible. Not to be stopped, I posited a second argument: software is free from physical constraints, thereby enlarging the solution space. A bridge’s incline, for instance, is limited by the coefficient of friction between the road surface and a car tire: vertical bridges may be possible, but they are not useful.
“Innovation in building has never been more rapid or unbounded,” she countered, “Computer modeling makes unprecedented structures possible today. Although inconveniences like gravity do limit design to an extent, they are probably no more limiting than the constraints imposed upon you by APIs and frameworks. Just look at the Walt Disney Opera House in LA.”
The abstract nature of software is therefore not a reasonable excuse for 99.9% of late software projects. Most of us aren’t “in technology”: we use tools and methodologies that academics developed years ago. You’re not on the bleeding edge. Get over yourself.
I finally advanced an enervated argument based on estimation: it’s hard to finish software on time because software design is difficult to estimate. How long will it take you to finish your math homework? How long does it take to solve a Sudoku puzzle? How long does it take to catch three fish? We’re trying to predict an unpredictable task riddled with risk.
“How long does it take to design anything new? Architects deal with unreasonable requirements, unreasonable customers, and unreasonable deadlines. How long will it take me to design a building down to its moldings and handrails? I’ll tell you how long: a lot of sleepless nights. You can only estimate something if you’ve done it before.”
And this is precisely McConnell’s thesis. Unfortunately, expert opinion is not a sufficient resource. Historical data, on the other hand, has been used in study after study to achieve reliably accurate software estimates. Productivity is an organizational thing, so one organization’s data may not apply to another organization’s projects. Do you think that expert who has worked at 10 different companies can give you a useful estimate based on judgment?
So why are software projects late? I see three problems:
Bad requirements–Ask yourself these questions: does your organization employ a professionally-trained requirements engineer? Hold requirements inspections? Version control requirements at the line-item level? Link those line-items to code? Maintain requirements throughout the entire system lifecycle, including maintenance?
Brooks’s Law–Adding people to a late project makes it later. Graph theory holds the proof to this axiomatic observation. Adding more nodes to a connected graph makes the edge count increase exponentially, not linearly. This is why the scheduling equation to convert staff months to schedule months–the most “agreed-upon” equation in software–has a coefficient and an exponent.
The Iowa Theory–Software engineers are unwilling to do the book-keeping work necessary to make large projects succeed because their brains are trained to look for optimal solutions. Associating requirements with code is tedious, but necessary. Drawing algorithm diagrams is tedious, but necessary. Managing software change is tedious, but necessary. But as software engineers, we think these tasks should be easier. Tools exist to make them effortless–the Telelogic Lifecycle suite, for example–but most organizations don’t invest money in these solutions. So software engineers gripe about tools, and go back to hacking. When we will admit that we’re part of the problem?
Consider the USS Iowa. Her keel was laid down in June 1940 and completed in August 1942. She was 890 feet long, could shoot 1225kg shells over 40km, and could cruise at 30 knots. She was built when the country’s survival was at stake. Does your project have that kind of schedule pressure? She was designed by hand, using pencils, paper, and slide rules. Think about that. All that complexity was managed with filing cabinets, folders, and blueprints. Are software engineers willing to exert that kind of effort?
The key to efficient programmer tasking involves telling programmers exactly what to do and then allowing them the space to do it. Practically, this means providing them with specific development tasks in a sequential order. If the project’s tasking model can achieve these mischievously difficult conditions, then programmers can enter the ‘Flow’, which is impossible with heavy context-switching.
Below, I describe the task model used in my team’s software process. An iterative software project has three task types:
An effective tasking model has several key requirements:
Finally, each engineer should have a personalized view of his work assignments that is not cluttered by unrelated tasks. He can immediately determine not only the current day’s objective, but also his future workload.
Organization
All tasks should appear in centralized container that is collectively owned, ie all programmers have write permissions to it. Most web-based project management tools (FogBugz, Basecamp) and enterprise portal platforms (OpenText Livelink, Microsoft Sharepoint) can satisfy this requirement. Tasks are organized in the “To do List” using the following abstractions:
If your project management tool is integrated with both your requirements management system and you change management (CM) tool, then you can use the same organizer for IRs and CRs. Otherwise, you must create a separate tasking model in your CM package.
What to Do Next
Programmers should receive an email notification after a task is assigned to them. They must then provide estimates:
IMPORTANT: Programmers should not change task due dates if they encounter a delay! The tasking activity has little benefit unless programmers improve their personal estimation skills. When an engineer changes a task status to ‘Completed’, the task manager will record the differential between the estimated ‘Due Date’ and the ‘Completed Date’. Programmers should challenge themselves to minimize this interval.
A Daily Regime…
Programmers start their days with the following process:
IMPORTANT: The task list has no purpose unless it actually reflects what the programmers are doing during the day. THEY are the source of project status, for management without data is nothing more than sorcery. Good estimates build credibility, which the team should gather rapaciously.
Powered by WordPress
Entries and comments feeds.
Valid XHTML and CSS.