A more complete definition would make a distinction between crowdsourcing and other forms of collective enterprise such as Wikipedia, Yelp, and the Linux kernel. Whereas compensation for these efforts emanates principally from enhanced reputation and self-satisfaction, crowdsourcing attempts to exploit the power of weak links. Success stories from InnoCentive—one of which is chonicled in Howe’s article—illustrate this objective. As one sardonic wag wryly noted, experts are people who know more and more about less and less until they know everything about nothing. By contrast, an amateur may bring a diverse set of loosely connected skills to bear on a problem. If innovation results from questioning assumptions, then amateurs have an ex officio advantage in that they often do not know which assumptions to make from the outset.

The question of payment remains unanswered (this is the sign of an immature market). Two approaches currently exist, but I suspect that neither of them will grow into a sustainable revenue source. Podcasts have long used crowdfunding to support themselves. The author solicits contributions from the more dedicated consumers who can then choose to donate money, usually through Paypal. Public media has supported itself in this manner for decades, but one could hardly point to that model as a lucrative enterprise. Other crowdsourcing services organize themselves as brokerages—Amazon Mechanical Turk, InnoCentive, and Crowdspring are representative cases. Consumers use these tools to locate suppliers; the clearinghouses charge a percentage of the transaction. eBay has built an empire using this model, but whereas eBay deals mainly in stuff, crowdsourcing companies sell intellectual property. The latter is much harder to evaluate, protect, and exchange. For instance, what is the legal process associated with InnoCentive solutions? The contractual process must resemble the IP agreements signed by new employees at technology companies, otherwise the “solvers” could market their wares without reservation. What then differentiates a “solver” from a “contractor”?

Several other issues remain:

• How do crowdsourcing companies achieve “critical mass”, e.g. what is the best strategy for seeding the site?
• How do consumers maintain a working relationship with suppliers during the course of a project? Amateurs may lose interest in the work or be overcome with the demands of their primary vocations. How can consumers mitigate these risks?

For two years, I lived in the UAE. This is the country that has spawned Dubai, that sprawling pearl on the arid Gulf coast. Living in the UAE was like standing atop a power plant: you cannot ignore the sound of the whirring dynamo under your feet. The leaders of that tiny federation have an unfettered desire to make their country the exclusive haven of the well-heeled. These sheikhs are rich and they are happy to divulge that information to the world. The resulting schemes, however indecorous they appear by ‘Old World’ standards, nonetheless challenge the imagination. One folk tale has it that Sheikh Zayed, envious of the sheet-shaped hotel on his northern border, commissioned some European engineers to study various methods for moving the edifice. Specifically, he wanted them to excavate its foundation and float it down the coast to Abu Dhabi. A year and $10 million were needed to conclude that this plan would not sail.

In Abu Dhabi, I was never at ease. It seemed to say: you can run faster. In those two years, I ran myself to exhaustion. I learned to dive. I climbed mountains. I became fluent in a language. I mastered new skills at work. I traveled. Those years have left an indelible imprint on my life.

In October, I moved back to Baltimore. In the ensuing eight months, I can scarcely identify one meaningful change in my life. I am not a better engineer, my Arabic has fallen into disrepair, and I sleep too much. It took little meditation to speculate that this course could not hold. When the next big decision presented itself this spring, I thus thought hard about where I wanted to live. An unusual job offer could have kept me in Baltimore, but I feared stagnation. Chicago was a possibility, but I felt crushed by its girth (and its winter). California, Joan Didion’s ‘Golden Land’, made its case one hour after I departed from the airport.

Famished after four hours on a poorly-stocked plane, I parked my car in front of the first restaurant that I saw on University Ave in Palo Alto. Several patrons milled near the drink dispenser, waiting on others to vacate the five tables arranged at angles in the main dining area. One party vacated the table nearest the door and, undeterred by the pile of fetid dishes left on the table, I sat down at it. A prematurely white-haired man lingered in the aisle, and I invited him to sit with me. Presently, we ascertained that we were both in software and had entrepreneurial ambitions. The notable exception was that he was running a software company, and I was still dreaming about such a project. You should talk to so-and-so, he said, and he recovered his BlackBerry from his jacket pocket and made a call. He’s a Stanford professor, my new friend said. The professor said: come to Stanford. My new friend, a Wharton graduate, said: this is where it’s happening. Come to Stanford. Everyone I met said the same thing: join us.

How many places in the world exude that level of energy?

This is why software development at large, legacy corporations can be so stultifying. In his prescient IEEE Computer article, Barry Boehm labels the “That’s How We’ve Always Done It” (THWADI) attitude as a paralyzing disability in the rapidly changing software world:

Software teams that grasp this reality and its implications “look” and “feel” different from the moribund organizations that churn out the same old Micro-crap. The Guardian’s web team is a recent example of the former that comes to mind.

These criticisms are unremarkable. Jeff Atwood, among others, labeled the multicore hype an extension of the clock-speed race in the late 90’s. 900MHz is better than 700Mhz, so four cores must be better than two. Right? So say the marketing panjandrams. Most consumers lack a rudimentary understanding of computer architecture, so marketers need a comprehensible “hook.” Core count, like clock speed, seems analogous to horsepower, torque, and other “power” metrics. Consumers need such a gimick.

The criticism of Agile is even less significant. Agile introduces practices that good programmers intuitively follow. Knuth is a good programmer, and he does not need advice.

What the masses seem to have missed was this insight into Knuth’s working habits:

This passage reveals the character of genius, which is realized through these disciplines:

• Real progress comes not from independent ideas, but from the careful accumulation and synthesis of quality information.
• Understanding comes through action. Knuth does not jsut write about algorithms. He programs them, dissects them, and then disseminates them.
• Distractions must be minimized. Complicating access to the internet during the work day is one of the best ways to start.
• Real genius cannot be separated from art, which has much to do with taste. In the 70’s, Knuth grew tired of visually imprecise electronic typesetting systems. His frustration must not have been unique, but his ability to identify, decompose, and solve it was.

Edward Said once wrote that the American university remains the only refuge for those interested in reflection and the refinement of the intellect. What must be considered is that personal habits–not just talent and environment–have much to do with the expansion of human knowledge.

I am trying to fathom what you have against an additional library into the architecture. The AJAX framework provided by MS$ is an additional library we have to use, there are Oracle libraries we have to use…what is the roadblock you have with an IronRuby, IronPython, or Lua library? Limiting the number of libraries, selective in the process is essential, but if to restrictive, can ignore industry standard flexibility in our system.

My reply follows:
I’m not limiting the inclusion of other libraries. I prefer to think about the problem first, then select the best method of expression. Language is simply that: a method of expression. It is almost always better to program in the language–namely, through the use of native syntax–than to program through it. The latter mode is a common mistake: have you ever seen someone write Java as if it were C?

An ad hoc approach is the alternative. In this case, we choose technologies first and solve problems later. This technique is used often in OSS. Here you must recall the essential difference between commercial software development and experimentation. Read this post.

We don’t exercise all of those steps here because we don’t do proper software development. The lesson, however, is this: the decisions that you make as a coder have lifecycle costs associated with them. Think about it: you include library X. We have to learn that technology. I&T has to test it. CM has to integrate it into the nightly builds. Maintenance has to update it, changing custom code as necessary. A whole succession of maintenance programmers for the next 5-10 years must follow this process.

You must also consider the risk of the technology disappearing during the software’s lifecycle. This is always a risk in software, but it can be approached intelligently. Microsoft has made a significant capital investment in .NET, and they have included AJAX in .NET 3.5. Moreover, other companies have staked their viability on .NET (incidentally, this is the anti-trust argument against Windows in the enterprise space: companies cannot afford to move away from it). Can you say the same for other technologies? Perl was the “next” silver bullet in 1995, but we are still waiting on Perl 6. In the meantime, it has been superseded by faster bullets: Python, Ruby, and Scheme.

I recommend that you read the book Beyond Software Architecture by Hohmann. Technology decisions cannot be made in isolation because they impact the business. The converse is also true. We must always be thinking not only in terms of “wow, this is cool”, but also in light of the question: “Does this make good, long-term business sense?”

I’m not trying to limit your creative freedom. I am trying to show you that we must make considered decisions at this point in the “cone of uncertainty.”

It is the closest thing to a perpetual motion machine, for its inertia alone seems sufficient to sustain it. In many ways, the modern engineering organization is no different than this assembly line. Whereas Ford has its conveyors and pneumatic arms, the large engineering company has its “Big M” methodologies. Use Python for a business system? Too risky. Compress the management hierarchy? Too controversial. Go on the offensive during requirements development? Too costly. Breaking free from this order takes a refractory personality. This is precisely Chouinard’s conclusion:

One of my favorite sayings about entrepreneurship is: If you want to understand the entrepreneur, study the juvenile delinquent. The delinquent is saying with his actions, “This sucks. I’m going to do my own thing.” Since I had never wanted to be a businessman, I needed a few good reasons to be one. One thing I did not want to change, even if we got serious: Work had to be enjoyable on a daily basis. We all had to come to work on the balls of our feet and go up the stairs two steps at a time. We needed to be surrounded by friends who could dress whatever way they wanted, even be barefoot. We all needed to have flextime to surf the waves when they were good, or ski the powder after a big snowstorm, or stay home and take care of a sick child. Breaking the rules and making my own system work are the creative part of management that is particularly satisfying to me.

Chouinard now has the luxury of reflecting on his ascent, which was fraught with challenges. At one point, he resorted to eating dog food when his money ran out. Such is the life of one who challenges convention, which by definition is a position arrived at by force. A terminal moraine, a huge stone pushed by a glacier, is a natural corollary:

Business, climbing, and even car detailing each have their customs that were developed through natural selection over extended periods. Objecting to an established position is no more palatable than exterminating a particular species of animal. This is why Machiavelli wrote in “The Prince”:

And it should be considered that nothing is more difficult to handle, more doubtful of success, nor more dangerous to manage, than to put oneself at the head of introducing new orders. For the introducer has all those who benefit from the old orders as enemies, and he has lukewarm defenders in all those who might benefit from the new orders.

We often forget that some of our greatest luminaries were not overnight successes, but lately recognized geniuses.

Solve the Problem
Every good system solves a problem, which is often elusive. Think about any famous product and try to describe it with a single sentence or a single image. What does the iPod do? It allows you carry digital audio with you. Google? It lets you find relevant stuff on the Internet. Linux? The world needs a good, free operating system. This is not a pedantic exercise. It took me three months of reading and observation to discover the purpose of a recent project. Not a single person in our organization could articulate the system’s raison d’etre clearly, and I found that when I could, my design work took on a new level of coherence. What does the system do and why does it do it? Why is it useful? Answering these questions can go a long way toward unifying the design process. If the questions can’t be answered, then it might be prudent to get real and kill the project.

Write It Down
Once the system concept becomes clear, write a detailed spec. Specs serve several purposes:

• A reason to think: Writing is a specific activity whereas conversation and drawing are often unspecific. If you write, “The system should have a login page,” and then reflect on that statement, then you immediately realize the need for more detail. What does the page look like? What screen elements does it have? What text does it display? The answers to these questions are all requirements that go into the spec.
• A common design decision repository: The login page’s design details need a persistent home. Engineers, just like football players, need to operate from a common playbook. This is why email, PowerPoint, and the lauded whiteboard are not suitable design media: they are not permanent.
• A context: Our system has more than a login page. What is the next page in the interface hierarchy? How does authentication work? Where is user information stored? It is impossible to construct software without knowledge of context.

Software specs exist in myriad degrees of formality, breadth, and depth. But the most important thing is that they exist at the beginning:

Writing a spec is a great way to nail down all those irritating design decisions, large and small, that get covered up if you don’t have a spec. Even small decisions can get nailed down with a spec. For example, if you’re building a web site with membership, you might all agree that if the user forgets their password, you’ll mail it to them. Great. But that’s not enough to write the code. To write the code, you need to know the actual words in that email.

I don’t advocate Joel’s informal approach to specs. His method may be sufficient for designing web sites and business systems, but it cannot be used for Space Shuttle avionics or air defense systems. You wouldn’t use the instructions that came with your Coleman tent to build the Empire State building. The IEEE830-1998 standard is a better reference.

Draw It
Any software architect will quickly learn that it is difficult to model a system in its entirety. The software blueprint will probably never exist, a conclusion reached by the Agile crowd a decade ago. Metaformats suffer from a variety of issues, including:

• Inability to represent the whole system: The UML As Blueprint people have in mind the “exploded” view of systems that is so useful to mechanical engineers. Software is not tangible, though, meaning that it can behave strangely. Try to model a recursive function in UML and you’ll notice the problem.
• A certain level of “OO-ness”: The existence of “Class diagrams” in UML reveals its intellectual heritage. But system architects shouldn’t think in terms of classes, which are like cells in a body. We must first decide if the body should have two arms or three, not the number of cells in each arm.
• Time to develop: Drawing diagrams takes time that could be used to actually write the code. This is particularly true of interfaces. A good Interface Control Document (ICD) and an interface definition are infinitely more valuable than a diagram. Once coding begins, diagrams are often discarded. Why were they created at all?

A multi-faceted approach to design seems more prudent. I prefer a combination of tables (for data definitions), sequence diagrams (for modeling interactions between systems), flow charts (for designing processes), schemas (for databases and XML formats) and natural language requirements. If done properly, the latter can be remarkably effective. Like a good specification, an effective natural language requirement should be:

Correct;
Unambiguous;
Complete;
Consistent;
Verifable;
Modifable;
Traceable

Good specs and designs do not guarantee success. As in the entrepreneurial world, good plans do not make rich men. Execution matters.

Build the Organization
This is the most misunderstood diagram in software development:

Systems engineers design the systems that developers implement. Developers should not make judgments about how the user should behave. Likewise, systems engineers should not decide how to implement code. These competing interests need an arbitrator. In this diagram, I have called him an Integrated Product Team (IPT) lead after the Chrysler convention. Microsoft calls him a PM. Whatever his label, he knows enough about systems and software to mediate between the engineering factions. He also understands the business objectives, and can make the difficult distinction between too much schedule pressure, which harms analysis, and too much analysis, which leads to paralysis. He becomes the technical authority, the “System Solon”. Most importantly, he is at the top of the triangle. If software engineering dominates, then cross-cutting attributes such as performance may not be properly evaluated. If systems engineering sets the project tone, then code-level technical insights–”bottom-up” analysis–may be ignored. The Solon is the bulwark against both outcomes.

“Are you telling me that your software is more complicated than the Burj Dubai? No one has ever built a building that tall. Moreover, the water table is one meter below the sand, so the whole structure is founded upon a massive concrete pad. Your software has greater complexity?”

Her response stymied me. Some pieces of software exceed that building in complexity by orders of magnitude–the Space Shuttle software, avionics controllers on the Airbus A380, Windows Vista–but how many of us work on those systems? Most engineers slave away on J2EE business platforms, or better yet, “In house” software [link] that solve mundane problems as inelegantly as possible. Not to be stopped, I posited a second argument: software is free from physical constraints, thereby enlarging the solution space. A bridge’s incline, for instance, is limited by the coefficient of friction between the road surface and a car tire: vertical bridges may be possible, but they are not useful.

“Innovation in building has never been more rapid or unbounded,” she countered, “Computer modeling makes unprecedented structures possible today. Although inconveniences like gravity do limit design to an extent, they are probably no more limiting than the constraints imposed upon you by APIs and frameworks. Just look at the Walt Disney Opera House in LA.”

The abstract nature of software is therefore not a reasonable excuse for 99.9% of late software projects. Most of us aren’t “in technology”: we use tools and methodologies that academics developed years ago. You’re not on the bleeding edge. Get over yourself.

I finally advanced an enervated argument based on estimation: it’s hard to finish software on time because software design is difficult to estimate. How long will it take you to finish your math homework? How long does it take to solve a Sudoku puzzle? How long does it take to catch three fish? We’re trying to predict an unpredictable task riddled with risk.

“How long does it take to design anything new? Architects deal with unreasonable requirements, unreasonable customers, and unreasonable deadlines. How long will it take me to design a building down to its moldings and handrails? I’ll tell you how long: a lot of sleepless nights. You can only estimate something if you’ve done it before.”

And this is precisely McConnell’s thesis. Unfortunately, expert opinion is not a sufficient resource. Historical data, on the other hand, has been used in study after study to achieve reliably accurate software estimates. Productivity is an organizational thing, so one organization’s data may not apply to another organization’s projects. Do you think that expert who has worked at 10 different companies can give you a useful estimate based on judgment?

So why are software projects late? I see three problems:

Bad requirements–Ask yourself these questions: does your organization employ a professionally-trained requirements engineer? Hold requirements inspections? Version control requirements at the line-item level? Link those line-items to code? Maintain requirements throughout the entire system lifecycle, including maintenance?

Brooks’s Law–Adding people to a late project makes it later. Graph theory holds the proof to this axiomatic observation. Adding more nodes to a connected graph makes the edge count increase exponentially, not linearly. This is why the scheduling equation to convert staff months to schedule months–the most “agreed-upon” equation in software–has a coefficient and an exponent.

The Iowa Theory–Software engineers are unwilling to do the book-keeping work necessary to make large projects succeed because their brains are trained to look for optimal solutions. Associating requirements with code is tedious, but necessary. Drawing algorithm diagrams is tedious, but necessary. Managing software change is tedious, but necessary. But as software engineers, we think these tasks should be easier. Tools exist to make them effortless–the Telelogic Lifecycle suite, for example–but most organizations don’t invest money in these solutions. So software engineers gripe about tools, and go back to hacking. When we will admit that we’re part of the problem?

Consider the USS Iowa. Her keel was laid down in June 1940 and completed in August 1942. She was 890 feet long, could shoot 1225kg shells over 40km, and could cruise at 30 knots. She was built when the country’s survival was at stake. Does your project have that kind of schedule pressure? She was designed by hand, using pencils, paper, and slide rules. Think about that. All that complexity was managed with filing cabinets, folders, and blueprints. Are software engineers willing to exert that kind of effort?