Chapter 5: Why Diversity Trumps Ability
There is a popular theory that maintains that a group of average people can outperform a group of experts when it comes to solving new problems. The reason for this is that people who are experts in a given subject are acculturated to the same knowledge - they have read the same books, taken the same classes, subscribe to the same theories, and so on. Experts are, in that way, a very homogenous group. They tend to agree to the same things and come up with the same ideas because they are bound to the same information and theoretical frameworks.
Meanwhile, a diverse group of people who are not institutionalized can think more diversely and consider different approaches: they don't know the "proper" way to go about solving a problem, so they attack it as best they can - and in so doing they can stumble upon things that the experts might have dismissed as being irrelevant or unlikely to succeed because they do not conform to known theories.
(EN: A common example of this is the computer hacker. A teenager who's just goofing around on a computer can defeat security systems designed by a team of credentialed professionals because the pros designed their defenses against a conventional attack, and were unable to predict the unconventional behavior of a novice. Meanwhile, the teen doesn't know the "proper" way to attack a system, and does things no expert would have thought of.)
He then strays into the dubious theory of "collective intelligence" - the notion that a group of people is possessed of an intelligence that no single individual in the group has, but which mysteriously manifests itself when they are in proximity to one another. (EN: This theory has been around for a while, and it remains entirely mystical, so I'm not along for the ride.)
Crowdsourcing is based on a democratic premise: one person's opinion is just as valid as any other, and the best ideas are those which are espoused by the most people.
Three forms of Crowdsourcing
The author suggests three classes of activity that take place in the crowd:
The first form is prediction - which is much like political polling or market surveys that ask people to speculate about their future behavior, whether to vote for a candidate or purchase a product. It is generally believed that people will be honest and accurate in describing their own future behavior, though this is far from infallible.
The second form is problem-solving. A person with a problem to solve broadcasts it to a large group of people, hoping that someone in the crowd will propose a solution. This, too, is similar to surveying in that the solution that gets the most "votes" is effective. But again, this has limited applicability because answers that rely on expertise are not often answered by the crowd - the advice of a million random people about treating a medical condition is much less reliable than the advice of one doctor.
The third form is brainstorming. This is like an internet-based suggestion box that calls upon people to provide ideas (not necessarily solutions to a specific problem). Otherwise it is the same as the former.
(EN: Curiously, all of this is about talking rather than doing - so he's likely missing a fourth form of collaborative work, in which people contribute effort to a task such as writing a software program or maintaining an online encyclopedia.)
Collaborative Competition
The author describes, in detail, a contest among programmers: The sponsor provides a problem to solve, such as plotting the most efficient route to travel between multiple locations. Competitors submit their solutions, but they need not be original - that is, a person can "steal" another person's program, make a few tweaks, and resubmit it as his own entry.
Conceded, this creates a bit of nerd rage when someone takes someone else's work and makes a few tweaks to pull into the lead - but what they found was that the original submitter will then take the plagiarized version, make a few more tweaks, and resubmit it.
In one instance, the code created in the competition was compared to the work of a professional programmer, and it was found to be better (though the criteria for evaluation are not disclosed), which shows that a legion of "script kiddies" can produce work that's better than that of an expert.
Crowds as Markets
The field of economics is well familiar with the function of crowds. A market is simply the aggregation of a large number of sellers and buyers, each pursuing their self-interest and making decisions independently. Statistics about the global markets reflect their aggregate behavior - and in most instances, the markets get along just fine: suppliers gauge what buyers want and the market works most efficiently when left alone to be guided by the "invisible hands" of the crowd.
FA Hayek considered the action of crowds in his paper, "The Use of Knowledge in Society," which remarks that each individual embodies a unique set of information - that is, everyone knows different things. A person may make a foolish decision, but people in aggregate are actually quite knowledgeable and society as a whole gets along just fine. The worst economic disasters are brought by regulation, one person in a position of power thinking he knows what's best for everyone else and forcing others to abandon their own beliefs and do as he demands.
One of the advantages of civilization in general is a diversity of knowledge. Other people know things we do not, and we generally learn by imitating the practices of others that achieve the best results, but always have the ability to experiment with doing things our own way. If it works out for the better, it's adopted by others. If it does not, we harm only ourselves.
There is also the notion that "we may well already possess the solutions to our greatest dilemmas" - that for any human problem, someone already knows the answer but is being prevented or at least discouraged from sharing what he knows with others.
Normalization and Groupthink
One of the problems of diversity is that it does not last long. Assemble a group of people with diverse ideas and opinions, and before long they become homogenized, forming into polarized camps who each espouse a specific set of opinions and beliefs.
This kind of normalization is itself completely normal. As social creatures, we tend to seek amicable agreements with one another, which often means suppressing our individualism. This may be a conscious choice to "go along to get along" but in time it becomes a completely unconscious process. When two minds know the same things and have the same beliefs, it is inevitable that they will come to the same conclusions even if they do not communicate with one another.
In that sense, a crowd is different to a group. It is never meant to have a single identify or a set of social norms, but is most effective when people don't seek to conform, imitate, and get along with one another. The more homogeneous it becomes, the less capable it will be because people adopt or imitate the ideas of others rather than contributing anything unique to the discussion.
Normalization, in fact, can be a dangerous quality. Consider that a crowd of people doesn't become a mob until it begins acting in unison. And once a crowd becomes a mob, it begins intentionally destroying its diversity and the individuality of members: those who do not accept the mob rule are told "get with the program" and threatened with "you're with us or against us" and discouraged to think as individuals.
Diversity Does Not Always Trump Ability
The author concedes that diversity and the crowd are not always a good thing.
For some things, there's a simple, straightforward, and single answer: no-one needs a crowd to tell them the sum of two numbers - and the opinion of many does not make the wrong answer right.
In other instances, the problem is more efficiently solved by a person who has a solution, rather than the lengthy process of debate and voting among a large number of people.
Sometimes, expertise is needed. A random survey of subway commuters cannot outperform a group of nuclear physicists at designing a more efficient reactor.
And lastly, some problems are just too large and poorly defined for a crowd to handle. A crowd of people can often guess the right answer to a trivia question, but they cannot write a novel.