2: Why Advancing Knowledge Is So Hard
The chapter opens with a narrative about a geneticist researching autism, a popular condition about which much remains unknown. This particular researcher has earned much renown for his worn in the area, and Martin was curious about his approach.
The geneticist was actually more of an analyst, conducting "trash can" research - which meant poring over the data that other researchers had recorded, but which was dismissed as irrelevant to their studies (which may have been into different topics completely). He applied little judgment so as to avoid preconceptions and instead considered everything he could get his hands on, and is particularly attentive to the bits of unusual data that are often purged from a study as anomalies or outliers.
The massive amount of data enables him to detect patterns that others have dismissed or overlooked, or are simply unable to perceive because their scope is limited to a single experiment rather than casting a wider net. It enables him to passively observe things that others disregarded in their quest to discover something specific.
Reliability and Validity
Reliability is the ability to get consistent results from an analysis - such that if a test were repeated a hundred times, the same result would occur. If a statistic is reliable, testing a small group will be true of the population. Validity is related, in that it assesses whether a statistical analysis is essentially correct and meaningful.
The problem occurs when validity is sacrificed for the sake of reliability: a hypothesis can be made reliable by throwing out factors that are not consistent among trials, which is to say by getting rid of meaningful but inconsistent observations.
Going back to our geneticist, he recognized that existing studies were ignoring a lot of information for the sake of reliability - and that there was tremendous potential to draw valid conclusions from the data that was excluded from other studies.
Ideally, a decision is based on data that is both reliable and valid, but when both are not possible, there is a consequence to choosing one over the other. Those who seek opportunity and tolerate risk are drawn to validity, and those who seek stability and cannot tolerate risk are drawn to reliability.
The problem is, as the geneticist has proven, that a reliable statistic doesn't render meaningful information and opportunities are overlooked when parameters that are merely inconsistent (but which are valid and meaningful) are excluded for the sake of certainty.
Why Reliability Rules
Martin recalls a conference presentation about the future of business, which proclaimed that computer technology would put an end to "instinct" and drive scientific management with a wealth of data and numerical analysis to give certainty to business decisions.
This is an argument in favor of reliability over validity - and while innovation hasn't been entirely crushed, it certainly took a back seat for many years as companies relied solely upon quantifiable analysis to optimize their businesses and failed to innovate.
Martin does not take the opposite extreme, as he recognizes the value of optimization - but instead maintains the position that neither extreme is correct. Innovation without optimization is death by inefficiency. Optimization without innovation is death by obsolescence.
The assumption that technology eliminates human error is a fallacy: a computer system is programmed by people, and as a result it thinks like people - specifically, the people who programmed it to think in a certain way. So a computer system is merely consistent in its logic, not merely right in its logic. And the one quality it lacks is the ability to think differently.
Thus technology can be rigged to make consistently bad decisions just as readily as it can be rigged to make consistently good ones. And even if its decisions are good at the time it is programmed, the world changes. A computer system programmed ten years ago is doing what made sense ten years ago, which may be different to what makes sense today.
Technology is also limited to those elements that can be quantified and measured, and will ignore any data that seems inconsistent even if it is valid and relevant. This is valuing reliability over validity, and results in a system that is out of touch with reality because it is rigged to ignore the factors that matter most in favor of those that are most easily measured.
The appeal of technology is often to escape the responsibility of judgment. A person who does what a computer tells him is not to blame. He is prevented from thinking for himself, but is merely following orders. And as the computer was programmed according to logic dictated by others, he is not following the orders of the machine but of the person (or committee) who determined how the machine was to be programmed.
In that way, technology also caters to megalomania. The person who programmed the system exercises great power over everyone whose daily activities are dictated by the system, for as long as the system is in use. Autocrats enjoy making rules for others to follow, and the ability to do so through a computer, to effectively control many others over a long period of time while meanwhile escaping the blame (their bad logic will be the computer's fault) is very attractive to cowardly and manipulative people.
That said, the flaws in the motives and methods of technology should not be used to dismiss it altogether. Technology is very good for some things, particularly when the most valid factors are, in fact, quantifiable and unlikely to change over long periods of time. The data they provide is invaluable to optimizing a process that is repeated frequently in a situation that will not change. The question, which is seldom raised, is whether the real-world phenomenon to which technology is applied actually meets those criteria. Very often, technology is the wrong tool for the job - and very often, it is used anyway.
Where the factors that are most influential to outcomes are qualitative, or where the elements and environment are subject to change, technology is a very bad solution - which puts a firm on rails to do the wrong course consistently, and never to discover a different course that would be more successful.
Why do businesses have such a pronounced preference for reliability? Because reliability suits the analytical-optimization mindset, which is preferred for reasons discussed in the previous chapter, plus a few that will follow.
The Persistence of the Past
The demand for proof one of the most powerful defenses against innovation. Proof, by its nature, seeks details from history and applies inductive or deductive logic to derive at a prediction from the future. Consequently, any analysis based on the past tends to favor continuing the past.
During the petroleum crisis of the 1970s, Japanese manufacturers were able to make inroads into the US market as customers turned to smaller and more efficient vehicles. American manufacturers, however, were slow to make the change because they relied upon analyses based on the past, prior to the staggering increase in gas prices, when consumers favored larger vehicles and were indifferent to fuel economy. Naturally, their analyses "proved" that there was little demand for smaller vehicles. And so they were left behind.
All such analyses are implicitly based on the assumption of continuity - that the future will be just like the past and that nothing significant will change. And while everyone claims to know that this is not true, and that change is constant and relentless, the models by which revenues and expenses are predicted for any proposed change are all based on past performance.
(EN: It is a curious thing, that in every finance class I have taken, that one of the first lessons is "past performance does not guarantee future results," and then the rest of the semester is spent learning rational analysis and technical analysis, both of which use historical data as a means of predicting future performance. Perhaps the reason nobody seems to have the stock market figured out is because they're using this very approach?)
The Attempt to Eliminate Bias
Another common problem is the desire to eliminate "bias" from decisions - which seems sensible enough, as it is believed that any subjective judgment is based on speculation and irrational impulses that may be based entirely on fallacy.
But again, this depends on quantifiable information - which ignores anything that cannot be expressed as a number. The irony of it is that translating qualitative data into numbers is itself a dodgy practice. To reduce customer satisfaction to a scale of one to ten is to overlook the true causes of satisfaction, which are entirely non-numeric. A satisfaction rating of "six" means nothing - it's a number picked arbitrarily by an individual who is being asked a qualitative question. The amount of money a customer figures he would be willing to pay for something is another instance of struggling to make a number out of something that is not perceived or conceived as numeric.
It can also be observed that bias-free systems are highly inaccurate. An algorithm that attempts to determine how much a person will pay for a product may take into account masses of data and subject it to rigorous statistical analysts and spit out a price that customers will reject. If bias-free systems were accurate, investing would be risk-free because it would accurately determine the exact price people will pay for a stock - but in spite of a massive amount of effort over many years, no-one has yet developed an accurate market forecasting model, though most models are entirely free of bias.
Martin also mentions the hapless attempts to use computers to predict consumer tastes. Firms like Amazon and Netflix have attempted to formulate algorithms to recommend books and movies to customers based on their past behavior compared to hundreds of millions of transactions. And still, their efforts have been comically inaccurate.
The Pressures of Time
A third reason that reliability trumps validity in business settings is time efficiency: it is a very simple matter to derive an algorithm for analyzing the vast stores of historical information that exists, but much more difficult to construct a logical model based on the probability of future events for which no data presently exists.
Martin mentions the example of asset-allocation systems at investment advisory firms: by asking a new client to complete a questionnaire and comparing a few numerical values to historical data, the program can graph the recommended mix of stocks, bonds, and other investment vehicles that is "right" for that client - not based on his individual needs or personal investment goals, but out of the data based on others who seem to be similar to him based on a few parameters - even if the other clients' are not satisfied with the performance of their portfolios. But the alternative, to talk to a client and determine his actual needs, takes much more time than handing them a form to fill out.
As an aside: the use of forms and databases are the hallmark of backward-thinking and the assumption of uniformity and consistency, and the desire for objective and reliable outcomes that may or may not be accurate and valid. The only way that such and approach will be valid is if the assumptions on which it is based are true: customers are homogenous (at least in a few narrowly-defined categories) and the future turns out just like the past.
The Pervasiveness of Reliability over Validity
Martin goes on a bit of a tare about the manner in which the cult of quantification analysis has become pervasive, both in the breadth of firms who rely upon it, as well as the degree to which it has spread within a firm to affect virtually every business decisions (planning, budgeting, hiring, training, marketing, etc.)
Virtually every course in a business curriculum has parted from teaching students to exercise judgment based on theory, to teaching them to rely on quantitative analysis. And trends such as "Six Sigma" encourages qualitative thinkers to abandon their subjective personal assessments in favor of managing to algorithms.
Six Sigma itself can be seen as the epitome of reliability and the death of validity. It begins by throwing away anything that can't be reduced to numbers, and then throwing away any numbers that aren't consistent, with a goal of arriving at a "goal" of proving the business is consistent and reliable exactly the way it is.
(EN: In operations management class, it was noted that many airlines consider themselves to be six-sigma operations, with 99.999% consistency. Given that flight delays were so common that there was little chance of arriving at your destination at the time that was predicted, and that the quality of service gave rise to the phenomenon of "air rage" among vexed passengers, one wonders what they were measuring to proclaims themselves near-flawless in their operations. They had obviously thrown meaningful data out of the analysis in order to get the numbers they wanted.)
Martin paints an even more sinister picture of the executives who gravitate toward quantitative analysis. Given the complexity of mathematics, and the ability to keep the algorithms by which a business operates secret from competitors, managers had the ability to set goals for themselves and then rig the numbers to prove their success, regardless of real-world evidence of stunning failure (such as dismal financial performance and plummeting market share). Thus, the reason executives of failing firms are routinely awarded outlandish bonuses for their quantified and proven management performance even while firms fell into bankruptcy.
In the market, customers care little about these metrics: they seek products that are effective at meeting their needs for a price they are willing to pay, and care little about the abstract metrics by which a firm considers its own operations to be efficient. As such, running a business "by the numbers" does not guarantee success, and often leads firms in the wrong direction.
Martin also mentions examples of organizations that have learned this lesson the hard way: the closer the firm works with a client, the more it comes to recognize the things that matter to the customer - and finds them to be qualitative assessments that do not yield to numerical analysis. These are the firms that topple the former industry giants, who are managed by the numbers.
The switch to innovation and service excellence requires firms to view numeric analysis with suspicion rather than blind faith, and to take a more intuitive approach that involves trial and error to determine what actually does work - rather than following the numbers down a path that "ought to" work, but often does not. Doing so means accepting uncertainty and taking risk - and those who are able to do so will reap the rewards.
SIDEBAR: Pressure from Capital Markets
While Martin has laid much of the blame on management, he concedes in a sidebar that they are in turn pressured by capital markets. In essence, those that lend or invest money in businesses demand "proof" that the firm will use those funds to generate sufficient revenues to provide the returns they seek.
The proof they seek is based on reliability measures. Talk about speculative undertakings in the future is far less compelling than rows and columns of numbers from the past, along with the suggestion that anything "new" will be like everything the company has done before.
Reliability provides what was done in the past, whereas validity can only be assessed by what will be achieved in the future - and has not been achieved in the past. And as an investor looking for returns in the future, their focus should be on the future - but ironically, they are generally more interested in the past performance of the firm, even knowing it is not a guarantee of its future.
Investors also seek to evaluate the variety of options available to them, and there seems to be no way of comparing visions of the future among different firms - but the numbers from the past can be easily placed in a spreadsheet for side-by-side comparison.
Like gold mining, there is no way to tell how much ore a new mine will produce before it's dug. But there is also no way to tell how much ore an existing mine will continue to produce - its production in the past is no indication of what it has yet to uncover.
Private capital markets, specifically venture capitalists who are willing to risk large sums of money on new businesses, are often highly profitable (EN: but also highly risky). Unlike the public markets in which investors rely on analysts to prove the potential performance based on history, the venture capitalist spends time listening to the details of the plan and evaluates a plausible projection of the future. It funds innovation, which the public markets shun.
Making Room for Validity
The author reiterates his premise: the both reliability and validity are important, but for different things: reliability helps to optimize current behavior to ensure efficiency, validity seeks to make a case for future behavior that will be more effective.
The typical method for balancing the two is for organizations to set aside some funds for a research and development department. While the rest of the organization busies itself with the business of the day, in operations that are expected to be unchanging and perpetual, the R&D area speculates about the future and tests out new ideas that have the potential to change the operations.
Unfortunately, R&D are treated as a luxury by many firms, and it's the first budget to be cut when the firm is facing financial difficulties or simply wants to devote more capital to existing operations. To the author's way of thinking, that is squandering the future for the past.