jim.shamlin.com

13 - Scientific Reasoning

The methods by which science seeks poof is by means of experimentation, modeling, and demonstration - but the underlying structure of science is largely the same as logic: it seeks to establish premises that lead to a conclusion.

Logic borrows back from science by adopting the method of hypothesis testing: proposing that something is true, then "testing" to determine if it is, then coming up with an alternate proposal and testing it, until we find the truth (or at least a proposal that proves out to be true).

Hypothesis Testing

The author suggests a four-step method for hypothesis testing:

• Define the hypothesis to be tested.
• Collect the evidence for and against the hypothesis.
• List all the alternative hypotheses.
• Rank them and pick the best one to accept.

None of this indicates where a hypothesis comes from in the first place, which seems to be entirely random. We are struck by a notion, possibly inspired by something that we observed, or saw in a dream, or whatnot. Science refers to the methodology that is applied after the moment of inspiration.

Step 1: Define the Theory to Be Tested

To be testable, a hypothesis must be sufficiently specific and clearly stated. Much is said about specificity: there are various kinds of "energy" in physics; to say something is a 'good investment' is almost meaningless for lack of detail such as return rate and time frame; and the scope of the theory can be problematic.

Step 2: Gather the Evidence for and against the Theory

When gathering evidence, focus on the two types:

1. Supporting evidence are facts that increase confidence the hypothesis is true
2. Counterevidence decreases confidence

Evidence can also differ in strength. Some evidence may suggest that the hypothesis is true/false without definitively proving it to be so. Much depends on the hypothesis: if your argument is that all swans are white, seeing a thousand white swans is evidence that bolsters confidence but does not prove it to be true; but seeing one black swan is definitive proof it is false.

In general, more evidence is better. To test something once or twice does not preclude the possibility it is an anomaly of a fluke, but multiple iterations of a test decrease that possibility.

Seeking counterevidence is particularly important to avoid bias. It is our natural inclination to want a statement to be proven true, and possibly ignore evidence that suggests it is false unless we make a point of seeking it.

Step 3: List all the Alternative Theories

"The world is a complicated place and things are often not what they seem" are two hackneyed phrases that represent a fundamental problem: there is more than one possible cause for a given outcome, and a tendency to latch onto the first and most obvious explanation leads us to ignore all others.

For that reason it is necessary to make a purposeful process of exploring alternative theories, to consider more than one explanation and then to explore the likelihood of many alternatives rather than attempting to prove the first that comes to mind.

Consider the panic over global warming (replacing the panic over global cooling) and the hubris of the assumption that mankind's behavior is to blame. There are alternative theories, which get less attention for their lack of drama, such as the longer-term fluctuation that cause the earth to warm or cool, around a given norm. It may just so happen we are in a warming cycle, and it has nothing to do with us.

Coming up with alternative theories is as imaginative a process as formulating the first hypothesis, made all the more different by human biases. Consider the influence of religion, and the tendency of certain people to seek a supernatural explanation to anything that is puzzling, then resolutely defend it against rational explanations. Others seem to have less supernatural defaults, but a favorite target nonetheless: that it is the faulty of society, the government, the wealthy, business, and so on.

Good scientific reasoning requires us to recognize our biases, pursue alternative theories, and perform a careful evaluation that results in the identification of the most plausible explanation.

Step 4: Rank the Theories and Pick the Best One

Once alternative theories are identified, evaluate them carefully to consider their premises and identify the one that is most plausible.

Theoretical Considerations

The last step in the previous process was to pick the best theory among alternatives - but how do we go about doing so? The author suggests a number of considerations.

Predictive Power

The predictive power of the premises that explain an outcome can be tested or observed, or where neither testing nor observation is possible, we may still assess by logic whether it might be true. Of importance is getting beyond blind faith.

Specificity is often a factor in predictive power - the more vague a prediction, the more likely it is to come true, but the less meaningful it is as a cause. An astrologist who suggests a 90-year-old man will die in 20 years gains no credibility if he dies in ten. One who predicts he will die on a specific date (June 12, 2019 at exactly 3:08 am) is less likely to be right, but highly credible if he is.

When a prediction fails to prove out, it may be possible to salvage the theory by challenging auxiliary assumptions: we suggest that water freezes at 32 degrees, but can observe water to freeze at 40 degrees - in which case we can challenge the accuracy of the thermometer.

However, we must also beware of an ad hoc premise that is tendered when a hypothesis has failed. Consider that psychics who fail to perform better than what could statistically be expected suddenly suggest that the reason for failure was the negative thoughts or disbelief of the subject whose mind they were asked to read. This was never mentioned as a criteria before the experiment failed.

While an ad hoc premise or an ad hoc hypotheses has some merit in that it may turn out to be true, they are so often used as dodges as to lack credibility, and should be used with reluctance and regarded with suspicion.

Mechanisms of Causality

This pertains to the notion that correlation is not causality. It can be demonstrated that there is a positive correlation between ice cream sales and shark attacks in Australia, but it cannot be concluded that eating ice cream makes a person more likely to be attacked by a shark. But this is merely because of a third factor, seasonality, in that in warmer weather people consume more ice cream and spend more time in the water.

In general, we should avoid the inclusion of premises that have no observable connection to the outcomes. This is how superstitions are born. When it does not make sense that one thing might cause another, but there is statistical correlation, it indicates the need for further consideration and the search for other causes.

Fruitfulness

An oblique reference is made to the fruitfulness of a hypothesis - whether the discovery of the answer yields practical information, or gives way to further investigation.

As an example, the theory of plate tectonics that indicates the outer crust of the earth consists of floating plates rather than a solid orb was first considered in the 1960s, and the very notion generated a host of new theories and explanations about geography, climate, and geography.

(EN: I'm not sure why the author mentions the concept of fruitfulness here, as it seems to be a prerequisite to the argument, deciding whether it is worth the time to investigate a hypothesis, or can be a happy accident. It does not help indicate whether a given hypothesis is better than others in an investigation.)

Coherence

The coherence of a theory is internal, in its logical consistency. A theory that is inconsistent is unlikely to prove true, so it requires improvement.

There is also the notion of external consistency, in that a theory is not in conflict with a body of knowledge, particularly with accepted theories. It is entirely possible for a theory that is inconsistent with the present theory to be right, and for it to cause us to question conventional knowledge, but this is also extremely rare. Going out on a limb to pursue a bizarre theory is very seldom productive, and should only be considered when more plausible ones have been exhausted.

Coherence is sometimes a matter of common sense. The folksy wisdom that "it something sounds too good to be true it probably is" very often prevents people from being taken in, or even wasting their time in considering the specious claims of advertisers - though the fact that advertisers are still using the same tactics suggests that there is a significant proportion of the population that are easily deceived.

Simplicity

Another common aphorism is that the simplest solution is usually the best, and this tends to be the case. Einstein is quoted: "Nature is the realization of the simplest that is mathematically conceivable."

It is disputable as to whether simple is universally good, or merely an oversimplification. However, simple theories have a number of advantages: they are easier to apply, they require less evidence to support, they are more broadly applicable, and they become the building blocks of more complex investigations.

Expert Opinion

It is a benefit to society that people can specialize in certain fields of study and become more knowledgeable than others: we can rely upon their expertise and defer to their wisdom. However, this does not mean that we should entirely abdicate the necessity to think for ourselves and place blind faith in intellectual leaders. Some are outright frauds, and even the best of them can make mistakes.

Some key questions to ask:

• What is the expert's opinion? Especially when you get this information second hand, particularly through the media, the opinion becomes distorted. A newspaper is fond of reporting that "A causes B" whereas a scientist will observe that there is "some correlation" between the two.
• Is the expert in his field? There is a tendency to grant too broad a respect to experts, and for expert status to go to their heads, such that they are consulted or weigh in on topics in which they are not qualified. The author gives the example of the German chemist who speculated that the supercollider might create black holes that would destroy the earth, which is a matter of physics rather than chemistry. To the media, a scientist is a scientist. (EN: Also consider that this is more broadly applied to celebrity. A distressing number of people follow the political remarks of musicians or actors because they are famous or talented as a performer, and know nothing of politics.)
• Is the expert reliable? There are many 'experts" who have a track record of getting things terribly wrong, laughingstocks in their own profession but who still have the credentials to present themselves as qualified to those who do not know their reputation. The same can be said of secondary sources of information, particularly those that do not cite their sources or do so with an anonymous citation such as "a leading researcher."
• Is the opinion in context? Quotes can be taken out of context to change their meaning, and it's also been observed that esteemed people often attempt to humanize themselves by joking around, make an off-the-record remark that ends up being recorded, or speak speculatively.
• Is their opinion accepted? The popularity of an opinion does not guarantee its accuracy, but when experts disagree with each other, it does indicate a lack of consensus that suggests that matters are not settled, and more consideration is necessary before siding with one of a number of conflicting theories.
• Is there a conflict of interest? Particularly in the present age, scientists and scholars are compromised by economic concerns: a study is commissioned by a patron who has a desire to have a given outcome (a tobacco company wants to disprove claims that smoking is unhealthy) and the expert who sides with them can count on continuous funding. We used to believe that science and academics were above such concerns, but they have lost their credibility in that regard. This does not mean the outcome of a study should be summarily dismissed, but viewed with great suspicion.
• Is there another source of bias? Aside of economic concerns, there are other reasons a person's opinion can be biased. For example, if an academic aligns himself with a given political party or a particular school of thought, it will color their perspective and bias their outcomes.

The author reiterates that these criteria are warning flags and not absolute laws. A misrepresented opinion that is taken out of context, attributed to a person who is not an expert in a given field and has a poor reputation, that is not widely accepted, and where there are many reasons for bias is not necessarily wrong.

Sidebar: Bertrand Russell

The author cites some basic principles suggested by philosopher Bertrand Russell in regard to deferring to expert opinion:

1. When the experts agree, the opposite opinion cannot be regarded as certain
2. When they are not agreed, no opinion can be regarded as certain
3. When the experts maintain there is not sufficient ground to come to a conclusion, the ordinary man would do well to suspend his own judgment