Scientific Data Demonstrating All Swans Are White (Fooled by Randomness, Part 2)

This is Part 2 of my cut & paste Kindle highlights of the wonderful book Fooled by Randomness, by Nassim Taleb]

Observational data is important to science, and Fooled By Randomness by Nassim Taleb cautions us that we should not take scientific data so seriously. We live in a random world, and data can only get us so far in life.

To understand the white swan problem, let’s start with the basics (actually, we’ll end with the basics too, but you know what I mean):

At some point in history, a biologist of the binocular sort (other biologists are of the microscope sort, right?) decided to prove once and for all that “all swans are white”.

He started proving “all swans are white” by observing lots of swans, and documenting his findings. The first year, he observed 1,000 swans, and all of them were white. Even with a thousand observations under his belt, he was not ready to publish his results in “Nature”, as more data was needed. So, he observes another 3,000 swans the next year. Same result–all the swans were white. The question is, how much data must this biologist collect before proving that all swans are white? And, as this biologist collects more and more data, how much more confident should we be in the theory? Will another 10,000 observations of white swans make us pretty sure that all swans are white? How about another 100,000 observations of white swans?

Taleb makes the point that “[no] amount of observations of white swans can allow the inference that all swans are white, but the observation of a single black swan is sufficient to refute that conclusion.” Taleb at location 2153 of 4891 (again, sorry my Kindle doesn’t give real page numbers)

“Consider the following statements: Statement A: No swan is black, because I looked at four thousand swans and found none. Statement B: Not all swans are white. I cannot logically make Statement A, no matter how many successive white swans I may have observed in my life and may observe in the future (except, of course, if I am given the privilege of observing with certainty all available swans). It is, however, possible to make Statement B merely by finding one single counterexample.” Id. at 2186. “The following inductive statement illustrates the problem of interpreting past data literally, without methodology or logic: I have just completed a thorough statistical examination of the life of President Bush. For fifty-eight years, close to 21,000 observations, he did not die once. I can hence pronounce him as immortal, with a high degree of statistical significance.” Id. at 2195.

The induction problem applies equally to scientists who insist their theory is correct due to repeated observations, but it is especially problematic for folks who observe stock market conditions. Taleb draws the following analogy: “Say the market went up every day for a month. For many people of inductive taste it could confirm the theory that it is going up every day. But consider: It may confirm the theory that it goes up every day then crashes–what we are witnessing is not an ascending market but one that ascends then crashes. . . .Accordingly, by such logic, the fact that the market went up all this time may confirm that it will crash tomorrow! It confirms that we are observing a rising-crashing market.” Id. at 4343.

Part of the problem with science, data, and numbers, involves our ability to falsify the theory. One black swan will falsify the theory that ‘all swans are white’. But what happens when you’re presented with a scientific theory that cannot be falsified? (I’m thinking of the multi-verse theory in physics, but that’s a discussion for another day) “[Karl] Popper’s idea is that science is not to be taken as seriously as it sounds (Popper when meeting Einstein did not take him as the demigod he thought he was). There are only two types of theories: 1. Theories that are known to be wrong, as they were tested and adequately rejected (he calls them falsified). 2. Theories that have not yet been known to be wrong, not falsified yet, but are exposed to be proved wrong. . . . A theory that falls outside of these two categories is not a theory. A theory that does not present a set of conditions under which it would be considered wrong would be termed charlatanism–it would be impossible to reject otherwise”. Id. at 2291 Taleb goes on to reason that “Newtonian physics is scientific because it allowed us to falsify it, as we know that it is wrong, while astrology is not because it does not offer conditions under which we could reject it. Astrology cannot be disproved, owing to the auxiliary hypotheses that come into play. Such point lies at the basis of the demarcation between science and nonsense (called “the problem of demarcation”). Id. at 2296.

Monkeys Who Can Type and Survivorship Bias

The white swan problem tells us that more data does not necessarily mean that we know “more” about a given problem. That being said, can we conceive of a problem in which sample size might make a difference? Taleb gives us an analysis of how, sometimes, sample size helps tease out the difference between random (meaningless) data from data that we should pay attention to. Here’s one of my favorite examples from the book:

“If one puts an infinite number of monkeys in front of (strongly built) typewriters, and lets them clap away, there is a certainty that one of them would come out with an exact version of the Illiad. . . . Now that we have found that hero among monkeys, would any reader invest his life’s savings on a bet that the monkey would write the Odyssey next?” Id. at 2372. A recurring theme in Fooled by Randomness involves how much faith can we place in past performance. “In this thought experiment, it is the second step that is interesting. How much can past performance (here the typing of the Illiad) be relevant in forecasting future performance?” Id. at 2373.

“The initial sample size matters greatly. If there are five monkeys in the game, I would be rather impressed with the Illiad writer, to the point of suspecting him to be a reincarnation of the ancient poet. If there are a billion to the power one billion monkeys I would be less impressed–as a matter of fact I would be surprised if one of them did not get some well-known (but unspecified) piece of work, just by luck (perhaps Casanova’s Memoirs of My Life).” Id. at 2386.

The infinite typing monkey problem reveals the fact that the folks who get lucky–like this novel writing monkey–draw all the attention to themselves (you’ve heard this analysis before in relation to 1,000 random coin flippers, there will be one person who flips ‘heads’ a 1,000 times and CNN will want to know how he did it. The answer is, it was randomness that did it, but CNN will air the coin flipper’s technique, diet, and explain that his mother was a coin collector, and so forth). The monkeys who failed to type an epic novel do not make the headlines, “in real life the other monkeys are not countable, let alone visible. They are hidden away, as one sees only the winners–it is natural for those who failed to vanish completely. Accordingly, one sees the survivors, and only the survivors, which imparts such a mistaken perception of the odds. . . This is known as the survivorship bias, arising from the fact that we see only winners and get a distorted view of the odds”. Id. at 2396.

Knowing how the survivorship bias works, I’ve had some white collar fraud clients play both sides of this, and Taleb does a good job of breaking down the scam: “The trick is as follows. The con operator pulls 10,000 names out of a phone book. He mails a bullish letter to one half of the sample, and a bearish one to the other half. The following month he selects the names of the persons to whom he mailed the letter whose prediction turned out to be right, that is, 5,000 names. The next month he does the same with the remaining 2,500 names, until the list narrows down to 500 people. Of these there will be 200 victims. An investment in a few thousand dollars’ worth of postage stamps will turn into several million.” Id. at 2668.

[In the “About the Author” section, it appears as though Taleb has more credentials than many scholars, yet refuses to list them because “Taleb believes that prizes, honorary degrees, awards, and ceremonialism debase knowledge by turning it into a spectator sport.” Couldn’t have said it better myself. I think ‘spectator sport’ is an understatement, ‘circle jerk’ is probably more accurate, albeit crude]