Wednesday, May 12, 2004
Black Swans and Randomness
Petra sent me this interesting piece from Edge -- an essay by mathematical trader, Nassim Taleb. He writes on outlier black swans, randomness and the 9/11 commission. I am certain many of you will enjoy reading this as much as I did. And perhaps the mathematicians, traders, behavioural economists and cognitive scientists among you could leave a few comments on what you think :)
A black swan is an outlier, an event that lies beyond the realm of normal expectations. Most people expect all swans to be white because that's what their experience tells them; a black swan is by definition a surprise. Nevertheless, people tend to concoct explanations for them after the fact, which makes them appear more predictable, and less random, than they are. Our minds are designed to retain, for efficient storage, past information that fits into a compressed narrative. This distortion, called the hindsight bias, prevents us from adequately learning from the past.
Consider: How would an understanding of the world on June 27, 1914, have helped anyone guess what was to happen next? The rise of Hitler, the demise of the Soviet bloc, the spread of Islamic fundamentalism, the Internet bubble: not only were these events unpredictable, but anyone who correctly forecast any of them would have been deemed a lunatic (indeed, some were). This accusation of lunacy would have also applied to a correct prediction of the events of 9/11 — a black swan of the vicious variety.
A vicious black swan has an additional elusive property: its very unexpectedness helps create the conditions for it to occur. Had a terrorist attack been a conceivable risk on Sept. 10, 2001, it would likely not have happened. Jet fighters would have been on alert to intercept hijacked planes, airplanes would have had locks on their cockpit doors, airports would have carefully checked all passenger luggage. None of that happened, of course, until after 9/11.
One can study randomness, at three levels: mathematical, empirical, and behavioral. The first is the narrowly defined mathematics of randomness, which is no longer the interesting problem because we've pretty much reached small returns in what we can develop in that branch. The second one is the dynamics of the real world, the dynamics of history, what we can and cannot model, how we can get into the guts of the mechanics of historical events, whether quantitative models can help us and how they can hurt us. And the third is our human ability to understand uncertainty. We are endowed with a native scorn of the abstract; we ignore what we do not see, even if our logic recommends otherwise. We tend to overestimate causal relationships. When we meet someone who by playing Russian roulette became extremely influential, wealthy, and powerful, we still act toward that person as if he gained that status just by skills, even when you know there's been a lot of luck. Why? Because our behavior toward that person is going to be entirely determined by shallow heuristics and very superficial matters related to his appearance.
A black swan is an outlier, an event that lies beyond the realm of normal expectations. Most people expect all swans to be white because that's what their experience tells them; a black swan is by definition a surprise. Nevertheless, people tend to concoct explanations for them after the fact, which makes them appear more predictable, and less random, than they are. Our minds are designed to retain, for efficient storage, past information that fits into a compressed narrative. This distortion, called the hindsight bias, prevents us from adequately learning from the past.
Consider: How would an understanding of the world on June 27, 1914, have helped anyone guess what was to happen next? The rise of Hitler, the demise of the Soviet bloc, the spread of Islamic fundamentalism, the Internet bubble: not only were these events unpredictable, but anyone who correctly forecast any of them would have been deemed a lunatic (indeed, some were). This accusation of lunacy would have also applied to a correct prediction of the events of 9/11 — a black swan of the vicious variety.
A vicious black swan has an additional elusive property: its very unexpectedness helps create the conditions for it to occur. Had a terrorist attack been a conceivable risk on Sept. 10, 2001, it would likely not have happened. Jet fighters would have been on alert to intercept hijacked planes, airplanes would have had locks on their cockpit doors, airports would have carefully checked all passenger luggage. None of that happened, of course, until after 9/11.
One can study randomness, at three levels: mathematical, empirical, and behavioral. The first is the narrowly defined mathematics of randomness, which is no longer the interesting problem because we've pretty much reached small returns in what we can develop in that branch. The second one is the dynamics of the real world, the dynamics of history, what we can and cannot model, how we can get into the guts of the mechanics of historical events, whether quantitative models can help us and how they can hurt us. And the third is our human ability to understand uncertainty. We are endowed with a native scorn of the abstract; we ignore what we do not see, even if our logic recommends otherwise. We tend to overestimate causal relationships. When we meet someone who by playing Russian roulette became extremely influential, wealthy, and powerful, we still act toward that person as if he gained that status just by skills, even when you know there's been a lot of luck. Why? Because our behavior toward that person is going to be entirely determined by shallow heuristics and very superficial matters related to his appearance.
