In 2011, IARPA, the research arm of the US intelligence community, launched a massive competition to identify cutting-edge methods to forecast geopolitical events. Four years, 500 questions, and over a million forecasts later, the Good Judgment Project (GJP) led by Philip Tetlock and Barbara Mellers at the University of Pennsylvania emerged as the undisputed victor in the tournament.
GJP’s forecasts were so accurate that they even outperformed those of intelligence analysts with access to classified data. One of the biggest discoveries of GJP were the Superforecasters: GJP research found compelling evidence that some people are exceptionally skilled at assigning realistic probabilities to possible outcomes even on topics outside their primary subject-matter training.
Philip Tetlock and his colleague Dan Gardner profile several of these talented forecasters, describing the attributes they share, including open-minded thinking, and argue that forecasting is a skill to be cultivated, rather than an inborn aptitude. In the 20 year study, Tetlock showed that even the average expert was only slightly better at predicting the future than random guesswork. Tetlock’s latest project, a government funded forecasting tournament involving over a million individual predictions has since shown that there are, however, some people with real demonstrable foresight.
The Good Judgment Project trained and tracked the performance of over 2,000 individuals, was able to systematically evaluate the factors that lead to better forecasting behaviors.
Foxes make better forecasters than hedgehogs.
Decades ago, the philosopher Isaiah Berlin wrote a much-acclaimed but rarely read essay that compared the styles of thinking of great authors through the ages. To organize his observations, he drew on a scrap of 2,500-year-old Greek poetry attributed to the warrior-poet Archilochus: “The fox knows many things but the hedgehog knows one big thing.” No one will ever know whether Archilochus was on the side of the fox or the hedgehog but Berlin favored foxes. I felt no need to take sides. I just liked the metaphor because it captured something deep in my data. I dubbed the Big Idea experts “hedgehogs” and the more eclectic experts “foxes.” Foxes beat hedgehogs. And the foxes didn’t just win by acting like chickens, playing it safe with 60% and 70% forecasts where hedgehogs boldly went with 90% and 100%. Foxes beat hedgehogs on both calibration and resolution. Foxes had real foresight. Hedgehogs didn’t.”
Hedgehogs are people who make predictions based on their unshakeable belief in what they see as a few fundamental truths. Foxes, by contrast, are guided in their forecasts by drawing on diverse strands of evidence and ideas. Hedgehogs know a few big things, foxes know lots of little things. When new information comes to light a fox is likely to adjust her forecast, where a hedgehog will likely discount the new data. For a fox, being wrong is an opportunity to learn new things.
The types of probabilities that that the participants in the Good Judgment Project were asked to calculate were all time-bound and measurable. They tended to focus on issues such as commodity prices, financial results, and political outcomes. Analysts performed only marginally better than chance, and often worse when they became too attached to a grand theory about how the world works.
That is where Berlin’s distinction between foxes and hedgehogs becomes useful, although arguably Berlin’s original essay remains the sharper and more enduring work because it was really describing a permanent tension in human thinking: the seductive appeal of elegant, totalizing explanations versus the messy reality of history and human behavior. Tetlock simply found a way to measure the consequences of that tension empirically.
The hedgehogs in the Good Judgment Project were often intelligent people trapped by the need to be right in a coherent, ideological way. They interpreted events through one dominant framework and protected that framework even when reality stopped cooperating. Foxes, by contrast, succeeded largely because they were less emotionally committed to certainty. They were willing to absorb contradiction, revise estimates, and admit they might be wrong. In other words, they treated the world as complicated instead of pretending it could be reduced to one clean theory.
The uncomfortable implication is that good forecasting may depend less on brilliance than on resisting the very habits that make public intellectuals and political commentators successful in the first place. Confidence sells. Simplicity persuades. Grand narratives are memorable. But reality stubbornly refuses to organize itself into a single elegant system. The foxes performed better not because they possessed mystical foresight, but because they were more honest about uncertainty.