Superforecasting: The Art and Science of Prediction by Philip E. Tetlock and Dan Gardner (Crown 2015. 352 pp. $28.00)

A bat and a ball cost $1.10 together. The bat costs a dollar more than the ball. Quick: How much does the ball cost?

If you answered five cents, you might be the kind of forecaster that psychological researcher and Wharton School professor Philip Tetlock profiles in Superforecasting: The Art and Science of Prediction. Written with journalist Dan Gardner, the book chronicles the four years Tetlock spent running a forecasting contest. By simply keeping score, he was able to find out who made the most accurate forecasts, and more importantly for readers, exactly how the people he calls “superforecasters” achieved such startling results.

The origin of the book dates back to 2010. The U.S. intelligence community was reeling from their prediction — ultimately proved false — that Saddam Hussein had stockpiled weapons of mass destruction in Iraq. Wondering how the art of forecasting could be scientifically improved, the intelligence organization IARPA asked Tetlock to enter a contest with a serious purpose.

The Good Judgment Project

The request led Tetlock to launch the Good Judgment Project with his research partner Barbara Mellers in 2011. They recruited more than 20,000 volunteers to answer questions about international events, such as, “In the next year, will any country withdraw from the eurozone?” Each question had a specific time frame and a closing date, and the volunteers answered with a probability. Over the four years of the tournament, nearly 500 such questions garnered over a million forecasts. Along the way, Tetlock not only studied the results but was allowed to run experiments to try to improve his team’s performance.

Tetlock’s volunteers were competing not just against each other, but also against other teams in the IARPA tourney — one of which was the U.S. intelligence community. But by the end of the tournament, Tetlock’s team had delivered results that no one had predicted. Using volunteers and a few algorithms to adjust their predictions, the team won the tournament, beating even the intelligence professionals (although the margin of their success against the professionals remains classified).

To start, the best forecasters were careful. The first thing they were careful about was falling into the trap of what’s called System 1 thinking — the kind of snap judgments that people make automatically to save time.

“I was very surprised,” says Tetlock. “I was really surprised that the best people were able to do so much better, even than professional intelligence analysts, who had access to classified information.” Overall, the superforecasters outperformed the other forecasters in the tournament by 60 percent.

Are Superforecasters Super Smart?

What made the superforecasters so good? The researchers began by looking at test scores. Tetlock found that while the average volunteer scored about 70 percent higher than the U.S. population on tests of intelligence and knowledge, the superforecasters scored about 80 percent higher. They were also comfortable with math and tended to be probabilistic thinkers, although as Tetlock points out, “the basic analytical tools that the best forecasters used to boost their accuracy are accessible to anyone who has very basic, intro college math.”

In the end, the researchers concluded it wasn’t so much who the superforecasters were, but what they did. “What matters most,” Tetlock writes, “is how the forecaster thinks.”

To start, the best forecasters were careful. The first thing they were careful about was falling into the trap of what’s called System 1 thinking — the kind of snap judgments that people make automatically to save time.

System 1 thinking is why most people get the bat and ball question wrong. At first glance, it seems obvious that if the bat and ball together cost $1.10, and the bat is a dollar more, the ball must cost ten cents. To get the question right, however, you need to slow down and use System 2 thinking — deliberate, conscious thought.

Avoid the Bait-and-Switch

Tetlock found the best forecasters consistently used a handful of methods to enhance their analytical reasoning. The first of these had to do with how they looked at the question itself. To get to an accurate forecast, it turns out, you first have to avoid the bait-and-switch trap.

Bait-and-switch is another automatic process, so it tends to go unnoticed until it’s too late. The process goes like this: Confronted by a very difficult question, most people get stymied and automatically substitute an easier one instead.

Tetlock recounts how the bait-and-switch problem caused superforecaster Bill Flack to flub the question, “Will Japanese Prime Minister Shinzo Abe visit the Yasukuni shrine in 2013?” Yasukuni honors Japan’s war casualties, listing the names of almost 2.5 million people. But among those listed are some 1,000 war criminals, so official visits to the shrine provoke protests from the governments of China and South Korea. Skipping the visit looked like the most rational course of action.

But counter to Bill’s prediction, Abe did visit the shrine. Reviewing his process, Bill said, “I think that the question I was really answering wasn’t ‘Will Abe visit Yasukuni?’ but ‘If I were PM of Japan, would I visit Yasukuni?’”

Channel Enrico Fermi

How many extraterrestrial civilizations exist in the universe? That’s not a question from the forecasting tournament, but rather one posed by physicist Enrico Fermi, who was famous for coming up with estimates for seemingly unanswerable questions. Fermi’s secret was to break the question into sub-problems, dividing the knowable from the unknowable. The best forecasters often used Fermi’s method, known as “Fermi-izing,” to approach tough questions.

As a quick lesson in how to Fermi-ize, Tetlock deconstructs a question Fermi often asked his students: How many piano tuners are there in Chicago? (Remember, this was before Google). To arrive at a good estimate, start by asking another question: What do I need to know to answer this question?

Tetlock keys on in four variables: the number of pianos in Chicago, how often pianos are tuned each year, how long it takes to tune a piano, and how many hours per year the average piano tuner works. By intelligently estimating each sub-question, it’s possible to come up with an answer that’s surprisingly close to the truth.

Only after searching for the best comparison class and locating any data did the best forecasters then proceed to the “inside view” — the specifics of the situation.

First the outside view, then the inside

Like actuaries, the superforecasters tended to answer a question by looking first for the outside view, in the form of statistics. To illustrate this, Tetlock describes hypothetical four-year-old Tommy Renzetti, who lives in a small house with his dad, a bookkeeper named Frank, and his mom, Mary. Here’s the question: Does Tommy have a dog?

If your first thought was look up the percentage of American families that own dogs, you’re following in the footsteps of the superforecasters. Only after searching for the best comparison class and locating any data did the best forecasters then proceed to the “inside view” — the specifics of the situation — such as the fact that Tommy is still too young to walk a dog by himself, hypothetical or otherwise.

In putting together a variety of views, Tetlock compares the best forecasters to Star Trek’s Captain Kirk. When Kirk makes decisions, he usually gets a super-rational opinion from Mr. Spock, followed by a more emotional viewpoint from Dr. McCoy. In the end, he considers both. Likewise, good forecasters synthesize different points of view.

Superforecasters at a Wedding

Synthesizing different perspectives is a skill that many people find counter-intuitive. “Superforecasters . . . have a rather distinctive way of thinking,” says Tetlock, “and some people find it a bit off-putting.” He explains his observation by imagining a superforecaster at a wedding.

“If you were to ask [superforecasters] how likely is it that the couple is going to stay together,” he says, “you wouldn’t get the usual reaction, which tends to be, ‘well, look at how happy they are!’ You’d get, ‘The right thing to look at is the base rate of divorce in their demographic category.’”

“Superforecasters . . . have a rather distinctive way of thinking,” says Tetlock, “and some people find it a bit off-putting.” He explains his observation by imagining a superforecaster at a wedding.

After starting their estimate with data, based on the best comparison class, the superforecasters would then adjust that estimate using any inside information about the relationship. “So if they happen to know that the groom is a psychopathic philanderer,” says Tetlock, “they’ll decrease [the probability of staying together] in response to the idiosyncratic information about the individuals involved.”

The Wisdom of Crowds

From the start, one of the goals of the project was to find ways to make forecasters better. During the first year of the project, Tetlock and his researchers ran a side experiment. They randomly assigned several hundred forecasters to work alone, and several hundred others to work on teams. Although the teams didn’t meet in person, members could communicate by email or Skype. By the end of the first year, the teams were 23 percent more accurate than the individuals.  For the best forecasters, the benefits of becoming part of a team were even clearer: The superforecasters from year one became 50 percent more accurate when placed on a team in year two.

What are the implications of Tetlock’s research for actuaries? You might expect actuaries to be excellent forecasters, and Tetlock says that during the tournament, several actuaries did make it into the ranks of the superforecasters. “It’s an interesting question of how far the rigor of actuarial thinking can be extended,” he says, “into the domains where you don’t have well-defined statistical distributions. And I think if there’s a big, big surprise, on a philosophical scale; it’s that it’s possible to take explicit probabilistic reasoning into domains where few people previously thought it possible.”

Will forecasting become a more scientific endeavor in the near future, as Tetlock hopes? You can always make a prediction.

Laurie McClellan is a freelance writer and photographer living in Arlington, Virginia. She is on the faculty of Johns Hopkins University, where she teaches in the M.A. in Science Writing program.