Assume you are considering investing $700,000 and that this is a large sum for you. Experts give you the names of two reputable investment houses of long standing.
When you visit the first of these, Benjamin Company, Inc., the manager recommends Bailout Bonds as an excellent investment for you. He gives you what sound like good reasons for his recommendation and reassures you that he is very confident in his advice, which is based on the analysis of top experts. He urges you to act swiftly to avoid missing out.
As a cautious investor, you visit the other investment house that was recommended to you, Franklin Company, Inc.. The manager advises you that Bailout Bonds would be a poor investment and that you would be better off holding on to your money than putting such a large amount into such a speculative investment. Like Benjamin’s manager, Franklin’s manager provides good reasons and she too is confident in her advice.
Given the conflicting advice, you decide to check the track record of each of these investment houses. Fortunately, numerous academic studies have been published on the accuracy of each firm’s investment recommendations since 1930. Each firm retains the advice of some of the best experts in the world. You find that on average, the Benjamin experts have been right for 50 percent of their forecasts — and wrong on 50 percent. Interestingly, the Franklin experts have exactly the same performance record.
What would you do? We suspect small business people would skip this investment opportunity.
The year 1930 is roughly the date of the first studies on the value of expert forecasts that relate to complex and uncertain situations. Contrary to popular opinion, the findings are consistent with our Benjamin versus Franklin story.
For example, Philip E. Tetlock’s 2005 book, Expert Political Judgment, describes how he recruited 284 people whose professions included “commenting or offering advice on political and economic trends.” He asked them to forecast the outcomes for various situations. By 2003, he had accumulated 82,361 forecasts.
The conclusions from the studies are that:
1) expert opinions are useless for forecasting related to complex and uncertain situations.
2) expertise does not help; college students do as well as seasoned experts at such forecasting.
3) experts’ statements about confidence have virtually no value. Indeed, if you put a group of experts in a room and have them make forecasts, their confidence goes up rapidly, but this has no relationship to accuracy.
The country faces a similar problem. But our leaders in Washington are not debating about their own investments. Instead, they are thinking about how to spend other people’s money. They have provided no scientific basis for their decision and they show no awareness of how one should properly approach such a forecasting problem.
There are ways to study this problem, but they should not be done in a rush, and they should not be done in group meetings.
We don’t know what will happen, but we do know what procedures to use to obtain scientific forecasts of the outcomes of various plans. Researchers in the field have been trying to spread the word on scientific (evidence-based) forecasting by making forecasting knowledge easily and freely available to others at http://forecastingprinciples.com.
The question for our leaders is whether they should invest $700 billion when, despite their confidence, they are completely ignorant of the outcome of the investment plan.
Dr. J. Scott Armstrong is a professor at The Wharton School, University of Pennsylvania. Dr. Kesten C. Green is with the Business and Economic Forecasting Unit at Monash University. This essay first appeared online at http://publicpolicyforecasting.com and is reprinted here with permission.