The other important thing I will say is, if Comey’s quote is true, then he actually had to listen good election forecasts that showed that number was more than 70 percent. So that becomes an argument for further forecasts.
So, what is a “good” prognosis? Going back to 2016, as you say, Nate Silver’s forecast gave Trump a 30 percent chance of winning. Other models projected Trump’s chances at more than 1 percent or in the low single digits. The sense is that because Trump won, Nate Silver was therefore “right”. But of course, we can’t really say that. If you say the chance of something happening is 1 in 100, and it does, that could mean you underestimated it, or it could just mean a 1 in 100 chance.
This is a problem in finding out whether pre-election forecasting models are correctly tuned to real-world events. If we go back to 1940, we have only 20 presidential elections in our sample. So there is no real statistical justification for the precise probability here. 97 vs. 96 — it’s insanely hard with our limited test size to know if these things are calibrated correctly to 1 percent. This whole exercise is far more uncertain than the press, I think, leads consumers of polls and forecasts to believe.
In your book, you talk about Franklin Roosevelt’s pollsters, who was an early polling genius—but even his career caught fire later, didn’t it?
This guy, Emil Hurja, was Franklin Roosevelt’s pollster extraordinaire and election prognosticator. He devised the first type of survey aggregate, the first tracking survey. A truly fascinating character in the polls story. It’s insanely accurate at first. In 1932, he predicted that Franklin Roosevelt would win by 7.5 million votes, even though other people predicted that Roosevelt would lose. He wins with 7.1 million votes. Therefore, Hurja was better calibrated than other pollsters at that time. But then he fails in 1940, and after that he’s basically as accurate as your average pollster.
In investing, it is difficult to beat the market over a long period of time. Similarly, with surveying, you must constantly rethink your methods and your assumptions. Although Emil Hurj was initially referred to as the “Wizard from Washington” and the “Crystal Gazer from Crystal Falls, Michigan”, his record slipped over time. Or he just got lucky early on. It is difficult to know afterwards whether he was really a genius predictor.
I bring this up because—well, I’m not trying to scare you, but it could be that your biggest screw-up is somewhere in the future, and it’s yet to come.
That’s kind of the lesson here. What I want people to think about is that just because the polls were biased one way in the last few elections doesn’t mean they will be biased the same way for the same reasons in the next election. The smartest thing we can do is read each individual survey with a view to how that data was generated. Are these questions correctly worded? Does this poll reflect Americans through their demographic and political trends? Is this outlet reputable? Is there something going on in the political environment that might cause Democrats or Republicans to answer the phone or respond to online surveys at higher or lower rates than the other party? You must think through all these possible outcomes before accepting the data. And that’s an argument for treating polls with more uncertainty than we’ve treated them in the past. I think that’s a pretty obvious takeaway from the past few elections. But more importantly, how pollsters arrive at their estimates is truer. These are uncertain estimates at the end of the day; they are not the ground truth about public opinion. And that’s how I want people to think about it.
