Don't let randomness make a fool of you

If you care about what happens in this election, don’t sweat the polls — go vote.

Almost exactly a year ago, I wrote a post entitled “The election model is a little too popular”. Basically, it was me being a drama queen about whether I wanted to run a public-facing version of an election model at all this year. 

Pros: I was certainly going to be highly interested in the election. And the Silver Bulletin model — what was formerly the FiveThirtyEight model — is the best way I know for how to analyze what the polls are telling us, especially about the respective candidates’ chances of winning. There are other models and polling averages, some of which are quite good. But this was the only model out there that has any sort of a long-term track record. And the marginal cost of booting it back up wouldn’t be so high. I’ve accumulated a lot of expertise in psephology over 16 years of running election forecasts, and I think there’s some social utility in being able to walk people through the numbers in a way that’s honest, informed and transparent when a lot of other people are losing their minds.

Plus, there’s always been a huge audience for the model. And, finally free from the constraints of Disney, I had an opportunity to build a real business around it. The notion of a subscriber newsletter — a.k.a. a Substack — had a lot of appeal from the start. For a lot of reasons, including having a Matt Yglesias-like ability to write cogent copy quickly, I knew from the first time I heard about the platform that it was a good fit for me. What better way to build up an audience than to hire some smart young journalist like Eli and cover one of the most important elections in the nation’s history?

Cons: Although the physical process of running the model every day isn’t so much work, it’s still a big leap to so completely immerse yourself in a story. You’re strapping yourself into a roller-coaster ride with no eject button and without a lot of ability to foresee what’s coming around the next turn.

And there was some reputational risk: if our forecast was 80/20 on Election Day and the 20 percent came up — as 20 percent probabilities have an uncanny knack for doing 20 percent of the time — it was going to negatively affect my life. Not to the point where I was putting my whole reputational bankroll on the line — but tangibly, especially in a world where many people don’t understand the nature of probabilistic forecasts. And I had a good alternative: write my book and ride off into the sunset.

That part, though, was a calculated gamble, one that lends itself relatively well to an expected value framework. To some extent, for instance, the fact that there have always been a lot of bad-faith interpretations of polls and forecasts was a reason to run the model. Trust me: the haters will hate on you whether you’re right or not. So why let them ruin a product that many people — the regular readers of this newsletter — evidently find valuable?

There was another downside that I forgot about, though. An election is not just an incredibly important story but a story that many people understandably feel incredibly emotional about. The extremely partisan political climate in the United States makes all of this worse and enables people to rationalize a lot of abusive behavior. Polls and models become a vehicle for what psychologists call transference: basically, people displace all their anxieties about the election onto the forecasts and the people who design them.

Have you ever been to Milwaukee? I’m probably going to get some unsubscribes from Milwaukee-area readers, so let me just say up front that I like Milwaukee. Milwaukee’s great. My parents met at the University of Wisconsin, so I’ve always felt some affinity for the Badger State. But when I used to live in Chicago, I found that I’d wind up going to Milwaukee only about once a year. The reason is that it took about a year to forget that the cost of going to Milwaukee — booking a train ticket and a hotel for a couple of nights, plus the time it took to get there — wasn’t quite worth it when Chicago was also pretty great, and great in somewhat the same ways that Milwaukee is. (For instance, having a lakefront and a lively bar scene.)

Similarly, it takes about three years to forget about the emotional cost of being an avatar for people’s election-related anxieties. That’s a little harder to place a price on. And conveniently — or inconveniently — elections only happen once every four years. (Midterms don’t count: people aren’t nearly so insane about them.)

I thought I could thread the needle with a subscriber newsletter, and particularly by putting the probabilistic component of the model behind a paywall. We’d keep the probabilities our little secret among a self-selected and sympathetic group. But that part didn’t really work.

Or maybe the problem is that it worked too well.

I never expected the sort of traffic or subscriptions that we’ve wound up getting at Silver Bulletin. We’re the #3 Substack in our category and probably pretty close to that in the overall Substack ecosystem since I’m reliably informed that our category (“U.S. Politics”) is by far the biggest category. Without getting into specifics, I’ve never been so widely read, and the conversion from drive-by readers to paid subscriptions has gone well, too.

But that means the self-selected, let’s-keep-it-between-ourselves plan was doomed from the start. The paid subscribers are generally a reasonably well-behaved group, but they make up a minority of the audience for any post. And that’s before we get into how the Silver Bulletin forecasts are discussed on social media. With the audience that we have, the paywalled components of the model frequently “leak” and there isn’t really enough time in the day in an election year to police this. On Twitter in particular, there can be complete context collapse. People treat probabilistic predictions as deterministic ones, e.g. if Trump goes from a 48 percent chance of winning Wisconsin to a 52 percent chance, you’ll get a lot of Nate Silver is calling Wisconsin for Trump!!! even though the forecast expresses a high degree of uncertainty and nothing in the model has really changed. And that’s on a good day. There are a lot of partisans — some acting in good faith and some not — who can become literally conspiratorial about the model.

Look, the election is probably going to be close. The Biden-Trump election might not have been close, but Democrats were smart enough to replace their candidate, and the Harris-Trump election probably will be close.

Tomorrow or Saturday, I’ll have a newsletter (likely for paid subscribers) about interpreting the cornucopia of polls we’ve seen over the past 48 hours. But these polls aren’t telling us anything we didn’t know. On November 5th, we’ll all wake up with a lot of uncertainty about who will win. And we might go to bed with a lot of uncertainty, too: if recent American elections are any guide, the outcome could take several days to resolve. (There’s even almost a 10 percent chance of a 2000-style recount in a decisive state, the model figures.) 

Anything I say about the election, or anything any poll says, isn’t going to change that. If you care about the outcome, then vote, donate, volunteer, or try to persuade your friends. But don’t be an asshole on social media because it isn’t going to help.

And keep in mind that polls come with a margin of error. Let’s say that if we had Nostradamus-like abilities, we knew that the true state of the race is that Kamala Harris would win Wisconsin by 1 percentage point in an election held today. A typical poll has about 800 respondents. Well, the margin of error in an 800-person poll is plus or minus 3.5 points. Except, that substantially understates the case because the margin of error pertains only to one candidate’s vote share. In an election like this one where third-party candidates play little role, basically every vote that isn’t a Harris vote is a Trump vote and vice versa. So the margin of error on the difference separating the candidates is roughly twice that: about 7 points.

That means if the true state of the race is Harris +1, you’ll get some Trump +5s and Harris +7s just from sampling error alone (from surveying a random sample of the population rather than every voter). And that’s before getting into all the other ways that polls can go wrong. Or the fact that the margin of error is only supposed to cover 95 percent of the cases: about 1 time in 20, you’ll get a true outlier — a Harris +10 or a Trump +8 — provided that pollsters are being honest, which they probably aren’t. (In practice, pollsters tend to herd toward the consensus and suppress outlier results.)

Averaging different polls together helps, but only so much. Let’s say you have four recent polls of Wisconsin, for instance: this is equivalent to having a sample size of 3200 voters. The margin of error on the difference separating the candidates is still 3 or 4 points, so even if the true state of the race is Harris +1, the average might come out with a Trump +2 or a Harris +4.

Models like the Silver Bulletin model take some further steps, like by making inferences about what the polls in one state say based on polls of other states. That’s some of their value. But their more important function is by accounting for another possibility: the chance of a systematic polling error — of the sort that, for instance, led polls to substantially underestimate Trump in both 2016 and 2020. That’s where the probabilities in the forecast come from: they’re derived from how accurate the polls really have been, in practice — and in practice, the margins of error are considerably wider than they are in theory.

If one candidate has a big lead, there’s a bit less mental strain from all of this. Harris +8 doesn’t feel that different from Harris +4, and an 85 percent probability of her winning doesn’t feel that different than 75 percent. But when the probabilities are in the vicinity of even, it can seem like you’re ping-ponging between radically different universes. A forecast showing Harris as a 55/45 favorite will be interpreted much differently from one showing Trump as a 55/45 favorite instead, even though they’re basically saying the same thing: the election is more or less a coin flip.

If you enjoy what gamblers call the sweat, navigating the vicissitudes of these slightly changing probabilities, I get it, and the model is here for you. But if you don’t, there’s another option beyond voting, donating and volunteering: chilling the eff out. The dunk you make on Twitter isn’t going to make Trump lose. The nasty comment you leave isn’t going to make Trump win. You can unskew the polls, but usually, you’re only fooling yourself. So lead your best life, and have the serenity to accept the things you cannot change.