SBSQ #12: Will the polls lowball Trump again?

Plus, the mistakes I made in the “data journalism” era.

Cheers from London, where I’m wildly jealous of how much better the Indian food is than in the States but still terrified I'll look the wrong way and get run over every time I cross the street. And welcome to the August edition of Silver Bulletin Subscriber Questions. Now that we’re back on track after some schedule weirdness, you can leave questions for the September edition in the comments below.

Starting this upcoming week through Election Day on Nov. 5, we’ll publish two paid Model Talk columns per week instead of one. We still hope to have plenty of free content, too, averaging 4-5 pieces per week total, as well as model updates 6 to 7 times per week.

Originally, we'd planned to turn monthly subscriptions off at this point, which have higher churn and make longer-term planning more difficult. Instead, however, there will be a price increase on newly-initiated monthly subs beginning tomorrow (Sept. 1). This will not affect people who are already subscribed on either the monthly or annual plans: our policy at Silver Bulletin is that you get to keep the price you signed up for indefinitely so long as you remained continually subscribed. It will also not affect annual pricing, which will remain unchanged at $95/year.

There is zero subtlety here: we're trying to incentivize annual subscriptions by offering them at a considerable discount. With that said, we understand there are people here for the election forecast and not much else. We sincerely appreciate your business whether you’re in that category, are around for the long haul, or even just enjoy the frequent free posts. Even for paid posts like this one, we work hard to provide some meaningful content before the paywall line so that we’re not clogging your inbox. We have email open rates that average around 60 percent, which is high for this business, and that’s a metric we look at carefully to ensure we aren’t wasting your time.

In this edition of SBSQ, let’s keep it to three questions since the final two answers are quite long. (Perhaps we can return to a more lighting-round format for September.) I’ll probably also steal a couple of questions that didn’t quite make the cut for future Model Talk columns.

• What keeps you up at night about the model — and what lessons might there be for 2028?

• Will polls be biased against Trump again?

• Was “data journalism” a failure? What went wrong at FiveThirtyEight @ Disney?

What keeps you up at night about the model — and what lessons might there be for 2028?

Let’s start with a question from Jack Mumma in the subscriber Chat:

What does this 'unusual ' election give you a chance to learn about the model?

In some ways, the unusual events of this election — the most important of which, of course, was Joe Biden dropping out midstream — make it hard to learn long-term lessons since the circumstances may not be replicated. However, every election presents new challenges. Four years ago, we inserted various special provisions for COVID, such as reducing the convention bounce adjustment (figuring that mostly virtual conventions wouldn’t really “pop”) and widening the error bars in the forecast, given that turnout would potentially be harder to predict in a pandemic. I suppose I feel OK about those changes — the convention bounces were indeed small, and the polls were quite far off the mark. But it’s hard to know: we’ll never get an adequate sample size for what was (hopefully) a once-in-a-century pandemic.

This year, by contrast, we’ve tried to stick to the model’s preexisting logic as much as possible, even though the way events have piled on top of one another — such as RFK Jr. dropping out the day after the DNC — is not ideal. One reason the model has moved so much on new Pennsylvania polling, for instance, is because since earlier polling occurred before what the model considers to be two “landmark events” — the convention and a candidate exiting the race — so the model looks at older polling as being really out of date. Might it be overdoing things a bit? Sure. But I don’t think there’s anything inherently wrong with how it’s handling these circumstances, either, and it’s also naturally self-correcting (e.g. if/when Kamala Harris gets some better Pennsylvania polls, her numbers will improve). The model has closely followed prediction markets throughout these periods, which seems like a good sign.

The convention bounce adjustment is one of those things that keeps me up at night, though. It can make the model's behavior counterintuitive, e.g., Harris gaining in the polling average while losing ground in the forecast. Given that convention bounces are smaller than they used to be, maybe we could treat them as a rounding error in future years, like by saying that Harris’s numbers were likely to be a little inflated for a few weeks in the narrative explanations we provide, but not making any specific adjustment for the convention in the model itself.

But our goal here is not to flatter our (probably majority Harris-voting) audience but to give you the best forecast we can. Trump and Harris both did get convention bounces, according to our polling averages, although there were complicating circumstances in each case. (The assassination attempt immediately preceding the RNC, perhaps enhancing Trump’s convention bounce, and RFK dropping out, perhaps mitigating Harris’s). And I don’t see anything inherently wrong with the model’s hypothesis that this ought to be a strong period for Harris in the polling, so if the Electoral College is a dogfight even now, that’s not a great sign for her. The theme is “trust the process” — but of course, we’ll want to see how the rest of the process will play out.

Will polls be biased against Trump again?

Jeremy asks:

Question for SBSQ12: Which direction do you think polling errors will go. I've heard the simple arguments. Trump is a unique candidate, and he outperformed in 2016 and 2020, and therefore is more likely to do so again. And the counterarguments, polling errors are hard to predict, equally likely to affect either party, pollsters updated methods after 2020, and had a great year in 2022 (and 2018).

I know this is a tough question, and it's hard to go deeper, but assuming the model stays in this 50/50 range (a very likely scenario), this is what will decide the election. Could you do a bayesian analysis, given what we saw with polls in 2016/2020, and in 2022?

There were many versions of this question, so I picked Jeremy’s out of a hat. (Congrats, Jeremy.) Let me start by articulating precisely what assumption is embedded in the model: The model assumes that, by Election Day, polls won't be predictably biased.

Those details are important. It means that we think the polls, over the long run, will aim toward the center of the target. But it does not imply that they will have an especially precise shot. Rather, we think the situation looks roughly like the bottom-left corner of this chart. In any given election, the polls could be off by several points in any direction. A polling error of a few points is normal, in fact — it’s years like 2008 when polls nail the outcome with incredible precision that are unusual.

And there's some chance the polls could be off by more than a few points. Our estimate of Election Day polling error is based on an analysis of all election cycles since 1936. Landline telephone penetration only reached 75 percent of the US population by the mid-1950s, and before then (think “Dewey defeats Truman”), polling was often a very rough enterprise. So the model is trained on plenty of examples where the polls were far off the mark.

Moreover, the model assumes that to the extent there’s a polling error, it probably will be systematic, at least in part. If Trump overperforms our final forecast in Wisconsin, for instance, there’s a strong chance he’ll also do so in Michigan. But the same is true for Harris.

The clause “by Election Day” is also important. Until one day before the election, the Silver Bulletin forecast is not a pure polls-based model. Instead, it combines the polls with a “fundamentals” prior based on incumbency and our economic index. Currently, this prior estimates that the national popular vote “should” be roughly a tie — not Harris ahead by 3 or 4 points, as in recent national polls. Although the fundamentals are gradually getting phased out, they still get about 20 percent of the weight for now, which is shaving a net of about four-tenths of a point off of Harris’s projected finish on Election Day.

Furthermore, the model’s convention bounce adjustment explicitly assumes that the numbers for the candidate who just held their convention are likely to be inflated for a few weeks. That’s having a reasonably big impact on Harris’s forecast right now.

The upshot is that for the time being, the model thinks Trump is more likely than not to do better than his current polling. That does not mean it assumes the polls would be predictably biased if you held an election today — but the election isn’t today. It thinks Harris is more likely to face headwinds than tailwinds from here forward.

The model's interactions between state and national polls are also complicated. As I wrote, the model defaults toward a “polls-only” view of the race by Election Day. Or at least, that’s mostly true: economic data and the fundamentals no longer have any influence, and the convention bounce adjustment will long have worked its way out of the system. But the model somewhat liberally interprets this “polls-only” mandate: it uses state polls to inform its estimates of the national popular vote and national polls to inform its state-by-state polling averages. And it uses information based on demographic data and past voting patterns to smooth out the state-by-state estimates and come up with plausible maps.

In fact, our forecast of the national popular vote is not actually based on national polls. Rather, we project the results in each state, and then sum up these estimates, weighted by the projected turnout in each state, to come up with a national number. National polls have a substantial impact on the forecast — but it’s indirect based on the various adjustments the model makes, like its trendline adjustment to bring older polling data up to date.

Furthermore, our state-by-state forecasts aren’t purely based on state polling, but instead a combination of state polls and inferences the model makes from polling in other states and national polls.

Let’s say, for example, that the only poll we have of Hawaii shows Harris ahead by just 5 points. The model will blend that with an estimate based on a regression analysis using polling from other states and national polls, informed by demographic data and past voting patterns. If everything else looks normal, for instance, the model will be skeptical that Harris is really going to win Hawaii by just 5; maybe its regression estimate will say she should win by 25 points or something instead. So it will compromise, especially if the lone Hawaii poll is out of date or comes from a pollster with a low rating. In fact, it might still forecast Harris to win Hawaii by 15 or 20 points. Put differently, the model will assume that this particular Hawaii poll is biased — very biased — against Harris. That is to say, statistically biased: we assume nothing about the intent or integrity of the pollster.

In more robustly polled states, the state polling averages get the majority of the weight, and the regression estimate gets less. Indeed, often the overwhelming majority — by Election Day, anywhere from 80 percent of the estimate in fringy swing states like New Hampshire to the high 90s in others like Pennsylvania will come directly from our state polling averages. But the model does hedge if the polls in one state seem out of line with others and there isn’t much polling. This is actually one subtle reason the model has gotten a little bit bearish on Harris lately. Since there hasn’t been a ton of great state polling since the DNC and RFK’s exit, it defaults somewhat toward its regression-based estimates, which reflect the relatively large gap between the Electoral College and the popular vote that hurt Democrats in 2016 and 2020.

Phew. I won’t pretend the process is simple; it’s several thousand lines of code. I still haven’t really answered your question, though, Jeremy. Let’s say that by Election Day, the model’s estimate of the national popular vote is Harris +3. Should you assume that estimate is biased — that the “real number” should be Harris +2 or something — given that Trump overperformed in 2016 and 2020?