Pollster ratings, Silver Bulletin style
They're back, baby — fully updated and ready to go for 2024.
I’m getting excited about the launch of the Silver Bulletin presidential election forecast later this month1 — and in preparation, I’ve recalculated the pollster ratings that the model employs. If you’re interested in signing up for the election forecast — the polling averages will be free, but the probabilities and other detail will be paywalled — you can sign up using the handy link below.
These ratings restore the methodology that I used for pollster ratings when I left FiveThirtyEight in 20232. I’ve also updated them with polls and election results since the last update, namely:
The 2023 gubernatorial elections;
Special elections to Congress;
The 2024 Republican presidential primaries3.
Because most of these elections weren’t very competitive and didn’t get polled all that much, there isn’t a whole heck of a lot of new data — only 63 polls since the last update. So the ratings have not changed very much. The top-rated pollsters — Selzer & Co. and The New York Times/Siena College — will be familiar to longtime readers. For the record, though, eligible polls in the 2024 presidential primaries missed the final margin between the top two candidates by an average of 9.7 percentage points.4 That isn’t very accurate, obviously — but it’s consistent with the long-term average error in presidential primary polls since 2000, which is 9.3 percentage points.
Since there aren’t any methodological changes, let’s just go ahead and get to the numbers. (If we’re being honest, I’m not too interested in engaging in public debates about the merits of individual pollsters anyway.) The columns in the pollster ratings table are as follows5:
An overall grade based on a pollster’s predictive plus-minus rating. Grades for pollsters with sparse data are rounded (e.g. to “A/B” rather than A-) and banned pollsters6 automatically receive a grade of F.
The predictive/plus minus rating itself, which is how accurate the model expects the pollster to be going forward based on a combination of its historical accuracy and its transparency/disclosure standards — pollsters get a bonus if they’re either a member of the AAPOR Transparency Initiative or share their data with the Roper Center archive. Negative plus-minus scores are good and imply that we expect the pollster to be more accurate than average going forward.
Mean-reverted bias — that is, a pollster's historical average statistical bias toward Democratic or Republican candidates, reverted to a mean of zero based on the number of polls in the database. This is calculated only for races in which exactly one Republican and one Democrat are the two leading candidates (so it doesn’t apply for presidential primaries, for instance). A score of "D +1.5", for example, means that the pollster has historically overrated the performance of Democratic candidates.
Finally, the number of polls included in the calculation. The database goes back to 1998, though polls from more recent years are weighted more heavily. In general, these ratings cover all polls in the 21 days prior to presidential, Congressional and gubernatorial general elections, and presidential primaries.
Finally, here are two files containing a substantial amount of additional data. First, here are more detailed stats for each pollster, such as how frequently they’ve “called” races correctly, and alternative versions of the plus-minus calculations.
And last, here is the raw data used to calculate the averages, including topline numbers from more than 11,500 polls. You are welcome to use this data free of charge for any purpose, but please attribute to Nate Silver and/or Silver Bulletin.
The next two weeks are going to be busy — I really want to get some version of the presidential forecast published before the debate, and I have to record the narration of my audiobook!7 — so posting volume may be sparse. Thanks for reading, and happy poll-watching.
The goal is to launch the forecast at some point during the week of June 24, as the first presidential debate will be held on June 27. There’s a fair amount of work still to do, but I think I’m on track to hit that deadline, with help from the fabulous new Silver Bulletin assistant elections analyst, Eli McKown-Dawson.
I know this may be a little confusing, but I continue to own the IP for all of the original FiveThirtyEight election models and forecasts. The remaining staff at 538 have created some cool new versions of their own, but I haven’t played a role in anything they’ve built since June 2023.
I had also originally endeavored to use data from the 2024 Democratic presidential primaries, such as they were. However there were very few polls for the Democrats — other than in New Hampshire, where Joe Biden was running as a write-in, a difficult circumstance for pollsters that I don’t think it’s fair to judge them on. And in other states, named candidates like Dean Phillips and Marianne Williamson often ran behind the “uncommitted” ballot line. Few if any of these polls would qualify by the various rules I apply to primary polls, so I just skipped the Democratic primaries entirely.
The polls also had a substantial pro-Trump bias, overestimating Trump’s margin against Nikki Haley by an average of 4.4 points.
Eli and I are in the midst of experimenting with Datawrapper for the tables and visualizations associated with the model this year — it’s a slightly tricky problem for a subscriber newsletter because about half of you read Silver Bulletin in an email client and the other half on the web. What’s supposed to happen is that people reading this in email will see a statistic graphic, whereas there’s an interactive component for the web version where you’ll be able to sort by different columns and so forth. (I’d always encourage email readers to click over to the web version to take advantage of the interactivity.) If this isn’t working for you — for instance, you’re not seeing any images at all in your email — please let us know, with as many details as you feel comfortable sharing about your email client or web browser setup.
I’m also reverting back any banned/unbanned decisions to the ones that were in place in spring 2023. Not in the mood to re-litigate these, sorry.
And it’s a long freakin’ book.
One thing about the tables is that when sorting by grade, A+ gets sorted after A, A-, and A/B. Not sure if there’s anything that can be done about that. Maybe adding spaces between the A and the plus?
Hey Nate, could you clarify what got the banned pollsters banned in the first place? Not looking for an itemized list explaining the offense of each one, just trying to understand what results in a ban.