Trust a pollster more when it publishes “outliers”

Just toss it in the average — because sometimes those outliers turn out to be right.

In late October 2022, the New York Times published a narrative-defying series of polls as part of its partnership with Siena College. The polls showed Democrats doing surprisingly well in four battleground House districts — including Kansas 3, the well-educated western suburbs of Kansas City in which the Democratic incumbent Sharice Davids was up by 13 points. The polls so defied expectations for a “red wave” — which had always been based more on the vibes than on the data — that even the Times’s writeup of the survey downplayed the findings.

You can probably guess what happened next. Democrats won all four districts, including Davids by a 12.1-point margin. The polls that had been slandered as outliers by Twitter armchair quarterbacks pretty much nailed it.

Lately, the Times/Siena surveys — the second-highest-rated outlet in the Silver Bulletin rankings — have also been defying expectations, although this time in a way that isn’t so welcome for Democrats. Their last two national polls showed Donald Trump ahead by one point and then a tied race, making them one of the few high-quality outlets with such mediocre numbers for Kamala Harris in the popular vote. Despite that, Harris was 4 points ahead in their survey of Pennsylvania. The good news for Democrats ends there, however. Trump led by 2 to 5 points in surveys of North Carolina, Georgia and Arizona that the Times released on Monday morning.

The polls had less of an effect on our model than I’d expected. Partly that’s because Harris was already narrowly behind in our projections of these states — and none of them are necessary for her minimum viable pathway to an Electoral College victory, which runs through Pennsylvania, Michigan and Wisconsin. Those states, plus the 2nd Congressional District of Nebraska — like Kansas 3, urban, white and well-educated — would suffice to give Harris 270 electoral votes. (A last-minute Republican push to revert Nebraska to a winner-take-all system looks to have fallen through.) And partly it’s that the Times/Siena Sunbelt polls were offset by a Wisconsin poll that showed Harris ahead by 6 to 7 points in the Badger State, which also came from a highly-rated firm, MassINC.

At least one of those polls is almost certain to be wrong. It’s not so unlikely that the election could tip heavily to Harris or Trump in the campaign's closing days. But you’d bet heavily against this combination of Democratic overperformance in Wisconsin and underperformance in Arizona. Out of the 40,000 simulations we ran on Monday morning, there were only 147 cases where Harris both lost Arizona by at least 5 points, matching the Times result, and won Wisconsin by at least 6, duplicating MassINC.

But that’s OK, because the model isn’t taking any of these numbers quite at face value, instead averaging them with other surveys of these states as well as its priors based on their demographics. On the basis of the new data, Harris’s win probability in North Carolina fell from 40 percent to 37 percent, in Georgia from 37 percent to 34 percent and in Arizona from 37 percent to 32 percent — but rose from 57 percent to 60 percent in Wisconsin. Outside of inflection points following major events like debates, changes in the model tend to be incremental.¹

Then again, what do we really know? Many polls have shown signs of racial depolarization this cycle, meaning that the white vote is getting bluer, while racial and ethnic minorities are growing more Republican. It probably won’t get to the point where Arizona and Wisconsin behave so disparately, but perhaps there will be more realignment in this race than we’re used to seeing. Some former swing states like New Hampshire and Florida could move completely off the map, while new ones like Alaska come into play.

And if that occurs, the place to look for hints of it will be in the polls — not the conventional wisdom about the race, which almost always rotely insists that what happened last time will happen again. The same types of people who are whining about the Times or MassINC polls are the ones who criticized Trump for campaigning in Wisconsin and Michigan in 2016 — oops! — or bought into the red wave narrative in 2022 that was never justified by the objective data.

In fact, when pollsters publish results like these, you should take that as a sign that they’re doing good and honest work. There’s frequently a lot of conventional wisdom baked into the polls, with pollsters either file-drawering seeming outlier results or massaging the data until they produce a number that’s socially acceptable. There’s clear statistical evidence of herding in the publication of polling results, meaning that pollsters tend to cheat by peeking off their neighbors and hedging toward whatever the polling averages say. (Because polling averages like Silver Bulletin potentially contribute to this problem, our pollster ratings actually punish polling firms that show clear signs of doing this.)

After all, it’s not fun to get yelled at by partisans on Twitter, who frequently have an amateurish understanding of statistical variation in polling. For instance, the Times polls of Georgia, Arizona and North Carolina had a combined margin of error of 2.5 percentage points on one candidate’s vote share — meaning that the margin of error is closer to 5 points on the difference separating the candidates. So if the polls average out to a Trump +3 race in the Sunbelt, that could easily enough result from a Harris +2 universe — or a Trump +8. (And one in 20 polls will fall outside the margin of error.)

Attempts to prove the polls wrong by diving into the crosstabs only makes matters worse. For instance, the Times poll of Arizona surveyed 149 Hispanic voters. That translates to a margin of error of 16 points on the difference separating Trump and Harris. 

Every poll will produce some unexpected results among small subgroups like these — unless the pollster is overly massaging the data, which some probably are. Some surveys are de facto closer to models than polls, in fact, with pollsters implicitly or explicitly importing a lot of priors into their numbers.

The problem with this is that while it may reduce the risk of an outlier from any one survey, it makes polling averages worse — in particular, they may miss the big demographic shifts that do occur in the electorate from time to time:

So perhaps Public Policy Polling sits at the opposite end of the spectrum from J. Ann Selzer. But herding may be hard to eradicate. The paradoxical-seeming reason is that herding can make the average poll more accurate even as it makes the polling average worse. (For economics nerds — this is sort of a tragedy of the commons problem.)

And over the long run, the pollsters that stand behind their seeming outliers are usually more accurate. Next to the Times, the pollster that has the biggest track record for conventional wisdom-defying polls is Ann Selzer. But those supposed outliers have often proven to be uncannily accurate, whether it’s Sezler’s survey that showed Barack Obama scoring a big win in the Iowa caucuses in 2008 or the landslide win she projected for Joni Ernst in the U.S. Senate race in 2014. That’s why Selzer is our highest-rated pollster — the only one to rank higher than the Times. Selzer’s most recent survey, showing Harris only 4 points behind in Iowa, was a bit of a surprise too — although it’s consistent with a story of racial depolarization.

Now, it’s true that you wouldn’t bet on Trump to win Arizona by 5 points — or Harris to win Wisconsin by 6 or lose Iowa by only 4. But you don’t have to engage in any elaborate unskewing or crosstab-diving to come to that conclusion. Just toss it in the averages.

Or if you don’t like the Silver Bulletin or 538 or RealClearPolitics averages, I’ll offer another alternative. Make your own average. Seriously, it’s not that hard. But I do have one stipulation: you have to publicly specify the rules ahead of time. I think you’ll find that when you’re forced to be consistent, to set standards that aren’t governed by your ad hoc sense of the vibes or by your partisan preferences, you’ll have a lot more sympathy for the polling aggregators — and you won’t be as surprised when one of the outliers turns out to be right.

1. An earlier version of this article imprecisely stated that Arizona has the fewest electoral votes of these states. It has fewer than Georgia or North Carolina — but more than Wisconsin.