Polling averages shouldn't be political litmus tests

And they need consistent standards, not make-it-up-as-you-go

I’m now a free agent! Yesterday was the last day of my contract with Disney, which is in the midst of a series of layoffs including most of the FiveThirtyEight staff. Having worked for the company for nearly 10 years, I wish them well and thank them for the opportunity.

I tried to avoid writing too much about politics or media while I was still cashing a Disney paycheck. And I don’t expect that to change too much in the short run around here. My main priorities remain a) finishing my book and b) figuring out the long term. There’s no news to share on that front just yet, though there are a lot of promising leads.¹ And yes, I’m very optimistic that the long-term will involve — among other things — finding a home for the sports models that FiveThirtyEight has stopped updating.²

But I did want to address something that happened last week, involving the scaled-down version of FiveThirtyEight that remains at Disney/ABC News. If you follow me, you probably saw my tweet about it so it won’t be news, but I’m going to articulate a more full-fledged version of my critique.

This past week, the new Editorial Director of Data Analytics at ABC News, G. Elliott Morris, who was brought in to work with the remaining FiveThirtyEight team, sent a letter to the polling firm Rasmussen Reports demanding that they answer a series of questions about their political views and polling methodology or be banned from FiveThirtyEight’s polling averages, election forecasts and news coverage. I found several things about the letter to be misguided.

For clarity, any methodological changes that Morris is planning will only affect the new averages and products that he’s creating for ABC News. I still own the original version of the FiveThirtyEight models that you’ve loved and/or hated for the past 15 years, and am planning on employing them in some productive capacity in 2024.³

So why should I care? I’ve tried over the past few years to disengage from “model wars” debates over polling aggregation and election forecasting topics. I don’t feel like I’m learning much from them and I don’t think they’re particularly high-stakes questions.

The Internet has a memory so there’s also not much point in hiding that I really don’t like either of the parties to the letter. I’m not a fan of Rasmussen controlling shareholder Ted Carroll, who has been nasty to me over the years.⁴ Nor am I a fan of Rasmussen’s leading questions related to the 2020 and 2022 elections, which seem like a way to provide validation to election conspiracy theories.

Morris, meanwhile, is one of only something like four people whom I have blocked on Twitter. This is an extremely high bar for me — it’s easier to just mute or unfollow someone. I only undertake after many rounds of going back and forth with someone when I feel like arguing with them has become an asymmetrical quagmire. So take that as a signal that I don’t intend this a back-and-forth.

Still, I felt like I needed to say something, and explain myself a little more thoroughly here than I originally did on Twitter. The reason is that Morris’s approach has the potential to create negative downstream consequences for me because of possible brand confusion. There are still people who think I write for The New York Times, which I last worked for 10 years ago. So if “FiveThirtyEight” is in the news because Morris or ABC News is making decisions I strongly disagree with, that creates a potential issue. (To be clear, unlike the models — which they merely licensed — Disney bought the FiveThirtyEight trade name from me. So I have no legal recourse. But I have the right to complain, like I’d complain if Disney introduced the FiveThirtyEight Swing State Swirl spinning teacup ride at the Magic Kingdom.⁵ Actually, that sounds kind of fun.)

Enough throat-clearing? Let’s hope so. My critique of Morris’s letter to Rasmussen is fairly straightforward.

First, I strongly oppose subjecting pollsters to an ideological or political litmus test. Look, there might be good reasons to exclude Rasmussen based on their methodology, although I’d note that their track record of polling accuracy is average, not poor. But that’s not the question that Morris leads with in his letter to Carroll. Instead, it’s this: one

First, Rasmussen must explain the nature of its relationship with several right-leaning blogs and online media outlets, which have given us reason to doubt the ethical operation of the polling firm. please tell us whether questions are ever suggested to Rasmussen from these outlets, including Fox News and "Steve Bannon's War Room", where Rasmussen's head pollster regularly appears, with the promise of coverage in return for "public" fieldwork? Do Rasmussen's pollsters work with anyone from these organizations on topics to consider polling, despite listing polls as un-sponsored or sponsored by other groups? Does the pollster have a close personal relationship with any of these figures that might cloud their judgement in the operation of a public poll? 

I’m trying to wrap my head around this. Why, unless you’re a dyed-in-the-wool left-leaning partisan, would having a “relationship with several right-leaning blogs and online media outlets” lead one to “doubt the ethical operation of the polling firm”? Many pollsters have strong political opinions — if they didn’t, they’d probably have chosen another profession — and they are friendly with political operatives of one stripe or another. Some of them, of course, poll for campaigns themselves.

If you wanted to argue that pollsters and other people working in public opinion should be more austere about disclosing their political opinions, I could maybe get on board with that. But if that were the basis for banning pollsters, you’d have to wipe out dozens of pollsters from the FiveThirtyEight ledger. (And frankly, it’s not the sort of philosophy that Morris tends to espouse.⁶) I guess I can’t know for sure, but I strongly doubt that Morris would view being friendly with left-wing political figures as something that might “cloud their judgement in the operation of a public poll”.

Besides, the proof is in the fucking pudding! Unlike most people in politics or journalism, pollsters have a track record that can be tested against actual results. They can be held accountable for political bias that might render their polls less accurate. Rasmussen has indeed had strongly Republican-leaning results relative to the consensus for many years. Despite that strong Republican house effect, however, they’ve had roughly average accuracy overall because polls have considerably understated Republican performance in several recent elections (2014, 2016, 2020).

Is that a case of two wrongs making a right — Rasmussen has had a Republican bias, but other polls have had a Democratic bias, so they come out of the wash looking OK? Yeah, probably. Still, there are ways to adjust for that — statistical ways like a house effects adjustment, not ways that involve acting like you’re the House Un-American Activities Committee.

Second, even if you’re going to remove Rasmussen from the averages going forward, it’s inappropriate to write them out of the past, as Morris has threatened to do.

Quoting from the letter again; emphasis mine.

I am emailing you to send a final notice that FiveThirtyEight is considering formally banning Rasmussen Reports from its coverage. Such a ban would result in being removed from listing on our main polls page and being excluded from all of our aggregation and election forecasting models. If banned, Rasmussen Reports would also be removed from our historical averages of polls and from our pollster ratings. Your surveys would no longer appear in reporting and we would write an article explaining our reasons for the ban. 

It’s bad practice to revise data that’s already been published, based on decisions you made long after that data was published. For one thing, it makes your numbers less reliable as a historical record. For another, it can lead to overconfidence when using that data to train or build models. If you clean up a past data set based on information that wouldn’t have been available to you at the time, it neglects the challenges you face when encountering new data, which will also be full of edge cases and minor errors that you’d have removed with the benefit of hindsight, but you won’t necessarily catch in advance.

There’s also an implicit conflict here about the degree to which journalists should gatekeep or shield the public from potential sources of “misinformation”. You’ll see my skepticism about this idea by the fact that I put the term misinformation in scare quotes. I’m skeptical about misinformation being a useful analytic construct, And I’m skeptical that any of this is helping restore to public trust in journalism; I think people have a pretty good eye for detecting censoriousness and become less trusting toward sources that engage in it. But that’s an argument for another day. I just wanted to flag that there are some parallels between debates about how to construct polling averages and broader journalistic arguments in the ether.

Third, I think it’s clear that the letter is an ad hoc exercise to exclude Rasmussen, not an effort to develop a consistent set of standards.

After scrutinizing his political relationships, Morris asks Carroll a series of methodological questions. But most of them are highly esoteric questions that don’t point toward anything unusual that Rasmussen is doing; they could be asked of many if not all pollsters. For instance:

Your methodology states, "Calls are placed to randomly-selected phone numbers through a process that ensures appropriate geographic representation." What is the process being applied? And what does "randomly-selected" mean here? If not RDD, where are you getting your call lists?

I’ve rarely seen a pollster describe its randomization process in this much detail, or be asked to do so.

The methodology mentions you weight by "age, race, gender, political party, and other factors." What are the other factors?

It’s fine to ask about this, but I’d imagine many pollsters have boilerplate language in describing their methodology (“other factors”, “and so forth”, “and similar factors”). This is normal and benign language.

The methodology also states, "For political surveys, census bureau data provides a starting point and a series of screening questions are used to determine likely voters. The questions involve voting history, interest in the current campaign, and likely voting intentions." Does this mean you are weighting first and screening second? If so, is there additional rebalancing for the LV sample? For example, women are less likely than men to say they're definitely going to vote, but they usually make up at least half of the electorate anyway.

This is very in-the-weeds, and I’m sure if Rasmussen hired Morris to improve its weighting methods, he’d have some good ideas. But this asks for well beyond the level of disclosure that a pollster typically provides, and a commercial polling organization would be within its rights to keep some of the details of its weighting algorithms proprietary.

Pretty much the whole series of questions is like this. They’re nitpicks. Some of them are nitpicks you could ask of almost any pollster. Some of them are more Rasmussen-specific nitpicks — but you’d also have some, say, Quinnipiac-specific nitpicks if you were asking questions of Quinnipiac.

The thing about running a polling average is that you need a consistent and legible set of rules that be applied to hundreds of pollsters you’ll encounter over the course of an election campaign. Going on a case-by-case basis is a) extremely time-consuming (don’t neglect how busy you’ll be in the middle of an election campaign) and b) highly likely to result in introducing your own biases, whether it’s the political outcome you’re rooting for or whatever you think will make your model look smart. That’s why, after 15 years of doing this, I’ve been a stickler for consistency, even if that means including some pollsters whom I subjectively don’t like, politically or methodologically.

Perhaps Morris’s questions were getting at some larger theme or more acute problem. But if so, he have should stated it more explicitly in his letter. Journalists, in most circumstances, shouldn’t act like Vincent D'Onofrio in Law & Order trying to sniff around for clues or throw a suspect off-kilter. Ask clear, concise questions that make your intentions clear.

Instead, this looks like a fishing expedition, with Morris hoping to catch Rasmussen in some sort of venial methodological sin that is probably fairly common within the industry. Or, because the questions are onerous, the tone of his email is hostile, and Carroll was only given a day-and-a-half to respond just before a four-day summer weekend, he was hoping that thy wouldn’t be answered at all — so he could say “See! They refused to answer my questions!”. Either way, this is the letter you get only once someone has already made their mind up.⁷

1. If you want to get in touch, I’m at NATE DOT SILVER DOT MEDIA AT GMAIL DOT COM.

2. To be more precise, there are some FiveThirtyEight sports models — like the NHL model — that I didn’t have a hand in creating and have no ongoing rights to. But I was a primary statistical author of many of them and have a copy that I can sell or license elsewhere.

3. That presumably means for public consumption although I’ve gotten some feelers about taking them private for trading/betting purposes. As a gambler at heart, it’s not not tempting.

4. Rasmussen’s eponymous founder, Scott Rasmussen, left the company in 2013.

5. Blame ChatGPT for this suggestion.

6. It’s my understanding that he’d like to nudge FiveThirtyEight in a more explicitly progressive direction.

7. My strong advice, as someone who is a journalist but has also sometimes been the subject of hostile journalistic coverage, is that when you encounter a situation like this, you probably shouldn’t respond. If you catch a whiff of bad faith, trust your instincts. Ignoring is often the best options. Or take it to your bosses, your lawyers, your PR people — or preempt it by front-running the story.