This is coming from a place of total ignorance, but has there been any effort by the polling industry to engage iPhone and Android teams to improve caller ID practices? Most don’t pick up calls from unknown numbers, but perhaps there would be a slight uptick if the call was marked “Survey” in some way? Have already started to see this with calls from doctor’s offices being marked as “Healthcare”, assume it could at least help response rates incrementally?
Nice job, Eli! Your writing's getting better. Not in a "your writing was bad" way, just.. it takes practice, and that practice is showing!
I deeply want to know the current demographic correlates on positive response to random digit dial sampling.
Hypothesis: landline-owning, caller-ID-not-using grandmothers who wish their grandkids would just, please, for once pick up the phone and give nana a ring. You don't even have to send a handwritten thank-you for that scarf she knit, just give her a call!
This article also suggests that pollsters should be a serious lobby for getting telemarketer scam calls under wraps. It's even incentive compatible for a politician trying to figure out what campaign to run! Know who the pollsters should send as their lobbyist? Ann Selzer.
PS not a dig on Selzer. Sometimes you swing and you whiff, but now small pollsters are just going to be more risk averse. Makes it hard to have an independent pollster that's not backed by a big institution (e.g The New York Times).
Suppose there were a movement to deliberately sabotage polls ... give wrong or random answers every time a pollster asks. How big a portion of the population would have to participate to put you all out of business?
10 - 15% of actual human beings, but it would likely get picked up earlier than people think because at that point the secret is nearly impossible to keep.
The bigger concern is polling cost vs. polling accuracy, which has been going up as data balkanization as started ramping up over the past 18 years, since the fall of the Landline telephone.
2008-2014 there was the belief that they had 'figured out' the issues, but that was when trolling and botting started driving people off of social media, which was always a bad standard to work from because of other factors.
There seems a persistent GOP, not just Trump, underestimation. After all the congressional generic was off about as much as the Presidential in the same direction. Nate had years back dismissed the shy Trump effect but I think it is more of a non responder Trump/Republican effect, where the GOP supporters are more anti institutional and more reluctant to answer pollsters.
It would seem that weighting for prior vote would help this, and I wonder if those polls that did weight for prior vote were any more accurate. That should not be hard to analyze.
2022 was a small bias towards Republicans over results,
which is why the democrats and many pollsters seemed more inclined to lean towards democrats in the final days of 2024
As someone doing the post game analysis on 2024 and being paid for the effort, I'd say it's less 'Shy Trump' voter and more 'Trump brings a different team to the yard.'
Trump voters are more politically agnostic, rebellious, and generally don't fit in the GOP or Democrat or even the independent box. They are, in effect, more the cynic then the true believers, and that messes with the way things are found and sorted.
They also aren't a large enough faction to easily nail down, probably about 3-4% of the voting population, with uneven distribution.
It's less 'getting them to talk to you' and more 'they don't fit the categories as designed'.
Which is why I think 2024 would be a dangerous base to build new models on, as Trump is unlikely to be directly on the ballot going forward, and most of his 'power' is in getting the Trump Voter into the voting booth in the first place.
As Nate outlined there is a big difference between midterms and presidentials. Your hypothesis sounds reasonable whatever you want to call those voters, but the one clear thing is they dont appear in the polls. btw a lot was made of 2022 because there were some high profile Senate Races that were called for the Rs and went D but overall the congressional generic was actually Dem biased by 1.6% where 538 had them Rs up by 1.2 and they won by 2.8.
I think a better model would locate these voters, and even if Trump is gone in 2028 I think those voters will still be in the GOP so if you miss them you will have the same problem
I don't work for 538, which is a model that had a liberal bias in 2022,
so don't lump me in with the hacks, please.
And I'll give you a comment taken from a Trump Voter, but is indicative of a larger chunk of Trump voters than you would think. He also votes Democratic when Trump isn't on the balance. Put these into your calculus for what happens after Trump:
"Your [Discussions Points] I think is the real reason the right won this election.
For this election the right was the side that was open to discussion.
And the left was the side of any discourse is hate speech.
And the side that I can talk to is more likely to do the things I want. Even if my values align with the opposite right now."
And
"Much of this administration is former leftist…" [In reference to Trumps choices]
And
"I generally think the left is more correct, as it is general progress.
Occasionally we need the right to comeback and undo the stupid the left does.
Occasionally we need the right to remind the left to stop being stupid.
Occasionally the left goes so stupid the left joins the right. Which is what also happened this past 4 years."
And his thoughts on who he's voting for in 4 years:
"I am going to guess only the left listens, but not well. In 2 years the right will make slight gains or maintain. Then the left will get their heads out of their asses make slight gains but not president, Biden will be too fresh. Then they will take things back."
For the record, I want to clarify that these are the OPINIONS of a VOTER and not the OPINIONS of myself, who they were discussing things with at the time.
He, and many of the voters who vote for Trump, do not view his as being Liberal or Conservative. They view Trump as the moderate who 'doesn't reward bad behavior.'
So why would they vote Conservative without Trump and without the "Abusive Democratic Institutions" to motivate them.
My impression of the "Trump effect" theory is that he's bringing out a class of voters that wouldn't come out for any other politician and doesn't respond to polling, but once they're in the voting booth they'll vote for the GOP down ballot. If that's correct (and assuming that Trump isn't up again in 2028 for some reason), then we would expect it to not apply for any future elections.
An alternative option is that Trump/the MAGA movement created this class of voters that is willing to come out for the GOP in presidential elections but isn't really willing to come out for midterms, in which case the effect will likely continue into the future.
I suspect some of both. Trump does have the edge on the less informed voters, who are also the ones who skip midterms, and by Trump I think more the MAGA movement which might linger past Trump.
I sort of agree with you: it could be Trump is a unicorn and the next GOP candidate won't have his star power.
You make sense about the danger of the 2026 midterms for the GOP: so I'd predict a push to get voters to the polls as a show of loyalty to Trump, so he can hope to keep the House.
Public political polling consistently favors Democrats because public political pollsters and their employees are Democrats who prioritize desire over accuracy. Accuracy only becomes an objective as the election approaches and pollsters wish to avoid looking incompetent. The underestimation of the Republican vote in 2016, 2018, 2020, 2022 and 2024 is not an anomaly. It is possible to achieve accuracy but not at the expense of the Democratic candidate. Polling is not science, it is marketing for the Democratic Party.
Just have to chuckle at all the self-serving commentary here. The mathematical wizards who run the polls just flat out miss the forest for the trees. Most people outside of the tiny little echo chamber of polling data nerds don't really care a rat's ass about margin of error, herding, etc. In the end, they care about the big picture, as in:
2016: Polls predicted Hillary by a landslide (WRONG)
2020: Polls predicted Biden by a landslide (WRONG)
2024: Polls predicted a razor thin photo finish (WRONG)
Other than that Mrs. Lincoln, everything is just fine. Who really cares if the results were within the "margin of error"?
Finally, the author ruminates about the Trump underestimation factor. It is well established now (well, I guess not among the pollsters who are curiously quiet about this) that the French Whale who won $85 million in the betting markets just simply outsmarted ALL the pollsters with his simple (yet profound) methodology to capture the impact of the "shy Trump voter".
Right. We lay people want to know who will win, and they messed that up big time. Over and over. So we don't care about how they almost did better, etc., etc.
Actually, it was pretty awful! I won't take any interest again. It's all to get the money from people who hire them, but it doesn't work. And often (not Nate, but many) it isn't even honest.
Great column. Nate is digging into a question I had which is why is so hard to eliminate a bias (against Trump in this case) in the polls. It was like Lucy and the football — every election we think the bias has been eliminated or even reversed but it remains. I am comparing this to measurements, manufacturing, and control, where eliminating biases is easier. The chart showing how polling methodology is in flux clearly points to one challenge — the “same” poll may be using different methodology in 2028.
This was an optimistic article about polling, but I am a total nihilist since about halfway through the last election. Now that we have the final results of the bad polling, I can decide polling is the pits and never to pay any attention to it again.
It's sad the worse polling gets, the more pollsters there are! I suppose that's the incessant demand for 24/7 "news" and the quality of the news simply doesn't matter: just how many clicks it gets. So I suppose there will be more and more and worse and worse polls till they are all that that Iowa poll that was 17 points off ----------------- against Trump, of course. And he's suing, and I think he should.
Of course given mass innumeracy (and particularly journalists innumeracy and general misunderstanding of probabilistic presentation [which one has to admit is not natural to human reflection so really hard to genuinely integrate]) the perception problem will continue from base misunderstanding of polls as oracular predictions, probably permanently so long as elections are very close such that the margin of the result is within the error cone.
If Eli is right, we should stop calling it polling and start calling it “estimating.” To most people (I think) poll means the unadjusted results of a survey of a representative group of people. I’ve learned from Eli and Nate that it doesn’t really work that way anymore (and hasn’t for quite some time) and that’s ok.
This is coming from a place of total ignorance, but has there been any effort by the polling industry to engage iPhone and Android teams to improve caller ID practices? Most don’t pick up calls from unknown numbers, but perhaps there would be a slight uptick if the call was marked “Survey” in some way? Have already started to see this with calls from doctor’s offices being marked as “Healthcare”, assume it could at least help response rates incrementally?
This already has been attempted,
but the results were people were *less likely* to pick up for a Survey than a random call.
People don't really want to spend the time normally to take a survey for some unknown cause, generally.
This is on top of the problem that the people most likely to respond are the people who were already easy to reach in general.
The psychology of people is a weird thing.
Nice job, Eli! Your writing's getting better. Not in a "your writing was bad" way, just.. it takes practice, and that practice is showing!
I deeply want to know the current demographic correlates on positive response to random digit dial sampling.
Hypothesis: landline-owning, caller-ID-not-using grandmothers who wish their grandkids would just, please, for once pick up the phone and give nana a ring. You don't even have to send a handwritten thank-you for that scarf she knit, just give her a call!
This article also suggests that pollsters should be a serious lobby for getting telemarketer scam calls under wraps. It's even incentive compatible for a politician trying to figure out what campaign to run! Know who the pollsters should send as their lobbyist? Ann Selzer.
PS not a dig on Selzer. Sometimes you swing and you whiff, but now small pollsters are just going to be more risk averse. Makes it hard to have an independent pollster that's not backed by a big institution (e.g The New York Times).
Thanks! And there's lots of interesting research on how survey mode affects the demographic composition of samples. Here are a few good places to start of you're interested: https://academic.oup.com/poq/article-abstract/73/4/641/1830107, https://www.pewresearch.org/methods/2015/05/13/from-telephone-to-the-web-the-challenge-of-mode-of-interview-effects-in-public-opinion-polls/, https://www.cambridge.org/core/journals/political-analysis/article/abs/nonresponse-and-mode-effects-in-self-and-intervieweradministered-surveys/F9F43D3FB6EDE5DA2294F21E385823D0
My Lazy Sunday reading! Thanks, Eli!
Suppose there were a movement to deliberately sabotage polls ... give wrong or random answers every time a pollster asks. How big a portion of the population would have to participate to put you all out of business?
Part of this would depend on how well pollsters were able to identify these responses. Good pollsters already take steps to weed out fraudulent responses (checking that the respondent lives in the state they claim to live in, including attention-check questions, various flags to remove bots, etc.). This sort of thing is another issue pollsters (especially those who poll online) are trying to fix: https://www.pewresearch.org/short-reads/2024/03/05/online-opt-in-polls-can-produce-misleading-results-especially-for-young-people-and-hispanic-adults/
10 - 15% of actual human beings, but it would likely get picked up earlier than people think because at that point the secret is nearly impossible to keep.
The bigger concern is polling cost vs. polling accuracy, which has been going up as data balkanization as started ramping up over the past 18 years, since the fall of the Landline telephone.
2008-2014 there was the belief that they had 'figured out' the issues, but that was when trolling and botting started driving people off of social media, which was always a bad standard to work from because of other factors.
Great post Eli, thanks!
There seems a persistent GOP, not just Trump, underestimation. After all the congressional generic was off about as much as the Presidential in the same direction. Nate had years back dismissed the shy Trump effect but I think it is more of a non responder Trump/Republican effect, where the GOP supporters are more anti institutional and more reluctant to answer pollsters.
It would seem that weighting for prior vote would help this, and I wonder if those polls that did weight for prior vote were any more accurate. That should not be hard to analyze.
I'd disagree,
2022 was a small bias towards Republicans over results,
which is why the democrats and many pollsters seemed more inclined to lean towards democrats in the final days of 2024
As someone doing the post game analysis on 2024 and being paid for the effort, I'd say it's less 'Shy Trump' voter and more 'Trump brings a different team to the yard.'
Trump voters are more politically agnostic, rebellious, and generally don't fit in the GOP or Democrat or even the independent box. They are, in effect, more the cynic then the true believers, and that messes with the way things are found and sorted.
They also aren't a large enough faction to easily nail down, probably about 3-4% of the voting population, with uneven distribution.
It's less 'getting them to talk to you' and more 'they don't fit the categories as designed'.
Which is why I think 2024 would be a dangerous base to build new models on, as Trump is unlikely to be directly on the ballot going forward, and most of his 'power' is in getting the Trump Voter into the voting booth in the first place.
As Nate outlined there is a big difference between midterms and presidentials. Your hypothesis sounds reasonable whatever you want to call those voters, but the one clear thing is they dont appear in the polls. btw a lot was made of 2022 because there were some high profile Senate Races that were called for the Rs and went D but overall the congressional generic was actually Dem biased by 1.6% where 538 had them Rs up by 1.2 and they won by 2.8.
I think a better model would locate these voters, and even if Trump is gone in 2028 I think those voters will still be in the GOP so if you miss them you will have the same problem
I don't work for 538, which is a model that had a liberal bias in 2022,
so don't lump me in with the hacks, please.
And I'll give you a comment taken from a Trump Voter, but is indicative of a larger chunk of Trump voters than you would think. He also votes Democratic when Trump isn't on the balance. Put these into your calculus for what happens after Trump:
"Your [Discussions Points] I think is the real reason the right won this election.
For this election the right was the side that was open to discussion.
And the left was the side of any discourse is hate speech.
And the side that I can talk to is more likely to do the things I want. Even if my values align with the opposite right now."
And
"Much of this administration is former leftist…" [In reference to Trumps choices]
And
"I generally think the left is more correct, as it is general progress.
Occasionally we need the right to comeback and undo the stupid the left does.
Occasionally we need the right to remind the left to stop being stupid.
Occasionally the left goes so stupid the left joins the right. Which is what also happened this past 4 years."
And his thoughts on who he's voting for in 4 years:
"I am going to guess only the left listens, but not well. In 2 years the right will make slight gains or maintain. Then the left will get their heads out of their asses make slight gains but not president, Biden will be too fresh. Then they will take things back."
****************************************************************************************
For the record, I want to clarify that these are the OPINIONS of a VOTER and not the OPINIONS of myself, who they were discussing things with at the time.
****************************************************************************************
He, and many of the voters who vote for Trump, do not view his as being Liberal or Conservative. They view Trump as the moderate who 'doesn't reward bad behavior.'
So why would they vote Conservative without Trump and without the "Abusive Democratic Institutions" to motivate them.
My impression of the "Trump effect" theory is that he's bringing out a class of voters that wouldn't come out for any other politician and doesn't respond to polling, but once they're in the voting booth they'll vote for the GOP down ballot. If that's correct (and assuming that Trump isn't up again in 2028 for some reason), then we would expect it to not apply for any future elections.
An alternative option is that Trump/the MAGA movement created this class of voters that is willing to come out for the GOP in presidential elections but isn't really willing to come out for midterms, in which case the effect will likely continue into the future.
I suspect some of both. Trump does have the edge on the less informed voters, who are also the ones who skip midterms, and by Trump I think more the MAGA movement which might linger past Trump.
I sort of agree with you: it could be Trump is a unicorn and the next GOP candidate won't have his star power.
You make sense about the danger of the 2026 midterms for the GOP: so I'd predict a push to get voters to the polls as a show of loyalty to Trump, so he can hope to keep the House.
Public political polling consistently favors Democrats because public political pollsters and their employees are Democrats who prioritize desire over accuracy. Accuracy only becomes an objective as the election approaches and pollsters wish to avoid looking incompetent. The underestimation of the Republican vote in 2016, 2018, 2020, 2022 and 2024 is not an anomaly. It is possible to achieve accuracy but not at the expense of the Democratic candidate. Polling is not science, it is marketing for the Democratic Party.
We have a "trust default" and it's almost impossible to see what you are saying. But I think you are right.
Just have to chuckle at all the self-serving commentary here. The mathematical wizards who run the polls just flat out miss the forest for the trees. Most people outside of the tiny little echo chamber of polling data nerds don't really care a rat's ass about margin of error, herding, etc. In the end, they care about the big picture, as in:
2016: Polls predicted Hillary by a landslide (WRONG)
2020: Polls predicted Biden by a landslide (WRONG)
2024: Polls predicted a razor thin photo finish (WRONG)
Other than that Mrs. Lincoln, everything is just fine. Who really cares if the results were within the "margin of error"?
Finally, the author ruminates about the Trump underestimation factor. It is well established now (well, I guess not among the pollsters who are curiously quiet about this) that the French Whale who won $85 million in the betting markets just simply outsmarted ALL the pollsters with his simple (yet profound) methodology to capture the impact of the "shy Trump voter".
https://www.thefp.com/p/french-whale-makes-85-million-on-polymarket-trump-win
Way too much statistical minutiae, and way too little common sense.
Right. We lay people want to know who will win, and they messed that up big time. Over and over. So we don't care about how they almost did better, etc., etc.
Actually, it was pretty awful! I won't take any interest again. It's all to get the money from people who hire them, but it doesn't work. And often (not Nate, but many) it isn't even honest.
Great column. Nate is digging into a question I had which is why is so hard to eliminate a bias (against Trump in this case) in the polls. It was like Lucy and the football — every election we think the bias has been eliminated or even reversed but it remains. I am comparing this to measurements, manufacturing, and control, where eliminating biases is easier. The chart showing how polling methodology is in flux clearly points to one challenge — the “same” poll may be using different methodology in 2028.
This was an optimistic article about polling, but I am a total nihilist since about halfway through the last election. Now that we have the final results of the bad polling, I can decide polling is the pits and never to pay any attention to it again.
It's sad the worse polling gets, the more pollsters there are! I suppose that's the incessant demand for 24/7 "news" and the quality of the news simply doesn't matter: just how many clicks it gets. So I suppose there will be more and more and worse and worse polls till they are all that that Iowa poll that was 17 points off ----------------- against Trump, of course. And he's suing, and I think he should.
Interesting review of the practices.
Of course given mass innumeracy (and particularly journalists innumeracy and general misunderstanding of probabilistic presentation [which one has to admit is not natural to human reflection so really hard to genuinely integrate]) the perception problem will continue from base misunderstanding of polls as oracular predictions, probably permanently so long as elections are very close such that the margin of the result is within the error cone.
If Eli is right, we should stop calling it polling and start calling it “estimating.” To most people (I think) poll means the unadjusted results of a survey of a representative group of people. I’ve learned from Eli and Nate that it doesn’t really work that way anymore (and hasn’t for quite some time) and that’s ok.
But it’s not a Poll. It’s an estimate.
Maybe it's just a Trump effect and will be gone going forward.