And is for the “overcompensating” theory, while it is not easy to prove this but I wouldn’t be too surprised if they worry way more about missing in the direction of understanding GOP than the other way around.
For one thing, Dems esp the ones who consume news tends to be the neurotic ones who is more paranoid about polls underestimating Trump than the other way around. They prob won’t yell and scream if the polls underestimate Dems.
For another Trump himself and Too Online MAGAs also scream when they are underestimated on the poll (like they literally sued Ann Seltzer) while they don’t call polls fake when GOP is overestimated.
So, underestimating GOP/Trump is almost a recipe for being yelled and heckled by those two groups while the other way around is not - and many polled at the end of the day are sponsored by the media, who have to cater to the demand of the audience as well as wanting to minimize the risk of unwanted lawsuits.
(The case for Trafalgar tho is prob somewhat different - I don’t think it is super far fetched to call them basically GOP hopium pollsters where the methodology is very opaque and have track record of wild misses in off year elections. Not quite sure if I go as far to call them making up numbers to match the narrative they try to push but I think there are some smokes out there?)
There are far more professional ramifications for underestimating the Republicans right now than the Democrats.
The irony was the 2024 elections were pretty solid by all measures, but people still complained that they underestimated conservatives. Thus, polling shifts to a more conservative baseline, which means that people begin making decisions on the belief that conservatives are stronger than they are and that Democrats are weaker than they are, leading to 'surprises' during the coming years.
And I think this prob does have negative consequences that they have not thought about.
This is by no means the only reason of not even the main reason but one reason I *think* why Trump and MAGAs (I guess honestly more so for MAGAs than Trump himself) has acted way more brazen than the actual election result suggests is, the polls have not shown as bad of a result. I confess I have no idea how “off” Trump approval rate is rn in comparison with NJ and VA gubernatorial polls but I also won’t be too surprised that is the case.
Ngl, I have been rather fascinated that RV polls of Trump approval have been a smidge better than all adults poll - and this is a bit counterintuitive considering how high engaging ones has become more Dems.
I can think of two not mutually exclusive theories
1. Low engaging voters actually soured on Trump a lot so the Dems edge in high engaging voter in comparison with low engaging one decreased since 2024 (which I think is confirmed indirectly by the fact that young ppl and racial minorities bounced hard back from Trump)
2. The voter model is off - they make the projected voter to be much more conservative than it actually is
I wonder both are happening more or less but I am eager to be proven wrong on this!
Rating polls by difference from election results may make sense for some purposes, but it ignores the polls' claimed standard errors. If a poll predicts a Republican victory by six percentage points, with a standard error of two percentage points, a Democratic victory is very strong statistical evidence that either the poll's estimate or its standard error were wrong. But with a four percentage point standard error, the result is consistent with an accurate poll.
For most of us not involved in polling, the most important information in a poll is the implied probability of the election result. A useful metric of relative poll accuracy is how much money you would make or lose betting using one poll's implied probabilities for payouts and the other for bets. You can also compare polls to prediction market prices and expert judgments.
Unfortunately, since polls are taken over somewhat different time intervals you can't always find direct head-to-head comparisons. But that's also true of the metric used in the post.
You're correct about the margin of error being important. The metrics we use for our pollster ratings (Advanced Plus-Minus and its simpler counterparts) do account for sampling error. Basically, we estimate how large a poll's error should be based on things like sampling error, when it was conducted, how hard the race is to poll, etc. and compare that to how large the error actually was. You can read more here: https://fivethirtyeight.com/methodology/how-our-pollster-ratings-work/
“It’s easy to imagine people being more upset with the 2025 polls if they once again underestimated Democrats. Instead, the polls were too bullish on Republicans, on average.”
I’m trying to understand what this pair of sentences means…can anyone help me out?
I'm pretty certain it's a typo and should say "once again overestimated Democrats" - because the polls have been overestimating Democrats for the last three presidential elections.
Nate love the model and evaluation posts! Not sure how your search for a copy editor is going but in the third graf below the first table I think you say the opposite of what you mean. Should be either again underestimate Republicans or again underestimate Democrats as a contrast to being too bullish on republicans in reality. Let me know 😀 that’s my job!
Am I misinterpreting or is there a significant error with how the House races have been coded?
There is a line in the article that says "Instead, the polls were too bullish on Republicans, on average." And then the associated footnote (#3) says "Technically, the two 2025 House polls we added to our database did underestimate Democrats by 8.4 points on average, but that mostly comes down to the blue team overperforming expectations in FL-06 and TN-07."
At some level, I don't understand the footnote -- if the the polls were too bullish on Republicans across the board (i.e. both governor and House), why is this footnote needed?
And then more importantly -- if the two House polls *underestimated* Democrats (as the footnote says), why does the table show a D+8.4 bias, which would suggest those polls *overestimated* Dems? Shouldn't that be a +8.4 R bias?
I'm ultimately confused by what this is trying to say -- were the House polls too good for Dems (and thus cut against the trend of governor polls being too good for the GOP), or were all the polls too good for the GOP?
That was a dumb typo on my part: underestimate should have been overestimate in the footnote. I've fixed that now. The reason I included the footnote is because the second chart shows a large Democratic bias in the 2025 House polls, and I wanted to clarify that it's being driven by only two polls.
The polling bias towards Democrats in presidential races has only been a regular thing since 2016. Polling underestimated Obama in 2012. That’s why I think it’s a Trump issue as he does well with low propensity voters.
He does well with voters on the outside of the two teams.
I could tell he was a danger in 2015 when members of my libertarian circles who I knew mostly voted to write in Mickey Mouse started telling me "I'm voting Trump, and you should vote Trump with me to stick it to the man."
Great read!
And is for the “overcompensating” theory, while it is not easy to prove this but I wouldn’t be too surprised if they worry way more about missing in the direction of understanding GOP than the other way around.
For one thing, Dems esp the ones who consume news tends to be the neurotic ones who is more paranoid about polls underestimating Trump than the other way around. They prob won’t yell and scream if the polls underestimate Dems.
For another Trump himself and Too Online MAGAs also scream when they are underestimated on the poll (like they literally sued Ann Seltzer) while they don’t call polls fake when GOP is overestimated.
So, underestimating GOP/Trump is almost a recipe for being yelled and heckled by those two groups while the other way around is not - and many polled at the end of the day are sponsored by the media, who have to cater to the demand of the audience as well as wanting to minimize the risk of unwanted lawsuits.
(The case for Trafalgar tho is prob somewhat different - I don’t think it is super far fetched to call them basically GOP hopium pollsters where the methodology is very opaque and have track record of wild misses in off year elections. Not quite sure if I go as far to call them making up numbers to match the narrative they try to push but I think there are some smokes out there?)
There are far more professional ramifications for underestimating the Republicans right now than the Democrats.
The irony was the 2024 elections were pretty solid by all measures, but people still complained that they underestimated conservatives. Thus, polling shifts to a more conservative baseline, which means that people begin making decisions on the belief that conservatives are stronger than they are and that Democrats are weaker than they are, leading to 'surprises' during the coming years.
Yes, I absolutely agree with what you said!!
And I think this prob does have negative consequences that they have not thought about.
This is by no means the only reason of not even the main reason but one reason I *think* why Trump and MAGAs (I guess honestly more so for MAGAs than Trump himself) has acted way more brazen than the actual election result suggests is, the polls have not shown as bad of a result. I confess I have no idea how “off” Trump approval rate is rn in comparison with NJ and VA gubernatorial polls but I also won’t be too surprised that is the case.
Ngl, I have been rather fascinated that RV polls of Trump approval have been a smidge better than all adults poll - and this is a bit counterintuitive considering how high engaging ones has become more Dems.
I can think of two not mutually exclusive theories
1. Low engaging voters actually soured on Trump a lot so the Dems edge in high engaging voter in comparison with low engaging one decreased since 2024 (which I think is confirmed indirectly by the fact that young ppl and racial minorities bounced hard back from Trump)
2. The voter model is off - they make the projected voter to be much more conservative than it actually is
I wonder both are happening more or less but I am eager to be proven wrong on this!
Rating polls by difference from election results may make sense for some purposes, but it ignores the polls' claimed standard errors. If a poll predicts a Republican victory by six percentage points, with a standard error of two percentage points, a Democratic victory is very strong statistical evidence that either the poll's estimate or its standard error were wrong. But with a four percentage point standard error, the result is consistent with an accurate poll.
For most of us not involved in polling, the most important information in a poll is the implied probability of the election result. A useful metric of relative poll accuracy is how much money you would make or lose betting using one poll's implied probabilities for payouts and the other for bets. You can also compare polls to prediction market prices and expert judgments.
Unfortunately, since polls are taken over somewhat different time intervals you can't always find direct head-to-head comparisons. But that's also true of the metric used in the post.
You're correct about the margin of error being important. The metrics we use for our pollster ratings (Advanced Plus-Minus and its simpler counterparts) do account for sampling error. Basically, we estimate how large a poll's error should be based on things like sampling error, when it was conducted, how hard the race is to poll, etc. and compare that to how large the error actually was. You can read more here: https://fivethirtyeight.com/methodology/how-our-pollster-ratings-work/
Thank you for the information.
“It’s easy to imagine people being more upset with the 2025 polls if they once again underestimated Democrats. Instead, the polls were too bullish on Republicans, on average.”
I’m trying to understand what this pair of sentences means…can anyone help me out?
Yep that should have been "overestimated." That typo has been fixed!
I'm pretty certain it's a typo and should say "once again overestimated Democrats" - because the polls have been overestimating Democrats for the last three presidential elections.
Nate love the model and evaluation posts! Not sure how your search for a copy editor is going but in the third graf below the first table I think you say the opposite of what you mean. Should be either again underestimate Republicans or again underestimate Democrats as a contrast to being too bullish on republicans in reality. Let me know 😀 that’s my job!
Am I misinterpreting or is there a significant error with how the House races have been coded?
There is a line in the article that says "Instead, the polls were too bullish on Republicans, on average." And then the associated footnote (#3) says "Technically, the two 2025 House polls we added to our database did underestimate Democrats by 8.4 points on average, but that mostly comes down to the blue team overperforming expectations in FL-06 and TN-07."
At some level, I don't understand the footnote -- if the the polls were too bullish on Republicans across the board (i.e. both governor and House), why is this footnote needed?
And then more importantly -- if the two House polls *underestimated* Democrats (as the footnote says), why does the table show a D+8.4 bias, which would suggest those polls *overestimated* Dems? Shouldn't that be a +8.4 R bias?
I'm ultimately confused by what this is trying to say -- were the House polls too good for Dems (and thus cut against the trend of governor polls being too good for the GOP), or were all the polls too good for the GOP?
Thanks in advance for digging into this.
That was a dumb typo on my part: underestimate should have been overestimate in the footnote. I've fixed that now. The reason I included the footnote is because the second chart shows a large Democratic bias in the 2025 House polls, and I wanted to clarify that it's being driven by only two polls.
Got it — but how does that square with the “blue team overperforming expectations”?
Doesn’t a Dem poll bias suggest that the blue team underperformed expectations? You may want to clarify that as well.
The polling bias towards Democrats in presidential races has only been a regular thing since 2016. Polling underestimated Obama in 2012. That’s why I think it’s a Trump issue as he does well with low propensity voters.
He does well with voters on the outside of the two teams.
I could tell he was a danger in 2015 when members of my libertarian circles who I knew mostly voted to write in Mickey Mouse started telling me "I'm voting Trump, and you should vote Trump with me to stick it to the man."