SBSQ #10: Everything you always wanted to know about the election model but were afraid to ask

For instance, why it so different from 538?

There’s no exact anniversary date for Silver Bulletin. Technically, this newsletter was launched on November 17, 2022, when I was in a panic that Twitter might collapse and wanted to establish this as a backup. (That Twitter/X hasn’t collapsed — and that the people who thought it would collapse are the sort of people who are wrong about everything — is something I should write about at some point.) But the first substantive post of any kind was published last May, the first post that roughly resembles the analysis I’m typically known for came last June, and paid subscriptions were launched last September.

But if you had to pick a single date, it would be 7/1/23, the first day I was no longer officially a “cast member” of the Walt Disney Co. (and responded with an appropriately cheeky/spicy take involving my former employer) — and today is the one-year anniversary of that. 

Substackers sometimes use anniversaries like this to post what are clearly (not-so) humblebrags disguised as “lessons for newsletter entrepreneurs” or whatnot, and I’m not going to do that. (Other than the extent to which this paragraph is a humblebrag: we’re getting very meta here.) We did hit some big round numbers recently. However, the economics of this particular newsletter are unusual and probably not easy to draw extrapolations from. Roughly half of paid subscriptions and 25 percent of total subscriptions came during one of two “model spikes”, one around the NCAA tournament and the other (still ongoing) around the presidential election forecast, which we launched last week. These are big revenue drivers — and I think they’re really good products — but also involve more churn than is typical of subscriber newsletters, so this just means I don’t take for granted as much that the current very good numbers are the new normal. Anyway, I deeply appreciate your being here, whether you’re mostly for the models or you’re an OG who’s been here from the start.

In any event, this is the monthly (paid) subscriber Q&A — and this particular edition focuses exclusively on questions related to the election model. I’d encourage you to leave questions on other subjects in the comments for the next edition, which we’ll catch up with at the end of the month.

In this edition:

• How would the model handle Biden dropping out?

• C’mon, what do you really think about the new 538 model?

• My quick updated thoughts on models vs. prediction markets

• How much do you expect the forecast to change between now and Election Day? What are the chances Biden is ahead by then?

• How do you forecast RFK Jr.’s vote? Can he really win a state?

• I have bad news about the Congressional forecast (though good news about the long run)

How would the model handle Biden dropping out?

Tom Iwancio asks:

Can the model be modified to quickly evaluate potential replacements for Biden on the assumption his replacement is one of the Governors?

Not exactly. Mostly because there isn’t anything resembling adequate polling for any of the Biden alternatives. But also because I’m not sure it’s an apples-to-apples comparison. Biden is actually running for president; the other matchups are hypothetical.¹ And many of the alternatives aren’t all that well-known nationally, which tends to depress their numbers.

But what if Biden does exit the race at some point? Traders at Polymarket think there’s a 40 percent chance of this, so it’s not an idle question. How would the model handle it?

In principle, it’s simple. We’d suspend the model, wait for a week or two until there were several national polls and at least a smattering of state polls, tell it that the Democratic nominee is, say, candidate_id 16661 (Kamala Harris) rather than candidate_id 19368 (Joe Biden) and turn it back on.

The “polls” part of the model was calibrated based on elections since 1968², which includes many cycles when polling data was much sparser, or didn’t begin until later in the campaign. So in theory, it should handle this fine. The model is designed to make some clever inferences — for instance, using data from states where there’s more polling to extrapolate the outcomes to states where there isn’t.

There are few things we’d need to think about. Would we use polls featuring Biden as a proxy until there was more data for the new candidate? I’m leaning “no”, but I’ll want more time to think about it. And would we use the handful of polls that have already tested candidates like Harris in hypothetical matchups? Probably not, because the numbers might change once they actually indicated their interest in the job.

We could insert some sort of additional uncertainty parameter — like we did for COVID — but I don’t think we’d need to. Instead, the uncertainty would naturally go up because of the “uncertainty index” that the model already calculates. For instance, the model would account for the fact that there were initially far fewer polls, and therefore more uncertainty. And the candidate switch would undoubtedly inspire days and days of banner newspaper headlines, another factor the uncertainty index considers.

It’s also possible that if Biden withdraws with a sufficient amount of time remaining before the convention, multiple Democrats would throw their hat in the ring and would presumably be polled in state and national surveys. In that case, we could simply add some code to loop through the model multiple times each day to generate a forecast for each candidate.

Now, in reality, we’d want to make sure the model was producing sensible results. Backtesting a model on 1968 data isn’t the same as publishing one in real time. And there might be some conceptual complications — implicit assumptions that were violated by a late replacement candidate — that we haven’t thought of yet. But we’d be prepared to get the model back up and running as soon as we thought it was responsible to do so.

C’mon, what do you really think about the new 538 model?

Brian asks:

Subscriber Question for July / SBSQ10: I would really like to know what the material differences are between the assumptions you’ve made at Silver Bulletin and the assumptions that the re-branded 538 and the Economist have made.

I don’t want model wars, but it would be interesting if you could lay out in some comparative form what you see as the key methodological differences and allow readers to decide which ones match their intuition, since you probably have much more insight than we would just reviewing the different methodology pages as a lay person.

I don’t want “model wars” either, but I’m getting enough versions of this question that I think I have to address it. Let me start by saying that I think all of the models I’ve currently seen out there are basically pretty reasonable. The ones that made truly indefensible assumptions were weeded out after 2016.

But let’s go behind the paywall, so I can get into What-Nate-would-say-after-half-a glass-of-wine mode — although this still hopefully falls very much into the category of constructive criticism.