I like the idea of combining options 3 and 4. License the full model to companies they need that level of precision, and also publish a simplified, less precise version of the model that’s refreshed weekly.
I think I started listening to the FiveThirtyEight podcast a little after the 2016 election, and started reading written content regularly in 2019 or so. I believe I got more value from the written/podcast content than the model alone because the contextualization helped me better understand the trends and larger political environment.
I feel like that's already a common business model in finance, so people that need it would have no qualms about subscribing. E.g. using Google I can get stock prices with a two minute delay, but I have to pay a bunch of money to get up-to-the-second prices. And there are plenty of people willing to pay for that.
This notion makes a lot of sense to me. Selfishly, I'm very interested in 3, and having followed you closely on FiveThirtyEight and... ahem... "X" for years now, it's plain that 4 would be really healthy and/or a relief for you. In fact, before you even got into the forward-looking part of this essay, I had the thought that 'man, I bet part of him wishes he'd sold Disney the model'. Moreover, for all of the reasons outlined here, covering this election seems particularly excruciating. Regardless, it seems like the bulletin is doing well, so I hope you're able to make the right decision (partially) unburdened by practical considerations.
The value in election forecasts is why the number is the way it is. Like I mentioned with Moody’s in 2016–it predicted Hillary was going to win because of many factors but the one that stuck out to me was low gasoline prices. Low gasoline prices was actually why Hillary lost because we had a mini recession because of low gasoline prices.
So the takeaway from Nate’s forecast should have been Get Out the Vote!!! Well in 2020 we maxed out GOTV and I think everyone knows we need to max out GOTV in 2024 in order to defeat Trump. If Nate were a partisan actor he wouldn’t bother with a forecast and just put the number at whatever would help maximize Democratic GOTV while reducing GOP GOTV.
I've never found anyone else I can trust other than Nate to run election models. Too many people think they know more than they do, and try to fit numbers to a narrative rather than the other way around. If I don't have Nate's model I'll feel completely in the dark, not much better informed than if I were only listening to pundits.
But I completely sympathize with everything in this post. I like the idea of not updating the model more than once a week or so, because yes it's too tempting to check it every day kind of obsessively, and I agree that's not healthy and doing so doesn't add any value. One concern about selling the model is that I don't know what the new owner might do with it? It's always helpful to have Nate's analysis and interpretation to go along with whatever the model spits out.
Nate, I hope you can find a balance where you can continue to provide this very valuable service without it becoming so much of a burden that it's a significant life stressor. We all only live once!
My gut feeling is that the most profitable way to sell access to your model would be to have three tiers:
1. A free, ad-supported tier with something like the lo-fi version that you suggested.
2. A premium hi-fi tier for political junkies, with a price in the ballpark of $100 per election cycle.
3. A much-more expensive tier that includes API access to the data, maybe sold through Quandl. This is your product for hedge funds and the like.
But you seem to be mixing this question up with a second one, which is what you feel like doing with the rest of your life. I can't answer that one for you, but one thing I can tell you is that you aren't irreplaceable. If you don't currently have anyone else who can do your job, go fix that — train an apprentice or three. And then once you aren't tethered to the needs of the business you'll be free to go touch grass or play poker or do whatever else gets you as far away from thinking about Donald Trump as remains possible within the confines of a single-planet civilization.
I think a lot of the value add is being both 1) temperamentally inclined to view this as a math problem instead of an exercise in partisanship, and 2) willing to spend so much time thinking about politics. Also 3) managing to signal (1), at least to some people
I suspect most people with sufficient skill to build comparable models are able to make more money doing something else, especially considering the barrier-to-entry of learning all the relevant object-level political science. The ones who apply their energy to political forecasting probably care about politics in a way that makes them inclined to partisanship. Then there's the actual object-level modeling skill, because Nate legitimately has a familiarity with political science that most quants (for example) don't, which creates a barrier to entry.
1. I can't imagine you can paywall this and expect it not to get screenshotted all over Twitter or whatever. I don't know how easy it is to suppress stuff like that on the modern internet, but I imagine that trying to get model shots taken down will be an enormous, continuous hassle. On a related note "put out model but only have non-idiots read it" is probably impossible.
2. I like the idea of limited updates (e.g. once per day at a fixed time). There's no reason anyone needs more than that. It's not your job to stop us from smoking crack, but you can decide not to be the source of crack. Rounding to nearest 5% feels a little silly though. People will just go nuts waiting for the round to change.
Agree with all of this except the rounding part. Part of my professional work is doing forecasts of things. We intentionally round like this in certain forums, and I’ve found that to be THE most effective way to drill home the uncertainty. The number one piece of feedback new hires get is, stop putting decimal points on estimates :). I’ve found it really does help people understand the uncertainty more than any disclaimer can. I agree about some people waiting for it to round, but most people’s reaction to that would not be “oh no the odds changed by 5% just now” it would be (IMO) oh, ok, so this model isn’t even pretending to be reliable at that level of sensitivity.
If Nate goes with a pricing model something like I suggested in https://www.natesilver.net/p/the-election-model-is-a-little-too/comment/40399100 then I don't think he should bother worrying about piracy. The people who care will still pony up the $100, and the people who don't probably wouldn't have anyway, with or without the piracy. The lesson of the '00s is that people will get their data from whatever source gives them the best user experience. Napster was killing the record labels not because it was cheaper than shopping for CDs, but because it was more convenient. And then iTunes and (later) streaming services came along, and they were more convenient than piracy, and piracy immediately fell by the wayside. I'm certain that Nate can offer a UX that beats searching Twitter for screencaps.
The question is whether he wants to paywall to make money, or paywall to have a dedicated audience of data nerds and be invisible to the wider media. I think only the first is possible.
For what it's worth, of all the various contextualization experiments over the years, I found using "1 in 7" rather than "14%" to be the most effective.
For what it's worth on my end, I notice people seem to find "1 in 7" as something that is more likely to happen than "14%", even if they wrongly don't think of the two as the same.
Yea, this. I think it's that for most people it's easier to visualize "1 in X".
(At least if we're talking about an X that's under 10 -- once you're in to double-digit values for X it tends to blur together for most people. "1 in 12" doesn't land any differently than "1 in 30" or whatever.)
Anything that's publicly available & constantly updating & concerning very important, deeply emotional issues will inevitably lead to public freakouts (Twitter, sports, Heard-Depp trial, etc.)
I think putting it behind a paywall, even a modest one, and updating less regularly would fix alleviatw a lot. I also think that being disassociated from Disney/ABC will naturally mean less eyes on the product, because it is no longer *the forecast*.
Selfishly, I've made money betting in 2016, 2018, 2020, and 2022 basically using just the model. So I would like to see it available.
Nate, big fan. Please keep the model somewhat around, maybe it would be a good change to orient the model towards the probability-math-modeling crowd rather than the pure politic pundit world. I do like the idea of changing the audience though! I'll go crazy if I have to talk about calibration plots with another 538 hater
Nate, great question. My advice is to look at return on investment. A good model is a lot of work. What's the payoff? For journalists, it's "eyeballs". My take is that fivethirtyeight over invested. Would making the model a few percentage points more accurate have boosted traffic very much? Is there enough of a payoff in political prediction markets? Maybe? That would be a paywall model with a pretty stiff price of entry. But probably not a lot of fun. To me, the best value you offer is insightful debunking of dumb ideas. You probably don't need a detailed model to do that.
I really fear I will have to cancel my subscription if you'll stop using your electoral model and analyzing the results! I'm sorry, I can appreciate your reasons for doing that, but I couldn't care - and understand - less about poker and NBA. So, please, bring back a (Silver)y Fox 🦊😊!
Put it out in all its glory and do your best. You made a good thing, and no matter what you do some people will abuse it. And yet many more will put it to good use. Obviously, do whatever you have to so that it doesn't negatively impact your life, but you can't fix people. And you especially can't fix how they're going to react to this election.
I'd support whatever you want to do, even it means no more public model. I certainly understand why you'd want to bow out from watching the constant mean tweets and negative press you get. FWIW I'd consider it a loss if it was not available to the public in any form at all, but I get it.
You may feel like your work is consumed in a way you don’t like by 9/10 people, but I think you should consider the good you do for the 1/10 who want to see falsifiable, data driven information about politics. It is precisely because of the credibility you established with the few who would appreciate this that made your model so popular with everyone else. There are so few sources the public has access to that calibrate and correct themselves. To that extent, I always found your model a great way to not fall for the media engagement hype cycle and stay grounded on where things were at based on an empirical standard.
I suppose what I’m saying is that I’d love to see the model either accessible publicly or for a small fee to subscribers. I would hate for it not to exist in the world. You can’t control how other people are going to use and interpret your work, but they were going to find something to sate themselves regardless. I think your work just makes the political news landscape a better place, and is a huge contribution to the discussion
The main public perception problem I see is people think a probability forecast is "wrong" if it gives X a probability greater than 50% and X doesn't happen. I think that's different from "people round 70% up to 100%".
To avoid obsessing over small changed in the precise numbers, I suggest giving a probability range (ex. Biden has a 63%-69% chance of winning) rather than rounding to the nearest 5%. The concern I have with rounding is that a small change in the raw number (say from 42% to 44%) will appear as a bigger change (40% to 45%).
I think a major part of the problem is that the nihilistic catastrophism that has long infected progressive thought about esoteric things like the climate and “racism” has reached elections themselves. There are deranged liberals who sincerely believe nutty conspiracy theories like “This is just like the end of Weimar Germany!” and “Democracy is on the ballot!”. As long as those hysterics with their existential house stakes are in your audience, you won't be able to have your audience treat the model with sufficient detachment. You need to find a way to gently put those people off, and I don't think paywalling it here is going to be sufficient: doomsday cultists are famous for their willingness to spend their life savings on hearing how the world will end in salacious detail.
If you do anything please bring back the electoral college snake! That was data visualization at its finest!
I like the idea of combining options 3 and 4. License the full model to companies they need that level of precision, and also publish a simplified, less precise version of the model that’s refreshed weekly.
I think I started listening to the FiveThirtyEight podcast a little after the 2016 election, and started reading written content regularly in 2019 or so. I believe I got more value from the written/podcast content than the model alone because the contextualization helped me better understand the trends and larger political environment.
I feel like that's already a common business model in finance, so people that need it would have no qualms about subscribing. E.g. using Google I can get stock prices with a two minute delay, but I have to pay a bunch of money to get up-to-the-second prices. And there are plenty of people willing to pay for that.
yes this is a good approach. i'd be interested in 1 or 2x per week analysis, but don't need it updated 24/7
This notion makes a lot of sense to me. Selfishly, I'm very interested in 3, and having followed you closely on FiveThirtyEight and... ahem... "X" for years now, it's plain that 4 would be really healthy and/or a relief for you. In fact, before you even got into the forward-looking part of this essay, I had the thought that 'man, I bet part of him wishes he'd sold Disney the model'. Moreover, for all of the reasons outlined here, covering this election seems particularly excruciating. Regardless, it seems like the bulletin is doing well, so I hope you're able to make the right decision (partially) unburdened by practical considerations.
The value in election forecasts is why the number is the way it is. Like I mentioned with Moody’s in 2016–it predicted Hillary was going to win because of many factors but the one that stuck out to me was low gasoline prices. Low gasoline prices was actually why Hillary lost because we had a mini recession because of low gasoline prices.
So the takeaway from Nate’s forecast should have been Get Out the Vote!!! Well in 2020 we maxed out GOTV and I think everyone knows we need to max out GOTV in 2024 in order to defeat Trump. If Nate were a partisan actor he wouldn’t bother with a forecast and just put the number at whatever would help maximize Democratic GOTV while reducing GOP GOTV.
I've never found anyone else I can trust other than Nate to run election models. Too many people think they know more than they do, and try to fit numbers to a narrative rather than the other way around. If I don't have Nate's model I'll feel completely in the dark, not much better informed than if I were only listening to pundits.
But I completely sympathize with everything in this post. I like the idea of not updating the model more than once a week or so, because yes it's too tempting to check it every day kind of obsessively, and I agree that's not healthy and doing so doesn't add any value. One concern about selling the model is that I don't know what the new owner might do with it? It's always helpful to have Nate's analysis and interpretation to go along with whatever the model spits out.
Nate, I hope you can find a balance where you can continue to provide this very valuable service without it becoming so much of a burden that it's a significant life stressor. We all only live once!
My gut feeling is that the most profitable way to sell access to your model would be to have three tiers:
1. A free, ad-supported tier with something like the lo-fi version that you suggested.
2. A premium hi-fi tier for political junkies, with a price in the ballpark of $100 per election cycle.
3. A much-more expensive tier that includes API access to the data, maybe sold through Quandl. This is your product for hedge funds and the like.
But you seem to be mixing this question up with a second one, which is what you feel like doing with the rest of your life. I can't answer that one for you, but one thing I can tell you is that you aren't irreplaceable. If you don't currently have anyone else who can do your job, go fix that — train an apprentice or three. And then once you aren't tethered to the needs of the business you'll be free to go touch grass or play poker or do whatever else gets you as far away from thinking about Donald Trump as remains possible within the confines of a single-planet civilization.
Nate’s modeling skills are not that special but his ability to critique and put models and results in perspective is invaluable
I think a lot of the value add is being both 1) temperamentally inclined to view this as a math problem instead of an exercise in partisanship, and 2) willing to spend so much time thinking about politics. Also 3) managing to signal (1), at least to some people
I suspect most people with sufficient skill to build comparable models are able to make more money doing something else, especially considering the barrier-to-entry of learning all the relevant object-level political science. The ones who apply their energy to political forecasting probably care about politics in a way that makes them inclined to partisanship. Then there's the actual object-level modeling skill, because Nate legitimately has a familiarity with political science that most quants (for example) don't, which creates a barrier to entry.
I have to agree with Nate—it’s “politics junkie”. I’ve always thought a “political junkie” is an opium addict that is a partisan political activist.
Two random thoughts (if you are reading, Nate):
1. I can't imagine you can paywall this and expect it not to get screenshotted all over Twitter or whatever. I don't know how easy it is to suppress stuff like that on the modern internet, but I imagine that trying to get model shots taken down will be an enormous, continuous hassle. On a related note "put out model but only have non-idiots read it" is probably impossible.
2. I like the idea of limited updates (e.g. once per day at a fixed time). There's no reason anyone needs more than that. It's not your job to stop us from smoking crack, but you can decide not to be the source of crack. Rounding to nearest 5% feels a little silly though. People will just go nuts waiting for the round to change.
Agree with all of this except the rounding part. Part of my professional work is doing forecasts of things. We intentionally round like this in certain forums, and I’ve found that to be THE most effective way to drill home the uncertainty. The number one piece of feedback new hires get is, stop putting decimal points on estimates :). I’ve found it really does help people understand the uncertainty more than any disclaimer can. I agree about some people waiting for it to round, but most people’s reaction to that would not be “oh no the odds changed by 5% just now” it would be (IMO) oh, ok, so this model isn’t even pretending to be reliable at that level of sensitivity.
If Nate goes with a pricing model something like I suggested in https://www.natesilver.net/p/the-election-model-is-a-little-too/comment/40399100 then I don't think he should bother worrying about piracy. The people who care will still pony up the $100, and the people who don't probably wouldn't have anyway, with or without the piracy. The lesson of the '00s is that people will get their data from whatever source gives them the best user experience. Napster was killing the record labels not because it was cheaper than shopping for CDs, but because it was more convenient. And then iTunes and (later) streaming services came along, and they were more convenient than piracy, and piracy immediately fell by the wayside. I'm certain that Nate can offer a UX that beats searching Twitter for screencaps.
The question is whether he wants to paywall to make money, or paywall to have a dedicated audience of data nerds and be invisible to the wider media. I think only the first is possible.
Oh, agreed. Nate is a marked man at this point, and internets gonna internet.
For what it's worth, of all the various contextualization experiments over the years, I found using "1 in 7" rather than "14%" to be the most effective.
(Fivey was the cutest)
For what it's worth on my end, I notice people seem to find "1 in 7" as something that is more likely to happen than "14%", even if they wrongly don't think of the two as the same.
Yea, this. I think it's that for most people it's easier to visualize "1 in X".
(At least if we're talking about an X that's under 10 -- once you're in to double-digit values for X it tends to blur together for most people. "1 in 12" doesn't land any differently than "1 in 30" or whatever.)
Anything that's publicly available & constantly updating & concerning very important, deeply emotional issues will inevitably lead to public freakouts (Twitter, sports, Heard-Depp trial, etc.)
I think putting it behind a paywall, even a modest one, and updating less regularly would fix alleviatw a lot. I also think that being disassociated from Disney/ABC will naturally mean less eyes on the product, because it is no longer *the forecast*.
Selfishly, I've made money betting in 2016, 2018, 2020, and 2022 basically using just the model. So I would like to see it available.
Nate, big fan. Please keep the model somewhat around, maybe it would be a good change to orient the model towards the probability-math-modeling crowd rather than the pure politic pundit world. I do like the idea of changing the audience though! I'll go crazy if I have to talk about calibration plots with another 538 hater
Nate, great question. My advice is to look at return on investment. A good model is a lot of work. What's the payoff? For journalists, it's "eyeballs". My take is that fivethirtyeight over invested. Would making the model a few percentage points more accurate have boosted traffic very much? Is there enough of a payoff in political prediction markets? Maybe? That would be a paywall model with a pretty stiff price of entry. But probably not a lot of fun. To me, the best value you offer is insightful debunking of dumb ideas. You probably don't need a detailed model to do that.
I really liked 538 for two things: the models, and the model-informed analysis articles.
I would be quite disappointed not to read any model-informed analysis here, it's the content I most looked forward to when deciding to subscribe.
So what I am saying is, please don't license the model in a way that would prevent you from writing about it.
Yes, this! I would hope that's not on the table.
I really fear I will have to cancel my subscription if you'll stop using your electoral model and analyzing the results! I'm sorry, I can appreciate your reasons for doing that, but I couldn't care - and understand - less about poker and NBA. So, please, bring back a (Silver)y Fox 🦊😊!
Put it out in all its glory and do your best. You made a good thing, and no matter what you do some people will abuse it. And yet many more will put it to good use. Obviously, do whatever you have to so that it doesn't negatively impact your life, but you can't fix people. And you especially can't fix how they're going to react to this election.
I'd support whatever you want to do, even it means no more public model. I certainly understand why you'd want to bow out from watching the constant mean tweets and negative press you get. FWIW I'd consider it a loss if it was not available to the public in any form at all, but I get it.
You may feel like your work is consumed in a way you don’t like by 9/10 people, but I think you should consider the good you do for the 1/10 who want to see falsifiable, data driven information about politics. It is precisely because of the credibility you established with the few who would appreciate this that made your model so popular with everyone else. There are so few sources the public has access to that calibrate and correct themselves. To that extent, I always found your model a great way to not fall for the media engagement hype cycle and stay grounded on where things were at based on an empirical standard.
I suppose what I’m saying is that I’d love to see the model either accessible publicly or for a small fee to subscribers. I would hate for it not to exist in the world. You can’t control how other people are going to use and interpret your work, but they were going to find something to sate themselves regardless. I think your work just makes the political news landscape a better place, and is a huge contribution to the discussion
The main public perception problem I see is people think a probability forecast is "wrong" if it gives X a probability greater than 50% and X doesn't happen. I think that's different from "people round 70% up to 100%".
To avoid obsessing over small changed in the precise numbers, I suggest giving a probability range (ex. Biden has a 63%-69% chance of winning) rather than rounding to the nearest 5%. The concern I have with rounding is that a small change in the raw number (say from 42% to 44%) will appear as a bigger change (40% to 45%).
I think a major part of the problem is that the nihilistic catastrophism that has long infected progressive thought about esoteric things like the climate and “racism” has reached elections themselves. There are deranged liberals who sincerely believe nutty conspiracy theories like “This is just like the end of Weimar Germany!” and “Democracy is on the ballot!”. As long as those hysterics with their existential house stakes are in your audience, you won't be able to have your audience treat the model with sufficient detachment. You need to find a way to gently put those people off, and I don't think paywalling it here is going to be sufficient: doomsday cultists are famous for their willingness to spend their life savings on hearing how the world will end in salacious detail.
Better safe than sorry. I know what I'm seeing out there.