Long time fan. I recently wrote a guide to AI catastrophe that I think is superior to existing offerings (it assumes less things, makes the case more rigorously, is intended for a lay audience, doesn't have weird quirks or science fiction, etc):
Many of your subscribers likely have a vague understanding that some experts and professionals in the field believe that AI catastrophes (the type that kills you, not just cause job loss) is likely, but may not have a good understanding of why. I hope this guide makes things clearer.
I don't think the case I sketched out above is *overwhelmingly* likely, but I want to make it clear that many of the top people who worked on AI in the past like Geoffrey Hinton and Yoshua Bengio think it's pretty plausible, not just a small tail risk we might want to spend a bit of resources on mitigating, like asteroids.
One thing I always think about is how much the current economy runs on many things we'd consider frivolous 50 years ago. The excess labor savings from automation has gone into building social media apps, creating+supporting prominent influencers, and many frivolities.
Somehow human beings find some 1% niche after 99% of their lives have been automated, and they find economic value there. Maybe for silly reasons like social status and influence, not due to deeper "this will feed my family" reasons.
As a programmer, I think the disruption is coming and coming quick. The latest Anthropic and OpenAI models literally can code faster and mostly better than me with skilled usage. It wouldn't take much for them to be able to code faster and better than me with unskilled usage.
Recursive self improvement isn't theoretical, it has already started. Both Anthropic and OpenAI's latest models were mostly coded by earlier versions of themselves.
The part that I'm most curious about is how well this generalizes, especially into relatively data-poor domains. I know there are several companies doing expert annotation and extensions on existing and simulated datasets for a bunch of fields, but for all kinds of reasons (among them the compositions of simulated data in some fields) I suspect that code writing (with decades of easily tokenized forum posts, to say nothing of github) is relatively easier than some other skills because the public dataset allone is much larger and better documented than in, say materials science. Not that AI of whatever type won't be able to automate taksks in those fields, but the scope of automation seems intuitively unlikely to grow as quickly as it did and will in software development.
What do you mean by "mostly coded"? Models are "trained", not "coded". What specific contribution do you have in mind, from the earlier versions to the latest version? Can you be more explicit about the precise content of this "mostly coded" that you have in mind?
Nate, I'm pretty sure the AI companies are “Theranos”ing all of us at this point (okay, that’s a little extreme, I don’t think it’s outright fraud, but definitely overoptimism for the sake of pulling in investments). Like obviously LLMs are real and tangible, but the progress is becoming increasingly slow and the AI companies aren’t eager to own up to that fact. To take a rather infamous old example, strapping a calculator to a model because it doesn’t know how to do math doesn’t fundamentally fix the underlying issue causing the model to suck at math. Are these models still getting appreciably better or are they just taping a band-aid over a gaping wound over and over again?
I use AI for literature searching, primarily because search engines have become so terrible that only the AI models built into the search engines can turn anything up reliably. EVERY TIME, there is obviously incorrect information in the write up. Just patently incorrect. Sometimes it doesn’t even follow the assertions made earlier. The papers it turns up are fine, summarising a single paper is fine, but the moment you ask an LLM from a synthesis of multiple sources, the wheels inevitably come off. The “check this response” feature won’t even turn up most errors. It’ll find one line it can verify and mark that and ignore the other factual assertions. How does this tech cause massive, long-term technological disruption if the credibility of its output is worse than a coin flip?
Well, there is one way, dependency. It doesn’t matter if the models are good or not if students are trained to be dependent on them, but that’s not singularity.
I’m also still waiting to hear what their plan is to create new training sets. Most of the improvements so far have been from sucking in more training data. The problem is at this point a large amount of new content online is AI generated. We know that, for some reason (actually a fairly well understood one), training an AI on AI output causes model collapse. If you can’t separate AI output from human output, how can you create a pure training set that you haven’t already used?
This “no-ceiling” rhetoric may end up being correct, but the negative case is compelling to me and is not frequently considered (beyond the shallow chat bot critique you call out). Obviously big tech isn’t interested in that argument because they’re financially entangled with the AI firms, that means that all of the “experts” aren’t really able to be objective on the question. (As an aside, when you frame the question as whether OpenAI will *announce* AGI, I think 16% is fair odds based on how many times in history the definition of AI has changed and the number of times a company has announced something they didn’t actually create)
Researching with an LLM is an under-appreciated skill. When you have to research something *that matters* and not just ask a casual question, you see the problems
* They want to confirm your priors, not challenge them. They're eager to please you. You have to work against that in your prompting
* They can be bad at evaluating evidence quality. They make very confident assertions based on what they find, unless you ask them to be systematic about evaluating this
* They can hallucinate even about the content they find (they might hallucinate in the voice of that content now)
* You still need to search and find novel evidence for it to consider
* After enough context the LLM takes on a kind of "personality" from all its context that it suddenly has its own strong, specific priors
If I were your editor, I would say this is a strong premise, but when you conjure “profound and unpredictable political impacts,” you owe the reader more specific consequences. This essay stops short of the good stuff. Like, the rise of a new party structure? Or the French Revolution? Unpredictable means nobody really knows, but we read you because we value your analytics based imagination. Complete this essay. Imagine more.
I think we'll switch from a service economy to a "responsibility" economy
As in its not nesc. my job to create code for some service at BigCorp. But I need to be the one responsible for it in a human accountability sense. If AI writes 100% of the code, I'm the one steering the AI. I would wear many hats (technical, product, on-call, etc for this service). I make sure we don't have a massive disaster due to unattended AI, and when all the agents stop churning on fixing bugs, I'm there to reorient them.
But even that's a massive hit to the economy, and I'm probably doing (with AI) what a team of 10 was doing pre Covid.
Regarding Nate's footnote #4, I highly recommend Iain M. Banks' The Player of Games, which has a very similar central idea. The main character is a master game player in Banks' post-scarcity, AI-steered, Culture who is recruited to play in what you might think of as the ultimate poker/chess/everything-else tournament for control of an interstellar empire:
As far as the broader subject of the article goes, I think Altman is right that humans will always find new things to want and new things to do, but I think Nate is right that the changes won't be gentle. As long as there is a gap between saying, "I want X," and having an AI being able to instantly deliver X, there will be a role for humans in trying to fill that gap -- whether to do something that AI can't do or in figuring out a way to make AI smarter. The trick will be finding the right spot in the value chain when the set of "things that AI can't do" is constantly changing and shrinking.
My favorite thing about the Altman quote about the industrial revolution is how it led to massive unrest, revolutions, and communism. If we were to have had the industrial revolution in the nuclear age the world would be over.
Hi Nate,
Long time fan. I recently wrote a guide to AI catastrophe that I think is superior to existing offerings (it assumes less things, makes the case more rigorously, is intended for a lay audience, doesn't have weird quirks or science fiction, etc):
https://linch.substack.com/p/simplest-case-ai-catastrophe
Many of your subscribers likely have a vague understanding that some experts and professionals in the field believe that AI catastrophes (the type that kills you, not just cause job loss) is likely, but may not have a good understanding of why. I hope this guide makes things clearer.
I don't think the case I sketched out above is *overwhelmingly* likely, but I want to make it clear that many of the top people who worked on AI in the past like Geoffrey Hinton and Yoshua Bengio think it's pretty plausible, not just a small tail risk we might want to spend a bit of resources on mitigating, like asteroids.
One thing I always think about is how much the current economy runs on many things we'd consider frivolous 50 years ago. The excess labor savings from automation has gone into building social media apps, creating+supporting prominent influencers, and many frivolities.
Somehow human beings find some 1% niche after 99% of their lives have been automated, and they find economic value there. Maybe for silly reasons like social status and influence, not due to deeper "this will feed my family" reasons.
As a programmer, I think the disruption is coming and coming quick. The latest Anthropic and OpenAI models literally can code faster and mostly better than me with skilled usage. It wouldn't take much for them to be able to code faster and better than me with unskilled usage.
Recursive self improvement isn't theoretical, it has already started. Both Anthropic and OpenAI's latest models were mostly coded by earlier versions of themselves.
The part that I'm most curious about is how well this generalizes, especially into relatively data-poor domains. I know there are several companies doing expert annotation and extensions on existing and simulated datasets for a bunch of fields, but for all kinds of reasons (among them the compositions of simulated data in some fields) I suspect that code writing (with decades of easily tokenized forum posts, to say nothing of github) is relatively easier than some other skills because the public dataset allone is much larger and better documented than in, say materials science. Not that AI of whatever type won't be able to automate taksks in those fields, but the scope of automation seems intuitively unlikely to grow as quickly as it did and will in software development.
What do you mean by "mostly coded"? Models are "trained", not "coded". What specific contribution do you have in mind, from the earlier versions to the latest version? Can you be more explicit about the precise content of this "mostly coded" that you have in mind?
Nate, I'm pretty sure the AI companies are “Theranos”ing all of us at this point (okay, that’s a little extreme, I don’t think it’s outright fraud, but definitely overoptimism for the sake of pulling in investments). Like obviously LLMs are real and tangible, but the progress is becoming increasingly slow and the AI companies aren’t eager to own up to that fact. To take a rather infamous old example, strapping a calculator to a model because it doesn’t know how to do math doesn’t fundamentally fix the underlying issue causing the model to suck at math. Are these models still getting appreciably better or are they just taping a band-aid over a gaping wound over and over again?
I use AI for literature searching, primarily because search engines have become so terrible that only the AI models built into the search engines can turn anything up reliably. EVERY TIME, there is obviously incorrect information in the write up. Just patently incorrect. Sometimes it doesn’t even follow the assertions made earlier. The papers it turns up are fine, summarising a single paper is fine, but the moment you ask an LLM from a synthesis of multiple sources, the wheels inevitably come off. The “check this response” feature won’t even turn up most errors. It’ll find one line it can verify and mark that and ignore the other factual assertions. How does this tech cause massive, long-term technological disruption if the credibility of its output is worse than a coin flip?
Well, there is one way, dependency. It doesn’t matter if the models are good or not if students are trained to be dependent on them, but that’s not singularity.
I’m also still waiting to hear what their plan is to create new training sets. Most of the improvements so far have been from sucking in more training data. The problem is at this point a large amount of new content online is AI generated. We know that, for some reason (actually a fairly well understood one), training an AI on AI output causes model collapse. If you can’t separate AI output from human output, how can you create a pure training set that you haven’t already used?
This “no-ceiling” rhetoric may end up being correct, but the negative case is compelling to me and is not frequently considered (beyond the shallow chat bot critique you call out). Obviously big tech isn’t interested in that argument because they’re financially entangled with the AI firms, that means that all of the “experts” aren’t really able to be objective on the question. (As an aside, when you frame the question as whether OpenAI will *announce* AGI, I think 16% is fair odds based on how many times in history the definition of AI has changed and the number of times a company has announced something they didn’t actually create)
Researching with an LLM is an under-appreciated skill. When you have to research something *that matters* and not just ask a casual question, you see the problems
* They want to confirm your priors, not challenge them. They're eager to please you. You have to work against that in your prompting
* They can be bad at evaluating evidence quality. They make very confident assertions based on what they find, unless you ask them to be systematic about evaluating this
* They can hallucinate even about the content they find (they might hallucinate in the voice of that content now)
* You still need to search and find novel evidence for it to consider
* After enough context the LLM takes on a kind of "personality" from all its context that it suddenly has its own strong, specific priors
I wrote about this
https://softwaredoug.com/blog/2025/08/19/researching-with-agents
I think its going to be a major issue. And we probably need to teach it as a skill. Not just assume the LLM is magic.
If I were your editor, I would say this is a strong premise, but when you conjure “profound and unpredictable political impacts,” you owe the reader more specific consequences. This essay stops short of the good stuff. Like, the rise of a new party structure? Or the French Revolution? Unpredictable means nobody really knows, but we read you because we value your analytics based imagination. Complete this essay. Imagine more.
I think we'll switch from a service economy to a "responsibility" economy
As in its not nesc. my job to create code for some service at BigCorp. But I need to be the one responsible for it in a human accountability sense. If AI writes 100% of the code, I'm the one steering the AI. I would wear many hats (technical, product, on-call, etc for this service). I make sure we don't have a massive disaster due to unattended AI, and when all the agents stop churning on fixing bugs, I'm there to reorient them.
But even that's a massive hit to the economy, and I'm probably doing (with AI) what a team of 10 was doing pre Covid.
Regarding Nate's footnote #4, I highly recommend Iain M. Banks' The Player of Games, which has a very similar central idea. The main character is a master game player in Banks' post-scarcity, AI-steered, Culture who is recruited to play in what you might think of as the ultimate poker/chess/everything-else tournament for control of an interstellar empire:
https://en.wikipedia.org/wiki/The_Player_of_Games
As far as the broader subject of the article goes, I think Altman is right that humans will always find new things to want and new things to do, but I think Nate is right that the changes won't be gentle. As long as there is a gap between saying, "I want X," and having an AI being able to instantly deliver X, there will be a role for humans in trying to fill that gap -- whether to do something that AI can't do or in figuring out a way to make AI smarter. The trick will be finding the right spot in the value chain when the set of "things that AI can't do" is constantly changing and shrinking.
My favorite thing about the Altman quote about the industrial revolution is how it led to massive unrest, revolutions, and communism. If we were to have had the industrial revolution in the nuclear age the world would be over.
I am quite curious about why ChatGPT was grumpy -- what did it have to say?