If I were your editor, I would say this is a strong premise, but when you conjure “profound and unpredictable political impacts,” you owe the reader more specific consequences. This essay stops short of the good stuff. Like, the rise of a new party structure? Or the French Revolution? Unpredictable means nobody really knows, but we read you because we value your analytics based imagination. Complete this essay. Imagine more.
It's the same kind of frustrating vagueness that the Douthat piece was talking about regarding people saying people need to "prepare for AI". Like what does that mean specifically?
I've been a software engineer for about 20 years. I'll say that, I'm actively embracing AI because, as I've gotten older, I actually kind of hate spending hours and hours writing code, or learning a new language, etc ... I used to like this, but after 20 years, it is boring and tedious, and I just want to build cool things. So on the one hand, I'm hugely excited. I dusted off one of my hobby projects that I started back 2020 and AI has super charged adding new features, fixing bugs, etc ... On the other hand though: I've seen how the trajectory capitalism in this country has gone. Globalization was supposed to make "everything cheaper" and it certainly did, but the cost came at the blue collar class's sense of dignity, their place in community and society, and their optimism about the future. It hollowed out the soul of many lower and middle class communities; you cannot replace that with a cheap TV from China. Now the same trajectory seems to be playing out for potentially upper middle class and anyone out side of high net worth groups. To think this won't have extreme, and possibly violent, political ramifications is not just naive; it's profoundly arrogant.
Globalization didn’t take blue collar jobs in the USA, robots did.
Those jobs aren’t coming back because labor is too expensive for manufacturing to work the way it used to. If we build up manufacturing from here, it will be automated.
As evidence, see how manufacturing grew continuously and then flattened out since around 2010, while manufacturing jobs have been decreasing since 1980, but particularly since 2000:
This is a long term positive trend (more production per unit of labor) that causes real costs to those who lose their jobs. I don’t know what replaces those jobs longer term, but the pain now is real, especially to communities that relied on factory employment.
How long until the pain subsides then? Its been nearly 3 or 4 decades of this decline at this point, no matter what the cause is, and all we've gotten out of the bargain is Donald Trump.
The big job losses have been since 2000, it was a slower decline from 1980 to then. Thats probably why it is coming to a head now.
As long as manual manufacturing jobs exist, they will keep getting replaced by robots or offshored.
One problem is that people want to remain living in their small towns (often for good reasons, like family) but there are few businesses that make sense in those areas. Another problem is that it is often too expensive for people to move from small towns to where jobs are because housing prices are out of control, which causes an inelasticity in the job market and prolonging the pain.
I don’t have solutions to these problem or a clear idea of when the job losses will end. The closest I can come is to suggest that we need to build more housing to bring prices down in areas where there are jobs.
The housing price problem was really only a recent phenomenon, many of these people don't have the skills to match jobs you're talking about; and the ones you are talking about are now on the hit list for automation as well. I bet you live in a city; and cannot imagine why other people would not want to -- and to suggest "they have to to be employable" is exactly the kind of thing that built resentment up in the people who've been electing people like Trump.
The housing price phenomenon goes back to well before the housing crisis in some states, including mine.
I do not live in a city, I live in a semi-rural suburb outside of a major city. I also spent a lot of my youth on a ranch and can understand exactly why people would prefer not to live in a city.
The reality is that there are fewer and fewer job opportunities in rural areas unless your job allows remote working. Automation has come for farming and manufacturing.
The lack of jobs isn’t going to change until someone comes up with jobs that can be done profitably far from urban centers. That forces people to either choose to live un(or under)employed or to move to where jobs exist. I’m sorry that people are offended by that, but it is how reality works.
"but the cost came at the blue collar class's sense of dignity, their place in community and society, and their optimism about the future."
True for some specific areas, false in aggregate. There are very few Americans that could get a better, higher paying job in 1970 then they could today. Maybe try not cycling through trite populist nonsense as if these were your original thoughts. Saying "you've seen trajectories" while embracing wrong and popular banal slogans.
Nate, I'm pretty sure the AI companies are “Theranos”ing all of us at this point (okay, that’s a little extreme, I don’t think it’s outright fraud, but definitely overoptimism for the sake of pulling in investments). Like obviously LLMs are real and tangible, but the progress is becoming increasingly slow and the AI companies aren’t eager to own up to that fact. To take a rather infamous old example, strapping a calculator to a model because it doesn’t know how to do math doesn’t fundamentally fix the underlying issue causing the model to suck at math. Are these models still getting appreciably better or are they just taping a band-aid over a gaping wound over and over again?
I use AI for literature searching, primarily because search engines have become so terrible that only the AI models built into the search engines can turn anything up reliably. EVERY TIME, there is obviously incorrect information in the write up. Just patently incorrect. Sometimes it doesn’t even follow the assertions made earlier. The papers it turns up are fine, summarising a single paper is fine, but the moment you ask an LLM from a synthesis of multiple sources, the wheels inevitably come off. The “check this response” feature won’t even turn up most errors. It’ll find one line it can verify and mark that and ignore the other factual assertions. How does this tech cause massive, long-term technological disruption if the credibility of its output is worse than a coin flip?
Well, there is one way, dependency. It doesn’t matter if the models are good or not if students are trained to be dependent on them, but that’s not singularity.
I’m also still waiting to hear what their plan is to create new training sets. Most of the improvements so far have been from sucking in more training data. The problem is at this point a large amount of new content online is AI generated. We know that, for some reason (actually a fairly well understood one), training an AI on AI output causes model collapse. If you can’t separate AI output from human output, how can you create a pure training set that you haven’t already used?
This “no-ceiling” rhetoric may end up being correct, but the negative case is compelling to me and is not frequently considered (beyond the shallow chat bot critique you call out). Obviously big tech isn’t interested in that argument because they’re financially entangled with the AI firms, that means that all of the “experts” aren’t really able to be objective on the question. (As an aside, when you frame the question as whether OpenAI will *announce* AGI, I think 16% is fair odds based on how many times in history the definition of AI has changed and the number of times a company has announced something they didn’t actually create)
Researching with an LLM is an under-appreciated skill. When you have to research something *that matters* and not just ask a casual question, you see the problems
* They want to confirm your priors, not challenge them. They're eager to please you. You have to work against that in your prompting
* They can be bad at evaluating evidence quality. They make very confident assertions based on what they find, unless you ask them to be systematic about evaluating this
* They can hallucinate even about the content they find (they might hallucinate in the voice of that content now)
* You still need to search and find novel evidence for it to consider
* After enough context the LLM takes on a kind of "personality" from all its context that it suddenly has its own strong, specific priors
I think a lot of what you’ve put here is broadly correct. LLMs are a productivity tool, like Word or Excel, and literacy in a tool you’re using is important. It’s exactly that reason why I wish the AI folks would be more honest in their assessments of their models. The problem is that their continued success, at this point, is derived from keeping the mood high and pulling in investors for as long as they can. LLMs are great at a certain subset of tasks and really terrible at others, but companies are all too happy to enable and encourage people to use them for the latter class of tasks. It’s hard to convince people that they shouldn’t use (or need to be careful using) a tool for a task that it says they can use it for.
You're conflating research with analysis. They're good at research (sifting through sources, identifying relevant information, summarizing) but I wouldn't use them for analysis (evaluating evidence, drawing conclusions, logical implications, etc.) Hallucinations happen, but IME (at least recently) they get facts literally wrong but 'semantically' correct. So they'll mis-name section (possibly improving it!) when summarizing a document.
Definitely not a magic answer machine, but also clearly a productivity enhancer for research.
If you haven’t read about the lawyers getting in trouble by using LLMs for research, you should.
It makes it clear that LLMs completely make up relevant information and sources fairly often, making them unreliable for anything that actually matters.
I too use AI for literature searching, but I don't agree there are errors every time, or even that often now. I am impressed that results are getting better and better. When I started with it last year, there were errors and incongruities more often, but now I am not seeing them much (you do have to watch, and importantly, you have to know a lot about what you are researching --- this is why young people are going to get caught by errors, they don't know things.). Yesterday I asked why the Helvetii REALLY left, well, Switzerland, to storm through allied and Roman lands 300,000 strong until Julius Caesar caught up with them. They burned all their own houses and supplies, too, so they couldn't come back. Caesar has them saying they did it for their young men to have a chance to show their valour! This was absurd, so I asked my AI why they really did it. It said warlike and vicious Germans had been invading and settling their land, after they had made the mistake of hiring some thousands of them as mercenaries, and those decided to bring their families and claim the land. This is the whole story of the Fall of Rome from then well into the seventh century, and longer in Britain, so good: and later in Book I of the Gallic Wars Caesar shows the elders of the Helvetii telling him the exact same thing, "weeping and throwing themselves at his feet" and forgetting all that nonsense about them emigrating to let the young men show their valour. I am usually finding the AI right about Latin and modern literature these days, much better than last year.
You do have to know your subject though, I would agree. And it doesn't hurt to check.
In my experience, the LLMs struggle with nuanced distinctions that prevent extrapolation from one area to another. Most of the errors I catch are misidentification of an equation as being appropriate when it’s assumptions aren’t met, inappropriate extrapolation from results in research that is related in methodology, but not in substance, and just plain filling in missing details with numbers not present in the cited works. Sometimes the errors are glaring, other times they’re highly nuanced, but the fact that there may be some unknown number of nuanced errors makes the output impossible for me to rely on outside of the sources it cites. I think it’s pretty good at turning up sources, but I just can’t trust it beyond that.
In history/literature it’s not surprising to me that these axes of failure would be less pronounced as the lines of when analogy and extrapolation are appropriate are much less clear.
Yeah ---- I have also seen some mistakes in plain numbers, if not as elaborate errors as the ones you are presumably working with. It is unnerving. I'm thinking 2026 will be used to get these models much improved, and then we'll see how they do. It could be awhile before we can really trust them, if we ever do.
We can hope! I get worried that “fixing error rate” isn’t splashy enough (or is too hard) for them to commit to, but I’d be pleased to see them take that seriously. Selfishly, I’d love if they’d just make a mode where I could pose a research topic and it gave me a list of papers broken down by topic area with short summaries of each instead of trying to synthesise a single answer from multiple sources, but that’s probably too niche for them to pursue
You could ask for exactly that in your prompt: AIs love elaborate prompts. This is the research topic, and I want a list of 15 papers on these five subjects pertaining to the general topic, three in each.
I know you don't need advice from me, but it was an irresistable prompt question.
Handing a calculator to a model because he or she isn't good at math is a perfectly reasonable decision that doesn't invalidate the idea that humans are intelligent..
Is it really unreasonable for AI to use a tool to solve a problem that isn't amenable to pure LLM techniques?
No, it’s not, but then presenting it as evidence of your model now being good at math is dishonest. LLMs being bad at basic arithmetic is evidence that they struggle with reasoning that we consider basic and fundamental and should temper our expectations of these models.
Presenting an AI system as good at math because the LLM has been programmed or trained to send math problems to a specialized engine seems perfectly reasonable.
AI is not a synonym for LLM, correct. However, the major improvements in AI technology have recently been from LLMs. The bullish predictions of continued AI improvements are rooted in optimism in LLM models continuing to improve. Strapping a calculator to an LLM may render that system serviceable for doing arithmetic. It doesn’t resolve the fundamental issues in reasoning that make those arithmetic issues arise in the first place.
My point is that we should reign in our expectations for where the ceiling of this technology is. Handing the AI a calculator is emblematic of this because a fundamental issue in LLMs had to be resolved in a method other than LLM improvement. That’s not to say that these systems won’t get better at all, but if your prediction is highly disruptive AGI in the very near future (or even substantial economic displacement due to AI), I’m not sure that there’s the evidence to support it. If you want to plan for it because it’s a low probability scenario with very severe consequences, sure, but let’s frame it that way instead of an inevitable outcome.
You’re completely missing the point: there is nothing wrong with strapping a calculator to a LLM to solve a math problem if your goal is specifically to have an LLM solve a math problem. The problem is that it does suggest a limit to rate of improvement of the underlying architecture. Unless you believe that the primary limiting factor in further AI improvement is not further improvements to LLM architecture and training sets, but access to subsystems.
The fact that letting the AI use a calculator fixed *an* issue is non responsive to my concern that the evidence that AI technology is approaching the point of undermining the current economic system is sorely lacking. I picked the calculator as an example where the solution couldn’t be attained by architectural and training set improvements, which is the assumption (in my eyes) underpinning the most bullish expectations of AI, but by tying in an already well known subsystem.
Long time fan. I recently wrote a guide to AI catastrophe that I think is superior to existing offerings (it assumes less things, makes the case more rigorously, is intended for a lay audience, doesn't have weird quirks or science fiction, etc):
Many of your subscribers likely have a vague understanding that some experts and professionals in the field believe that AI catastrophes (the type that kills you, not just cause job loss) is likely, but may not have a good understanding of why. I hope this guide makes things clearer.
I don't think the case I sketched out above is *overwhelmingly* likely, but I want to make it clear that many of the top people who worked on AI in the past like Geoffrey Hinton and Yoshua Bengio think it's pretty plausible, not just a small tail risk we might want to spend a bit of resources on mitigating, like asteroids.
One thing I always think about is how much the current economy runs on many things we'd consider frivolous 50 years ago. The excess labor savings from automation has gone into building social media apps, creating+supporting prominent influencers, and many frivolities.
Somehow human beings find some 1% niche after 99% of their lives have been automated, and they find economic value there. Maybe for silly reasons like social status and influence, not due to deeper "this will feed my family" reasons.
New Rule: You only get one of these "AI will change everything. You need to be worried about it" columns. It's fine - I don't disagree with a premise as uncontroversial as "if AI is half as important as Big Tech says it is, it will be a big deal." I agree! Big Tech compares AI to fire and the wheel all the time - if it is half as important as those two, it will definitely shake things up.
So you've played this card now. Ross Douthat has played it several dozen times. The next step is for you, Nate, to give us some ideas on how AI will change things, how the economy will function with all knowledge workers laid off (and how this will not somehow drag down company valuations), the probability of AGI by 2027, and what Democrats should do about any of this. I need my intelligentsia - the Yglesiases, the Silvers, and the Douthats to give me something more concrete and actionable and valuable than this "well it will probably change everything, but I won't even hazard a guess as to when and how" laziness. Frankly, I expect you guys to be guiding us and figuring this stuff out - not Hakeem Jeffries and Mike Johnson.
As a programmer, I think the disruption is coming and coming quick. The latest Anthropic and OpenAI models literally can code faster and mostly better than me with skilled usage. It wouldn't take much for them to be able to code faster and better than me with unskilled usage.
Recursive self improvement isn't theoretical, it has already started. Both Anthropic and OpenAI's latest models were mostly coded by earlier versions of themselves.
The part that I'm most curious about is how well this generalizes, especially into relatively data-poor domains. I know there are several companies doing expert annotation and extensions on existing and simulated datasets for a bunch of fields, but for all kinds of reasons (among them the compositions of simulated data in some fields) I suspect that code writing (with decades of easily tokenized forum posts, to say nothing of github) is relatively easier than some other skills because the public dataset allone is much larger and better documented than in, say materials science. Not that AI of whatever type won't be able to automate taksks in those fields, but the scope of automation seems intuitively unlikely to grow as quickly as it did and will in software development.
I said as much. It really doesn't answer the question of how long it will take to automate these fields to the extent that code generation is automated now.
I imagine faster than people taking comfort in that bottle neck would like to believe; it's more a raw capital expense with less physical constraints (like the ones building the data centers). The only one is probably number of people you can hire for the job, and then after that is "how much money" you have to pay them for the work.
What do you mean by "mostly coded"? Models are "trained", not "coded". What specific contribution do you have in mind, from the earlier versions to the latest version? Can you be more explicit about the precise content of this "mostly coded" that you have in mind?
If we take Sam Altman’s example of the Industrial Revolution, there were worker unions pushing back and demanding a fair piece of the growing economic pie. Today the worker union memberships are in constant decline and we have undermined worker laws. I think strengthening the horizontal connections, weakened by the shift from manufacturing to service-sector jobs, is the first step to securing our future.
I have two main thoughts about AI & politics: The people in control of AI platforms & models are getting extremely wealthy off of it, and that this technology will probably tear apart several nations (or reduce them to some form of totalitarianism) very violently, including ours.
When people like Altman & Musk talk about how great the world will be with AI and a "post-scarcity" society I think about history. I can't remember where I heard the line, but the best way I've seen it put is this: When has a very small group of people gathered up almost all of the wealth & power in a society, before voluntarily redistributing it in an equitable way? Never. It doesn't happen, and these authority figures have been anything but deserving of the kind of trust required for something like that to even plausibly work.
On the 2nd point, I fear the ability of AI models to create images, audio, and (soon) video that's so realistic, it's indistinguishable from reality. We're about to enter a time when anybody can conjure anything they want out of nothing. They can make people say things they never said. They can create video of people doing things they've never done.
How does society, any society, function like that?
I mean, imagine someone like Trump. He's already shown a willingness to alter images with these tools to affect his message. What if he were also willing to make it all up out of whole cloth? Or think back to Qanon -- The conspiracy theorists who believe the redhats are fighting against a global cabal of baby cannibals. Imagine nutjobs like that flooding social media with photorealistic videos of Biden killing babies, and MAGA running with it.
Maybe that's the level we find out Trump won't stoop to, but soon someone will. What if someone much worse, like Stalin was, gets an audience using this? And that's all ignoring the effect foreign adversaries will have by doing the same in a much more organized, targeted manner.
I worry that challenging any sort of authority or power will become impossible. We'll be so fractured into our information silos that any sort of organized anything on the part of the public becomes so difficult that even the most repressive, evil regime is too powerful to overcome.
I really like your point about AI being able to make up reality we cannot tell from real. MAYBE that's not so different from the propaganda and false reporting we've had thruout the ages -- no politicians or generals ever told the truth, did they? But we are dependent now on pictures, videos, audios that we think are real, and of course within months, now, that will break, I think.
Your first point I don't worry about much because we have always forever had individuals who had quantillions more money than anyone else. That's not new. It is inevitable around any new tech (railroads for Carnegie and Rockefeller, etc.) and whole economic systems, such as the Latifundia system in ancient Rome, when a few families had farms measured in dozens of square miles, and that was what produced all the wealth in the civilization, including that supporting the military. A few people having MOST of the wealth of a society is bog normal, as the British say, and that is not what caused the uproar and distress of the early and middle industrial revolution. The common people didn't want the mill owner's riches! They just wanted their old jobs back, where they were doing well in a predictable way. Then everything changed and they had to go to work and send their children to work in dark factories 12 hours a day, and they didn't like it. I hope that won't happen now, but if there are pitchforks and torches, it won't be about inequity. It would be about massive unemployment and worsening conditions.
Sure, lying isn't a surprise or unexpected. It's the absolute volume that can be churned out now. Making realistic edits is a relatively time consuming process that someone who knows a program like Photoshop would need to do themselves. A handful of images could be analyzed & scrutinized by experts who could point out irregularities.
Let's say one of these provocative images/videos/audios takes a model 5 minutes to make passable. Passable as in it's good enough to cause serious doubt. There's 525,960 minutes in a year. That's 105,120 images/videos/audios per year -- 288 a day. If we maxed out a 1000W PSU, running this machine 24/7/365 would then use 24kWh of power a day. At an electric price of $0.25/kWh, which is pretty high, that puts an upper cost in the $2200/yr range.
So, for about $6, anybody can churn out close to 300 made up images every 24 hours. For a government or megacorp, that's a pittance. For a middle-upper class person in a wealthy nation that's not a difficult cost to overcome. Also, don't forget that's per machine, and each of these PCs could be running multiple models/agents at once. If Putin mastered the "firehose of falsehood," then AI is about to put a firehose in the hands of every person of middling means with an internet connection. For wealthy entities, it's about to allow them to create a deluge of misinformation akin to Genesis.
I think you are right. We designed this Internet system to tell us what is REAL, but suddenly the photos and videos and speeches we recognized as real can be falsified by anyone, cheaply and quickly. I'm going to start watching out for this in my life: the Big Lies may soon be available to anyone with a little money and no honor, and there are a lot of people like that.
Thank you! So Mr. Silver is just credulously reporting whatever garbage hype Sam Altman is troweling out? Open AI will be bankrupt in a year, no more than 2. Along with Oracle and good riddance to both. Ed Zitron has been the lone journalist actually reporting on the reality of the LLMs, and is currently the most vindicated person on Earth.
I decided to stop taking Nate seriously on AI when he thought an LLM could be capable of playing poker. They can’t do simple math, puzzles or play checkers but he thought it was worth his time to see if it could play poker? They may not be entirely hallucination prone chat bots but Nates overeagerness for AGI to happen seems to make him just naive on this topic as the left, just in the opposite direction.
This is what I don't understand. If ai is going to inevitably cause massive job losses. ( e.g . How many drivers of trucks, buses etc will be put out of work when self driving vehicles are the norm?) If massive job losses occur, who then will be able to afford buying the stuff that ai will make?
One of the ways to evaluate the near-term future of AI is to look at health care. AI is entering the health care professions under the aegis of physicians, nurses, pharmacists, and other experts. The emphasis is on the last word in that sentence. AI’s contributions are being rigorously tested and compared with large bodies of evidence as to the accuracy and success of human experts. Under the dictum of do no harm, and with people’s lives literally at stake, AI does not get a free pass. So far, it is a very useful tool to augment clinical work, but is far from substituting for human experts.
That said, in the US, we do not have a health care system. We have a health care market. The clinical experts are only in charge at the lowest level. The payers (the federal government and private insurance companies) are really in charge. Coupled with large-scale hospital networks seeking ever cheaper ways to deliver care, we may see extremely strong temptations to bring these platforms into the clinical setting to cut costs and increase profits. Even non-profit entities must compete on the same playing ground as for-profits.
So, the future of AI health care is both sunny and cloudy.
I love your saying we only have a health care market. I don't think the medical industry could get much worse than it is here ---- oriented to get the most money out of patients for the least service with lots of "care" that people would be better off without, such as vaccinations that don't work.
Regarding Nate's footnote #4, I highly recommend Iain M. Banks' The Player of Games, which has a very similar central idea. The main character is a master game player in Banks' post-scarcity, AI-steered, Culture who is recruited to play in what you might think of as the ultimate poker/chess/everything-else tournament for control of an interstellar empire:
As far as the broader subject of the article goes, I think Altman is right that humans will always find new things to want and new things to do, but I think Nate is right that the changes won't be gentle. As long as there is a gap between saying, "I want X," and having an AI being able to instantly deliver X, there will be a role for humans in trying to fill that gap -- whether to do something that AI can't do or in figuring out a way to make AI smarter. The trick will be finding the right spot in the value chain when the set of "things that AI can't do" is constantly changing and shrinking.
I think we'll switch from a service economy to a "responsibility" economy
As in its not nesc. my job to create code for some service at BigCorp. But I need to be the one responsible for it in a human accountability sense. If AI writes 100% of the code, I'm the one steering the AI. I would wear many hats (technical, product, on-call, etc for this service). I make sure we don't have a massive disaster due to unattended AI, and when all the agents stop churning on fixing bugs, I'm there to reorient them.
But even that's a massive hit to the economy, and I'm probably doing (with AI) what a team of 10 was doing pre Covid.
Modest suggestion: Claude for copy editing, rather than GPT.
After 500 quant solo dev experiments with GPT, several thousand deep conversations, and a few dozen experiments done by other (developer) colleagues ($200k in API tokens) - GPT Pro is backup only for very niche cases. For copyediting important stuff like pitch decks to investors, compliance doc evidence for regulatory stuff, and just deep nested complex writing in general, Claude Op 4x has been solid, no drama with rewriting unexpectedly, and way way lower rate of hallucinating.
If I were your editor, I would say this is a strong premise, but when you conjure “profound and unpredictable political impacts,” you owe the reader more specific consequences. This essay stops short of the good stuff. Like, the rise of a new party structure? Or the French Revolution? Unpredictable means nobody really knows, but we read you because we value your analytics based imagination. Complete this essay. Imagine more.
No one has any real idea what will happen and the unpredictability is more profound than in all prior events.
It's the same kind of frustrating vagueness that the Douthat piece was talking about regarding people saying people need to "prepare for AI". Like what does that mean specifically?
I've been a software engineer for about 20 years. I'll say that, I'm actively embracing AI because, as I've gotten older, I actually kind of hate spending hours and hours writing code, or learning a new language, etc ... I used to like this, but after 20 years, it is boring and tedious, and I just want to build cool things. So on the one hand, I'm hugely excited. I dusted off one of my hobby projects that I started back 2020 and AI has super charged adding new features, fixing bugs, etc ... On the other hand though: I've seen how the trajectory capitalism in this country has gone. Globalization was supposed to make "everything cheaper" and it certainly did, but the cost came at the blue collar class's sense of dignity, their place in community and society, and their optimism about the future. It hollowed out the soul of many lower and middle class communities; you cannot replace that with a cheap TV from China. Now the same trajectory seems to be playing out for potentially upper middle class and anyone out side of high net worth groups. To think this won't have extreme, and possibly violent, political ramifications is not just naive; it's profoundly arrogant.
Globalization didn’t take blue collar jobs in the USA, robots did.
Those jobs aren’t coming back because labor is too expensive for manufacturing to work the way it used to. If we build up manufacturing from here, it will be automated.
As evidence, see how manufacturing grew continuously and then flattened out since around 2010, while manufacturing jobs have been decreasing since 1980, but particularly since 2000:
https://fred.stlouisfed.org/series/INDPRO
https://fred.stlouisfed.org/series/manemp
This is a long term positive trend (more production per unit of labor) that causes real costs to those who lose their jobs. I don’t know what replaces those jobs longer term, but the pain now is real, especially to communities that relied on factory employment.
How long until the pain subsides then? Its been nearly 3 or 4 decades of this decline at this point, no matter what the cause is, and all we've gotten out of the bargain is Donald Trump.
The big job losses have been since 2000, it was a slower decline from 1980 to then. Thats probably why it is coming to a head now.
As long as manual manufacturing jobs exist, they will keep getting replaced by robots or offshored.
One problem is that people want to remain living in their small towns (often for good reasons, like family) but there are few businesses that make sense in those areas. Another problem is that it is often too expensive for people to move from small towns to where jobs are because housing prices are out of control, which causes an inelasticity in the job market and prolonging the pain.
I don’t have solutions to these problem or a clear idea of when the job losses will end. The closest I can come is to suggest that we need to build more housing to bring prices down in areas where there are jobs.
The housing price problem was really only a recent phenomenon, many of these people don't have the skills to match jobs you're talking about; and the ones you are talking about are now on the hit list for automation as well. I bet you live in a city; and cannot imagine why other people would not want to -- and to suggest "they have to to be employable" is exactly the kind of thing that built resentment up in the people who've been electing people like Trump.
The housing price phenomenon goes back to well before the housing crisis in some states, including mine.
I do not live in a city, I live in a semi-rural suburb outside of a major city. I also spent a lot of my youth on a ranch and can understand exactly why people would prefer not to live in a city.
The reality is that there are fewer and fewer job opportunities in rural areas unless your job allows remote working. Automation has come for farming and manufacturing.
The lack of jobs isn’t going to change until someone comes up with jobs that can be done profitably far from urban centers. That forces people to either choose to live un(or under)employed or to move to where jobs exist. I’m sorry that people are offended by that, but it is how reality works.
It seems they also have the choice to make politics a blood sport as well, and have been. Hows that been going for us?
"but the cost came at the blue collar class's sense of dignity, their place in community and society, and their optimism about the future."
True for some specific areas, false in aggregate. There are very few Americans that could get a better, higher paying job in 1970 then they could today. Maybe try not cycling through trite populist nonsense as if these were your original thoughts. Saying "you've seen trajectories" while embracing wrong and popular banal slogans.
Nate, I'm pretty sure the AI companies are “Theranos”ing all of us at this point (okay, that’s a little extreme, I don’t think it’s outright fraud, but definitely overoptimism for the sake of pulling in investments). Like obviously LLMs are real and tangible, but the progress is becoming increasingly slow and the AI companies aren’t eager to own up to that fact. To take a rather infamous old example, strapping a calculator to a model because it doesn’t know how to do math doesn’t fundamentally fix the underlying issue causing the model to suck at math. Are these models still getting appreciably better or are they just taping a band-aid over a gaping wound over and over again?
I use AI for literature searching, primarily because search engines have become so terrible that only the AI models built into the search engines can turn anything up reliably. EVERY TIME, there is obviously incorrect information in the write up. Just patently incorrect. Sometimes it doesn’t even follow the assertions made earlier. The papers it turns up are fine, summarising a single paper is fine, but the moment you ask an LLM from a synthesis of multiple sources, the wheels inevitably come off. The “check this response” feature won’t even turn up most errors. It’ll find one line it can verify and mark that and ignore the other factual assertions. How does this tech cause massive, long-term technological disruption if the credibility of its output is worse than a coin flip?
Well, there is one way, dependency. It doesn’t matter if the models are good or not if students are trained to be dependent on them, but that’s not singularity.
I’m also still waiting to hear what their plan is to create new training sets. Most of the improvements so far have been from sucking in more training data. The problem is at this point a large amount of new content online is AI generated. We know that, for some reason (actually a fairly well understood one), training an AI on AI output causes model collapse. If you can’t separate AI output from human output, how can you create a pure training set that you haven’t already used?
This “no-ceiling” rhetoric may end up being correct, but the negative case is compelling to me and is not frequently considered (beyond the shallow chat bot critique you call out). Obviously big tech isn’t interested in that argument because they’re financially entangled with the AI firms, that means that all of the “experts” aren’t really able to be objective on the question. (As an aside, when you frame the question as whether OpenAI will *announce* AGI, I think 16% is fair odds based on how many times in history the definition of AI has changed and the number of times a company has announced something they didn’t actually create)
Researching with an LLM is an under-appreciated skill. When you have to research something *that matters* and not just ask a casual question, you see the problems
* They want to confirm your priors, not challenge them. They're eager to please you. You have to work against that in your prompting
* They can be bad at evaluating evidence quality. They make very confident assertions based on what they find, unless you ask them to be systematic about evaluating this
* They can hallucinate even about the content they find (they might hallucinate in the voice of that content now)
* You still need to search and find novel evidence for it to consider
* After enough context the LLM takes on a kind of "personality" from all its context that it suddenly has its own strong, specific priors
I wrote about this
https://softwaredoug.com/blog/2025/08/19/researching-with-agents
I think its going to be a major issue. And we probably need to teach it as a skill. Not just assume the LLM is magic.
I think a lot of what you’ve put here is broadly correct. LLMs are a productivity tool, like Word or Excel, and literacy in a tool you’re using is important. It’s exactly that reason why I wish the AI folks would be more honest in their assessments of their models. The problem is that their continued success, at this point, is derived from keeping the mood high and pulling in investors for as long as they can. LLMs are great at a certain subset of tasks and really terrible at others, but companies are all too happy to enable and encourage people to use them for the latter class of tasks. It’s hard to convince people that they shouldn’t use (or need to be careful using) a tool for a task that it says they can use it for.
You're conflating research with analysis. They're good at research (sifting through sources, identifying relevant information, summarizing) but I wouldn't use them for analysis (evaluating evidence, drawing conclusions, logical implications, etc.) Hallucinations happen, but IME (at least recently) they get facts literally wrong but 'semantically' correct. So they'll mis-name section (possibly improving it!) when summarizing a document.
Definitely not a magic answer machine, but also clearly a productivity enhancer for research.
If you haven’t read about the lawyers getting in trouble by using LLMs for research, you should.
It makes it clear that LLMs completely make up relevant information and sources fairly often, making them unreliable for anything that actually matters.
I too use AI for literature searching, but I don't agree there are errors every time, or even that often now. I am impressed that results are getting better and better. When I started with it last year, there were errors and incongruities more often, but now I am not seeing them much (you do have to watch, and importantly, you have to know a lot about what you are researching --- this is why young people are going to get caught by errors, they don't know things.). Yesterday I asked why the Helvetii REALLY left, well, Switzerland, to storm through allied and Roman lands 300,000 strong until Julius Caesar caught up with them. They burned all their own houses and supplies, too, so they couldn't come back. Caesar has them saying they did it for their young men to have a chance to show their valour! This was absurd, so I asked my AI why they really did it. It said warlike and vicious Germans had been invading and settling their land, after they had made the mistake of hiring some thousands of them as mercenaries, and those decided to bring their families and claim the land. This is the whole story of the Fall of Rome from then well into the seventh century, and longer in Britain, so good: and later in Book I of the Gallic Wars Caesar shows the elders of the Helvetii telling him the exact same thing, "weeping and throwing themselves at his feet" and forgetting all that nonsense about them emigrating to let the young men show their valour. I am usually finding the AI right about Latin and modern literature these days, much better than last year.
You do have to know your subject though, I would agree. And it doesn't hurt to check.
In my experience, the LLMs struggle with nuanced distinctions that prevent extrapolation from one area to another. Most of the errors I catch are misidentification of an equation as being appropriate when it’s assumptions aren’t met, inappropriate extrapolation from results in research that is related in methodology, but not in substance, and just plain filling in missing details with numbers not present in the cited works. Sometimes the errors are glaring, other times they’re highly nuanced, but the fact that there may be some unknown number of nuanced errors makes the output impossible for me to rely on outside of the sources it cites. I think it’s pretty good at turning up sources, but I just can’t trust it beyond that.
In history/literature it’s not surprising to me that these axes of failure would be less pronounced as the lines of when analogy and extrapolation are appropriate are much less clear.
Yeah ---- I have also seen some mistakes in plain numbers, if not as elaborate errors as the ones you are presumably working with. It is unnerving. I'm thinking 2026 will be used to get these models much improved, and then we'll see how they do. It could be awhile before we can really trust them, if we ever do.
We can hope! I get worried that “fixing error rate” isn’t splashy enough (or is too hard) for them to commit to, but I’d be pleased to see them take that seriously. Selfishly, I’d love if they’d just make a mode where I could pose a research topic and it gave me a list of papers broken down by topic area with short summaries of each instead of trying to synthesise a single answer from multiple sources, but that’s probably too niche for them to pursue
You could ask for exactly that in your prompt: AIs love elaborate prompts. This is the research topic, and I want a list of 15 papers on these five subjects pertaining to the general topic, three in each.
I know you don't need advice from me, but it was an irresistable prompt question.
Handing a calculator to a model because he or she isn't good at math is a perfectly reasonable decision that doesn't invalidate the idea that humans are intelligent..
Is it really unreasonable for AI to use a tool to solve a problem that isn't amenable to pure LLM techniques?
No, it’s not, but then presenting it as evidence of your model now being good at math is dishonest. LLMs being bad at basic arithmetic is evidence that they struggle with reasoning that we consider basic and fundamental and should temper our expectations of these models.
AI is not a synonym for LLM.
Presenting an AI system as good at math because the LLM has been programmed or trained to send math problems to a specialized engine seems perfectly reasonable.
AI is not a synonym for LLM, correct. However, the major improvements in AI technology have recently been from LLMs. The bullish predictions of continued AI improvements are rooted in optimism in LLM models continuing to improve. Strapping a calculator to an LLM may render that system serviceable for doing arithmetic. It doesn’t resolve the fundamental issues in reasoning that make those arithmetic issues arise in the first place.
My point is that we should reign in our expectations for where the ceiling of this technology is. Handing the AI a calculator is emblematic of this because a fundamental issue in LLMs had to be resolved in a method other than LLM improvement. That’s not to say that these systems won’t get better at all, but if your prediction is highly disruptive AGI in the very near future (or even substantial economic displacement due to AI), I’m not sure that there’s the evidence to support it. If you want to plan for it because it’s a low probability scenario with very severe consequences, sure, but let’s frame it that way instead of an inevitable outcome.
And my point is that most humans need a calculator too.
Airplanes don't fly by flapping their wings like birds.
An LLM that correctly parses a math question and hands it to an expert subsystem is still an AI system solving a problem.
You’re completely missing the point: there is nothing wrong with strapping a calculator to a LLM to solve a math problem if your goal is specifically to have an LLM solve a math problem. The problem is that it does suggest a limit to rate of improvement of the underlying architecture. Unless you believe that the primary limiting factor in further AI improvement is not further improvements to LLM architecture and training sets, but access to subsystems.
The fact that letting the AI use a calculator fixed *an* issue is non responsive to my concern that the evidence that AI technology is approaching the point of undermining the current economic system is sorely lacking. I picked the calculator as an example where the solution couldn’t be attained by architectural and training set improvements, which is the assumption (in my eyes) underpinning the most bullish expectations of AI, but by tying in an already well known subsystem.
Hi Nate,
Long time fan. I recently wrote a guide to AI catastrophe that I think is superior to existing offerings (it assumes less things, makes the case more rigorously, is intended for a lay audience, doesn't have weird quirks or science fiction, etc):
https://linch.substack.com/p/simplest-case-ai-catastrophe
Many of your subscribers likely have a vague understanding that some experts and professionals in the field believe that AI catastrophes (the type that kills you, not just cause job loss) is likely, but may not have a good understanding of why. I hope this guide makes things clearer.
I don't think the case I sketched out above is *overwhelmingly* likely, but I want to make it clear that many of the top people who worked on AI in the past like Geoffrey Hinton and Yoshua Bengio think it's pretty plausible, not just a small tail risk we might want to spend a bit of resources on mitigating, like asteroids.
One thing I always think about is how much the current economy runs on many things we'd consider frivolous 50 years ago. The excess labor savings from automation has gone into building social media apps, creating+supporting prominent influencers, and many frivolities.
Somehow human beings find some 1% niche after 99% of their lives have been automated, and they find economic value there. Maybe for silly reasons like social status and influence, not due to deeper "this will feed my family" reasons.
"The excess labor savings from automation has gone into building social media apps, creating+supporting prominent influencers, and many frivolities."
Yeah and we're all more anxious and depressed than ever. Because that 1% is increasingly hollow and vapid.
Let me channel a commentator long past his prime.
New Rule: You only get one of these "AI will change everything. You need to be worried about it" columns. It's fine - I don't disagree with a premise as uncontroversial as "if AI is half as important as Big Tech says it is, it will be a big deal." I agree! Big Tech compares AI to fire and the wheel all the time - if it is half as important as those two, it will definitely shake things up.
So you've played this card now. Ross Douthat has played it several dozen times. The next step is for you, Nate, to give us some ideas on how AI will change things, how the economy will function with all knowledge workers laid off (and how this will not somehow drag down company valuations), the probability of AGI by 2027, and what Democrats should do about any of this. I need my intelligentsia - the Yglesiases, the Silvers, and the Douthats to give me something more concrete and actionable and valuable than this "well it will probably change everything, but I won't even hazard a guess as to when and how" laziness. Frankly, I expect you guys to be guiding us and figuring this stuff out - not Hakeem Jeffries and Mike Johnson.
AI is neither fire nor the wheel.
It is a system for collecting and communicating information.
It is more like writing than a specific tool.
OK! If AI is half as important as writing itself, it'll definitely shake things up. Now let's make some predictions.
As a programmer, I think the disruption is coming and coming quick. The latest Anthropic and OpenAI models literally can code faster and mostly better than me with skilled usage. It wouldn't take much for them to be able to code faster and better than me with unskilled usage.
Recursive self improvement isn't theoretical, it has already started. Both Anthropic and OpenAI's latest models were mostly coded by earlier versions of themselves.
The part that I'm most curious about is how well this generalizes, especially into relatively data-poor domains. I know there are several companies doing expert annotation and extensions on existing and simulated datasets for a bunch of fields, but for all kinds of reasons (among them the compositions of simulated data in some fields) I suspect that code writing (with decades of easily tokenized forum posts, to say nothing of github) is relatively easier than some other skills because the public dataset allone is much larger and better documented than in, say materials science. Not that AI of whatever type won't be able to automate taksks in those fields, but the scope of automation seems intuitively unlikely to grow as quickly as it did and will in software development.
There are already companies starting up, hiring people to annotate / train these systems on those tasks for this exact reason.
I said as much. It really doesn't answer the question of how long it will take to automate these fields to the extent that code generation is automated now.
I imagine faster than people taking comfort in that bottle neck would like to believe; it's more a raw capital expense with less physical constraints (like the ones building the data centers). The only one is probably number of people you can hire for the job, and then after that is "how much money" you have to pay them for the work.
https://www.wsj.com/tech/ai/training-ai-job-seekers-contractors-1a7bd492?st=95TPLN&reflink=desktopwebshare_permalink
What do you mean by "mostly coded"? Models are "trained", not "coded". What specific contribution do you have in mind, from the earlier versions to the latest version? Can you be more explicit about the precise content of this "mostly coded" that you have in mind?
Models are coded, then trained.
More specifically, there is software running on the AI hardware, and there are weights that the software uses to generate the results.
And of course training is different software than inference.
For that, I will direct you to the blog posts where they announced the new models. I can't explain it any better than the companies themselves.
https://openai.com/index/introducing-gpt-5-3-codex/
https://www.anthropic.com/news/claude-opus-4-6
If we take Sam Altman’s example of the Industrial Revolution, there were worker unions pushing back and demanding a fair piece of the growing economic pie. Today the worker union memberships are in constant decline and we have undermined worker laws. I think strengthening the horizontal connections, weakened by the shift from manufacturing to service-sector jobs, is the first step to securing our future.
I have two main thoughts about AI & politics: The people in control of AI platforms & models are getting extremely wealthy off of it, and that this technology will probably tear apart several nations (or reduce them to some form of totalitarianism) very violently, including ours.
When people like Altman & Musk talk about how great the world will be with AI and a "post-scarcity" society I think about history. I can't remember where I heard the line, but the best way I've seen it put is this: When has a very small group of people gathered up almost all of the wealth & power in a society, before voluntarily redistributing it in an equitable way? Never. It doesn't happen, and these authority figures have been anything but deserving of the kind of trust required for something like that to even plausibly work.
On the 2nd point, I fear the ability of AI models to create images, audio, and (soon) video that's so realistic, it's indistinguishable from reality. We're about to enter a time when anybody can conjure anything they want out of nothing. They can make people say things they never said. They can create video of people doing things they've never done.
How does society, any society, function like that?
I mean, imagine someone like Trump. He's already shown a willingness to alter images with these tools to affect his message. What if he were also willing to make it all up out of whole cloth? Or think back to Qanon -- The conspiracy theorists who believe the redhats are fighting against a global cabal of baby cannibals. Imagine nutjobs like that flooding social media with photorealistic videos of Biden killing babies, and MAGA running with it.
Maybe that's the level we find out Trump won't stoop to, but soon someone will. What if someone much worse, like Stalin was, gets an audience using this? And that's all ignoring the effect foreign adversaries will have by doing the same in a much more organized, targeted manner.
I worry that challenging any sort of authority or power will become impossible. We'll be so fractured into our information silos that any sort of organized anything on the part of the public becomes so difficult that even the most repressive, evil regime is too powerful to overcome.
I really like your point about AI being able to make up reality we cannot tell from real. MAYBE that's not so different from the propaganda and false reporting we've had thruout the ages -- no politicians or generals ever told the truth, did they? But we are dependent now on pictures, videos, audios that we think are real, and of course within months, now, that will break, I think.
Your first point I don't worry about much because we have always forever had individuals who had quantillions more money than anyone else. That's not new. It is inevitable around any new tech (railroads for Carnegie and Rockefeller, etc.) and whole economic systems, such as the Latifundia system in ancient Rome, when a few families had farms measured in dozens of square miles, and that was what produced all the wealth in the civilization, including that supporting the military. A few people having MOST of the wealth of a society is bog normal, as the British say, and that is not what caused the uproar and distress of the early and middle industrial revolution. The common people didn't want the mill owner's riches! They just wanted their old jobs back, where they were doing well in a predictable way. Then everything changed and they had to go to work and send their children to work in dark factories 12 hours a day, and they didn't like it. I hope that won't happen now, but if there are pitchforks and torches, it won't be about inequity. It would be about massive unemployment and worsening conditions.
Sure, lying isn't a surprise or unexpected. It's the absolute volume that can be churned out now. Making realistic edits is a relatively time consuming process that someone who knows a program like Photoshop would need to do themselves. A handful of images could be analyzed & scrutinized by experts who could point out irregularities.
Let's say one of these provocative images/videos/audios takes a model 5 minutes to make passable. Passable as in it's good enough to cause serious doubt. There's 525,960 minutes in a year. That's 105,120 images/videos/audios per year -- 288 a day. If we maxed out a 1000W PSU, running this machine 24/7/365 would then use 24kWh of power a day. At an electric price of $0.25/kWh, which is pretty high, that puts an upper cost in the $2200/yr range.
So, for about $6, anybody can churn out close to 300 made up images every 24 hours. For a government or megacorp, that's a pittance. For a middle-upper class person in a wealthy nation that's not a difficult cost to overcome. Also, don't forget that's per machine, and each of these PCs could be running multiple models/agents at once. If Putin mastered the "firehose of falsehood," then AI is about to put a firehose in the hands of every person of middling means with an internet connection. For wealthy entities, it's about to allow them to create a deluge of misinformation akin to Genesis.
I think you are right. We designed this Internet system to tell us what is REAL, but suddenly the photos and videos and speeches we recognized as real can be falsified by anyone, cheaply and quickly. I'm going to start watching out for this in my life: the Big Lies may soon be available to anyone with a little money and no honor, and there are a lot of people like that.
Fuck "AI" and all its evangelists. Go sell your tulip bulbs to some other mug.
This just reads like Nate desperately trying to pump up his portfolio so he can get out unscathed before the bubble inevitably bursts.
I subscribe for data-backed political analysis, not this "vibe-coded' bullshit i just wasted 5 minutes of my life with.
Thank you! So Mr. Silver is just credulously reporting whatever garbage hype Sam Altman is troweling out? Open AI will be bankrupt in a year, no more than 2. Along with Oracle and good riddance to both. Ed Zitron has been the lone journalist actually reporting on the reality of the LLMs, and is currently the most vindicated person on Earth.
Respectfully, I find the AI takes here overly credulous and misinformed by media hype. It's a serious weakness.
AI is overhyped and mostly useless. And doesn't make money. And as to the so-called LLMs, no idea how they could conceivably make money.
I decided to stop taking Nate seriously on AI when he thought an LLM could be capable of playing poker. They can’t do simple math, puzzles or play checkers but he thought it was worth his time to see if it could play poker? They may not be entirely hallucination prone chat bots but Nates overeagerness for AGI to happen seems to make him just naive on this topic as the left, just in the opposite direction.
This is what I don't understand. If ai is going to inevitably cause massive job losses. ( e.g . How many drivers of trucks, buses etc will be put out of work when self driving vehicles are the norm?) If massive job losses occur, who then will be able to afford buying the stuff that ai will make?
One of the ways to evaluate the near-term future of AI is to look at health care. AI is entering the health care professions under the aegis of physicians, nurses, pharmacists, and other experts. The emphasis is on the last word in that sentence. AI’s contributions are being rigorously tested and compared with large bodies of evidence as to the accuracy and success of human experts. Under the dictum of do no harm, and with people’s lives literally at stake, AI does not get a free pass. So far, it is a very useful tool to augment clinical work, but is far from substituting for human experts.
That said, in the US, we do not have a health care system. We have a health care market. The clinical experts are only in charge at the lowest level. The payers (the federal government and private insurance companies) are really in charge. Coupled with large-scale hospital networks seeking ever cheaper ways to deliver care, we may see extremely strong temptations to bring these platforms into the clinical setting to cut costs and increase profits. Even non-profit entities must compete on the same playing ground as for-profits.
So, the future of AI health care is both sunny and cloudy.
I love your saying we only have a health care market. I don't think the medical industry could get much worse than it is here ---- oriented to get the most money out of patients for the least service with lots of "care" that people would be better off without, such as vaccinations that don't work.
Regarding Nate's footnote #4, I highly recommend Iain M. Banks' The Player of Games, which has a very similar central idea. The main character is a master game player in Banks' post-scarcity, AI-steered, Culture who is recruited to play in what you might think of as the ultimate poker/chess/everything-else tournament for control of an interstellar empire:
https://en.wikipedia.org/wiki/The_Player_of_Games
As far as the broader subject of the article goes, I think Altman is right that humans will always find new things to want and new things to do, but I think Nate is right that the changes won't be gentle. As long as there is a gap between saying, "I want X," and having an AI being able to instantly deliver X, there will be a role for humans in trying to fill that gap -- whether to do something that AI can't do or in figuring out a way to make AI smarter. The trick will be finding the right spot in the value chain when the set of "things that AI can't do" is constantly changing and shrinking.
I think we'll switch from a service economy to a "responsibility" economy
As in its not nesc. my job to create code for some service at BigCorp. But I need to be the one responsible for it in a human accountability sense. If AI writes 100% of the code, I'm the one steering the AI. I would wear many hats (technical, product, on-call, etc for this service). I make sure we don't have a massive disaster due to unattended AI, and when all the agents stop churning on fixing bugs, I'm there to reorient them.
But even that's a massive hit to the economy, and I'm probably doing (with AI) what a team of 10 was doing pre Covid.
You’re describing a reverse centaur btw
Modest suggestion: Claude for copy editing, rather than GPT.
After 500 quant solo dev experiments with GPT, several thousand deep conversations, and a few dozen experiments done by other (developer) colleagues ($200k in API tokens) - GPT Pro is backup only for very niche cases. For copyediting important stuff like pitch decks to investors, compliance doc evidence for regulatory stuff, and just deep nested complex writing in general, Claude Op 4x has been solid, no drama with rewriting unexpectedly, and way way lower rate of hallucinating.