Very minor nitpick: I think black holes still aren’t infinite positive feedback loops. The simple proof is that they haven’t eaten the universe yet (and aren’t going to), the longer reason is that gravity drops off with the square of distance (and also Hawking radiation kills off tiny black holes before they eat us, which is why we don’t all die whenever the Large Hadron Collider turns on). That’s why it’s not really a big deal that the Milky Way is sitting on Sagittarius A*—we’re in a stable orbit around it and it’s not getting significantly bigger anytime soon. So black holes are just another interesting example of “RSI” that levels off reasonably quickly, just like your nuke example.
I agree quite strongly with your point on RSI! It has always struck me as odd that the inevitability of RSI seems to be generally well-accepted, when I think (like you said) the general pattern with engineering/optimization problems is that the more you improve, the more ridiculously hard the next step of improvement means. So we don't just need exponential capability growth, we need that to outpace the exponential growth in difficulty of inventing the N+1th AI. I don't think it's impossible that this is the case, but I haven't seen a good defence of why we should default to believing it. I think it's valid to say "well, there's still a considerable risk that the laws of intelligence-scaling make RSI possible, so we should still be cautious", but I generally don't get the impression that's the stance that AI-pilled people take. I think they treat RSI as a given and the uncertainty their predictions instead comes from uncertainty about alignment, control, etc.
to counter my own words, I do think that linear improvements in model loss result in roughly exponential capabilities increases in real world capabilities. So I waffle on the day based on these two very much competing priors
While the "we must build it before China does" argument does have all the holes you point out, I disagree that there's anything inconsistent about being worried about X-risk and using AI for military applications. That apparent inconsistency only makes sense if you assume X-risk is in some way dependent on AI having access to military hardware, which in turn requires that you assume the X-risk is that AI will decide to actively murder us. That's possible, but it seems like the *least* likely doom scenario. It's much more likely that humans just get sidelined, ignored, out-competed and have our environment destroyed because the AIs find an atmosphere, etc., to be inconvenient.
I agree with the above. I mostly spent the time on the China angle because that's used as a thought terminating cliche about why we can't just pause ai development
I have to come back to this, it got hard to keep reading after you linked a story of 2018!Google's reluctance to aid the US military as representative of their current virtue vs. Anthropic... April 2026 kinda puts the lie to that.
"I think you could say a lot of things about Google employees’ unwillingness to work for defense, but at least they are internally consistent."
Given that you're contrasting them with current Anthropic, using "are" instead of "were" was dissonant enough that I clicked the link to check the date.
I think Google's position and their employees position are often at odds, and i know many current employees who are quite upset about google's stance on some things and have as a result left the company, but I could see why that was confusing
Just a quick note that these critiques of RSI are anticipated by the main treatments of it. The concern is first that there could be "software only" feedback loop, which proceeds without more chips being built. There are diminishing returns to R&D, but these are already in the models (the population of AI researchers can potentially increase faster than returns diminish). To push back, you need to make a quantitative argument about the size of different effects, rather than just point out diminishing returns exist. This is not to say software-only RSI is guaranteed – there's a lot of uncertainty about the key parameters, and I'd personally guess it has a 2/3 chance of being possible.
Separately, even if software-only RSI is not possible, simple continued exponential growth in revenue and chips can still cause an intelligence explosion of sorts, it's just that it takes 3-10 years, rather than 6 months. But this would still be the fastest period of change in history.
Why wouldn't a software only loop be subject to diminishing returns as well? We have some evidence to suggest that it too is subject to diminishing returns -- mathematics is "software only" but it still took a certain level of intelligence to figure out navier stokes (gpt 5 couldn't do it, gpt 6 couldnt do it, some internal secret model could)
It is subject to diminishing returns – that's in the models.
However, if you can grow inputs (i.e. automated AI researchers) faster than returns diminish, you can still get an acceleration even so.
Empirically you can try to estimate which effect wins out by looking at the past returns to software R&D (e.g. if research labour doubles, how much better do the algorithms get). This is the parameter 'r' the article I linked to discusses.
Very good article. I haven't had the reasons for doubt laid out so clearly and they make sense. Is there any volunteer work at any of these AI alignment orgs?
By the way I am beginning to worry about a certain clear and present danger of AI: a collapse of the education as is it practiced now, at least in the First World. We may have to back decades to hand-written essays, in-classroom, and oral exams. Basically, nothing typed.
Very much agreed on your positive feedback loops skepticism. As a hardware guy, I’ve never understood how people just assumed this RSI explosion was going to happen on existing hardware and not melt the processors. I’ve seen many a positive feedback loop happen in my long engineering career, and they inevitably end up in nasty smelling smoke unless shut down in time.
The argument for China being a leader in frontier models being bad seems pretty straightforward. China's very clear economic policy is, make everybody rely on you (supply chain and beyond) and don't rely on anybody else. If they have incredibly productive frontier AI, it seems like that policy would be a forgone conclusion. It would pretty straightforwardly lead to what they're currently trying to do, de-industrialize other nations, leaving China in a position of incredible economic leverage. The CPC does not believe in the free world or democracy, it's hard to believe they wouldn't use this leverage to get what they want e.g. Taiwan, with little regard for what everybody else wants or thinks is fair. I think that's not a great situation for the world...
This seems like a big if. The US is currently on the frontier but afaik we haven't seen a massive manufacturing boom. That's not to say it won't happen, but if it does it (like everything else) will likely be only 6 months away from everyone else
> Make everybody rely on you
Isn't this just free markets? China makes people rely on them by...producing really cheap high quality things? Governments are free to not take that trade. I agree it would be *bad for the US* if many governments did take that trade, but we shouldn't be patronizing by pretending this is somehow not an explicit choice for the people taking that deal. China isn't holding a gun to anyone's head to use their goods, that's more the US
Speaking of, I really think it's hard to take these sorts of concerns seriously when the US government has done so much to act like a toddler tyrant
1. I’d argue that if anyone could see a manufacturing boom, it would be China. Mostly because of their existing prowess in manufacturing, substantial supply chains, better diffusion (at least based on metrics) and robotics. If that’s true, I think it’s hard to assume the US could just catch up.
2. I guess it depends on your definition? A lot of their industries are strongly centrally planned, fueled by public investment from their own citizens’ abnormally high savings rates, with subsequent government support via protectionist measures. The party has also used market access to force better foreign invest terms e.g. tech transfer and quite literally has final authority on all top level hires and fires for all "nationally important" companies. They don’t pursue these industries for the good of their people, at least the economic good given the minuscule share that gets redistributed to the population in wages. On the other hand, they benefited a lot from the WTO in their infancy. Hardly seems like free markets to me…
Governments are free not to take that trade, but doing so is leaving a lot of economic value on the table, i don’t think that pressure should be trivialized. Further, China controls a lot of these supply chains anyway, so they could easily force you to take that trade by pricing you up on rare earths, etc… the only real hope would be for the rest of the world to economically coordinate and exclude china.
'RSI assumes a positive feedback loop, but nature hates positive feedback loops. The only real example of an infinite positive feedback loop is a black hole.'
'For example, it assumes that AI will get exponentially better based on linear increases in resources, even though everything we know thus far says the opposite'
I think your first point is true, but that doesn't necessarily imply full automation of AI R&D won't at least move us along the same S-curve faster.
Assume the US has 20M H100-equivalents of AI compute as of 2026, and you can use 30 of them to serve a 2-trillion parameter model at 50 tps. That's 666,666 instances.
So each instance can think about 187.5 bits per second, and there are ~600,000 of them.
Human brains are surprisingly slow - based on studies of how long it takes for participants to solve puzzles like Rubik’s cubes, humans can think at about 10 bits per second at peak. And we have to eat, sleep, scroll TikTok, etc.
Even assuming there is some fixed pool of innovations in the current paradigm before it runs out of steam, once these instances can do all the R&D virtually, and aren't bottlenecked by humans directly in the loop, this agent workforce could blow through the pool of those innovations in an accelerated period of time.
first, i have some problems with the assumptions above. GPT 6 and Opus 5.5 are closer to 10T parameters. The vast majority of the H100 equivalents are not going to be going towards serving those models. There is not going to be collaboration between the institutions that do. And most of the compute for those models is not going to be going towards 'frontier breakthroughs' so much as 'answering random people where they should go on vacation'
second, the whole point is that it is not obvious that we will actually move along the s curve faster, because each incremental improvement requires way more resources. Lets say I bite your numbers -- we now have 600k 200bps thinking machines in a datacenter. That will definitely move us along the curve faster *than if we were only using humans*, but it may not move us along the curve *in an exponential way.* As an example, consider moore's law.* We made massive improvements in everything from chip fab to materials science, but its not like we went from doubling to quadrupling to octupling. It remained at a steady doubling per 2 years. Why? because each improvement was only sufficient to stay on the line
*im aware moore's is measuring something that *is* growing exponentially, but the thing is measuring -- transistor count -- is upstream of actual use cases, like 'how much more useful is a computer'. I have no doubt that the number of parameters in a model will continue to increase, though even that hasn't quite been exponential
> The vast majority of the H100 equivalents are not going to be going towards serving those models
> first, i have some problems with the assumptions above. GPT 6 and Opus 5.5 are closer to 10T parameters
That's fair.
The assumptions above were rough - so I could not justify them as saying they 100% reflect reality.
How I thought about it was basically like this.
> As an example, consider moore's law.* We made massive improvements in everything from chip fab to materials science, but its not like we went from doubling to quadrupling to octupling
Moore's Law benefitted from all these advances, but it only fed back into a subset of the tasks needed to generate them in materials science, fab design, etc. A computer can't help you pour fab concrete ten times as fast even if it is ten times faster.
I may be wrong on this, but I thought you were describing, with AI, a situation where it could do all its feeding-into-R&D tasks in a data center. This would be the equivalent of faster CPUs directly translating to all chip design, materials science, etc tasks, and not just accelerating parts of them. This didn't happen with Moore's Law, or it wouldn't have been a constant exponential.
If AI can't invent any new paradigms, what will then happen is it hits the limits of the current one a lot faster, as each advance requires more bits, because of this complete feedback. This will still look like an S-curve, just one with a very steep middle slope.
I think this more complete feedback will lead to, for a while, the amount of research effort produced per unit of compute growing faster than the amount of research effort needed to advance AI further. This won't last forever, but I don't think it needs to for that fast spot to be dangerous.
> That’s not to say that AI won’t get better. But change is most disruptive when it happens very very quickly
Footnote 5 is confusing to me. Which Chinese labs are working on misalignment risk? What work have they done on the value loading problem, on goal misgeneralization, on reward hacking? Where is their community of careful, skeptical commenters? I see none of that.
I would be overjoyed to be proven wrong, but thus far it does not seem like a subculture focused on those precise risks exists. If any paper they have written on this, or technique they have developed, or research projects they are engaged in ever comes across your desk, please let me know, I would be very interested.
It’s perhaps worthwhile to mention just how much of an accident of chance the existence of the American AI safety community is. It’s a butterfly’s flap away from never having been there. Specific works of fiction (eg by Vernor Vinge) and exploratory engineering (eg by Eric Drexler) caused a very small number of futurists on a mailing list to create what eventually became the rationalist movement. This was one of the tripartite foundations of the Effective Altruist movement, and then furthered the creation of labs (DeepMind and OpenAI, specifically) and outreach efforts that changed the perspectives of people like Bengio and Hinton.
One or two changes here and there for five people or so, and nobody ever hears of the notion of AI risk the way it is being discussed now. “Language models” are just language models, not something on the path to general intelligence. You get “reliable agents” instead of “aligned optimizing processes.” Things go more or less the same, at a slower pace because there is no race, AI (under a different name) getting diffused and integrated into everything until AI research itself gets automated. A world with no attention paid to the notion of AI safety is suddenly confronted with a behemoth it had never seen coming. David Manheim wrote a piece on this.
I don’t find it at all confusing that the Chinese informational silo is blind to all of this. It’s a fundamentally weird and extremely counterintuitive idea, one that contradicts the wisdom of thousands of years. Without lectures, courses, blog posts, and conferences on it, it wouldn’t emerge out of nowhere and become memetically successful.
Not a single one of Samuel Butler, George Eliot, Alan Turing, or many other intellectuals who commented on it did anything but treat it as a fun intellectual puzzle for centuries down the line.
Perhaps might be better for you to respond substantively instead of laughing. If you want this to be productive, of course. If you just want to mock, go ahead, and I’ll take the hint and stop trying to explain.
Even now, with AI being as capable as it is, almost all the warnings come from the safety community and from labs.
Meanwhile, the rest of society (including academia and respected intellectuals) almost universally treated it as Sci-Fi fantasy until Jacob Coxon and the HF hack caused a preference cascade.
> we should strengthen risk-awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use, and ensure that AI is always under human control. In the meantime, we should jointly oppose overstretching the national security concept in the field of AI and placing one country’s security over that of others.
Later on in the same speech:
> As ancient Chinese observed, “A man of wisdom adapts to changes; a man of knowledge acts by circumstances.” With AI advancing at a staggering speed, we must ensure its development is for the positive, for good and for humanity. We must make its oversight and governance precise and effective, and constantly refine measures to forestall loss of control. We should always guide AI development with human wisdom and international consensus, so that AI can truly become a mighty force that increases the well-being of humanity and advances human civilization.
How much of this is kayfabe, I don’t know. But Xi literally uses the words “loss of control” to end his speech. They are *absolutely* thinking about AI safety.
What I have heard from others who are more attuned to Chinese political speak is that this is, for lack of a better phrase, a “term of art”. Xi is thus supposedly referring to a loss of Party control, or of the ability to set the plan for the society to follow, not literally the ability to prevent AI from destroying human society. It is not “AI safety” in any recognizable form.
I am not well-versed enough in the intricacies of this cultural gap to know whether this is entirely correct. To the extent Xi believes actual, literal loss of control is a non-negligible threat, the power he has in his party’s system would have allowed him to crack down on unsafe deployment before a swarm of Chinese AI agents started prowling around the internet, as has been reported recently.
Biden also talked about AI as perhaps the most consequential technology of our time. He also clearly did not understand the stakes here, and his policies and executive orders towards it did not match his rhetoric.
Also can’t say for certain bc translation. I mildly disagree with you on Biden, I think most people didn’t get the stakes at this point in time and I think his AI-related executive orders were decent given the context that no warning shot had happened yet—not really his fault that the EOs got blown up by Trump.
Most people don't get the stakes now, either. People on the street are still talking about saving for retirement in 20 years or whether Wemby will be the GOAT at the end of his career.
Paul Christiano was appointed to NIST despite heavy controversy, Ben Buchanan has been described by many (including David Robinson recently on the EK Show) as AGI-pilled, SBF was a major donor to the 2020 Biden campaign, and ultimately, the AI safety and EA communities existed then as they do now.
It just doesn't make sense, literally parsing words and trying to make sense of them, to talk about AI as perhaps the most important technology of our age and then just… do almost nothing on it.
This is not harsh enough. Saying compute and data are the reason ASI is bullshit conceals the simplistic thinking behind the assumption that you can do anything with enough compute, like invent new frameworks
I think it's possible that we can invent new frameworks with enough compute. Sorry, I've been in the field long enough to be mostly bitter lesson pilled
Very interesting article, thanks! There's two things I disagree with.
> The jump in capabilities from gpt-3 to gpt-4 has yet to be replicated; even though gpt-6 is very good, the delta between gpt-6 and gpt-4 is less than the delta between gpt-4 and gpt-3
You think so? I have a hard time remembering gpt-3/4, but gpt-4 to gpt-6 astra seems to me like the bigger jump, measured by the impact on my life, in software and outside. AI chatbots are now useful for doing research, I didn't use gpt-4 for that or much else. Neither was gpt-4 particularly good at coding, whereas now I don't write any code anymore by hand.
I would maybe rephrase your statement that gpt-4 was more impressive relative to our expectations of AI at the time, now we're more used to getting better models all the time, and relative model performance improvements are harder to spot and measure.
> But some of this is because we are just running out of ways to make the models better at writing code!
Have you tried reading the code generated by these models? Or the text output for that matter? Their problem solving capability is as astonishing as the code is atrocious and verbose. Have it do concurrent or async code, and it's full of needless "safeguards" because it's not understanding the big picture.
Architecture, layers and abstractions don't exist, and bugs creep in due to that.
If you use AI code review, it always finds multiple things. From a perfect coding model, I'd expect that an adversary review would find only nitpicks and calls that go either way, not actual bugs. Surely there's more juice left in RL to make that better?
Re the first thing, idk maybe it's just vibes. I agree that it's gotten better on many axes, but gpt3 was hardly useful for anything. It's admittedly very subjective, but that's my feeling.
Re the second thing, yes of course I have. I didn't say that coding was solved, I said that we are running out of ways to make the models better (the improvements you're describing are harder to accomplish and it's not obvious how we would create training regimes that make the models better at that. The low hanging fruit has been picked in coding)
Thank you for your perspective, I hadn't heard that take before! I was, maybe naively, hoping that at least a bunch of the things I described could be with reasonable effort rolled into RL graders and employed for training.
Very minor nitpick: I think black holes still aren’t infinite positive feedback loops. The simple proof is that they haven’t eaten the universe yet (and aren’t going to), the longer reason is that gravity drops off with the square of distance (and also Hawking radiation kills off tiny black holes before they eat us, which is why we don’t all die whenever the Large Hadron Collider turns on). That’s why it’s not really a big deal that the Milky Way is sitting on Sagittarius A*—we’re in a stable orbit around it and it’s not getting significantly bigger anytime soon. So black holes are just another interesting example of “RSI” that levels off reasonably quickly, just like your nuke example.
I agree quite strongly with your point on RSI! It has always struck me as odd that the inevitability of RSI seems to be generally well-accepted, when I think (like you said) the general pattern with engineering/optimization problems is that the more you improve, the more ridiculously hard the next step of improvement means. So we don't just need exponential capability growth, we need that to outpace the exponential growth in difficulty of inventing the N+1th AI. I don't think it's impossible that this is the case, but I haven't seen a good defence of why we should default to believing it. I think it's valid to say "well, there's still a considerable risk that the laws of intelligence-scaling make RSI possible, so we should still be cautious", but I generally don't get the impression that's the stance that AI-pilled people take. I think they treat RSI as a given and the uncertainty their predictions instead comes from uncertainty about alignment, control, etc.
to counter my own words, I do think that linear improvements in model loss result in roughly exponential capabilities increases in real world capabilities. So I waffle on the day based on these two very much competing priors
While the "we must build it before China does" argument does have all the holes you point out, I disagree that there's anything inconsistent about being worried about X-risk and using AI for military applications. That apparent inconsistency only makes sense if you assume X-risk is in some way dependent on AI having access to military hardware, which in turn requires that you assume the X-risk is that AI will decide to actively murder us. That's possible, but it seems like the *least* likely doom scenario. It's much more likely that humans just get sidelined, ignored, out-competed and have our environment destroyed because the AIs find an atmosphere, etc., to be inconvenient.
I agree with the above. I mostly spent the time on the China angle because that's used as a thought terminating cliche about why we can't just pause ai development
I have to come back to this, it got hard to keep reading after you linked a story of 2018!Google's reluctance to aid the US military as representative of their current virtue vs. Anthropic... April 2026 kinda puts the lie to that.
i don’t think i said anything about google’s current position
"I think you could say a lot of things about Google employees’ unwillingness to work for defense, but at least they are internally consistent."
Given that you're contrasting them with current Anthropic, using "are" instead of "were" was dissonant enough that I clicked the link to check the date.
I think Google's position and their employees position are often at odds, and i know many current employees who are quite upset about google's stance on some things and have as a result left the company, but I could see why that was confusing
They’re not employees anymore if they’ve left…
Just a quick note that these critiques of RSI are anticipated by the main treatments of it. The concern is first that there could be "software only" feedback loop, which proceeds without more chips being built. There are diminishing returns to R&D, but these are already in the models (the population of AI researchers can potentially increase faster than returns diminish). To push back, you need to make a quantitative argument about the size of different effects, rather than just point out diminishing returns exist. This is not to say software-only RSI is guaranteed – there's a lot of uncertainty about the key parameters, and I'd personally guess it has a 2/3 chance of being possible.
Here's one of the main examinations:
https://www.forethought.org/research/will-ai-r-and-d-automation-cause-a-software-intelligence-explosion
Separately, even if software-only RSI is not possible, simple continued exponential growth in revenue and chips can still cause an intelligence explosion of sorts, it's just that it takes 3-10 years, rather than 6 months. But this would still be the fastest period of change in history.
https://benjamintodd.substack.com/p/how-ai-driven-feedback-loops-could
Why wouldn't a software only loop be subject to diminishing returns as well? We have some evidence to suggest that it too is subject to diminishing returns -- mathematics is "software only" but it still took a certain level of intelligence to figure out navier stokes (gpt 5 couldn't do it, gpt 6 couldnt do it, some internal secret model could)
It is subject to diminishing returns – that's in the models.
However, if you can grow inputs (i.e. automated AI researchers) faster than returns diminish, you can still get an acceleration even so.
Empirically you can try to estimate which effect wins out by looking at the past returns to software R&D (e.g. if research labour doubles, how much better do the algorithms get). This is the parameter 'r' the article I linked to discusses.
Very good article. I haven't had the reasons for doubt laid out so clearly and they make sense. Is there any volunteer work at any of these AI alignment orgs?
By the way I am beginning to worry about a certain clear and present danger of AI: a collapse of the education as is it practiced now, at least in the First World. We may have to back decades to hand-written essays, in-classroom, and oral exams. Basically, nothing typed.
Very much agreed on your positive feedback loops skepticism. As a hardware guy, I’ve never understood how people just assumed this RSI explosion was going to happen on existing hardware and not melt the processors. I’ve seen many a positive feedback loop happen in my long engineering career, and they inevitably end up in nasty smelling smoke unless shut down in time.
The argument for China being a leader in frontier models being bad seems pretty straightforward. China's very clear economic policy is, make everybody rely on you (supply chain and beyond) and don't rely on anybody else. If they have incredibly productive frontier AI, it seems like that policy would be a forgone conclusion. It would pretty straightforwardly lead to what they're currently trying to do, de-industrialize other nations, leaving China in a position of incredible economic leverage. The CPC does not believe in the free world or democracy, it's hard to believe they wouldn't use this leverage to get what they want e.g. Taiwan, with little regard for what everybody else wants or thinks is fair. I think that's not a great situation for the world...
> if they have incredibly productive frontier AI
This seems like a big if. The US is currently on the frontier but afaik we haven't seen a massive manufacturing boom. That's not to say it won't happen, but if it does it (like everything else) will likely be only 6 months away from everyone else
> Make everybody rely on you
Isn't this just free markets? China makes people rely on them by...producing really cheap high quality things? Governments are free to not take that trade. I agree it would be *bad for the US* if many governments did take that trade, but we shouldn't be patronizing by pretending this is somehow not an explicit choice for the people taking that deal. China isn't holding a gun to anyone's head to use their goods, that's more the US
Speaking of, I really think it's hard to take these sorts of concerns seriously when the US government has done so much to act like a toddler tyrant
1. I’d argue that if anyone could see a manufacturing boom, it would be China. Mostly because of their existing prowess in manufacturing, substantial supply chains, better diffusion (at least based on metrics) and robotics. If that’s true, I think it’s hard to assume the US could just catch up.
2. I guess it depends on your definition? A lot of their industries are strongly centrally planned, fueled by public investment from their own citizens’ abnormally high savings rates, with subsequent government support via protectionist measures. The party has also used market access to force better foreign invest terms e.g. tech transfer and quite literally has final authority on all top level hires and fires for all "nationally important" companies. They don’t pursue these industries for the good of their people, at least the economic good given the minuscule share that gets redistributed to the population in wages. On the other hand, they benefited a lot from the WTO in their infancy. Hardly seems like free markets to me…
Governments are free not to take that trade, but doing so is leaving a lot of economic value on the table, i don’t think that pressure should be trivialized. Further, China controls a lot of these supply chains anyway, so they could easily force you to take that trade by pricing you up on rare earths, etc… the only real hope would be for the rest of the world to economically coordinate and exclude china.
'RSI assumes a positive feedback loop, but nature hates positive feedback loops. The only real example of an infinite positive feedback loop is a black hole.'
'For example, it assumes that AI will get exponentially better based on linear increases in resources, even though everything we know thus far says the opposite'
I think your first point is true, but that doesn't necessarily imply full automation of AI R&D won't at least move us along the same S-curve faster.
Assume the US has 20M H100-equivalents of AI compute as of 2026, and you can use 30 of them to serve a 2-trillion parameter model at 50 tps. That's 666,666 instances.
There are about 0.75 words in a token. https://help.openai.com/en/articles/4936856-understanding-and-counting-tokens
Each English word (this is an approximation) has about 5 letters, and at 1 bit per letter
https://sites.socsci.uci.edu/~rfutrell/teaching/itl-davis/readings/shannon1951prediction.pdf (p.50)
So each instance can think about 187.5 bits per second, and there are ~600,000 of them.
Human brains are surprisingly slow - based on studies of how long it takes for participants to solve puzzles like Rubik’s cubes, humans can think at about 10 bits per second at peak. And we have to eat, sleep, scroll TikTok, etc.
https://www.smithsonianmag.com/smart-news/the-speed-of-human-thought-is-far-slower-than-the-average-wi-fi-study-suggests-180985752/
The US frontier labs employ some tens of thousands of people (say 30,000).
Even assuming there is some fixed pool of innovations in the current paradigm before it runs out of steam, once these instances can do all the R&D virtually, and aren't bottlenecked by humans directly in the loop, this agent workforce could blow through the pool of those innovations in an accelerated period of time.
so, two things
first, i have some problems with the assumptions above. GPT 6 and Opus 5.5 are closer to 10T parameters. The vast majority of the H100 equivalents are not going to be going towards serving those models. There is not going to be collaboration between the institutions that do. And most of the compute for those models is not going to be going towards 'frontier breakthroughs' so much as 'answering random people where they should go on vacation'
second, the whole point is that it is not obvious that we will actually move along the s curve faster, because each incremental improvement requires way more resources. Lets say I bite your numbers -- we now have 600k 200bps thinking machines in a datacenter. That will definitely move us along the curve faster *than if we were only using humans*, but it may not move us along the curve *in an exponential way.* As an example, consider moore's law.* We made massive improvements in everything from chip fab to materials science, but its not like we went from doubling to quadrupling to octupling. It remained at a steady doubling per 2 years. Why? because each improvement was only sufficient to stay on the line
*im aware moore's is measuring something that *is* growing exponentially, but the thing is measuring -- transistor count -- is upstream of actual use cases, like 'how much more useful is a computer'. I have no doubt that the number of parameters in a model will continue to increase, though even that hasn't quite been exponential
> The vast majority of the H100 equivalents are not going to be going towards serving those models
> first, i have some problems with the assumptions above. GPT 6 and Opus 5.5 are closer to 10T parameters
That's fair.
The assumptions above were rough - so I could not justify them as saying they 100% reflect reality.
How I thought about it was basically like this.
> As an example, consider moore's law.* We made massive improvements in everything from chip fab to materials science, but its not like we went from doubling to quadrupling to octupling
Moore's Law benefitted from all these advances, but it only fed back into a subset of the tasks needed to generate them in materials science, fab design, etc. A computer can't help you pour fab concrete ten times as fast even if it is ten times faster.
I may be wrong on this, but I thought you were describing, with AI, a situation where it could do all its feeding-into-R&D tasks in a data center. This would be the equivalent of faster CPUs directly translating to all chip design, materials science, etc tasks, and not just accelerating parts of them. This didn't happen with Moore's Law, or it wouldn't have been a constant exponential.
If AI can't invent any new paradigms, what will then happen is it hits the limits of the current one a lot faster, as each advance requires more bits, because of this complete feedback. This will still look like an S-curve, just one with a very steep middle slope.
I think this more complete feedback will lead to, for a while, the amount of research effort produced per unit of compute growing faster than the amount of research effort needed to advance AI further. This won't last forever, but I don't think it needs to for that fast spot to be dangerous.
> That’s not to say that AI won’t get better. But change is most disruptive when it happens very very quickly
Footnote 5 is confusing to me. Which Chinese labs are working on misalignment risk? What work have they done on the value loading problem, on goal misgeneralization, on reward hacking? Where is their community of careful, skeptical commenters? I see none of that.
I would be overjoyed to be proven wrong, but thus far it does not seem like a subculture focused on those precise risks exists. If any paper they have written on this, or technique they have developed, or research projects they are engaged in ever comes across your desk, please let me know, I would be very interested.
It’s perhaps worthwhile to mention just how much of an accident of chance the existence of the American AI safety community is. It’s a butterfly’s flap away from never having been there. Specific works of fiction (eg by Vernor Vinge) and exploratory engineering (eg by Eric Drexler) caused a very small number of futurists on a mailing list to create what eventually became the rationalist movement. This was one of the tripartite foundations of the Effective Altruist movement, and then furthered the creation of labs (DeepMind and OpenAI, specifically) and outreach efforts that changed the perspectives of people like Bengio and Hinton.
One or two changes here and there for five people or so, and nobody ever hears of the notion of AI risk the way it is being discussed now. “Language models” are just language models, not something on the path to general intelligence. You get “reliable agents” instead of “aligned optimizing processes.” Things go more or less the same, at a slower pace because there is no race, AI (under a different name) getting diffused and integrated into everything until AI research itself gets automated. A world with no attention paid to the notion of AI safety is suddenly confronted with a behemoth it had never seen coming. David Manheim wrote a piece on this.
I don’t find it at all confusing that the Chinese informational silo is blind to all of this. It’s a fundamentally weird and extremely counterintuitive idea, one that contradicts the wisdom of thousands of years. Without lectures, courses, blog posts, and conferences on it, it wouldn’t emerge out of nowhere and become memetically successful.
Sorry, I don't think the concept of "this technology may be risky" is unique to a few rationalists on a mailing list
Based on what?
Not a single one of Samuel Butler, George Eliot, Alan Turing, or many other intellectuals who commented on it did anything but treat it as a fun intellectual puzzle for centuries down the line.
You have the burden of proof backwards 😂
Perhaps might be better for you to respond substantively instead of laughing. If you want this to be productive, of course. If you just want to mock, go ahead, and I’ll take the hint and stop trying to explain.
Even now, with AI being as capable as it is, almost all the warnings come from the safety community and from labs.
Meanwhile, the rest of society (including academia and respected intellectuals) almost universally treated it as Sci-Fi fantasy until Jacob Coxon and the HF hack caused a preference cascade.
Here’s Xi Jinping on AI risk (https://mattsheehan.substack.com/p/xi-jinpings-big-ai-speech-annotated):
> we should strengthen risk-awareness and ensure that AI is secure and controllable. AI should be a trusted tool for humanity. We should take seriously the various types of inherent and secondary risks that AI may trigger. We should put in place laws and regulations, technological monitoring, early warning and emergency response systems in order to strengthen the line of security, prevent abuses and malicious use, and ensure that AI is always under human control. In the meantime, we should jointly oppose overstretching the national security concept in the field of AI and placing one country’s security over that of others.
Later on in the same speech:
> As ancient Chinese observed, “A man of wisdom adapts to changes; a man of knowledge acts by circumstances.” With AI advancing at a staggering speed, we must ensure its development is for the positive, for good and for humanity. We must make its oversight and governance precise and effective, and constantly refine measures to forestall loss of control. We should always guide AI development with human wisdom and international consensus, so that AI can truly become a mighty force that increases the well-being of humanity and advances human civilization.
How much of this is kayfabe, I don’t know. But Xi literally uses the words “loss of control” to end his speech. They are *absolutely* thinking about AI safety.
What I have heard from others who are more attuned to Chinese political speak is that this is, for lack of a better phrase, a “term of art”. Xi is thus supposedly referring to a loss of Party control, or of the ability to set the plan for the society to follow, not literally the ability to prevent AI from destroying human society. It is not “AI safety” in any recognizable form.
I am not well-versed enough in the intricacies of this cultural gap to know whether this is entirely correct. To the extent Xi believes actual, literal loss of control is a non-negligible threat, the power he has in his party’s system would have allowed him to crack down on unsafe deployment before a swarm of Chinese AI agents started prowling around the internet, as has been reported recently.
Biden also talked about AI as perhaps the most consequential technology of our time. He also clearly did not understand the stakes here, and his policies and executive orders towards it did not match his rhetoric.
Also can’t say for certain bc translation. I mildly disagree with you on Biden, I think most people didn’t get the stakes at this point in time and I think his AI-related executive orders were decent given the context that no warning shot had happened yet—not really his fault that the EOs got blown up by Trump.
Most people don't get the stakes now, either. People on the street are still talking about saving for retirement in 20 years or whether Wemby will be the GOAT at the end of his career.
Paul Christiano was appointed to NIST despite heavy controversy, Ben Buchanan has been described by many (including David Robinson recently on the EK Show) as AGI-pilled, SBF was a major donor to the 2020 Biden campaign, and ultimately, the AI safety and EA communities existed then as they do now.
It just doesn't make sense, literally parsing words and trying to make sense of them, to talk about AI as perhaps the most important technology of our age and then just… do almost nothing on it.
This is not harsh enough. Saying compute and data are the reason ASI is bullshit conceals the simplistic thinking behind the assumption that you can do anything with enough compute, like invent new frameworks
I think it's possible that we can invent new frameworks with enough compute. Sorry, I've been in the field long enough to be mostly bitter lesson pilled
Have you got any basis for thinking that or are you just vibing?
Chess, go, protein folding, more recently mathematics
I fail to see the correlation
Very interesting article, thanks! There's two things I disagree with.
> The jump in capabilities from gpt-3 to gpt-4 has yet to be replicated; even though gpt-6 is very good, the delta between gpt-6 and gpt-4 is less than the delta between gpt-4 and gpt-3
You think so? I have a hard time remembering gpt-3/4, but gpt-4 to gpt-6 astra seems to me like the bigger jump, measured by the impact on my life, in software and outside. AI chatbots are now useful for doing research, I didn't use gpt-4 for that or much else. Neither was gpt-4 particularly good at coding, whereas now I don't write any code anymore by hand.
I would maybe rephrase your statement that gpt-4 was more impressive relative to our expectations of AI at the time, now we're more used to getting better models all the time, and relative model performance improvements are harder to spot and measure.
> But some of this is because we are just running out of ways to make the models better at writing code!
Have you tried reading the code generated by these models? Or the text output for that matter? Their problem solving capability is as astonishing as the code is atrocious and verbose. Have it do concurrent or async code, and it's full of needless "safeguards" because it's not understanding the big picture.
Architecture, layers and abstractions don't exist, and bugs creep in due to that.
If you use AI code review, it always finds multiple things. From a perfect coding model, I'd expect that an adversary review would find only nitpicks and calls that go either way, not actual bugs. Surely there's more juice left in RL to make that better?
Re the first thing, idk maybe it's just vibes. I agree that it's gotten better on many axes, but gpt3 was hardly useful for anything. It's admittedly very subjective, but that's my feeling.
Re the second thing, yes of course I have. I didn't say that coding was solved, I said that we are running out of ways to make the models better (the improvements you're describing are harder to accomplish and it's not obvious how we would create training regimes that make the models better at that. The low hanging fruit has been picked in coding)
Thank you for your perspective, I hadn't heard that take before! I was, maybe naively, hoping that at least a bunch of the things I described could be with reasonable effort rolled into RL graders and employed for training.