I agree quite strongly with your point on RSI! It has always struck me as odd that the inevitability of RSI seems to be generally well-accepted, when I think (like you said) the general pattern with engineering/optimization problems is that the more you improve, the more ridiculously hard the next step of improvement means. So we don't just need exponential capability growth, we need that to outpace the exponential growth in difficulty of inventing the N+1th AI. I don't think it's impossible that this is the case, but I haven't seen a good defence of why we should default to believing it. I think it's valid to say "well, there's still a considerable risk that the laws of intelligence-scaling make RSI possible, so we should still be cautious", but I generally don't get the impression that's the stance that AI-pilled people take. I think they treat RSI as a given and the uncertainty their predictions instead comes from uncertainty about alignment, control, etc.
to counter my own words, I do think that linear improvements in model loss result in roughly exponential capabilities increases in real world capabilities. So I waffle on the day based on these two very much competing priors
I have to come back to this, it got hard to keep reading after you linked a story of 2018!Google's reluctance to aid the US military as representative of their current virtue vs. Anthropic... April 2026 kinda puts the lie to that.
"I think you could say a lot of things about Google employees’ unwillingness to work for defense, but at least they are internally consistent."
Given that you're contrasting them with current Anthropic, using "are" instead of "were" was dissonant enough that I clicked the link to check the date.
I think Google's position and their employees position are often at odds, and i know many current employees who are quite upset about google's stance on some things and have as a result left the company, but I could see why that was confusing
Very minor nitpick: I think black holes still aren’t infinite positive feedback loops. The simple proof is that they haven’t eaten the universe yet (and aren’t going to), the longer reason is that gravity drops off with the square of distance (and also Hawking radiation kills off tiny black holes before they eat us, which is why we don’t all die whenever the Large Hadron Collider turns on). That’s why it’s not really a big deal that the Milky Way is sitting on Sagittarius A*—we’re in a stable orbit around it and it’s not getting significantly bigger anytime soon. So black holes are just another interesting example of “RSI” that levels off reasonably quickly, just like your nuke example.
I agree quite strongly with your point on RSI! It has always struck me as odd that the inevitability of RSI seems to be generally well-accepted, when I think (like you said) the general pattern with engineering/optimization problems is that the more you improve, the more ridiculously hard the next step of improvement means. So we don't just need exponential capability growth, we need that to outpace the exponential growth in difficulty of inventing the N+1th AI. I don't think it's impossible that this is the case, but I haven't seen a good defence of why we should default to believing it. I think it's valid to say "well, there's still a considerable risk that the laws of intelligence-scaling make RSI possible, so we should still be cautious", but I generally don't get the impression that's the stance that AI-pilled people take. I think they treat RSI as a given and the uncertainty their predictions instead comes from uncertainty about alignment, control, etc.
to counter my own words, I do think that linear improvements in model loss result in roughly exponential capabilities increases in real world capabilities. So I waffle on the day based on these two very much competing priors
I have to come back to this, it got hard to keep reading after you linked a story of 2018!Google's reluctance to aid the US military as representative of their current virtue vs. Anthropic... April 2026 kinda puts the lie to that.
i don’t think i said anything about google’s current position
"I think you could say a lot of things about Google employees’ unwillingness to work for defense, but at least they are internally consistent."
Given that you're contrasting them with current Anthropic, using "are" instead of "were" was dissonant enough that I clicked the link to check the date.
I think Google's position and their employees position are often at odds, and i know many current employees who are quite upset about google's stance on some things and have as a result left the company, but I could see why that was confusing
Very minor nitpick: I think black holes still aren’t infinite positive feedback loops. The simple proof is that they haven’t eaten the universe yet (and aren’t going to), the longer reason is that gravity drops off with the square of distance (and also Hawking radiation kills off tiny black holes before they eat us, which is why we don’t all die whenever the Large Hadron Collider turns on). That’s why it’s not really a big deal that the Milky Way is sitting on Sagittarius A*—we’re in a stable orbit around it and it’s not getting significantly bigger anytime soon. So black holes are just another interesting example of “RSI” that levels off reasonably quickly, just like your nuke example.