Anthropic has recently become the bad boy (negative) in Silicon Valley, in large part because the company cannot really muster enough enthusiasm about open source models to write a convincing letter about not wanting to ban them. To their credit, they tried! They wrote a letter on the subject! Unfortunately I think that made things worse, because in said letter Anthropic managed to mention China or the CCP about 12 times, and then kinda sorta argued that they did want to ban open source models after all.
This letter managed to anger approximately all of the commentariat on hackernews. I think the top two things the average HN commenter hates is 1) hypocrisy and 2) the feeling of being condescended to. And unfortunately for Anthropic, the contradictions between ‘we do not want to give our technology to bad people’ and ‘we are ok with our technology being used in a war of aggression in the Middle East’ were pretty hard to ignore.
A few days ago, Dario (CEO of Anthropic) posted a follow up on Twitter.
I think at this point I’ve mostly made my opinions about Twitter known? In case you’re new here, I find it a horrible place, an evil equivalent to a horocrux or the one ring in terms of its ability to warp and destroy even the best of men, and I wish with all my heart that the tech world would simply pick up and move to a different platform.
Among other things (and there are many other things) I’m always impressed by the lack of real thought that seems to go into the average Twitter conversation. I suppose that’s a given, it comes with the platform. The bird app was a thing conceived in the fires of ‘140 characters,’ and though you can put lipstick on a pig and add the ability to write longform articles, a trace of the true self exists in the false self. But man, even knowing all that, I am constantly surprised at how loud the self inflated shrieking can become.
(Also, the agents will eventually syndicate this to Twitter, where I’m sure it will be met with a reasonable and well proportioned response if the post does well)
Anyway, one line in Dario’s message caught my eye that I think mostly goes unaddressed. Dario writes:
Overall my view is that AI is structurally a technology that tends to concentrate power, for reasons that have nothing to do with regulation.
This is basically just stated and is then, based on my extremely limited time on Twitter, never spoken of again. But this is actually a very large claim, isn’t it!
First, it’s worth thinking about the background concept of the idea that technology can structurally be political at all. This is a very unintuitive idea. Most people think that technology is a tool, like a shovel or a trowel, and that it is incoherent to ask about the politics of a tool. What are we going to do next, ask about the politics of the air? It’s all inanimate! To the extent that tools have politics, they are rounded off to the politics of the user of said tool. You see this a lot in discussions about guns. Guns are good in the hands of police protecting an old woman from being mugged and bad in the hands of school shooters, that sort of thing.
The mistake is thinking about a single object, an instance of a thing instead of the class of the thing. Any individual gun is, in fact, an inanimate object. It will not campaign, or vote, or make extremely awkward remarks at the Thanksgiving dinner table. But guns in general are absolutely political. Yes, obviously there are gun owners who have a lot of identity wrapped up in the thing, and some very powerful gun lobbyists, and the talking heads do like to bring the subject up quite a bit around election time. And all of these people will campaign and vote, of course. But that’s not quite what I mean. If you zoom out a bit further, the existence of guns implies many things about society: that many more people can leverage violence, that the state will be pressured to become strong / more invasive / more aggressive to maintain control, that we are willing to accept that it is easier to escalate conflicts in exchange for individual choice. Given all of the above, I view guns as a structurally decentralizing technology.
Guns are obviously a hot topic, but this sort of logic also applies to things that seem pretty neutral day to day. For example, many of the parkway bridges in and around Long Island are built so low that they prevent buses from reaching the beaches and parks. On face, this seems like an unfortunate coincidence. But if you do a bit of research, you’ll find that the guy who was responsible for building the bridges, Robert Moses, was racist, and he did not want the primarily minority inner city families that relied on public transport to go to the city parks.
If I pointed at any Long Island bridge and went ‘that bridge is racist’ you would rightfully think I was crazy. After all, it is a bridge. It is a collection of stone and asphalt and metal with zero intent or consciousness whatsoever.1 What are we even talking about? And yet, the bridge was created by a racist person for racist reasons, and to this day the bridge prevents people who live in the inner city from accessing beaches and parks.2
One of the most influential papers I have ever read about technology and its relationship to the world is Langdon Winner’s 1980 paper, “Do Artifacts Have Politics?” This paper ruthlessly tears apart the idea that technology, especially supposedly neutral technology, has no political impact. Once you see it, you see it everywhere. Benches that have dividers so that people can’t lie down on them. Designing cities around cars, which makes car ownership a requirement, which makes more people become car owners, which makes them more likely to vote in ways that benefit car owners. Ramps, railings, and other forms of ADA accommodations as an explicitly political statement about what we value as a society. And so on. We leverage technology to enforce our political norms, and we embed our political norms in our technology, and technology cyclically plays on what our political norms even are.
So, back to Dario.
Overall my view is that AI is structurally a technology that tends to concentrate power, for reasons that have nothing to do with regulation
What are these reasons?
I can come up with a few.
Creating a frontier AI model requires a massive amount of compute / capital expenditure. That in turn requires things like land and energy, but more importantly it requires an organization that is capable of leveraging all of those things. This is hard to do, so AI is centralizing.3
It is also often difficult to run models locally. Most of the time, AI usage occurs through some kind of inference server. The organization that is running the inference server therefore has full visibility over everything the user is doing with AI. That level of visibility naturally lends itself to surveillance, which is centralizing.
The design choices of an AI model are inherently patterned off a small group of people and then reused in many places. This means that a small group of people have outsized say in the downstream uses of the tool. So AI is centralizing.4 From this perspective, social media is centralizing, as is Salesforce, as is AWS. Really most most mass market software.
Globally, AI tokens are a scarce resource. The monthly cost of AI in many white collar settings approaches hundreds of USD per person per year. This is an impossible cost for many people in the developing world. Assuming that AI can meaningfully accelerate a company’s output, companies that cannot afford to use AI will have their lunch eaten by those that can. This consolidates the market. So AI is centralizing.
But I think Dario misses the main way that AI is decentralizing: it is now impossible to tell the difference between good and bad behaviors on the web.
A captcha is meant to distinguish between a human and a bot. Thanks to AI tools, there is no captcha on Earth that can do this. Anything that is hard enough to block AI is going to inevitably block a lot of people too. Soon it will be the opposite — things that are too hard will only block humans. And even if you could find some kind of ‘is it AI’ signal, it is trivially easy for a human to go past the captcha and then let the AI take over afterwards. So moderators and administrators have no way to tell the difference between legitimate (human) usage and bot spam. And users are more than happy to take advantage of the commons,5 with bot scrapers becoming so prolific that many websites have buckled under the fully automated onslaught.
Anthropic may be able to build classifiers that prevent the models from talking about bioterrorism, which is great. But they will never be able to stop randos from slop-posting onto reddit subs. There’s no way to build a classifier that can catch that kind of behavior even if Anthropic wanted to do so, which they almost certainly don’t because this sort of thing is how they make money. The end effect is ruining those communities / making it unprofitable to publicly host those resources.6
Decentralizing technologies aren’t necessarily bad, and centralizing technologies aren’t necessarily good. There is always a trade-off. So I don’t want people to walk away thinking that I’m taking a stand one way or another. But when we’re thinking about AI policy, we need to account for the ways in which AI is actually used, and it seems to me that Dario doesn’t really get (or is purposely not engaging with) how AI can hollow things out.
In part, this is structural. Anthropic is coming into this with a very idealized perspective, one where the organization that ‘owns’ the AI holds the reins of the future, where the masters of the technology can grant or withhold access at whim. They kinda have to have this perspective, because their business and access to capital depends on that model of the world being true, and it’s very hard to convince others of something you don’t believe in yourself. Notice how concerns about open weights are always framed as if ownership of the weights is what allows for bad behavior.
But there are two huge problems with this world view. First, the market pushes Anthropic and everyone else to make tokens as widely accessible as possible. Prevent people from using AI? Anthropic can’t even stop subsidizing its most expensive models!
And second, most of the bad things people are concerned about can already be done with Claude today!
Together, this makes any complaints about open weights ring totally hollow.
From where I sit, it seems that AI is a massively decentralizing technology, putting a ton of power in the hands of individuals at the expense of institutions. And as a result, all of the safety things that Anthropic et al are worried about — things like bioweapons — feel totally disconnected from the problems people actually see. I think the average person would rather have better tools to detect and prevent bot spam, or more infrastructure that clearly demarcates when a “user” is actually AI, or even bans and legal liability for inference providers. Approximately no one cares about open models, because only a tiny tiny fraction of the population ever interacts with them, and an even smaller fraction of the population cares about things like “who owns the tokens.” You don’t need to go to a Chinese lab to generate botspam, Claude will happily do that today.
By making all of the regulation conversations about open models, Anthropic comes across as a company that is leveraging its power to cut the knees off competition, rather than one that is seriously attempting to grapple with the implications of AI. Until that changes — until Anthropic stops thinking about things exclusively from the lens of a model provider — the AI world in general and the frontier labs in particular won’t be able to change the overwhelmingly negative public opinion about the nature of this technology.
some people argue that some combinations of stone and asphalt and metal are conscious, but mostly that is when they are in the shape of a data center and not a bridge.
I think one of the biggest failures of political discourse in the 2010s - 2020s was failing to communicate this sort of thing. Activists would argue ‘the highways are evil!’ and they would sound ridiculous, even if they were ~right. Simplifying a really complicated nuanced topic for mass consumption is always hard, ofc, but also I’m not sure many people really tried very hard when it was so much easier to just yell at people.
Though there are open collectives that have trained extremely large models, such as Eleuther, they actually have a pretty clear social structure too. Someone owns the keys to the compute. And I don’t think any open collective has managed to train a truly frontier model in a long time.
Though there are ways of rewriting these decisions, sometimes cheaply, e.g. with finetuning open weights in a variety of ways.
Many do not even realize that that is what is happening!
This could end up producing more paywalls / walled gardens, which is again a centralizing outcome. But the image I have in my mind is a bit ‘mad max’ — you have pockets of structure and a vast ecosystem of low quality bots.
Very thoughtful take! A nuance that maybe you didn't consider, maybe didn't make it into the post, or maybe just isn't very useful: technologies can be centralizing or decentralizing in different contexts, especially at different points in time. Nowadays guns are largely a decentralizing technology (e.g. it's easier for a gang to wage war on a police force than it was for medieval European peasants to wage war on plate-armored knights), but initially they were highly centralizing (only trained armies with access to the capital and expertise requires to smith guns). I wonder if AI proliferation will turn an initially centralizing technology decentralizing? Idk
If you haven't read Brynjolfsson's "Use of knowledge by AI in society", it basically argues that in a Hayekian world, knowledge is difficult to define, and inherently local, so distributed systems tend to work better. But in a post-Hayekian world, as more knowledge moves from tacit to codified, then it's easier to pool information, which removes barriers to centralization.
The precise point that happens is the balance between how much is gained by having "one single planner" who can take advantage of all that knowledge, versus how much benefit accrues to individuals by NOT sharing their knowledge. I can imagine that open-weight models increase local knowledge, but also that pooling info produces better results. So I dunno if I have a clear call on which it is.
https://www.nber.org/system/files/chapters/c15303/c15303.pdf