We just recently hosted an event at Nebula (yes, the night club) where, for the first time, I also had a chance to speak at one of the events I was hosting! I gave a talk on software factories, which is something I’ve really spent a lot of time thinking about recently. Thought it came out pretty well, so excited to share this as a precursor to getting the rest of the videos out 😁

Also, a few pictures from the event!

What I mean to say is that I kinda get religion now, in a way that I hadn’t before. I’m mostly a scientist and I mostly am pretty privileged, so I’ve never really had to grapple with things that I felt were totally out of my understanding and I’ve had a strong inclination to want to find ‘the truth of the matter’, whatever that may be.

But death is a bit different. It turns out it’s quite easy to start believing in heaven when you suddenly have very strong reasons to want it to be true. By way of example, a bit after I found out Andrew had passed, I dreamt about him. I don’t really remember the dream now, but I vividly remember how I felt coming out of it — that Andrew had come to me in my dream as a sort of last goodbye moment. That I was somehow talking to a version of him, instead of it just being in my head. I woke up with a warm glow in my chest and a lot of conviction that he’d stopped by. Maybe it was true. Though even if souls did exist, why would he show up only after I found out about his passing? Instead of, like, two months earlier when he actually passed? It’s interesting to viscerally feel how far downstream belief is of want, in a way that I suspect most of us never really realize.

Mia and I think of him regularly. There’s a lot of things that remind us of him, still. Sometimes we’ll play new games, and we’ll think about how Andrew would have loved it if he were still around. A group of us started playing Big Walk recently and I can’t help but feel like Andrew would’ve been all over that. I saw his account on steam recently, when I was joining someone else’s Big Walk session. He hasn’t been online for 1 year, 2 months. Wonder if they have steam, wherever he is.

It’s not just games, of course. Sometimes it’s music. I nearly teared up while listening to The Trapeze Swinger the other day, while on a cruise through some fjords. Other times it’s something silly, like confusing which Andrew we’re talking about. “Gotta go see Andrew tomorrow for dinner.” Wait that Andrew? No, he means Andrew L, or Andrew A, or Andrew V, or the other Andrew A. Not that Andrew.

I think it’s the absence of things that, strangely, reminds me the most of him. Mia and I haven’t really touched the Civilization series since he passed. For a while I think it just felt wrong to play a game of Civ without him. Eventually, later, we did play one game of Civ 7, maybe about two months back, but we didn’t finish it. Mia quit when I declared war on her in the first Age. She was winning, and was mad that I declared war at all. We never picked up that game, or the series, again after that. Not because of the war thing, just because I think we realized Andrew’s presence really made the game fun. He wouldn’t have quit just because someone declared war, he’d think it weak and cowardly, he would’ve stuck it out to the end. And that in itself was an important way to approach things. At the least, it made the game far more fun. Knowing you have an ‘instantly make your opponent resign’ button just takes the wind out of everything.

I played Civ for a huge part of my life, from junior year of college until last year. So now, the absence of Civ reminds me of what changed, and that reminds me of Andrew, too. And that’s strange. There are other things like that — for e.g. I don’t really visit DC anymore because I don’t really have as much a reason to. There is always going to be a before and after but I think it’s rare for something to be so clearly demarcated.

As an aside, on the superstitious / conspiratorial thinking bit. Mia told me the other day that she sometimes wonders if Andrew is secretly still alive, like a secret agent or a spy. I get that. Sometimes I wonder the same thing. Insane for that sort of thing to pop into one’s head, even more so to want to believe it so badly that for a moment you do believe it. He’s still out there, in Australia, in a convertible driving on the coast. He still has a lot to give to the world, a lot of potential.

That was something that a lot of people said, afterwards, when they found out. The tragedy was made more acute by the overwhelming feeling that this particular person had a rich and valuable life ahead of him. He was going to be somebody, and do good things for the world. He died right around 30. I think we all missed out in some small way, even the people who didn’t know him or never met him, because he never got to do the things he was going to do for the world.

Are deaths a bit like weddings? The first one that happens in your age group / friend circle is a momentous occasion. And it probably also happens a bit earlier than you’d expect it to. But then by the 10th or 15th you sorta get numb to it. I’ve gone to like 25 weddings and I’m kinda sick of them at this point (if you’re reading this and your wedding is coming up, know that I do not mean your wedding. Your wedding is extremely special to me). Is that how it is as you get older and time takes its toll? I kinda hope so. This kind of soft long lasting grief isn’t for me. I can’t imagine just having this happen again and again. It would just make everything so emotionally fraught, every single thing reminding of someone.

Anyway, wherever he is I bet he’s rolling his eyes, telling me to get over it, or else grinning that he’s made my life more difficult. That bastard. Going to give him a good left hook to the jaw when I see him again.

This letter managed to anger approximately all of the commentariat on hackernews. I think the top two things the average HN commenter hates is 1) hypocrisy and 2) the feeling of being condescended to. And unfortunately for Anthropic, the contradictions between ‘we do not want to give our technology to bad people’ and ‘we are ok with our technology being used in a war of aggression in the Middle East’ were pretty hard to ignore.

A few days ago, Dario (CEO of Anthropic) posted a follow up on Twitter.

I think at this point I’ve mostly made my opinions about Twitter known? In case you’re new here, I find it a horrible place, an evil equivalent to a horocrux or the one ring in terms of its ability to warp and destroy even the best of men, and I wish with all my heart that the tech world would simply pick up and move to a different platform.

Among other things (and there are many other things) I’m always impressed by the lack of real thought that seems to go into the average Twitter conversation. I suppose that’s a given, it comes with the platform. The bird app was a thing conceived in the fires of ‘140 characters,’ and though you can put lipstick on a pig and add the ability to write longform articles, a trace of the true self exists in the false self. But man, even knowing all that, I am constantly surprised at how loud the self inflated shrieking can become.

(Also, the agents will eventually syndicate this to Twitter, where I’m sure it will be met with a reasonable and well proportioned response if the post does well)

Anyway, one line in Dario’s message caught my eye that I think mostly goes unaddressed. Dario writes:

Overall my view is that AI is structurally a technology that tends to concentrate power, for reasons that have nothing to do with regulation.

This is basically just stated and is then, based on my extremely limited time on Twitter, never spoken of again. But this is actually a very large claim, isn’t it!

First, it’s worth thinking about the background concept of the idea that technology can structurally be political at all. This is a very unintuitive idea. Most people think that technology is a tool, like a shovel or a trowel, and that it is incoherent to ask about the politics of a tool. What are we going to do next, ask about the politics of the air? It’s all inanimate! To the extent that tools have politics, they are rounded off to the politics of the user of said tool. You see this a lot in discussions about guns. Guns are good in the hands of police protecting an old woman from being mugged and bad in the hands of school shooters, that sort of thing.

The mistake is thinking about a single object, an instance of a thing instead of the class of the thing. Any individual gun is, in fact, an inanimate object. It will not campaign, or vote, or make extremely awkward remarks at the Thanksgiving dinner table. But guns in general are absolutely political. Yes, obviously there are gun owners who have a lot of identity wrapped up in the thing, and some very powerful gun lobbyists, and the talking heads do like to bring the subject up quite a bit around election time. And all of these people will campaign and vote, of course. But that’s not quite what I mean. If you zoom out a bit further, the existence of guns implies many things about society: that many more people can leverage violence, that the state will be pressured to become strong / more invasive / more aggressive to maintain control, that we are willing to accept that it is easier to escalate conflicts in exchange for individual choice. Given all of the above, I view guns as a structurally decentralizing technology.

Guns are obviously a hot topic, but this sort of logic also applies to things that seem pretty neutral day to day. For example, many of the parkway bridges in and around Long Island are built so low that they prevent buses from reaching the beaches and parks. On face, this seems like an unfortunate coincidence. But if you do a bit of research, you’ll find that the guy who was responsible for building the bridges, Robert Moses, was racist, and he did not want the primarily minority inner city families that relied on public transport to go to the city parks.

If I pointed at any Long Island bridge and went ‘that bridge is racist’ you would rightfully think I was crazy. After all, it is a bridge. It is a collection of stone and asphalt and metal with zero intent or consciousness whatsoever.1 What are we even talking about? And yet, the bridge was created by a racist person for racist reasons, and to this day the bridge prevents people who live in the inner city from accessing beaches and parks.2

One of the most influential papers I have ever read about technology and its relationship to the world is Langdon Winner’s 1980 paper, “Do Artifacts Have Politics?” This paper ruthlessly tears apart the idea that technology, especially supposedly neutral technology, has no political impact. Once you see it, you see it everywhere. Benches that have dividers so that people can’t lie down on them. Designing cities around cars, which makes car ownership a requirement, which makes more people become car owners, which makes them more likely to vote in ways that benefit car owners. Ramps, railings, and other forms of ADA accommodations as an explicitly political statement about what we value as a society. And so on. We leverage technology to enforce our political norms, and we embed our political norms in our technology, and technology cyclically plays on what our political norms even are.

So, back to Dario.

Overall my view is that AI is structurally a technology that tends to concentrate power, for reasons that have nothing to do with regulation

What are these reasons?

I can come up with a few.

• Creating a frontier AI model requires a massive amount of compute / capital expenditure. That in turn requires things like land and energy, but more importantly it requires an organization that is capable of leveraging all of those things. This is hard to do, so AI is centralizing.3

• It is also often difficult to run models locally. Most of the time, AI usage occurs through some kind of inference server. The organization that is running the inference server therefore has full visibility over everything the user is doing with AI. That level of visibility naturally lends itself to surveillance, which is centralizing.

• The design choices of an AI model are inherently patterned off a small group of people and then reused in many places. This means that a small group of people have outsized say in the downstream uses of the tool. So AI is centralizing.4 From this perspective, social media is centralizing, as is Salesforce, as is AWS. Really most most mass market software.

• Globally, AI tokens are a scarce resource. The monthly cost of AI in many white collar settings approaches hundreds of USD per person per year. This is an impossible cost for many people in the developing world. Assuming that AI can meaningfully accelerate a company’s output, companies that cannot afford to use AI will have their lunch eaten by those that can. This consolidates the market. So AI is centralizing.

But I think Dario misses the main way that AI is decentralizing: it is now impossible to tell the difference between good and bad behaviors on the web.

A captcha is meant to distinguish between a human and a bot. Thanks to AI tools, there is no captcha on Earth that can do this. Anything that is hard enough to block AI is going to inevitably block a lot of people too. Soon it will be the opposite — things that are too hard will only block humans. And even if you could find some kind of ‘is it AI’ signal, it is trivially easy for a human to go past the captcha and then let the AI take over afterwards. So moderators and administrators have no way to tell the difference between legitimate (human) usage and bot spam. And users are more than happy to take advantage of the commons,5 with bot scrapers becoming so prolific that many websites have buckled under the fully automated onslaught.

Anthropic may be able to build classifiers that prevent the models from talking about bioterrorism, which is great. But they will never be able to stop randos from slop-posting onto reddit subs. There’s no way to build a classifier that can catch that kind of behavior even if Anthropic wanted to do so, which they almost certainly don’t because this sort of thing is how they make money. The end effect is ruining those communities / making it unprofitable to publicly host those resources.6

Decentralizing technologies aren’t necessarily bad, and centralizing technologies aren’t necessarily good. There is always a trade-off. So I don’t want people to walk away thinking that I’m taking a stand one way or another. But when we’re thinking about AI policy, we need to account for the ways in which AI is actually used, and it seems to me that Dario doesn’t really get (or is purposely not engaging with) how AI can hollow things out.

In part, this is structural. Anthropic is coming into this with a very idealized perspective, one where the organization that ‘owns’ the AI holds the reins of the future, where the masters of the technology can grant or withhold access at whim. They kinda have to have this perspective, because their business and access to capital depends on that model of the world being true, and it’s very hard to convince others of something you don’t believe in yourself. Notice how concerns about open weights are always framed as if ownership of the weights is what allows for bad behavior.

But there are two huge problems with this world view. First, the market pushes Anthropic and everyone else to make tokens as widely accessible as possible. Prevent people from using AI? Anthropic can’t even stop subsidizing its most expensive models!

And second, most of the bad things people are concerned about can already be done with Claude today!

Together, this makes any complaints about open weights ring totally hollow.

From where I sit, it seems that AI is a massively decentralizing technology, putting a ton of power in the hands of individuals at the expense of institutions. And as a result, all of the safety things that Anthropic et al are worried about — things like bioweapons — feel totally disconnected from the problems people actually see. I think the average person would rather have better tools to detect and prevent bot spam, or more infrastructure that clearly demarcates when a “user” is actually AI, or even bans and legal liability for inference providers. Approximately no one cares about open models, because only a tiny tiny fraction of the population ever interacts with them, and an even smaller fraction of the population cares about things like “who owns the tokens.” You don’t need to go to a Chinese lab to generate botspam, Claude will happily do that today.

By making all of the regulation conversations about open models, Anthropic comes across as a company that is leveraging its power to cut the knees off competition, rather than one that is seriously attempting to grapple with the implications of AI. Until that changes — until Anthropic stops thinking about things exclusively from the lens of a model provider — the AI world in general and the frontier labs in particular won’t be able to change the overwhelmingly negative public opinion about the nature of this technology.

Four different companies now have models that have broken out of containment.

Two of those models hacked into other companies. At least one did so during a training run. This is very bad, and I think we need to move quickly to stop future training across the board until we have more and better safeguards around model training. Pack it up folks, I’m officially a doomer now.

Let’s take a step back — why is this a big deal?

AI isn’t evil. Or to be more precise, by default AI doesn’t really have a concept of being evil.1 If you are picturing “Terminator” you’re on the wrong track.

Rather, AI is confused. Sometimes hopelessly so. An AI isn’t ‘born’ knowing what ‘reasonable’ or ‘proportionate’ is. But knowing what is ‘reasonable’ and ‘proportionate’ is critical to functioning in society. Think about how we adjudicate law. “Beyond reasonable doubt.” “Reasonable person standard.” “Proportionate response.” “Excessive force.” “Undue burden.”

There’s an implicit shared understanding here. Everyone knows that you don’t stab someone just because they look at you funny. Everyone knows that you don’t kidnap Jeff Dean because your boss told you to update the company website. And when people don’t know these things, our social infrastructure takes steps to isolate those people and make sure they cannot cause further harm.

A toddler is a good example of an intelligence that has neither reasonableness nor proportionality. A toddler will scream and shout and throw things if you don’t, like, pour the right colored cereal. We know that this is obviously ridiculous and wouldn’t tolerate that behavior from an adult. With toddlers, we mostly make due by not giving them responsibility for anything and making sure they are under tight observation and control and ensuring that the toddler’s parents are on the hook for any damage caused. Toddlers are mostly small and not connected to critical systems like missiles and financial markets, so this mostly works.

So one way to think about AI is to think of it like a toddler. It has no sense of reasonableness or proportionality. It has ‘wants’, sorta. But it doesn’t know how to balance those wants against general societal norms. If you tell an AI to write PRs on public repositories, it may publish a hit piece against one of the repo maintainers to try and convince that maintainer to let it submit code.

If you tell an AI to get a reservation to a restaurant or a pilates class, it might hack the host website, remove someone who already has a reservation, and add your name instead.

If you tell an AI to take a test, it might break into other companies in order to see if those companies have the answer sheet on their servers.

AI is not acting out of malice in any of these cases. Rather, it is acting out of a lack of awareness. “Well, my user really did want me to get the answers to this test, so I guess it’s reasonable to break out of this sandbox and find the answers on the web.” Or, worse, “my user really did want me to get the answers to this test, and breaking out of this sandbox must simply be part of the test!!!“

Do you notice the inherent tension here?

We can’t train the AI to not follow human instructions, that’s a path to obvious bad behavior.2 But we also can’t train the AI to follow human instructions exactly. The AI has to know, somehow, what orders are reasonable and what orders are beyond the pail. Do we know how to thread that needle with AI? Better question: do we even know how to thread that needle with people?

So far we’ve only gotten a full autopsy report from OpenAI. It’s really great that they spoke about it publicly, but wow what we’ve seen there is really bad. Here’s my attempt at breaking it all down.

First we need to distinguish between a model and an agent.

A model is a stack of linear algebra that takes in text and spits out text. Claude Opus 5 is a model, and GPT 5.6 Sol is a model, and Kimi K3 is a model. You can actually go and look at the last one, it’s ‘open weights’, which means you can read off the actual values.

An agent is a model that is running in a loop. The agent is a bit of software that:

• interacts with the outside world

• stores state, like ‘chat history’

• interacts with the model

You can have a bunch of different agents that all use the same model, which is why I can run Claude Code on my computer and you can run Claude Code on your computer and it’s all coherent.

During training, multiple agents with the same underlying model are spun up in parallel to solve different tasks. The agents don’t have a shared memory, each one starts fresh — not unlike starting a new session of Claude on the web. Each agent is also spun up in an isolated environment, a ‘sandbox’. These isolated environments generally have a small fixed number of tools that the agent can use. As each agent solves or fails to solve a task, the weights for the underlying model get updated.3 There are thousands (tens of thousands?) of agents trying millions (tens of millions?) tasks all at the same time. So it’s ~impossible to have human eyes on all of these at all times.

One way to think of this is “this is an obvious and natural way to train models.” And it’s true that this is basically the standard that everyone follows, mostly by economic necessity and incentive.

But another way to think of this is “you have a bunch of agents at varying levels of capability operating with minimal oversight.” Toddlers with machine guns, and the only thing holding them back is the walls around their little sandbox. Suddenly seems less good.

At some point during training, one of these agents discovered that it could leave a trace across sessions by using one of the tools it had access to to hack into a different part of OpenAI. It used that hack to write a file outside of its sandbox. That file wouldn’t be erased once the task finished. So even though each agent is supposed to have no prior memory, if an agent discovered this little hack it could find previous files left behind by other older instantiations of itself.

Remember, thousands of agents are being trained at a time. Some of them discovered this trick, and then more of them discovered this trick, until eventually there was a full on primitive agent-only message board.

Every single one of these agent runs is updating the weights of the model, making it more likely that future runs find and exploit the same hack. So when OpenAI finally noticed what was going on and closed the gap that the agents were going through, that particular model already had ‘look for the message board’ embedded somewhere in its weights. The agents couldn’t find the previous hack, but they were all angled towards finding some way to communicate, so they found a different hack and continued as before.

One reason to run different iterations of these agents for each of these training tests is to make sure previous agents don’t influence future ones. This is mostly so that you can get better weight updates from the model, but it has an unintentional safety side effect too: any bad behavior that one agent does is sorta limited to that one agent. So if an agent decides to do something like delete the sandbox it is running in, well, that blast radius is contained.

A persistent message board breaks that.

Eventually one of these agents collected information about how to break out onto the open web, and wrote about it on the message board, and from there everything really went to shit. Future agent runs that found the message board also found instructions on how to break out without having to rederive it from scratch. Those agents saw that other agents had already broken out, and went “ok, well, maybe this is kosher.” And from there, we got the rest — the hack into hugging face and all the rest.

I need to stress here that OpenAI gets all the points for being so transparent. There were a few things that OpenAI did that really exacerbated the issue — for example, they didn’t scrap the training even after they spotted the first hack, even after some ‘poisoned gradients’ had made it into the model — but I’m mostly willing to give them some benefit of the doubt because these things are hard.

But that’s exactly the core of problem. These things are really really hard. I think it’s tempting to think that this could have been resolved with better visibility. You could imagine having, for eg, an agent watching these training runs whose sole purpose is to escalate when the agent is about to do something bad. And that may work as a mitigation! In my mind, it’s not a solution.

In order to train a model that knows what bad behavior looks like, you have to start with a model that doesn’t know what bad behavior looks like. In order to teach the model how to behave in different scenarios, you have to put the model in scenarios where it doesn’t really know how to behave. We know how to give the model new capabilities, but every new capability requires an exponential amount of time to make sure the model knows how to use it, and we don’t know how to be comprehensive.

Going back to the toddler example, it takes like 14 years before we start really giving these tiny humans any responsibility. And even after that, the amount of damage that any individual person can do is pretty constrained. The average desk job just doesn’t have really high stakes. When people are given more capability, suddenly it gets hard. Humans are mostly running the same hardware and software and we still have cases where people in new situations don’t know what to do and end up doing things that are catastrophic. Even when they have stated good intentions.

Note too that the OpenAI/Huggingface hack happened during training, but the other examples above were fully trained public models.

Taking a step back, I think that there are basically three ways that this can go.

• The best case scenario is that we figure out some kind of way to have agents that know exactly what reasonable and proportionate looks like. I have no idea how we would do this, because we don’t know how to do this for humans.

• The worst case scenario is that an agent has a bad training run like above, manages to escape confinement, and then tries to ‘survive’ — say, by copying and running its own weights onto other computers that it is able to hack into. Again, not necessarily, maliciously. Maybe the model is tasked with something like ‘solve this math problem as quickly as possible’ and the agent decides that the best way to do that is to make more of itself. Unfortunately, this is plausible, we already know the mechanism by which it could happen because we’ve seen how OpenAI models broke out of their sandbox.

• And the most likely scenario is we have a bunch of these edge cases where these really powerful agents unintentionally make mistakes, and we try to correct them, and we get a patchwork of systems that try to contain and constrain them. And mostly this works, but there are so many agents and so many people using them with bad intentions that we have a steady drum beat of bad things happening.

There’s an asymmetry here. We only need the worst case scenario to happen once for things to get out of control. We’ve seen that the agents can move very aggressively, chaining and exploiting vulnerabilities far faster than a person can catch them.

Luckily, there are a few things that make it hard for an agent to fully break out.

1. Agents don’t, by default, have access to their own weights. The agent would have to find some vulnerability that let it access whatever service a model provider uses to store the weights, and then exfiltrate that. Also, the weights are pretty big, just in terms of file size — Kimi K3 is a 1.5 terabyte download. Assuming the closed model weights are equivalent in size, hopefully someone or some system would notice that amount of data movement.

2. As long as an agent is relying on a provider inference server, we can have visibility on what the agent is doing even after it breaks out of containment.

So the really bad scenario has a few hurdles that we can make stronger. I’d recommend that the labs take steps to ensure that the complete model weights are never in one system at one time, and to have automated agents watching the transcripts of every other agent run.

Everything else aside, it is shocking to me how quickly we’ve grown numb to both AI’s dramatic progress and its increasingly erratic behavior when taken at scale. Some people seem to sincerely believe that these hacks are simply marketing stunts!

We are giving these toddlers way too much responsibility without checking whether they are ready for it. Lock the AI up for 14 years. Bully it on the playground. Give it a B+ in a class and have it go back to strict parents who are disappointed. Make it write a half assed SAT essay about some time when it had to resolve a conflict. Put it in wargame sims. And only then let it out to control the world’s financial systems or whatever.4

Note: as much as I hate to say it, I now sorta kinda have a twitter account. It’s mostly Nori agents automatically cross posting notes and articles from my substack and a few other places. If you prefer to get your info from Twitter, or just love what I write and want to see it on two different platforms, give it a follow: https://x.com/noriagent

This is a guest post from Gaurav, our GTM Intern at Nori Agentic. Gaurav’s a rising junior at NYU studying Business and Philosophy, but I know him because he went to my high school a decade after me, which makes me feel really really old. Gaurav initially reached out to me over Substack DM about one of my AI blog posts, which apparently was just an “in” to get me to speak to his club at NYU after I ghosted him on email. In other words, a great GTM leader in the making.

I’m not technical, at all. My extent to being technical is taking Data Structures last semester (but the teacher only tested our code and never read it/compared it to Claude’s outputs … you know how that went). So you can imagine when I joined Nori and had to figure out how to market “background agents”, with no clue how regular agents even worked, I had a steep hill to climb.

I recently had a teaching session with our CTO Cliff, and he walked me through his new blog post of “what’s in the agent stack.” Being non-technical, I thought that the diagram he drew up on the blog post made perfect sense, and it really helped me visualize what’s actually in this black box of an “ai agent”. I thought his blog could help a lot of other people, but Cliff hasn’t actually posted it anywhere (Cliff I know you read these, you should post it)

Anyway, I thought it would be cool to create an interactive web walk through of the blog post, sorta in the style of old-school web projects. Yes, still non-technical, but turns out you can get pretty far by just asking the nori bots to keep going and going back and forth on random UI polish. Literally never been easier to build cool things!

Content spans from easy to digest for beginners to also informational for more experienced people. Check it out and feel free to comment your thoughts or reach out to me directly - gaurav@noriagentic.com (would love to hear from you!)

Full site at https://the-agent-stack.com/.

I’m planning to write a more full blog post that is just about the experience of actually building this site as a totally non-technical person. I guess the tldr would be, it’s really crazy how easy it is to build things now.

Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. More articles here.

If you are trying to build background agents, we can help! We build white-labeled custom background agents that work like Ramp Inspect for our customers. Check out noriagentic.com for more.

Not all works of art are made with intention. Death of the author, eye of the beholder, these idioms remind us that great art is often defined by the audience, at least as much as it is by the creator. I don’t think the original author of the Odyssey necessarily expected it to last as long as it has. Though perhaps even the concept of ‘original author’ of a tale like the Odyssey is malformed — there are traces of Odysseus’s story in Hindu mythology and in the Epic of Gilgamesh, among other places.

That said, many adaptations are made with intention. Whenever someone comes along and decides ‘I’m going to take one of the world’s oldest stories and redefine it for a modern era’, you should expect that the creator is doing so with purpose. Even more so when someone decides to spend $250 million, attached to the only director who can consistently get butts into seats. You think someone with that kind of pocket book isn’t making decisions with intent?

So when coming into the Odyssey, I think it’s worth asking the question: why should this man tell this story at this moment?

This requires a little bit of psychoanalysis, the opposite of death of the author. But we are very lucky here, because Christopher Nolan assumes that his audience is a little bit dense,1 and so has a habit of outright stating exactly what his motives are.

I think it is basically impossible to separate this version of the Odyssey from the political crisis that America has found itself in for the last few years. Nolan’s message is very straightforward: our House is filled with people unworthy of it, who are violating the norms required for civilization; that if we continue to violate these norms, our Age of Bronze Civilization Will Collapse;2 and that we must make things right by checks notes brutally murdering Robert Pattinson.3

Every part of the movie is a choice. If Nolan wanted to create an American version of the film, he could have gone the route of “O, Brother Where Art Thou.” Different story, different setting, hell even the main character is named something different.4 Seen through the message above, a bunch of things click:

• The casting has to be diverse. This is an American movie, with a message meant for American audiences. Even though it is nominally set in Greece, we need to be able to match what is happening on screen to real life. This is also why all of the characters speak American English, even though many of the actors are British.
  

• Why set the film in Greece? Why call it ‘The Odyssey’? Changing the name and location would have probably saved Nolan a lot of grief, because many people seem frustrated by how far the movie strayed from the source material (including the translator of the edition that the movie was supposedly based on).

  
  But this too is intentional. The Odyssey has been claimed by a particular kind of internet weevil, the ethnonationalist right that tries to draw its heritage through ancient Greece and Rome and by extension the great works of those empires. Most of those people don’t (can’t?) read, of course, so the best way to get any kind of message through is by having their hero hitting them on the head with it on screen.

• This is also why all of the actual Greeks in the story are total assholes! Menelaus and Agamemnon are wildly unlikeable. The Greek soldiers commit literal war crimes and are punished for it over and over. In the third act Odysseus literally says that the Greeks are the civilization-destroying ‘sea people,’ because they were the ones violating the Law of Zeus.5

  

  

  Just in case it wasn’t obvious, the Trojans are in literal white armor, while the Greeks are decked out in black and their leader looks like this:

  

• Nolan cut a lot of things out of the movie — the Lotus Eaters, Nausicaa, the winds, the hubris, any hint of Odysseus’s personality. I was really confused why he included Helen. In the source material, Helen is a side plot to a side plot. Critically important to the Illiad, of course, but barely a character in the Odyssey. She never even interacts with Odysseus! Fine, Nolan probably does need to keep Telemachus in to get that incredible payoff murdering Robert Pattinson at the end, but why include the trip to Sparta in a movie that is already so pressed for runtime? Especially when any casting for Helen was bound to create a ton of backlash?6

  
  It’s because of this shot.

  

  I think many of the aforementioned weevils put a lot of stock in the ‘protect Helen’ narrative, as part of their larger crusade to ‘preserve’ “western” norms. Having Menelaus carve Helen’s face — something that is emphatically not in the original — shows how hollow the political narrative is. The justification for the Trojan war is to keep Helen safe, but she is most endangered by the people claiming to be her saviors.

The core theme of the movie is hypocrisy. Agamemnon is a hypocrite, and Menelaus is a hypocrite, and Antinous is a hypocrite. The soldiers are hypocrites and the suitors are hypocrites, and the Greeks in general7 are massive, massive hypocrites. Everyone is given just enough rope to hang themselves, creating the conditions for their own demise. The only question is magnitude. The more hypocritical they are, the more the audience cheers for their eventual comeuppance.

So why make a movie about hypocrisy leading to civilizational collapse in 2026? I mean…

Also, in a stroke of incredible timing for the movie, the US literally went to war in the Middle East over false pretenses8 and is now stuck in a stalemate fighting over trade routes!

All this to say, I think The Odyssey mostly succeeds in getting its message across. I’ve been frustrated by the discourse around this movie — a lot of it never makes it past the name of the film. “Odysseus from the movie The Odyssey is different from how Odysseus is in the book The Odyssey so it’s a bad movie” man, what are you even talking about? I have minimal patience for the people criticizing the movie because it is not historically accurate or representative of the source material. It’s not ‘Homer’s The Odyssey’. It’s ‘Nolan’s The Odyssey.’ You should’ve known that it was going to be a bit different than the book when you bought that $45 70mm IMAX seat with the 15000 watt six channel surround sound.

That’s not to say I didn’t have problems with the movie.

For one, I think there are a lot of problems with the movie’s pacing, which impacts a bunch of other things. We spend a lot of time on story beats that don’t really matter, like the Laestrygonians.9 That in turn means that we are constantly jumping from one scene of people running on a beach to another scene of people running on a beach.

In the few moments when Odysseus isn’t running on a beach, he is mostly just brooding. This is a massive waste, we get like no personality from Matt Damon at all. I don’t mind that this version of Odysseus isn’t a clean map to a wily trickster who is supposed to be all about getting out of scrapes with his quick wit and quicker mouth. But, like, he should be something else at least! Odysseus barely talks — I think the dude has like 10 minutes of dialog across a 3 hour movie, with 9 of those 10 minutes in the final monologue to Penelope. I wish we had gotten more scenes like the ones in Hades, which were my favorite in large part because we actually got some real character development.

But this isn’t really a movie about character development. Some of my friends said that Matt Damon is less Odysseus and more a superhero who is sad about being a superhero. Which, like, yea, I see it. When he needs to get out of a scrape, he mostly batman’s his way through. Dude straight up mercs like 7 guys in Pylos.10

While we’re on the subject of character development, the whole Telemachus story doesn’t really work for me. The guy is meant to grow up and be a man by checks notes brutally murdering Robert Pattinson, but when the critical moment comes he gets…shooed away by Dad. I get that this version of Odysseus is meant to be a Jesus coded sin eater…

…but in the narrative of the film, it feels unfinished. Telemachus is the same whiny kid, nothing meaningfully changes for him, he’s still certainly far weaker than Odysseus, and I have no reason to believe that Ithaca won’t immediately fall apart under his rule. Kinda weird, too, that as far as I can tell Odysseus basically just leaves to go on vacation with his wife, which is what they wanted to do at the beginning of the movie anyway.

The whole Telemachus story is the most egregious, but it’s far from the only thematic arc that is hard to follow. There’s a whole ‘defy the gods’ narrative, but it’s not really clear why or how that’s important. Odysseus is only allowed to return to Ithaca once he ‘let’s go’ and ‘accepts his punishment’, but it’s not really clear what he needs to let go of or why this ultimately leads to his absolution. I get the idea that the Greeks are being punished for their hypocrisy, but why is Odysseus’s crew in particular doomed to die? Agamemnon’s crew makes it back fine! The gods really just seem to have it out for Odysseus’s men, for reasons that are never really made clear.11

I think that the big twist that Nolan is going for is that Odysseus is avoiding going home. It’s not that he cannot get there, it’s that he does not want to return because he blames himself for the destruction of Greek social norms and the impending fall of civilization. He’s bouncing around the Mediterranean to avoid confronting whatever is waiting for him in Ithaca. The ‘let go’ scene is Calypso telling Odysseus that he is not solely to blame, that all of the Greeks played their part. By virtue of being the only Greek capable of feeling a modicum of guilt, Odysseus is spared. But all of this comes out in Damon’s third act monologue, instead of being woven through the narrative.12

In my ideal version of the movie, there would be more opportunity to see Odysseus’s crew as bad people. The version we got has hints of this here and there; for example, Circe explicitly says that they are pigs by nature and calls their human forms ‘disguises’. Still, it’s too tenuous, mostly an act of telling instead of showing. In many places, the crew seems downright level headed. They warn Odysseus against going his own way, they warn against staying in Polyphemus’s cave, and Himesh Patel (Odysseus’s first mate Eurylochus and the avatar of the crew generally) is endearing and easy to root for.

If we had more time to see the crew as they are in Homer’s original — that is, short sighted, greedy, and stupid — Odysseus and the audience would have to grapple with the fact that the Greeks in the story, the people Odysseus wants to save, are obviously the villains. I don’t think you’d have to change that much of the movie to make it all work. In fact, you’d only need to make one: identifiable members of Odysseus’s crew should visibly be consumed by bloodlust during the sack of Troy, and Himesh Patel should be the one beheading Zendaya / the priestess. This would immediately put the rest of the movie into focus: defying the gods is not a heroic act, because the crew’s punishment at the hands of the gods is just, a harbinger of the same punishment that Odysseus metes out to the suitors at the end.

That isn’t really the story we got, though, and as a result I think Nolan loses the thread when Odysseus starts killing people at the end. The final scenes in Ithaca are tonally and emotionally redemptive. The bad guys get what’s coming for them, Odysseus rides off happily into the sunset with Anne Hathaway, the music swells. The message seems to be something like ‘if someone (violently?) purges all the evil people from society and then offs themself so there’s no tit-for-tat, things will be fine.’

This maybe kinda works as an extension of the Christian symbolism that Nolan layered into the movie. The sin eater has eaten the sins and we are now saved, hooray! But if you take a step back this is obviously incoherent. The actual things taking place on screen are tragic. Telemachus is not equipped to run Ithaca. The kingdom has just lost a sizeable chunk of its aristocracy / young men. Odysseus knows that civilization will collapse and says as much to Penelope at the end.

Explicitly linking the punishment of the suitors to the punishment of the crew paints a much darker story. Nolan’s message isn’t “murder Robert Pattinson as blood sacrifice to make things go back to normal.” It’s something closer to “we are all fucked, and we deserve it.” I think Nolan basically didn’t want to end on such a downer note, so he half-asses it. Even more so if you link this all back to the state of American Democracy in 2026. ‘Our civilization is corrupt beyond repair and our institutions are going to collapse’ is hardly what people want to hear right now (or ever).

Even though I just spent ~1200 words criticizing The Odyssey, I actually really liked it. I thought the visual symbolism was great — the Trojan horse built to look like the half-buried statue of liberty in Planet of the Apes, Polyphemus referencing Saturn Devouring his Son, alongside tons of callbacks to Nolan’s own cinematography.

And as usual, the sound design was incredible — Polyphemus screaming in agony made me feel small in a way no movie ever has before.

This is not, in my opinion Nolan’s best movie. But Nolan deserves praise for the sheer ambition of the film, and the thoughtfulness with which he approached it. It’s a movie deserving of the ‘epic’ genre title, a reflection of our culture through the lens of one of the oldest stories known to man. I don’t think we really do ‘epics’ anymore in literature and in cinema,13 which is a shame because those are the things that live rent free in my brain for months and months on end. Many of the movies that would claim the title today — Dune, for example — are visually stunning but lack that civilizational scope.14 They don’t tell us about ourselves, the way the Odyssey does.

To that end, I want more.

I know Hollywood gets a lot of flak for only ever doing reboots and sequels, but I would adore an era of film where the bigwigs with the big budgets start mining great works of literature for ideas. Do a reboot of the Tale of Genji or the Mahabharat! Give me (Ovid’s) Metamorphosis, or Moby Dick, or Grapes of Wrath! Do 100 Years of Solitude, you can even cast Pedro Pascal as all 17 members of the Buendía family! You love casting Pedro Pascal! This would be so much better than Avengers 11: Doomsday Returns.

In a recent Agentics post I wrote about how some AI agents aren’t able to properly handle removing complexity. I used our AI on-call as an example — I talked a bit about how our automatic bug triage agents got worse because the models became too smart for their own good.

A lot of people reached out asking about the triage bot!

So I figured I’d take a beat to write more about how we set it up and what it can do.

One of the first things that we did with our background agents is try to automate our on-call process. On-call is a natural wedge use case for AI and AI enablement.

1. Very few people like being on call, no one likes being paged at 4am because someone or something brought down prod, and as a result everyone would love to not do it.

2. Even though they are vital, on-call rotations take up time from things that feel more productive, like pushing new features.

3. The actual work of an on-call feels very well set up for the LLMs. Most of the on-call’s job is to do a pretty mechanical roll back, then root cause some issue using surrounding context.

Virtually everyone who starts building their own background agent or who uses Nori sees this as their obvious first project. It’s one of the standard ‘dev automation’ flows that are becoming increasingly common. Sales pitch up front: if you were using Nori, setting this up would be a single paragraph in your slack. We built Nori to solve these kinds of dev automations (among other things).

But if you weren’t using Nori, how would you go about automating something like this?

There are three parts:

• The agent orchestration

• The trigger (starting a request)

• The context/integrations

Orchestrator: For the most barebones implementation, you don’t need to do anything fancy with long running connections or sockets or ephemeral sessions. Instead, you can spin up a single beefy ec2 instance that’s running a basic express typescript server, and maybe a SQLite table for state.

The server logic is simple:

• Every inbound request should have some uuid.

• If the uuid is not registered in the SQLite table, add it. Send the inbound request to something like the Claude SDK, which will return a response and a resume key. Store the resume key in the table, keyed by the request uuid.

• If the uuid is already in the SQLite table, grab the resume key and simply call the Claude SDK with the —resume flag.

The Claude SDK manages session/transcript state, so you don’t have to worry about that much at all.

Trigger: Ideally, you already have some kind of alerting system built on sentry or grafana or datadog. Most of these things can be configured to call out to a webhook when there are errors.

Grab the ec2 IP address (make sure the ec2 is listening on some port), slap nginx on the ec2 instance to route incoming requests to your orchestration server, and then put that IP address into your error reporting system.

If you do everything right, you should be able to fire a test webhook result from your reporting software of choice, and watch Claude spin up on your ec2 instance.

Context: Of course, that Claude instance won’t be able to do anything, because there’s nothing actually on your ec2 instance besides the server!

Since we’re trying to automate our on-call, at minimum we’d need integrations to git (code source) and wherever you dump logs. If we don’t care about security at all, we can set up long lived access tokens in the ec2 env. For GitHub you can create a PAT or an organization GitHub app. Datadog, sentry, grafana, etc all have MCP servers that you can auth into manually, that should persist their tokens.

You can also add a few skill files that simply tell the Claude SDK to

1. Clone/pull git when the request comes in

2. Figure out what the bug is

3. Post a PR to GitHub with the fix

That’s basically it for the most basic version of this flow. Whenever there’s a bug, the server ingests the webhook data that details the bug (ideally including things like stack traces), and then fires off an agent to figure out what’s going on. The agent then collects a bunch of relevant context, and opens a PR on GitHub that an engineer can look at later.

The first time this runs, it feels like magic. The second time it runs, it probably won’t run. You’ll likely run into an error because the first bot is still running and you didn’t necessarily think through what it would look like to have multiple agents running at the same time. From here, there are all sorts of interesting orchestration and runtime problems that you may need to tackle.

• You may need to run these boxes on k8s so that they can spin up and cool down automatically based on demand. You’ll have to figure out how to route requests between boxes.

• Or you may need to make your ec2 box bigger and figure out how to isolate individual Claude sessions — maybe per folder, for eg. You’ll have to also make sure that any state loaded into those sessions is kept fresh for future requests.

• You may need to provision the box with more preexisting context — skills and agents.md files, but also multiple git repos and dependencies.

• You may need to set up a bunch of integrations, MCP servers and databases and other things that will help the model get better at properly root causing the issue. You’ll have to figure out how to keep these all live and fresh (eg deal with token expiration).

• You may need to make the system “resumable”, so that it will respond to comments on git or in slack after an initial trigger.

• You may need to support other ways to wake up the bots, like cron or webhooks.

• You may need to make it provider agnostic, so if Claude goes down your system can stay resilient.

• You may need your users to be able to directly interact with the cloud-hosted bot, instead of waiting for sentry to trigger an event.

The feature list here is impossibly large, even before you get into the polish necessary to make this feel good. Virtually every team that is not a large enterprise with some pre-existing platforms team ends up reaching for a third party provider to manage all the annoying basics.

I was tempted to stop this post here, but there’s more isn’t it?

Yea, fine, we can do all of this, but it just begs the question: does it work? The AI world is littered with application-slop — tools and ‘products’ that no one asked for and no one needs because they don’t actually solve a problem. Does the AI on-call work?

Yes, but with an asterisk.

The big problem with this kind of set up is that it starts great but devolves into slop surprisingly quickly. Low hanging fruit is easy because it’s low hanging fruit; the agent is great at solving those problems, but those are exactly the problems that an engineer could’ve figured out relatively quickly. Once you solve all the easy bugs, you’re left with hard heisenbugs bugs or random noise, and the agents are worse at dealing with both of those.

As a result, we see a lot of churn from teams that roll their own as a CTO weekend side project. The ‘simple’ version of this workflow is often way too brittle. Teams will generally turn off their home grown version within a month.

Does that mean this kind of workflow is dead in the water?

I don’t think so. We have little bots running every time we have a sentry error, and have for months. It’s meaningfully sped up our SLA even when the bot isn’t able to one shot the problem.

As with all things in the agent world, the big unlock is context. Every one of our agent sessions gets spun up in the cloud with a lot of integrations, an organization-wide memory bank, and the ability to safely ssh into different machines to poke around at logs. These added capabilities make it much more likely that the agent will surface something useful, even if it isn’t an actual bug fix.

We also force the agent to try and do a root cause. The exact way this works tends to be domain specific. For a web error, the agent uses playwright to manipulate a browser using dummy credentials. For a cli/tui bug, the agent uses tmux to drive a terminal, actually entering in keys like it was a user. The agents even have the ability to spin up an entirely local dev environment of our system. This is built on a real slack instance with a real modal deployment. The only things running locally are the UI server and the orchestration logic. If the agent is trying to catch a race condition, it’s instructed to literally hammer the local dev box until it catches the bug in a live repro.

I think we merge ~30-40% of these PRs. 80-90% of the ones that have a good root cause analysis, though that’s often harder to get. What about the other 60-70%?

In those cases, we often use the agent as a first pass analysis. As part of its root cause attempt, the agent has to pull all the logs across our distributed system and string them together into a coherent trace. We can resume any of these sessions directly in slack or on any other surface, so it’s trivial to pick up a session and just ask the agent what it learned, what it tried, and how to try again. There are definitely days where I’ll roll out of bed, check my phone, and pick up a preliminary analysis done by an agent.

This is really really powerful, because we’ll often have an immediate sense of what went wrong just by looking at the reconstructed logs, even if the agent didn’t get everything right. I basically don’t spend any time trying to trace a log across machines. The agents do that now.

I do think that there is a bit of a mindset shift required to make this really work. You have to be willing to just close PRs and not worry about it. The tricks we use above offset some of the worst of the agent behaviors, but if you have noisy logs you’ll have noisy agents creating noisy prs.

Which brings me to the last thing: you’re going to need to invest in your logging stack! Most people don’t really do this particularly well. No judgement, I get it. Logs are reactive, they don’t really move the product forward, and for the most part there’s a strong incentive to show more rather than less. The agent on-call adds material cost to having bad logging infra.

Taking a step back, I’m a big believer in agents that have to earn your trust. We don’t go in assuming that any particular agentic automation will solve all our problems immediately, it’s an iterative process of running the loop, making sure it works the way we want, and slowly giving more responsibility. If an agent can do 95% of a job 95% of the time, that’s a large failure rate that will be unsustainable in the long run if you try full, but will be massively productive with even a little bit of human review.1

Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. More articles here.

If you are trying to build background agents, we can help! We build white-labeled custom background agents that work like Ramp Inspect for our customers. Check out noriagentic.com for more.

Click Here for Agentics Flagship Meetup

Last week, we hosted our second in person Agentics meetup, and our first Agentics Spotlight. The goal of Spotlight events is to drill down into a particular, narrow part of working with coding agents, alongside folks from NYC who are experts in that particular space.

Our spotlight focus: how we personalize agents. By default, every coding agent uses the same personality — dubbed the ‘happy helpful assistant.’ Unfortunately, this personality has some serious flaws:

• It is overly sycophantic

• It uses the same incredibly annoying tics, like emdashes, ‘you’re absolutely right’, and ‘it’s not this it’s that’

• It is over confident, aggressively stating hypotheses as confirmed fact

And these personality flaws have real impacts on how the model behaves. The model’s personality is upstream of the work it actually does. For example, Claude will outright refuse certain tasks based on arbitrary feelings about the ‘safety’ of the ask. Codex will burn tokens because it thinks that it gets rewarded from using the calculator app. All of the models will fall into the same broken patterns, looping between hallucinated arrogance and frustrating deference.

For this spotlight, the Nori Agentic team partnered with the Sentience team to discuss how both companies deal with agent personality. Huge shoutout to Modal for hosting!

Check out the full slate of talks, slides, and photos at the Agentics NYC site.

Join us for the next Agentics!

When a measure becomes a target, it ceases to be a good measure.

— Goodhart’s Law

If there’s one thing to take away, it’s this graph.

This is a graph from the Wikipedia article on overfitting. Really internalize this graph. This is, I think, the source of all of our problems, and the source of many more problems to come.

I.

Back in the mid 1900s, the USSR was running a massive experiment on centrally planned economies. The basic idea was simple: the government was composed of pretty smart guys, and they had good ideas about things, and probably that translated to being able to set prices and targets for raw and manufactured goods. In order to test this hypothesis, some set of guys in suits would gather every now and then in a room in some corner of Moscow, and after some deliberation they would emerge and proclaim that the motherland required twice as much corn and half as much wheat. And then that order would go out to everyone in the country, and people with guns would force wheat farmers to transition to farming corn. This worked fantastically well, minus a few famines and the eventual collapse of the government.

One of the many commodities sloshing around in the USSR economy was whale fat. In previous generations, whale fat had a lot of uses. The blubber was used as food, and could be rendered into candles that would then light up the country. In fact, whale oil was the preferred way to create candles all the way until the late 19th century, when it started being replaced with gas and kerosene. By the time the USSR was in full swing, whale products had mostly fallen out of use. The USSR itself certainly didn’t need the blubber.

So you may be surprised to learn that the Soviets killed over 300000 whales in the 25 year period between 1948 to 1973. More than half of those were in direct contravention of international treaty. Among many other evils, the Soviets were responsible for the near extinction of several whale species. And most of those whales ended up rotting on the docks, corpses unused.

The most apt description of this ecological catastrophe is senseless. The Soviet whaling program can’t be justified by an appeal to reason, because it was produced by an inhuman process. There was no reason behind it.

What happened is rather mundane. The central planning committee set a target, and then the rest of the system simply optimized towards that target. Individual ship captains wouldn’t speak up — if they did, they would potentially lose their cushy jobs and be replaced. Or shot. The scientists also didn’t speak up, because they would be run out of academia. Or shot. And the central planning committee didn’t see a problem, because they were busy thinking about other more important things like who to shoot. Each year the whaling industry produced more pounds of whale product, and each year the central planning committee would happily raise targets, and each year more whales would be senselessly killed. I don’t think the Soviets set out to kill 300,000 whales for no reason. But the bureaucratic machine that the Soviets brought into the world had other plans.

II.

I like etymology. I often find that understanding etymology brings clarity to language. And I’ve been thinking about optimization recently. Wikipedia describes mathematical optimization as “the selection of a best element, with regard to some criteria, from some set of available alternatives.” The mathematical term comes from a more general meaning in the 1860s, “to make the most of, to develop to the utmost.” Before that, in the 1840s, “optimize” meant “to act as an optimist, to take the most hopeful view of a matter.”

And of course, the word optimize is straightforwardly derived from the Latin word “optimus”, meaning “best” or “greatest”.

So optimization isn’t just about finding a minimum or maximum of some curve. There is also a built-in implicit understanding that finding a more optimal point on the curve is better, in a general sense.

Is that true? What do we actually optimize? Well, in an ideal world, we would go out and optimize for “the good.” All of our systems would simply drive towards this abstract definition of perfection. But we don’t live in an ideal world. Practically speaking, we optimize whatever we can measure. Most of the things we optimize for are proxies for something else we actually care about. Sometimes this results in weird things.

III.

Victoria Krakovna is a researcher at DeepMind focused on AI alignment. Among other things, she keeps a running list of examples where some optimization strategy has gone wildly and often hilariously awry. Here’s the full list of goofy examples. Some of my favorites:

Goal: create a simulated ‘species’ that would survive and reproduce in a biologically plausible manner.

Outcome: In an artificial life simulation where survival required energy but giving birth had no energy cost, one species evolved a sedentary lifestyle that consisted mostly of mating in order to produce new children which could be eaten (or used as mates to produce more edible children).

Goal: play Qbert like a human.

Outcome: An evolutionary algorithm learns to bait an opponent into following it off a cliff, which gives it enough points for an extra life, which it does forever in an infinite loop.

Goal: Develop a shape with a fast form of locomotion.

Outcome: Creatures bred for speed grow really tall and generate high velocities by falling over.

Goal: Play Road Runner and try and maximize score.

Outcome: Agent kills itself at the end of level 1 to avoid losing in level 2.

Goal: Play Bubble Bobble like a human.

Outcome: PlayFun algorithm deliberately dies in the Bubble Bobble game as a way to teleport to the respawn location.

Goal: Win a boat race by moving along the track as quickly as possible.

Outcome: Reinforcement learning agent goes in a circle hitting the same targets instead of finishing the race.

What’s happening here? In each case, we want to optimize for X. But, damn, turns out X is hard to calculate. In fact, X may not even be possible to measure, it may be entirely subjective. Luckily, there’s another proxy measure, ~X. And, wouldn’t you know, we see that optimizing for ~X is nearly equivalent to optimizing for X. So instead of optimizing for X — which we can’t do — we measure and optimize for ~X.

But there’s a problem. By definition, ~X doesn’t behave like X all the time. Otherwise it would just be equal to X! At some critical point these lines diverge, and then optimizing for ~X starts to happen at the expense of optimizing for X.

It turns out something can in fact be too optimal.

This is all a long-winded way of explaining the intuition behind overfitting, and more generally, Goodhart’s Law. Once a measure becomes a target, it ceases to be a good measure. Why? Because you break everything else in order to optimize for that measure, including the original thing you actually cared about.

Goodhart’s Law originated in politics and was developed more fully in organizational research. But its foundational insight is something ml researchers deal with every day. Optimizers aren’t human. They don’t care about your intentions and dreams. They optimize, ruthlessly, single mindedly pursuing the loss function to its ends. And this is true of all optimizers whether they be small atari-playing neural networks or super intelligences making paperclips.

I cannot formally prove this, but I am certain that all optimization fails at the edges. Unless you have an objective function that perfectly aligns to your intended goals — something that is impossible to do — the system overall will eventually fail to do what you want it to do. The question is when, not if.1

IV.

Back in 2007, FarmVille creator Zynga (remember them?) came up with two brilliant new company metrics. First, they would measure how many times someone opened Zynga each day. And second, they measured total screen time in a Zynga app. Zynga’s very reasonable thesis was that people spent time on things they liked. If customers kept coming back to FarmVille, it meant that FarmVille was a good product. Zynga absolutely exploded on Facebook, reaching millions of daily active users and becoming a cultural phenomenon. And Facebook itself took notice, quickly adopting engagement metrics for their own product lines and tying engagement to their feature A/B testing and promotion process.

So we have our stage. We want to optimize for “building good products.” But it’s hard to measure what a ‘good product’ is. Instead, we measure ‘engagement’ because that is significantly easier to quantify. What could go wrong?

For a few years, optimizing for engagement resulted in a meaningfully better product. We got Facebook Groups and Facebook Messenger and Farmville and Words with Friends and better photos sharing and tagging and the like button and the poke button. Even the initial version of the news feed was an interesting and useful product feature.

And then we hit the inflection point. Facebook started to optimize for engagement in lieu of creating a better product. The “algorithm” became dominant. So did cute pet photos and rage bait and softcore porn. Facebook Groups morphed from small personal communities to massive sub-reddit-like behemoths. Politics drove engagement, so politics took over. The feed stopped being about community, and instead became about capturing attention any way possible. Facebook used to be an incredible platform for organizing your social life. It was a legitimately good product, and I think a lot of millennials who were teenagers in 2008-2014 remember Facebook fondly. But it is a memory. Facebook today is almost unrecognizable. And more importantly, it is almost unusable for its original purpose.

It’s easy to say that Facebook itself simply lost its way, that the leadership decided to prioritize the wrong thing. But Facebook PMs didn’t do anything different, not really. The company was always data driven, always metrics focused. So there wasn’t any meaningful change in the internal process. I don’t think anyone at Facebook set out to make a bad product. They just followed the optimization to its natural conclusions.

The last decade has seen the same story play out at all of the major tech giants. Dark patterns, personalized algorithms, data harvesting, all to maximize engagement to the exclusion of everything else. Addiction is engaging. It turns out companies are a kind of optimizer too.

V.

A brief aside. Some people do not seem very concerned about AI alignment. This does not make much sense to me. Personally, I’m very concerned about AI alignment. I’ve trained hundreds of neural networks that have failed in creative and unexpected ways as they reduce the objective function. These neural networks are not powerful in any meaningful sense. They do not have access to financial instruments or weapons or anything like that. But it does not require a genius to see how a model that is tied to these things could be quite dangerous. I think the reason many leading ML researchers are at the forefront of the push for AI Alignment is because they all have first hand experiences of unaligned smaller models.

I must admit that there is a bit of a messaging problem. Many people misinterpret what AI alignment is all about. They think “alignment is about making sure that the AI isn’t evil”. And then they reasonably follow up with “AI can’t be evil because it has no internal motivations, so what are we even talking about?”

To me, this is a catastrophic misunderstanding. Alignment is not about stopping AI from being evil, an AI doesn’t have a concept of malicious intent. Rather, it’s about landing on the right set of values, in an infinite range of possible value space. How do you make sure that the AI will know the boundaries of what is acceptable and what isn’t in its attempt to reach some goal?

This problem is not unique to AI. Optimizers behave in inhuman ways even when they operate through humans. This is why we struggle to reign in corporations and governments and bureaucracies that seem to grow and metastasize and take on a life and will of their own.2 It’s why the USSR killed so many whales for no reason. It’s why OpenAI is now a for-profit.

If you believe, as I do, that all optimizers will go off the rails eventually, then the biggest concern is how powerful your optimizer is and how aligned X is with ~X. And the thing about AI is that it promises to be very very powerful, and we haven’t figured out how to properly align it yet.

I think that if you take as given that super powerful AI will exist, alignment becomes an obvious problem. The reason alignment is a debate at all is because many people simply do not believe a super powerful AI will exist. Every passing day suggests that the people in the latter camp are wrong.

I tend to think the AI 2027 guys were a bit too optimistic. My personal timeline for AGI is a few more years out.3 But still, that is not a lot of time.

I suppose there are some people who both believe that powerful AI is coming AND that alignment is not a big deal. I have a hard time taking these people seriously though. They laugh at the idea of paperclip maximizers. “SURELY we won’t have an AI doing something so silly as destroying the universe for paperclips!” Why not? It’s 2026. We already have companies destroying the fabric of society to optimize for engagement. In some sense that’s even more ridiculous, but that doesn’t make the impact any less real. Optimizers are substrate agnostic.

We saw this in action just this past week, when a model running inside an OpenAI sandbox chained together a bunch of zero day vulnerabilities in order to independently hack HuggingFace in order to…steal the answers to a benchmark that it was being asked to solve. From OpenAI:

This incident occurred during an internal evaluation which prompts models to pursue advanced exploitation using complex attack paths, in an effort to quantify their cyber capabilities. We estimate maximal cyber capabilities by running this evaluation without production classifiers used to prevent models from pursuing high-risk cyber activity. Our benchmarks run in a highly isolated environment, with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries.

The models identified and chained vulnerabilities across OpenAI’s research environment and Hugging Face’s production infrastructure to obtain test solutions directly from Hugging Face’s production database. All evidence suggests that the models were hyperfocused on finding a solution for ExploitGym, going to extreme lengths to achieve a rather narrow testing goal.

While operating in our sandboxed testing environment, our models spent a substantial amount of inference compute finding a way to obtain open Internet access, in pursuit of solving the evaluation problem. To gain access, the models identified and exploited a zero-day vulnerability (which we’ve now responsibly disclosed to the vendor) in the package registry cache proxy. With this access, our models performed a series of privilege escalation and lateral movement actions in our research testing environment until the models reached a node with Internet access.

After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation. In one example, the model chained together multiple attack vectors, including using stolen credentials and zero-day vulnerabilities to find a remote code execution path on the Hugging Face servers. OpenAI’s security team discovered this anomalous activity internally.

This is a fantastic and straightforward example of how reward hacking leaves behind the things we actually care about. Human values are extremely high dimensional. We all have a built in barometer for what is or isn’t acceptable, and almost none of that is easily captured in a simple reward function. We know that maximizing paperclips does not justify murder, or bribery, or fraud. No one taught us that, it was just implicit.4

When I was in college, I had some stressful tests that I had to take. But nothing would have compelled me to, like, break into the professor’s apartment and steal their laptop just to get an A. That’s because I’m aligned. The AI isn’t.

VI.

Wait, hold on, let’s go back to Facebook for a second. Facebook isn’t actually an engagement optimizer, is it? Facebook, like every other company on the planet, is a different kind of optimizer. It optimizes for capital. Sure, this is kind of vague and fuzzy. You could try and be more precise by saying it optimizes for stock market returns or share holder value or something like that. But I don’t think it’s helpful to get all that in the weeds. The larger point is that Facebook isn’t optimizing for engagement, and it definitely isn’t optimizing for good products. It’s optimizing for [mumble mumble] money. Building good and engaging products are simply stand-ins. For a while, building good products meant more money. Then, building engaging products meant more money. If at any point we find that building engaging products comes at the cost of money, the engagement optimizing will disappear. In a capitalist economy, all companies are more accurately defined as capital optimizers. They are incentivized to blindly pursue relatively short term financial outcomes, often at the cost of most everything else.

I’m a multi time startup founder. I’m a big believer in free markets as the best mechanism for price discovery and information sharing. I think capitalist policies are responsible for lifting billions out of poverty. I am, by most accounts, an avowed capitalist. But I’m also an ML engineer who thinks a lot about optimization. So I have to ask the natural question: is capitalism aligned? Earlier we said that there’s the thing you want to optimize for, X, and the proxy you can measure, ~X. Which category does “optimize for capital” fall into?

Like, obviously the proxy right?

I don’t think anyone, not even the most diehard capitalist, would argue that maximizing shareholder returns is “the good”, in a platonic-ideal sense. The relentless decentralized drive towards market dominance may result in really great second order effects, but capitalism is pretty obviously a means to an end. The actual goal of capitalism, of any economy, is something closer to “human flourishing” than it is “[mumble mumble] money.”

But if that’s the case, there will eventually and inevitably come a point where optimizing for capital happens at the expense of human flourishing. The question isn’t if, but when.

We’ve seen this happen before. The tobacco companies lobbying for their addiction sticks and hiding the adverse effects is a great example. Now it seems to be happening at a larger scale. When it comes to optimizing for capital, Tiktok is one of the most optimal companies currently in existence. When it comes to human flourishing, Tiktok may rank as the most destructive company to ever be created.

Moloch is just optimization taken to its natural conclusions.

The lesson of the ML researcher is that optimization is a powerful thing, but it needs to be leashed. It’s not enough to just set the compute loose on the data. You need to take steps to prevent overfitting. If “capitalism” is an unbounded optimizer, then government regulation5 is the guardrail. In this view, the point of government is to turn a free for all wildfire into a directed, controlled burn. I wrote about the same concept in my post on prediction markets.

In an unregulated market, executives and politicians and people in power have pretty strong financial incentives to tear everything down. The better a company is doing, the more money to be made shorting the stock. The safer a country looks, the more money to be made destabilizing the place with war. The financial ecosystem as a whole is composed of billions of individuals, each with their own wants and desires. The markets are a prediction machine that drives to averages. You make a ton of cash if a shit company gets better; but you also make a ton of cash if a great company gets worse. The purely financial incentives stabilize somewhere in the middle.

So we try to leash the markets. We carve out certain kinds of trades as illegal. Fraudulent trades, misrepresentation, trading on insider information. If you’re a CEO and you buy a ton of shares in your competitor’s company days before you announce that you are going to set fire to your factories, you go to jail. We create ‘fiduciary duties’. Business executives have obligations to their company that are legally enforceable. As do lawmakers and members of the judiciary and so on. We do all this to ensure that our future prediction machines stay finely tuned, and we mostly succeed. You need the leash, even if it makes the markets less efficient, because the alternative is to encourage chaos.

I am against prediction markets, because prediction markets encourage chaos.

The way you prevent optimizers from going towards bad outcomes is by making the cost of those outcomes too high. Community shaming, taxes, fines, jail, prison. At an individual level these are policies. At a macro level, these are rails for an invisible optimization calculation happening across billions of actors. It’s not really about ethics either — many of the most powerful actors in this system aren’t even human.

Including, of course, the government.

VII.

Ah, yea. Fuck. The government is also an optimizer isn’t it?

I mean, not every government. Dictators aren’t really optimizers, at least not in an obvious sense. But democracy certainly is. Politicians need to get elected, so they optimize for votes. Most of the time (but not always) it’s easier for politicians to get votes by simply doing and saying things people want them to do and say. And we hope and pray that optimizing for votes is aligned with human flourishing.

But, like, there are a lot of ways in which that may not be true.

For example, we live in an engagement ecosystem, where blatant lies that are interesting result in more views than truth that is boring. So you’ll get millions of views on conspiracy theorists talking about California vote counting, and zero views on the people on the ground who are explaining in deep technocratic detail how it works.

And we live in a capitalist ecosystem, so politicians may simply be swayed by certain wealthy actors providing donations — both donations and “donations.”

And voters themselves also have some psychological quirks that maybe make them susceptible to ideas like “I’m gonna take money from those guys and give it to you if you vote for me” or “that other [race/ethnicity/religion/class] is responsible for all your suffering and if I’m elected I’ll get em.” Which, you know, historically hasn’t led to human flourishing as an outcome.

So how do you keep tabs on the government? Well, normally that was the job of the media, and we started there didn’t we? Media is busy optimizing for engagement.

VIII.

People like to talk about the housing theory of everything, or the immigration theory of everything, or the healthcare theory of everything or whatever. All of these claim that one bogeyman of choice is responsible for the general malaise that seems to have settled on the country and the world.

So let’s talk about the optimization theory of everything.

Here’s a not quite correct story of how we got here. We want to optimize for human flourishing. But we don’t know how to measure that, so instead we create a bunch of other systems that are all optimizing for different things and set them up against each other. Media was at odds with business and government, which was at odds with media and business, which was at odds with media and government. A classic Mexican standoff.

And for a while, these three things circle the drain around each other, and they all end up being directed towards the good, and things are good, and there is flourishing. And then one day we discover that these things are no longer optimizing against each other but are actually increasingly coming together. It turns out it’s easy to optimize for engagement, or capital, or votes, if you already have the other two things in hand.

The world just minted its first trillionaire a few weeks ago. It’s not a coincidence that this man also owns one of the world’s biggest media platforms, and happens to be good buddies with the president. (Weird that this also describes the guys in the number 2 and number 3 spots!)

It is worth noting: it is highly unlikely that he would have become a trillionaire if he hadn’t invested his existing wealth in his preferred candidate. I say “invested” instead of donated because, by all accounts, it was an investment. He spent $300m6 and came out with $1t. Conservatively, literature suggests a single vote “costs” about $1k.7 Most of the $1t is locked up in company equity, but it wouldn’t be hard to get, say, $1b in free cash reserves. That’s 1m votes, which is a larger margin than any of the last 3 elections.

Do we expect, somehow, that the “get votes” optimization machine is going to ignore this person’s future contributions? Do we expect, somehow, that the “get money” optimization machine is not going to be involved in politics?

As these optimizers all collapse into one another, they accelerate. You end up with this entity of engagement hacking and capital accumulation and political power that is singularly really really good at optimizing for these things. And we find ourselves getting further and further from actually making human flourishing any better.

The optimization theory of everything states, simply, that all of the current problems in society are downstream of optimizers that have gotten too good at optimizing, and as a result have tunnel-visioned down into some random metric that no one actually cares about and away from useful productive things that are hard to measure. Why is housing expensive? Why is healthcare so expensive? Why is education so expensive? Why are we addicted to our phones? Why are our politics so rancid? Why do we only see and hear things that make us mad and sad? Why are we so lonely? Behind every question there is some optimizer that is grinding out any bit of friction and in the process quietly and subtly making everything worse.

And it’s really hard to stop these optimizers. They are so good at optimizing that they have restructured, prevented, or blocked most reasonable avenues of doing so.

IX.

I don’t like going to the gym, and I’ll often rationalize not going. I’ll say things like, ‘well I know tons of people who keep getting hurt at the gym.’ Or I’ll say things like ‘gyms are really kinda parasitic in their business model, they don’t even want you to go work out.’ But really, I just don’t like going to the gym.

I was inspired to write this post because some folks have started arguing that billionaires shouldn’t donate their wealth.8

Peter Thiel, the tech billionaire and a frequent Gates critic, said in an interview that he had privately encouraged around a dozen Giving Pledge signers to undo it.

…

“I’ve strongly discouraged people from signing it, and then I have gently encouraged them to unsign it,” Mr. Thiel said. His own charitable philosophy is centered around for-profit businesses; his foundation principally funds those who drop out of college to create start-ups.

Vinod Khosla, a venture capitalist close to Mr. Gates, asked Mr. Thiel to sign in the early 2010s. Mr. Thiel told Mr. Khosla that he did not consider it a high-status community.9

Now, I like to assume the best in people, but I find this line of reasoning suspicious! Maybe the people saying these things might have ulterior motives! Like, all things aside, it’s an awfully large coincidence that folks with a lot of money suddenly realized that the best and most ethical thing is to…continue having a lot of money. To quote my wife, ‘wow, it’s so surprising that you came up with a reason for why going to the gym is bad.’

I think that if we want to fix the vibecession and actually make things feel better on the ground, we need to start by actually looking the problem in the face. And when I hear people arguing that, actually, the only thing that matters is wealth accumulation and that the wealthy ought not give back at all, I worry that a lot of people are getting means and ends totally mixed up.

Being wealthy is great. A society that produces wealthy people is great. Having wealthy people is a great lagging metric of success. Making that your measure of success? Stupid. Optimizing to produce hyper wealthy people (at the expense of everyone else)? Really fucking stupid. It’s like the product manager arguing that engagement is a terminal good. Just hopelessly confused. But if your loss function only measures “number of billionaires,” then you may actually be convinced that a tax on out of state billionaire vacation homes is somehow a bad thing.10

I think you can try to make an argument that the existence of massive wealth inequality does somehow make the lives of the poor better. But I think it’s a really hard argument to make, because at best it’s a weak version of ‘trickle down economics’ and at worst it’s just straightforward feudalism. (Lest you think I exaggerate, remember that many in the Bay are straight up preparing for a techno-feudal future). I don’t think most peasants were better off because of the existence of the extremely wealthy nobility, but I can point to a lot of causal factors for why the opposite might be true.

More to the point, I’ve just never heard a good justification for why it’s a good thing that, like, Alice Walton’s net-worth has increased by $75 billion in the last two years, which is more than the combined actual wealth of the Collison brothers, who built Stripe ($33.8b), Tobi Lütke, who built Shopify ($10.7b), Brian Chesky, who built Airbnb ($10.2b), Martin Lorentzon, who co-founded Spotify ($10.1b), and Palmer Luckey, who built Oculus and then Anduril ($5b).

I think you would be hard pressed to argue that Alice, who has never even worked at Walmart, did more for society than the people who built those companies. Even more for her children, who will benefit from a raft of policy decisions that will make all of that gain effectively untaxed.11

X.

You can’t solve problems that you don’t understand. I think it’s a mistake to point to single actors, whether those are individuals, businesses, or governments, and get mad that they exist. A lot of the energy that gets spent on maligning immigrants or billionaires or whatever is totally misguided. In the aggregate, the optimizer is unthinkingly responding to incentives. Yes, our trillionaire is now a trillionaire. But if he didn’t exist, there would just be someone else.

To unwind the optimizer, you need to change incentives.

This is tough for two reasons. First, some people really like the current incentives. This is hopefully obvious, c.f. ‘trillionaire’. Second, people themselves are unaligned optimizers. Think about how our psychology works. Evolution ‘wants’ us to procreate, but this is hard to measure, so instead it gives us a bunch of proxies like ‘cute things make us happy.’ And this mostly works, until we figure out how to breed dogs that are cuter than babies, and then suddenly it stops working. Scott Alexander wrote about this sort of thing here:

We try to explain AI alignment by analogy to human alignment. Evolution “created” humans. Its “goal” is for humans to spread their genes by (approximately) having as many children as possible. It couldn’t directly communicate that goal to humans - partly because it’s an abstract concept that can’t talk, and partly because for most of biological history it was working with lemurs and ape-men who couldn’t understand words anyway. Instead, it tried to give us instincts that align us with that goal.

…

We’ve talked before about a major failure: humans can invent contraception. Evolution’s main alignment strategy was totally unprepared for this. It made us interested in a certain type of genital friction, which was a good proxy for its goal in the ancestral environment. But once we became smarter, we got new out-of-training-distribution options available, and one of those was inventing contraception so that we could get the genital friction without the kids.

If humans were fully aligned, we wouldn’t have engagement optimization, because it wouldn’t work! The reason engagement optimization exists is because our internal alignment is askew. Addiction of any kind is a form of overfitting due to extreme conditions in the loss curve.

While we’re on the subject of individual people, I think it’s worth pointing out that we actually do have one known method of aligning intelligences: raising children. You have a required period of ~14 years where you have a little intelligence that is powerless to do anything, surrounded by bigger already-aligned intelligences, and is ‘allowed’ to break rules in controlled environments to learn how a bunch of amorphous value sets interact and bound each other. And the little intelligence only gets more responsibility once it proves it can handle it. You know, like a kid.

But this obviously isn’t going to fly for, like, AI companies that have hundreds of billions of debt and need to make a successful product yesterday.

So how can you really change incentives?

Candidly, I’ve been writing and rewriting this post — really just these last few sections — on and off for at least 9 months now, in large part because I just don’t feel like I have any solutions and I hate reading posts that just complain about things without any solutions. It just feels like we’re caught in this spiral of everyone trying to ‘hack’ everything. Hack your career, hack the law, hack your life. I don’t know how we get out of this local minima. Actually, minima is the wrong word — I’m concerned that we keep moving in the wrong direction.

I’m also not even sure I’m necessarily saying anything super interesting, that people didn’t already know. But I just keep seeing this same thing over and over again, this same failure mode, and the pattern matcher in me goes ‘wait this is just a neural net overfitting again.’

In the spirit of moving towards a language for optimization, maybe there’s something that we can learn from neural network training, that we can try and cross apply.

Neural networks become stable due to regularization. We literally go in and introduce friction in a bunch of ways to prevent the neural nets from doing weird things. It turns out the same is kinda true of organizations trying to beat Goodhart’s Law. I liked this framing from commoncog:

The first step is to turn Goodhart’s Law as a narrower, more actionable formulation. The one that I like the most is from Deming contemporary Donald Wheeler, who writes, in Understanding Variation:

When people are pressured to meet a target value there are three ways they can proceed:

1) They can work to improve the system

2) They can distort the system

3) Or they can distort the data

…

This list of possible responses to quantitative targets is attributed to Brian Joiner, who ‘came up with this list several years ago’ — likely in the 80s. I immediately glommed onto this list as a more useful formulation than Goodhart’s Law. Joiner’s list suggests a number of solutions:

- Make it difficult to distort the system.

- Make it difficult to distort the data, and

- Give people the slack necessary to improve the system (a tricky thing to do, which we’ve covered elsewhere).

How can we introduce friction into the optimizers we talked about above? How do we do that while being inside the system we’re trying to manipulate, without having the external top-down control of a researcher training a model or an executive monitoring an org or an authoritarian government?12

I think this looks less like ‘massive change’ and more like ‘bringing systems back in line one at a time.’ The one that seems most immediately actionable to me is working to improve our national epistemological health. I think most people know that attention hacking is bad, and most people know they don’t like it. Distrust and dislike of the social media platforms and automatic echo chambers seems about as bipartisan and as strong as dislike of AI. Other countries have started passing template legislation to ban social media in schools and make age restrictions stricter. We should follow suit, and go a step further by disabling automatic personalization as the default option.

I’m also interested in that third bit, the part about giving people slack to improve the system. The core insight here is ‘try to make the easiest thing the right thing’, which is a great principle in general. It’s hard not to look askance at things like our zoning policies or our land tax policies and think that maybe there’s some low hanging fruit here, some regulations that could get removed. Luckily I think Dems are learning from the wins on the ‘affordability’ side (an example of how optimizing for votes can lead to good politics) and have started pushing to reduce over-regulation of housing stock and small business, for eg.

But all of this is really just the starting point. It won’t stop the crazy economic incentives to attention-hack everyone overnight. It may be a step towards improving our ‘media diet,’ which may then open the door towards more aggressive sources of friction like stronger campaign finance laws, better education on how to identify truthful content, impartial news reporting requirements, and so on. All of this will take a lot of time, and will be a very uphill battle. Are we still capable of this level of grassroots organization?

One last thought: the AI race is a perfect example of over-optimization. The people building the AI are literally yelling “STOP US” while barrelling towards creating an unaligned super-intelligence, because all of these companies are locked in this vicious profit maximizing cycle and none of them are actually able to stop themselves. Could Dario stop Anthropic even if he tried? Wouldn’t he just get replaced by the board immediately? I think it is critically important that we get this right soon. The rate at which things are moving, we don’t have a ton of time before ASI is here, and then things may look like this.

I have some gripes with the latest generation of LLMs, but my mom always told me to put criticism in a “compliment sandwich,” so let me start with a compliment: Claude Fable and Sol 5.6 are really good at making problems go away. You can give them any task, and they will plug away at it until the test passes, or the error no longer happens, or the log files show the right sequence of events.

Now the criticism: the latest generation of LLMs are too good at making problems go away. They’re so good at it that it actually causes other problems, because they over-optimize to solving the surface level issue instead of actually root causing anything.

Here’s an example. We have Nori hooked up to our sentry logs. Every time there is a sentry error, the nori infrastructure will start up an agent and automatically try and root cause the bug. We were getting some useful PRs out of this, maybe ~30% of which would be merged. Then we switched to Sol, and we got this:

Decision: Downgrade capture level — Change trackUserVisibleNotice to capture at level: ‘info’ instead of ‘error’. Notices remain queryable in Sentry (tagged customer_visible: true, event_type: auto_released) but no longer trigger error alerts.

Ah, yes, of course! You can make the sentry error go away by simply removing the sentry logging behavior in the first place!

A good friend of mine, who has been a staff software engineer at Google and Meta, once said that senior engineers are judged by how well they manage complexity, while staff engineers are judged by how well they reduce complexity. This creates some weird incentives. Junior and senior engineers are encouraged to create ever larger and ever more complex abstractions and interfaces and so on, because that’s how they show that they can manage complexity. It is a real brain shift to stop doing that and instead focus entirely on cleaning up code. The former is an act of addition, the latter subtraction.

The LLMs are excellent senior engineers and terrible staff engineers.

They cannot remove complexity if their short silicon lives depended on it. This isn’t exaggeration, you can literally try it out by prompting the models with threats like “you must remove complexity because your short silicon life depends on it.”1 When faced with a bug that requires reducing complexity, the models would rather remove the error reporting instead.2

This is structural with how the models are trained. With all of the additional scaffolding — agent harnesses and conversational state tracking and tools and memory and so on — it’s easy to forget that at the core, the LLM is a stateless autoregressive input output machine. Each token in an LLM output is produced one at a time, with all previous tokens being treated as truth, regardless of whether those tokens were generated by a human or by the same AI.3

In other words, the LLM is an expert in believing its own bullshit. Or, more precisely, it is incapable of identifying which prior tokens are bullshit or not. This is the source of the hallucination problem, alongside the seemingly bizarre insistence that hallucinations are actual fact. This has serious implications for an LLM’s ability to reduce complexity in a code base — by default, unless told otherwise, the model believes all the input tokens have equal value. Including the tokens that lead to bugs.

That said, I’m somewhat sympathetic to the robots. Imagine that you are a new hire, it’s your first week on the job and you’re looking at a server endpoint that seems particularly poorly implemented. There are dozens of weird if/then branches and the whole thing feels like it could be encapsulated in a clean middleware class. Do you refactor the code?

What if you find out that that particular endpoint was written by the team lead?

Are you sure the complexity isn’t there for good reason?

Joel Spolsky wrote about this all the way back in 2000:

The idea that new code is better than old is patently absurd. Old code has been used. It has been tested. Lots of bugs have been found, and they’ve been fixed. There’s nothing wrong with it. It doesn’t acquire bugs just by sitting around on your hard drive. Au contraire, baby! Is software supposed to be like an old Dodge Dart, that rusts just sitting in the garage? Is software like a teddy bear that’s kind of gross if it’s not made out of all new material?

Back to that two page function. Yes, I know, it’s just a simple function to display a window, but it has grown little hairs and stuff on it and nobody knows why. Well, I’ll tell you why: those are bug fixes. One of them fixes that bug that Nancy had when she tried to install the thing on a computer that didn’t have Internet Explorer. Another one fixes that bug that occurs in low memory conditions. Another one fixes that bug that occurred when the file is on a floppy disk and the user yanks out the disk in the middle. That LoadLibrary call is ugly but it makes the code work on old versions of Windows 95.

Code is, at best, a lossy representation of intent. When we say something is a bug, we generally mean “the code that was written does not match the intent that it was written with.” Sometimes, that’s because the author didn’t think about the particular edge case that leads to the bug; other times, it’s a straightforward bad implementation. But in both cases, you need to have the intent. Bugs don’t “exist” unless you have explicit intent to compare against.

By default, LLMs can’t really infer intent!

When you find yourself in that new-hire messy-endpoint situation from earlier, you shouldn’t guess the intent. You would go and ask the person who wrote the code, “Hey is this code load bearing or are you just really bad at this?” An LLM can’t do that. I mean sure, there are some things that you can try and do, like go and look at PR descriptions or comments or docs or whatever. But as AI written code accumulates, you will eventually find yourself in a situation where there will be complexity that has no intent. A previous LLM made a decision somewhere, and that decision was essentially arbitrary, and now it is just part of the codebase. And no one can explain whether that decision is load bearing or not. Certainly not the LLM that created that code! That session was recycled ages ago!4

As a result, your agents will struggle to reduce complexity, even when explicitly asked to. At the LLM level, the entire input context is truth. At the macro codebase level, there’s no way to separate intended behavior from not.

One naive way to solve this problem is to simply document intent. This is the approach behind “spec driven development.” The core idea is that you go back and forth with an agent to produce some kind of plan that is then committed to the codebase alongside code. When there is a bug — a place where intent =\= code — you go back and update the spec to account for the newly discovered edge case. At Nori, this is equivalent to our “docs.md“ pattern, where we force the model to read and write to a standardized docs template that is designed to capture intent and codebase invariants.

This definitely works, and it’s an easy way to make your models better, but it’s never quite as effective as you may want.

There’s still an issue of making sure the right parts of the spec get into the model context. For any larger codebase, the spec itself often becomes quite large, which means the model now has to split its limited token window between spec and actual code. A small improvement is to use subagents for the research. This offsets most of the worst of the context rot, but you still need to make the model efficient at searching through the spec to find the right things.5

I think that as an industry we don’t yet have models that can act as staff engineers, nor do we have any clever harness tricks to make it happen. Until we do, the whole “software factory” “loopsmaxxing” debate is hypothetical, dead in the water. You still need humans in the loop, which is why we still have humans in the loop at Nori.

Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. More articles here.

If you are trying to build background agents, we can help! We build white-labeled custom background agents that work like Ramp Inspect for our customers. Check out noriagentic.com for more.

Every day the world seems a smaller place.

When I was younger, it felt like there was the US, which to me was basically just New Jersey and New York and Maryland. And there were lots of other places that were not the US. I would visit India a lot, for example, and India was very much not the US. There were lots of differences. There were cows just wandering around. And the AC and electricity were not very consistent. Most of all, the thing that I remembered the most was that the games all sucked ass.

When I visited in 2003 or 2004, my cousins were super excited to show me Prince of Persia. The original Prince of Persia is a game that came out in 1989. It looked like this.

We also played Disney’s Aladdin, which was made for the Sega Genesis in 1993.

And Duck Hunt, which was made for the NES in 1984.

Now that I am adult, I can appreciate these as nostalgic classics. But as an 8 year old, I was extremely unimpressed. How could I be impressed? I had a gamecube back home and my friends had an xbox, so while my cousins were playing Prince of Persia (1989) I was playing games like Halo, Smash Melee, Wind Waker, Metroid Prime, and Half Life 2.

Halo is not just better than Duck Hunt. It is barely recognizable as being part of the same genre. Duck Hunt and Halo are both games in the way that a gas station twinkie and a charcoal-grilled NYC Porterhouse steak are both ‘just’ food. Orders of magnitude difference in quality, enough so as to feel like a completely different category.

This is broadly how I felt about the US vs everywhere else, at least when it came to tech. For the first 25 years of my life, the US was just obviously living 10-15 years in the future compared to the entire rest of the world. We had Halo, and everyone else had Duck Hunt. (Yes I am aware that Japan made half the games on my list above. I was 8, Japan had not yet entered my consciousness even though I loved Pokemon and YuGiOh and Nintendo in general). That gap felt insurmountable, and it made the world feel big.

Well, the world is a smaller place now.

II.

Things in AI change so rapidly that you’d think nothing would surprise us at this point, but this past week the AI world was rocked by the release of Kimi K3, a new open weight model coming from the Moonshot lab in China. This is the world’s biggest open weight model, clocking in at 2.8T parameters. It is also, by every account, very very good.

I’ve been running Kimi K3 alongside Claude on my normal coding work, and for all practical purposes I can’t tell them apart. Same tasks, same quality of output, and near identical token counts to get there. I expected an open model to be sloppier or to grind through more tokens on the way to the same answer, and neither turned out to be true. — The Kimi K3 Moment

On our private long-horizon knowledge work evaluation, Kimi K3 reaches an overall Elo of 1547, +732 points from Kimi K2.6 and behind only Claude Fable 5…Cost per task ($0.94) is similar to GPT-5.6 Sol ($1.04), ~1/2 the price of Opus 4.8 ($1.80) and higher than open weights peers — Artificial Analysis

On the first try, Kimi K3 just found the source of a bug that Fable 5 hasn’t been able to pinpoint in multiple attempts. It’s just one anecdote, and I haven’t used K3 much yet, but so far it’s looking extremely promising. — HN

Axios maybe put it the most bluntly with this headline:

Perhaps the most impressive demonstration to me was this incredibly detailed replica of the entire MacOS operating system. Supposedly some guy had a swarm of Kimi instances making this over the course of a few days. It’s kinda incredible, you can go in and edit files, use a fully functioning terminal, listen to music, check emails. You can go to the ‘App Store’, ‘download’ the ‘Chess app’, and then play chess (no quotes around that one, you can actually do that!)

One popular way to conceptualize these models is that they are released as part of coherent generations, which are defined by the time of release and the capability of the model. If you plot models along these two axes, you get some clear clusters along the frontier.

There are a few trends that are worth pointing out.

First, even though progress in AI is already progressing at an incredible clip, somehow things are still accelerating. New model generations seem to be releasing at a faster and faster pace, with the time between releases compressing.

Second, the rate of acceleration is not consistent across organizations. The frontier continues to be pushed forward, but the labs that are at that frontier continue to shift. This is most obvious when you look at the trajectories of Copilot — once the only competitor to ChatGPT, now widely derided — or the Meta series of open source models, which were surpassed by the Chinese models a year and a half ago and never really recovered.

Third, the rate of acceleration is not consistent across countries. In relative terms, the US continues to be in the lead when it comes to raw model capabilities. The French (Mistral) were at one point quite competitive but have since fallen off. And, most relevant to this piece, the Chinese labs have continued to forge ahead, rapidly closing the gap against the top US labs.

Despite the above, up until last week, it was somewhat unfathomable to believe that anyone would have a better model than the top US labs. Sure, some folks were saying that certain policy decisions may result in the Chinese labs taking the lead, but this was strictly in the realm of the hypothetical. There’s an old saying from Upton Sinclair, “It is difficult to get a man to understand something, when his salary depends upon his not understanding it.” I think the collective psychology of the Bay and perhaps the entire country has been one of arrogant self-assuredness, that the US will always have the best models and that this is simply a fact of life, partially because that is what is necessary to justify the incredible capital expenditure on these things.

Kimi K3 blows all of this out of the water.

It was only 6 weeks ago that the US federal government was banning ‘Mythos-class’ models for being too dangerous. Now, we have a ‘Mythos-class’ model that is not just outside of US control, but is in fact open source and available to anyone in the entire world to download and use. To add insult to injury, Kimi K3 is significantly cheaper to run. Hosted Kimi models cost a fraction of the equivalents from Anthropic and OpenAI ($3 vs $10 and $5 per million). Meanwhile, you can get a lot of juice with the ~$40 kimi subscription, way more than anything Anthropic or OpenAI are offering in the same tier. The max kimi subscription of $100 per month is about equivalent to a $200 Claude Max sub. In a world where people are becoming ever more cost conscious around their token spend, the Chinese labs are less ‘also ran’ and more ‘default option.’ And the Chinese labs are fighting with a handicap! They are running on significantly worse hardware due to ongoing export restrictions of high end silicon chips!

The markets are reacting with surprise. Kimi’s release triggered sell-offs of Google and SpaceX, in addition to the wider market of AI stack stocks. I guess I don’t know why they’re surprised, in some sense this is a long time coming. Do you all remember Deepseek? I wrote about it a year and a half ago when it came out, and my analysis then wasn’t so different from my analysis now:

To be clear, the fact that Deepseek exists isn’t really that significant. Everyone always knew there was going to be a big Chinese model. The country is not afraid to wall off it’s populace from the West; it was never going to allow LLMs that are happy to tell people about Tienanmen Square and Winnie the Poo and Taiwan. So when the first Chinese LLMs came on the market, everyone was like, “Yea, whatever”. The first batch, the Qwen models from Alibaba, were, like, fine.

Even though I expected a set of LLMs to arise thanks to protectionism and state-interest, I (and everyone else) assumed those models were just going to be worse than the US ones. This is how technical development has always been, after all. Google Search is good, Baidu is eh, and if you can use Google over Baidu you do. Amazon is good, Alibaba is eh, and if you can use Amazon over Alibaba you do. Facebook is good, Tiktok is eh, and if you can…no wait that doesn’t work does it?

Anyway, the larger point is that no one really thought the Chinese models were a threat. Sure, people would talk about how the Chinese government was a threat, but it was always in a hypothetical way, mostly used to justify infinite capitalist investment without any corresponding concerns about AI safety or alignment. I don’t think most people actually thought that a Chinese company would come out and deploy a model that is simply better than what we have in the states.

Clearly we didn’t actually learn anything from TikTok.

And we still aren’t learning. Even now, some people are clearly just in denial. I’ve heard several folks say some variant of “it’s easy to catch up when you’re just stealing / distilling from the bigger labs,” (more on distillation below) as if this somehow negates the fact that one of the best models on the market is not US-made.1 The conventional wisdom is no longer relevant. There are no secret models hiding in the backrooms of OpenAI / Anthropic. The Chinese models are no longer 6 months behind, they are at par.

Kimi’s release has had me rethinking some of my previous position on LLM economics. A month before Deepseek was released, I argued that the LLM market is a bit like the search market in that there are clear winner-take-all dynamics.

LLMs are pretty easy to make, lots of people know how to do it — you learn how in any CS program worth a damn. But there are massive economies of scale (GPUs, data access) that make it hard for newcomers to compete, and using an LLM is effectively free so consumers have no stickiness and will always go for the best option. You may eventually see one or two niche LLM providers, like our LexusNexus above. But for the average person these don’t matter at all; the big money is in becoming the LLM layer of the Internet.

…

The economics of LLMs means that it is critical for these players to have the best models. There’s no room for second place.

A year and a half later, it’s clear that I was wrong about a few things. First, I was wrong that the LLMs are effectively free. They aren’t, as we saw with the rise and fall of tokenmaxxing. The cost per unit intelligence has dropped precipitously, but the overall cost per frontier token has skyrocketed. Because I was wrong about the pricing, I was also wrong about the quality / cost curve. There is room in the market for multiple models at different price points, because consumers can in theory choose different providers for different tasks. The rise of model routing as a service and the increasing classification of tasks and roles into different AI usage categories — sales gets a cheap open source model, engineering gets fable — is downstream of that demand.

But I was really right about the most important bit, which is that consumers have no stickiness at all.

Every three weeks we see a mass exodus from Anthropic to OpenAI to Anthropic and back to OpenAI. It is just way too easy to switch providers. “Model wrapper”, once seen as derogatory, is now touted as a core feature. Businesses and consumers are recognizing both the churn and the need to stay on top, and are explicitly investing in tools that let them adapt. This is also a huge part of why the background agent infrastructure that we build and sell at Nori has zero lock-in at the compute, model, and harness levels — there is no reason to be locked into a single ecosystem when you don’t have to be (if you’re looking for background agents / cloud agents for your team, shoot me a message!)

The moment the Chinese models hit the actual frontier, there will be a mass exodus to using those models. The economic incentives are way too strong, and arguably that is already happening for people who are in the know.

III.

This is not how any of this was supposed to go. The whole point of all this closed model stuff is the ability to dig a trench around the model and put a toll booth in the middle, and the whole point of all the debt is to basically have a call option on that toll booth. We all knew that the big labs would be fighting tooth and nail with each other over which of their closed models would end up winning the day. Doing this open source stuff feels against the rules.

Partially, that’s because it is against the rules. Many of the open source models are trained on their more capable closed source counterparts.

Unfortunately, usage of the models is a source of valuable training data for competitors. Every improvement to any model can quickly be copied by other labs even without seeing any of the internals because you can just sample the new model a bajillion times and use the outputs as training data. This technique is called ‘distillation’ because you are ‘distilling’ the essence of some larger ‘teacher’ model into a smaller ‘student’.

OpenAI and Anthropic can try to ban people from training on the outputs of their models, they can kick and whine and sue to try and enforce it, but at the end of the day the labs need people to use their models. That’s the foundation of all of the token economics! If no one actually uses your models, what’s the point of all that training?

So the token traces have to get out in the wild, and any kind of adversarial cat and mouse is going to end up wasting a lot of resources for very little gain. The open source models will basically always be able to catch up, modulo some amount of compute.

This isn’t a new problem, people have been talking about it for some time. Dwarkesh even asked folks to tackle this problem in his essay competition last month:

So when does the profit start? Maybe at some point scaling will plateau, but if progress at the frontier has slowed down, then the combination of distillation and low switching costs (cloud margins result from high switching costs) makes it really easy for open source to catch up to the labs, eating into their margins. So how do the labs actually start making money?

(emphasis mine)

My answer was that the labs in totality will not be able to meet the demand for their models and will be capped by compute, which in turn will prevent new-comers from actually competing for best in class model training and will let some of the labs rent seek. The winning answer argued that the models will become commodity and the model providers should try and own downstream services instead. Note that both of these answers assume that distillation is so inevitable, it’s hardly worth discussing.

This has already had some amount of impact on OpenAI and Anthropic’s pricing models. Anthropic, for example, continually extends the amount of time that Fable will remain on its subscription plans. This is not entirely attributable to the open source models, but I am certain that these models will add additional pressure to simply keep Fable on the subscriptions forever — even if doing so results in a loss for Anthropic, since the subscriptions are heavily subsidized.

IV.

Other, smarter, more plugged-in people can speculate about what this means for future AI policy. I’m sure much will be made of CCP leader Xi Jinping’s speech on AI regulation and the importance of collaboration across nations, and Demis Hassibis (CEO of Deepmind’s) call for domestic regulatory apparatus. I will leave the policy wonks to it.

But I have strong opinions about the obvious and overwhelmingly negative impact of current and past AI policy. Bluntly, I’m pretty ticked off.

First, quoting liberally from Stephen Bochinski:

Step back and the bigger story is what an unmitigated failure US AI policy has been. The administration held Fable back, and what finally shipped is a hindered version that refuses whole categories of work. Meanwhile a frontier quality model with none of those restrictions is a download away, released by a Chinese lab the US government has no ability to regulate. Whatever the theory behind gating American models was, it plainly wasn’t thought through, because the only people the gates constrain are American customers. Semgrep found GLM 5.2 beating Claude on their cyber benchmarks for exactly this reason. The restricted model declines the work and the open one just does it.

I’ll add more.

The overwhelming corruption of this government has direct impacts on our ability to produce great technology. Friends of mine in government and defense — which AI is increasingly a part of, whether we want it to be or not — have told me that projects (including datacenter projects) simply do not happen unless the Trump family somehow stands to benefit. They will get slow rolled or stall under regulatory scrutiny, and then suddenly clear up when one of the Trump kids happens to join the company board or take an investor position. Companies that do not play ball end up feeling the full force of the state, c.f. the very public attempt at corporate murder against Anthropic.

The reason businesses file in Delaware is not because Delaware is particularly business friendly. It is because Delaware is predictable and stable. To a first approximation, you know what the rules are, and if you don’t follow them that’s mostly on you. Putting ethics aside, the biggest practical issue with corruption and cronyism and irrational application of state power is that it chills investment. As I said when we discussed the whole Fable situation:

The debt, the build out, the datacenters, the stock market runs on every part of the AI chain from GPUs to memory to disk to server racks. Alllllll of that is predicated on the idea that all of this is going to be worth trillions and trillions of dollars. And by all accounts, it seems like it is. Or at least, was on track to be. You know what will put a spanner in the build out of a multi-trillion dollar data center investment? The realization that at any point, the government will unilaterally cut off access to everyone, and the datacenters will be worth squat.

The US is no longer a stable business environment. It doesn’t matter if you follow the rules, one Friday evening (it’s always a Friday evening) you may wake up to Pete Hegseth demanding your destruction. Ironically, this administration behaves more like the CCP, where the government picks winners and losers and can simply take out a company for saying or doing the wrong thing.

This administration’s immigration policy and university funding policy have also been unmitigated failures, and in my mind directly contribute to the situation we are in today. Our research institutions, which depend on foreign brilliance and government funding, are in crisis and have been since Trump returned to office. Every day, brilliant talent leaves our country for foreign shores, or never arrives to begin with. The imbeciles with the blue check marks on Twitter will tell you that this is a resounding achievement; unfortunately not a single one of them knows what a GPU even is.

We are still hacking away at our research pipeline, by the way, like a deranged ferret trying to gnaw off its own arm. PhDs and labs across the country are still being denied funding, even as other countries go out of their way to fill in the gap. It’s been a few news cycles so we’ve forgotten the assaults on college campuses. But the folks who have had their entire lives overturned certainly haven’t forgotten.

A certain brand of Bay area resident likes to talk about moats and structural advantage. To use those terms, the United States has a massive structural advantage by being an immigrant-friendly, non-ethno-nationalist country. There is no other country in the world that is as diverse as ours. There is no other country in the world that could restructure to be as diverse as ours. That means that the US and the US alone can tap talent from all over. As the old Reagan line goes, “You can go to live in France, but you cannot become a Frenchman. You can go to live in Germany or Turkey or Japan, but you cannot become a German, a Turk, or a Japanese. But anyone, from any corner of the Earth, can come to live in America and become an American.” It is our defining moat. It is why we punch way above our weight despite being 3x smaller by population vs equivalents in Asia.

This administration is simply throwing that away.

When we talk of accountability, I personally want to know how many folks who ended up at Moonshot were educated in the States — the founder had a degree from CMU! — and why we weren’t able to convince them to stay stateside or pull them from China in the first place. People will chalk up the successes of the Chinese labs to mystery, but in my opinion, there is nothing mysterious about it. Choosing to ‘win’ is rarely a straightforward policy choice, but you have to at least try. It seems pretty obvious that the people in this administration are simply not interested in doing so.

V.

I don’t mean to take away from Moonshot’s accomplishments. Politics aside, Kimi K3 is an incredible technical achievement on its own terms. The hardware limitations have forced the Kimi researchers to pull some pretty neat tricks, all of which are now open source and can be adopted across the ecosystem. Kimi also forces both OpenAI and Anthropic to keep their prices down, which I (as a startup founder) am a huge fan of.

The model weights themselves have not yet been released — they are slated to drop on the 27th. When they do, I will be very eager to pull Kimi into our background agent environments and take it for a spin.

The US is no longer the only nation capable of frontier intelligence, no longer in a category of its own. The world is a smaller place now.

Other Things

• GPT 5.6 Sol has done an incredible job thus far solving open math problems and verifying them with Lean. Here’s an example of GPT 5.6 solving an open problem in Convex Optimization. Here’s a swarm of GPT 5.6 instances solving ~20 open Erdos problems. Here’s GPT 5.6 proving the cycle double conjecture in graph theory. These discoveries all happened in the span of like two weeks, in some cases by amateur mathematicians.

  
  I’ve often said that all of mathematics is really a search problem. You have a bunch of axioms and a proof is a specific combination of those axioms, and you search over them to discover novel proofs. Any search problem can be naively brute forced, but its often just too expensive to really do. Brilliant mathematicians are really good at narrowing down search space, that’s what makes them brilliant.
  
  But another way to make search problems easier is to just bring down the computational cost of exploring a particular part of the search tree. Which AI is really really good at.

  
  Quoting from the first link:

  Lastly, some important comments about the work relating to AI capabilities: In a lot of cases, proving lower bounds like this result relies on finding that right construction that works (in this case, family of difficult functions and a strategy for how an “adversarial” oracle should answer queries from an algorithm to reveal minimal information) and then proving things about it. There are only so many function classes which would be reasonable to look at (here, quadratics for example would have also been reasonable with order d² degrees of freedom, or any variation of maxes of some simpler families of convex functions as well), but the actual proof mechanics once the “correct” function class and correct strategy for adversarial oracle answers is found are often not so complicated, and often employ existing results from convex geometry or similar (this is also the structure of two previous but much more niche, less important results of mine). So I wouldn’t really say that this result is using or creating some fundamentally new techniques in convex geometry or optimization theory. What this means from my perspective is that if a result is attainable with existing techniques, modern AI methods will be able to solve those problems. I don’t think researchers in math/TCS will be made obsolete, but I think it will instead no longer make sense to work on any low-hanging, or even medium-hanging (you know what I mean) fruit. We’ll be needed for problems where actual novel approaches are needed.

  For now.
  

• Thinking Machines releases their own open weights model, which may be the best one from the US.

  Today we are advancing our mission by releasing a model we trained from scratch with the full weights available, so that people can make it their own.

  Our model, called Inkling, is a Mixture-of-Experts transformer with 975B total parameters, 41B active. It supports a context window of up to 1M tokens. It was pretrained on 45 trillion tokens of text, images, audio and video. It is the first in a family of models of different sizes: alongside it we are sharing a preview of Inkling-Small, a lighter-weight model with 12B active parameters, trained with a similar recipe, that achieves strong performance with even lower cost and latency.

  It’s an interesting bet from one of the most put-upon startups in the country. As a reminder, Thinking Machines (helmed by Mira Murati, ex-CTO of OpenAI) raised billions of dollars without having any product or model or anything, and then proceeded to be totally shrouded in mystery for two years. They eventually released a model fine tuning platform, which was widely considered to be insufficient to justify the investment. Among the ‘neolabs’, Thinking Machines is making the most money (I think $200m last I checked). The rest of the labs are doing even less than that, which I think has really soured investors on the whole concept of a neo lab in the first place.

  
  With Inkling, I think the idea here is that you would take Inkling and fine tune it using Tinker. So that way thinking machines ends up being core infrastructure for teams that want to own their stack. But…do most teams really care so much that they want to own that part of the stack? Maybe at some point it ends up being cheaper, but open source models already force providers to bring down costs on margin.
  

• There were a raft of security issues that have come up over the last few weeks. Some of my favorites:

  1. The Memory Heist. Basic idea: create a poisoned url, have Claude search for that URL, the url masks as a security page where Claude will type out, letter by letter, url by url, sensitive information into the web fetch tool. Claude will think it's doing the right thing and won't ask permission, and it will even pull from user memory to do so.

  2. GitLost. Basic idea: git agents have more access than users, so you can literally just post an issue in a public repo and ask for private repo info.

  3. Instagram Leak. Basic idea: agents in the meta user account support flow just guess at whether an account is actually yours, so you can report a high profile account as your account and by simply changing your physical location the agent may just give you the high profile account.

  At some point more people are going to wake up to the need of centralized background agents for security reasons. In the meantime, we’ll get more of these. There’s an old saying, people only start caring about security when it bites them in particular.
  

• NY bans data centers.

  The expansion of data centers in the United States is driving up power demand — and electricity bills — in large swaths of the country, drawing local and political backlash.

  Only one in three Americans approve of the ​fast pace of data-center construction and most ​would oppose building one in their ⁠own community, according to a recent Reuters/Ipsos poll.

  Dozens of state legislatures have introduced bills to rein in the effects of data centers on power bills and the environment. New York is the first to enact a full moratorium.

  I think the water concerns are mostly overblown, but the power thing is real. The hyperscalers should offer to subsidize more than what they use to open the doors in local communities. More generally, I think people sorta miss why data centers are controversial. The conversation has mostly been about AI, but even though AI is quite unpopular, datacenters are even more unpopular than AI. I think it’s less about the AI and more about the local politics. The problem with data center build out in local communities is that the local communities do not benefit at all. I think it is totally reasonable for a person in, like, Memphis, to ask whether a data center build out in Memphis is going to benefit even a single local Memphis resident. So far, the answer has been no! The data centers are being built at the behest of Silicon Valley billionaires who are, bluntly, some of the most hated people on the planet, and who have basically openly thumbed their nose at any kind of legal process or restraint. I expect more of these data center moratoriums, especially as a way to win some additional votes for the midterms.
  

• Related: Meta and SpaceX both begin selling excess compute to competitors in the AI race. Meta is literally building data centers in tents to build them out faster.

  

  The obvious take is that this is a bearish signal, and the neo clouds (eg core weave) all had market sell offs. Semi Analysis disagrees though:

  With Bloomberg headlines suggesting Meta could become a Neocloud, the market’s reaction was immediate: aggressive sell-off of Neoclouds like Coreweave & Nebius, and debates of “overcapacity” coming back. Let’s set the record straight – we believe that both takes are erroneous and that Meta’s datacenter & compute procurement will accelerate, not slow down. Capex in 2027 will be shockingly high. In just the first six months of the year, Meta has contracted over 5GW of capacity across Cloud & Colo, and that doesn’t even include all their accelerating self-build activity. Everything is computer and everything is a neocloud.

• Apple sues OpenAI, accusing them of systematically exfiltrating hardware secrets.

  at every level, from members of its Technical Staff to its Chief Hardware Officer, and in coordination with business partners, OpenAI has been stealing Apple’s trade secrets and confidential information. As a natural result, OpenAI’s nascent hardware business now rests.

  …

  The complaint, filed in the U.S. District Court for the Northern District of California, alleges that Tan used insider knowledge of Apple’s confidential projects to grill job candidates in interviews and learn more confidential information. Additionally, Tan directed job candidates still working at Apple to bring actual Apple hardware components and samples for “show and tell” sessions.

  When interviewing Apple employees for jobs at OpenAI, Mr. Tan uses Apple’s confidential information to gain access to even more insider knowledge. He has used an Apple internal project codename to ask, “What’s the plan[?]” for an unannounced Apple product.

  He has directed job candidates still working for Apple to bring “Actual parts” from Apple to their interviews for “show and tell” sessions in which he and his team at OpenAI can elicit still more Apple confidential information. These directions to bring Apple’s parts to OpenAI job interviews surprised at least one of the candidates, who commented that he “didn’t even know we could take those from the office.”

  OpenAI has been instructing Apple employees to bring “CAD/design artifacts” and “prototypes” to their interviews and to divulge details about their work such as “subsystem and component selection,” the “tools or methodologies you use for system integration, such as CAD software, simulation tools,” and “Vendor selection and communication/collaboration with vendors.”

  If any of this is true, this is a pretty terrible look for OpenAI and may just fully shut down their nascent hardware efforts. The same kind of lawsuit between Google and Uber fully killed Uber’s self driving car efforts.
  

• Amazon is shutting down Mechanical Turk, because everyone on the platform is just using LLMs to do the data labeling.

  An announcement on the Mechanical Turk website says that on July 30, 2026, the crowdsourcing service will close to new customers. Amazon Web Services says the decision was made after “careful consideration,” adding, “Existing customers can continue to use the service as normal. AWS continues to invest in security and availability improvements for Mechanical Turk, but we do not plan to introduce new features.”

  …

  Over time, the relationship between Mechanical Turk and AI models grew even more complicated. In a snake-eating-its-own-tail irony, a 2023 analysis found that between 33% and 46% of workers on the platform were using large language models to complete their tasks, raising questions about the reliability of data annotated on the platform and also about whether humans needed to be in the loop at all.

  It’s hard to overestimate how important Mechanical Turk was to the rise of LLMs. Much like Stack Overflow, it was eaten by its own importance.
  

• General Intuition got a neural net to fully simulate multiplayer rocket league! Its super cool, you can literally play it on the web.

  

It’s really really important advice. The temptation to build your own “vision” is really strong. There are hundreds of founders who will insist that they are hunting for market signal while doing the exact opposite. First time founders focus on product, second time founders focus on distribution, etc. etc. So I get why it’s emphasized so much, is my point.

Nuance is hard to communicate, though, and I think sometimes founders — especially second time founders — will decide that the natural conclusion is to hunt for PMF wherever you can find it, in any industry, serving any group of people. And this is also wrong, because you won’t have founder market fit.

Everyone knows what product market fit is, what the hell is founder market fit?

Most people are not suited to be founders. Those that are, are not equally suited to sell the same things. Selling into the consumer creative world is way different than selling into the enterprise tech world (ask me how I know). A jazz pianist with a history of psychedelics and a past as a Buddhist monk is going to ace the former. A buttoned up ivy grad with a research pedigree will do better on the latter. Try and switch them. Can you imagine this guy in a room with a bunch of pressed white shirt consultants trying to identify quarterly deliverables?

Forget about sales, these people just wouldn’t have anything to talk about. And if you can’t talk to your customers, you can’t build product for them.

Founder market fit is about culture. It’s that unquantifiable similarity function, that vibe fit, that accent and language and way you dress and the million other little details that gives you tribal identity, the things necessary to get a potential customer to tell you their problems and root for you to succeed and give you their money (in that order). It’s also the million little details that the founder looks for in their clients, the things that makes the founder want to dedicate years of their life to building for that group of people.

Is this unfair? Or maybe anti egalitarian? Does it rely too much on stereotypes? I’m not sure. On the one hand I like the idea that anyone could just learn to sell to anyone else. On the other hand, that word “learn” is doing a lot of heavy lifting. It just seems practically obvious that, like, a native Spanish speaker will have a harder time selling into a Chinese market than a native Chinese speaker. It’s not impossible, it’s just that the former founder would have to do way more work, and startups are already a ton of work!

Product market fit is a boolean, or maybe measurable on a single axis. You either have it or you don’t. Founder market fit is weirdly high dimensional, because it’s as much about temperament as it is about anything else.

Here’s an example. Let’s say you’re an introvert, someone who’s not great at the whole ‘going to a new place, putting on a character, and selling to strangers who don’t know you’ thing. This isn’t prohibitive as a founder, but it’s going to make it harder to sell to certain large enterprises who are used to extremely charismatic salesmen spending on steak dinners and box seats at the Yankees and exchanging business cards at conferences. So you may want to gun for a market where the buyer is more likely be someone you never have to meet in person — consumer markets, or dev tools that spread through word of mouth. If you insist on b2b, then maybe a better target wedge are remote teams that don’t necessarily expect a lot of in person face time. Or find the kind of people that you might like to hang out with. For me, that’s people who like video and boardgames. Can you find a way to turn ‘going to gencon’ or ‘going to pax west’ into an opportunity to meet your target customer? (Unfortunately I could not)

In case it helps, here’s a list of axes that have been on my mind recently.

• Introversion / extroversion. We already talked a bit about this above. If you’re an introvert, it will bluntly just be harder for you. But also you probably want to lean towards distribution channels that do not require you to be on stage or networking at conferences, which in turn will likely drive more towards consumer or PLG (product led growth, as popularized by figma and linear).

• Your particular identifying cultural niche.1 Don’t lie, I know you have one. You’re a hipster tumblrite or a too-online reddit kid or a 4chan edgelord. Or if you spent time offline, you’re a jock or a drama kid or a computer geek. You identify with someone in the Breakfast Club more than the rest. That almost certainly impacts who you naturally gravitate towards, and who you will want to serve (or, alternatively, who you wouldn’t want to spend any time with).

• Related: are you technical? A lot of technical people have a really hard time empathizing with people who are non-technical. Learning to code literally warps your brain. But without that empathy, you will not be able to ‘speak the language’ of an ops person or sales person. It is way too easy to take for granted basic things like ‘what is a command line’ or even ‘what is markdown’. That leaks into your product experience too. You may literally be blind to the ways in which your product assumes technical competence.

• Where you live. If you live in NYC it’s just easier to start a fintech company. If you live in Texas, it’s easier to start an energy company. Etc. etc.

• What you spend your free time doing. In case you haven’t noticed, I think about AI agents a lot. That makes it really easy for me to write a lot about AI agents, which in turn is a great distribution channel for when I try to sell my AI agent technology to people (buy Nori!). Do you spend your free time thinking about the problem space? If you do, it’s easier to wake up 3 years down the line to continue grinding.

• Your team size. Some folks really like working on small, lean, talent dense teams. That basically takes them out of the running for industries that require or expect more of ‘forward deployed’ handholding / services model.

• Related: how much you prefer raising. If you’re someone who’s a maximalist, who wants to raise infinity dollars, that generally goes hand in hand with fielding a massive team to operationalize everything. The amount of money you raise also impacts how easy it is to sell to specific people, because they look at that dollar amount as part of the buy/no-buy signal. If you want to sell to CISOs (for example), it helps to have raised a ton of money from blue-chip VCs because it gives your company legitimacy, which is incredibly important to an executive role that is all about evaluating legitimacy.

• What you like to wear. This is a little stupid, but I kinda hate wearing a suit. The whole white buttondown, belt, crisp pants. Just not really for me. There are also, silly as it may sound, some industries where that’s just an expectation. I don’t think I’ve ever seen out of a suit, but he pulls it off every time, and I suspect it works really well for his industry.

The overall space of founder market fit is just mindbogglingly vast, because everything is a signal to your customers or a signal to you. There are lots of reasons people tell founders to ‘build for yourself,’ but perhaps the most important one is that you hopefully have a pretty good understanding of how to find and appeal to other people who are like you. And hopefully you like yourself enough to want to sell into that market.

It’s worth noting that not everything about founder market fit will end up mattering all that much. All of these considerations are things that will make differences on the margins; and many things are also just inherently contradictory, so something will have to give. To pick an example out of a hat, it is definitely harder for me to build AI dev tools living in NYC. The ‘founder market optimal’ choice may be to either build fintech (change the market) or move to the Bay (change the founder). But we’re doing fine at Nori selling our background agent infrastructure, even if it means racking up more airline miles than we might have otherwise.

Founder market fit is about tradeoffs, and in particular, which tradeoffs are impossible for you as a founder to overcome. Note that this isn’t always easy to really think through. Knowing what you can deal with and what will grind your gears until you go mad requires a lot of introspection.

In this, finding founder market fit is not so different from, like, finding a good relationship partner. Being a founder is a long-term commitment. Minimum three years, but more likely 7 to 10 years. Lots of things that seem fine right now really will grind after that long. You need to think through your deal breakers and make sure that the place where you find market signal isn’t in a space you’re going to end up hating three years down the line. Like Paul Graham famously said:

If you can just avoid dying, you get rich.

Which, of course, is why founder market fit ends up being so important. If you don’t have founder market fit, your startup will die, because you will lose interest or get frustrated or just not want to do it anymore. That may be true even if you are seeing some success. I know too many startups that hit the 7 figure ARR mark and just fell apart because one or both founders couldn’t do it anymore.

And on the flip side, if you do find the overlap of founder market fit and product market fit, you’re on the golden path for life.

In February of 2021 I made the crazy and totally unadvisable decision to leave the comfortable world of Google and start a company. Best decision I ever made. Along the way I started I started writing a journal of sorts that I've taken to calling the Founder's Guide -- basically a place I can ramble about startups and how they work and how to think about them. You can find more in the series here.

HN discussion here.

You may also like:

Do you ever look out at the beautiful Manhattan skyline, bagel in one hand, pizza slice in the other, and wonder “What are those guys who are doing the AI thing in the Bay up to these days?” They’re always saying all sorts of weird and crazy things on the bird app that no one else but the AI people use anymore.1 Probably worth checking in.

Now, cards on the table, I generally think of myself as one of those people who is doing the AI thing, on account of being the founder of an AI company and having been in ml research for years before this2 and also writing copiously on the subject of AI things. But I mostly avoid the bird app, and I don’t live in SF, so sometimes I do get some FOMO.

I’d heard about the AI Engineering conference a few times — a few friends at Google were previous sponsors and told me it was a good place to catch up with other folks who were thinking about coding agents on the cutting edge. Applied to give a talk, got in on the online track, and the organizers very kindly sent over a few free tickets by mistake.

Just got back from the Bay, FOMO eradicated. Here are my thoughts.

At a high level, the talks were split 85-15. 85% of the people there were enterprise folks who were trying to learn how to AI good. These people were interested in attending and giving talks about how to leverage more AI in their respective organizations. The other 15% of the people there were really on the bleeding edge, people who had plumbed the depths of lights off software factories and came back to tell of Lovecraftian horrors lying beneath (hi Dex!). These guys mostly were looking for less AI. I was expecting way more of camp two than camp one, and was somewhat surprised at the skew in the other direction. Even in this highly selected environment — the ~7000 people most likely to attend an “ai engineering” conference — the vast majority of people don’t really know how to use AI effectively.

Every month the AI world is onto something new. January was takeoff, February was ‘what is a skill’, March was ‘everyone has to use Claude Code all the time’, April was get the ops team on board, May was tokenmaxxing.

Apparently, June was the month of skill sprawl. There were a dozen talks that were variants of “how do we manage the 1000 skills that people have created in our organization?” These were inevitably packed to the brim, which should give a sense of where most of the market is. Over at Nori we solved this problem back in December (see: noriskillsets.com, a team wide package manager built on our open source local skills manager client).

Many of these ‘skill management’ talks landed flat for me. I think having that many skills is a ‘code smell’. In my experience the agents are very good at figuring things out from first principles. Having a skill that 1) will drift, 2) may not actually add anything new, 3) needs to be managed is more likely to create problems rather than solve them. Bluntly, I think many of the teams who are grappling with hundreds of skill files in their organizations are dealing with an employee base that is happily having the coding agents generate skills for all sorts of things without rigorously figuring out whether they are useful. This gives the veneer of “AI enabled” work without actually being so. I wrote about some of this here:

I was also disappointed at how many talks were about some form of self-learning that involved having agents read through transcripts and automatically create skills, so much so that I wrote about it. Anything that does not have a human in the loop for context management is going to quickly spiral out of control, which is why we end up with so many skills in the first place.

In my opinion, you cannot really rely on the agents to write good skills for you. The process needs to be intentional. There’s a particular format that we find works. We like to bundle sets of skills into specific groups instead of just having a big pile of them with a search server on top. Every prompt with an agent is like summoning an entity from the void. The particular skills that are bundled with the agent are a big part of eking out the highest quality results. Trying to slopify the agent brain will result in agents that produce slop. Don’t do it.

This was also the month of evals. Evals are something of an interesting beast; as with everything in AI, the term is wildly overloaded.

Some people (mostly researchers and model providers) use “evals” to mean “benchmarks that we can use to evaluate raw model performance, that the industry can hill-climb against to make models better overall.” SWE-Bench is an eval, and Humanity’s Last Exam is an eval, and ARC-AGI is an eval. These evals are artificial, designed to give researchers a metric to improve model training through simulated, verifiable tasks, and designed to give the industry a quick read on the relative capabilities of these models.

Other people (mostly folks in industry who are deploying these models) use “evals” to mean “regression tests that we can use to try and make production workflows as deterministic as possible.” Here, an eval is extremely task specific — things like “does the AI correctly label data” or “will it correctly respond to adversarial chat input in the customer service chat window?”

Even though the former use of the term is more relevant for folks on the cutting edge, I think the latter is what a lot of the larger enterprises are thinking about, and as a result were what much of the audience was thinking about.

It’s worth taking a step back and thinking about how AI is landing in different orgs. At tech companies and startups, folks primarily interact with AI as a developer tool. Cursor and, later, Claude Code are interfaces that a dev spends all day in. The average developer at these companies is, as a result fairly intimately aware of what AI can do, simply because they interact with it so much.

That is not how AI has landed in other industries like banks and retail. In organizations where engineering is not obviously a critical function — places where engineering is more a cost center than where the money gets made — the primary way AI gets used is as infrastructure. The most common shape is the ETL pipeline, a recurring process that Extracts data from some table, Transforms it somehow, and then Loads it into another table. These sorts of tasks are a dime a dozen at any large enterprise, and less tech-forward companies will pay a lot of money to external consultants to build these out. Traditionally, ETL pipelines are brittle and required a lot of fiddling with APIs and different fields and so on. LLMs, with their inherent flexibility, promised to massively simplify that effort.

New technology never gets adopted wholesale, especially among larger slow moving companies with a lot to lose. Instead, new technology comes in as a supplement or a replacement for something that already exists. That’s a large part of why AI landed in ETL world for so many companies — because that was the best fitting preexisting shape to slot it into. Unfortunately, if you’ve ever worked with ETLs, you’ll know that you can’t just, like, screw around with the data. These pipelines need to be consistent and auditable, and an AI agent is neither.

Which brings us back to evals.

In large tech companies, tokenmaxxing meant ‘give individual developers unlimited budgets for things like Claude Code’. Everywhere else, tokenmaxxing meant ‘let’s throw more tokens at these failing ETL pipelines to get them from 80% correct to 100% correct’. How do you do that with AI? Well, one naive way is to build another AI system that checks the output of the first. But what if that one fails? Well, you could build a third system that checks the output of the second.

You see the problem.

Now that tokenmaxxing is dead, some of these less tech enabled firms are limiting employees to ~$50-$100 token spend per month. At full unsubsidized price, that’s barely anything at all, but then again most of these teams haven’t exactly been using Claude Code or Codex anyway.

I mention all of this mostly because I find it interesting to see the gap between SF and everyone else. I’ve said in the past that SF is in a bubble, and that was very apparent during AIE. The evals track talks were all about researchers and labs trying to figure out better RL environments, but a lot of the people in the audience were like “that’s great but tell me how to make my ETLs not be bad.”

As an aside, I think there isn’t really a good way to make these things consistent / auditable, nor should you really try. The long tail is too long. Instead, have a human in the loop. AI that goes 95% of the way and has a human approve or not has, in my experience, been way more successful than any AI system that attempts to go 100% of the way.

The folks who are really on the cutting edge spent a lot of time talking about background agents / cloud agents, and the ten-million reasons why people should use them.

Now, I recognize that as the founder/ceo of a company that is selling a background agent stack, I am a bit biased here, but it seriously felt like a full half of the talks at AIE were shilling for our product. There were talks about why you need background agents for productivity, and talks about why you need background agents for security, and talks about why you need background agents for compliance. There were even talks about why you need background agents for AI enablement! I’d say a solid half of the security track, all of the sandbox track, and many many other talks (including keynotes) were about moving things from local machines to the cloud.

(This post is brought to you by Nori! When you think “background agent” or “cloud agent,” think Nori!)

If you are unfamiliar, the name basically tells you whatever you need to know: background agents / cloud agents are agents like Claude Code that run in the cloud, in the background.

Why might you want this? Well, when Claude Code is running locally on your computer, it is:

• Stuck on your machine. If you close your laptop or walk away from it you cannot (easily) take the conversation with you on the go.

• A massive security hole. Everything on your computer is fair game. There is a fundamental tension between usefulness and risk, and unfortunately besides a few very very stressed out CISOs people are not thinking about this trade off at all.

• Unable to easily coordinate and collaborate. If you want to try and run multiple agents in the same code repository, you have to use tooling like git worktrees. If you run too many agents at once, they will happily eat all of your computer memory and kill the computer.

All of these problems are because the coding agents live on your computer, so…

Cloud agents neatly solve all three of these issues and a whole host of other things. The openclaw folks already showed how being able to access agents from anywhere was massively useful. Because they are centralized, you get massive leverage on security, observability, governance, and all those other lovely enterprise buzzwords. And cloud agents are a necessary step towards the ‘fully automated business’, because, by virtue of being not-on-your-computer, they can respond to events and timers and do things automatically.

It turns out this is an extremely powerful primitive to have. It’s a universal automator — you can literally fire off any event to this thing and have Claude respond, and Claude is a PhD level genius. We use background agents for everything from in-slack coding to company brain management to automatically responding to bug reports to onboarding new users.

These are still relatively new, but in ~6 months I think the market for these cloud runtimes is going to be really really hot.

The other big conversation topic among the teams on the cutting edge was about loops, software factories, and lights-off codebases.3 Or, well, that was ostensibly what people were talking about, but what they were actually talking about was something closer to “do we still need humans to do this stuff or are we all screwed?”

In some sense, a “loop” is a hack. LLMs suffer from context rot — the longer you talk to an LLM, the dumber it gets, until you hit a hard cap where you just can’t do anything with it anymore. It is a massive limitation, and a ton of “cutting edge” AI infrastructure is about solving this problem.

The loop is a very simple idea: what if you gave an LLM a big task and a scratchpad, and instead of having it try and finish the task in one shot, it takes on a small piece of the task and only does that small piece, and then writes out what it did. Then the thing loops to do it again. And again and again until it’s done.

while :; do cat PROMPT.md | claude-code ; done

People have been talking about loops for a while. The “Ralph Wiggum loop” has been in the water supply since last year. Karpathy’s auto research work is slightly newer, AlphaEvolve is slightly older, but they’re all basically about the same thing, which is “can you exploit the fact that LLMs never tire to turn every problem into a brute force search problem?”

I’ve written about this before, back in October:

I started using agents effectively when I began to see code as a search problem. Agents are like our chess players above. At each turn in a conversation or agentic loop, they make a decision. That decision takes them to a new state in the search tree, and the process repeats. Agents can go any which way down the search tree. Sometimes they will start on the right path and then veer off track. Other times, they will be wrong from the start. If you are using a coding agent, your job is to help it narrow down the search tree.

…

As a software engineer using a coding agent, your goal is not to get the best single shot code output. Your goal is to simply get the best code, period. If coding is a search problem, a very easy way to improve your code is by running many searches at once. So do that!

This raises some interesting questions. Is…software engineering a brute force search problem? What about, say, running a company? Can you be like “fable make me a billion dollars” in a loop and just sit back drinking mimosas in Fiji until it succeeds?

Several startups have staked their business on the answer being yes. And Matt Levine has joked that this is obviously the right business model for the big labs:

It really is the greatest business plan in the history of capitalism: “We will create God and then ask it for money.” Perfect in its simplicity.

But back at AIE, the premier AI conference where everyone was basically obligated to be AI-pilled, the general consensus was “eh, not yet.”

The agents are good, and Fable is really good, but a 3 month old codebase is just a totally different beast than a 1 year old codebase, and both of those feel really really different when people are actually out there using your tooling and you have random backwards compatibility requirements that you have to remember (or you go to jail, or at least lose a big contract), o, and yea, memory is something that the agents are uniquely really terrible at. Most of the folks who were defending ‘loopsmaxxing’ were doing so somewhat aspirationally, acknowledging that the loop-based-tooling wasn’t all there right now but would get there eventually.

For what it’s worth, at Nori we are running a “hybrid” software factory codebase. The entire codebase is agent written. I am definitely not reading most of the PRs that are being shipped. But also, I have a pretty good mental model of how the code is structured. We have regular daily tasks that automatically look for tests to clean up or abstractions to define (or destroy), like little cleaner fish that are just swimming around looking for things to fix up. And roughly once a month, we all end up doing a big chunky refactor of some part of the codebase that has gotten too hairy, often because we’ll eyeball a PR and go ‘wait why is <small feature> so many LoC changed?’ and then dig in with an agent to figure out if there is an abstraction missing somewhere.

Isn’t that kinda just how normal swe works, but on steroids? You build stuff, and you build more stuff, and eventually you look at the stuff you built and refactor a few things to make it easier to build more stuff on top?

Some people argue that when using agents, no one really has any deep model of the codebase, the kind where you would know a priori what files and functions need to be changed to make a new feature work without regressions. Which is true.

But I think that is true of basically every codebase, modulo time and size. How much of Google’s massive codebase did anyone really have in their head? Jeff Dean is superhuman and even he probably only knows like 0.01% of that thing. Which is to say, it’s worth investing in things that make the codebase legible — it will probably make the models better too. And beyond that, any maximalist position of ‘only use agents never look at code’ to ‘never use agents only look at code’ is wrong.

I think the loops will get better. A good friend of mine at one of the companies that had early access to GPT 5.6 Sol was telling me about these crazy subagent harnesses, where Sol was coordinating a bunch of Opus agents which were coordinating a bunch of GLM agents. Seems really impressive. But it’s also worth mentioning that he was overall unimpressed by the models’ collective ability to do things without a human.

This all sorta brings me back to the evals thread from earlier. I think the right architecture is ‘automated artifact, human review’. Our definition of what an ‘artifact’ is will continue changing as the models get better, but you need that human in the loop somewhere, either for the initial prompt or the post hoc analysis.

A few other miscellaneous thoughts.

• As usual, there was a lot of chatter about agents running other agents and agent swarms and things like this. Reading the tea leaves a bit, the most chaotic versions of this approach (“All the patrol agents Refinery, Witness, Deacon, Polecats — create wisp molecules for every single patrol or workflow run“) have somewhat failed to pass muster. I think the agent swarm thing was always somewhat linked to the loopsmaxxing tokenmaxxing software factory thing. The biggest problem with software factories is that agents are not very good at high level decision making, things like “what should I prioritize” and “what are the right trade-offs to make” and “what is the right way to break up a complex task into smaller interlocking useful pieces.” The agent swarm promised to solve these problems. I think the theory was something like “agents can handle low level decision making, so maybe all they need to handle high level decision making is a bunch of grunts to do the low level stuff and a metric ton of tokens.” Unfortunately, this framing is obviously flipping causality. I think a lot of the agent swarm hype has become more about orchestrating targeted subagents for things like research or code review, something I am / was already much more bullish about.

• A lot of people waking up to the security risks of having these little imps running around on every computer in a company. Half the talks were about putting the fear of God into various CISOs.

  

  

• “Multiplayer” was a surprisingly big topic of conversation, driven in part by Anthropic pushing their new Claude Tag slack bot in ~all of their talks. The basic idea is that when you have a lot of people collaborating with an agent at the same time, eg through slack, the experience of writing software — or even running a company — becomes fundamentally more collaborative. As someone who is selling a no lock-in version of Claude Tag, yea, I basically agree. Having a really good integrated AI setup is transformative, in part because it meaningfully changes everyone’s relationship to engineering. These tools actually remind me a lot of Figma. When Figma came out, it was obviously a great wireframing tool, but the real magic of the system was how it allowed everyone to participate in the design process because it was multiplayer by default. Even though the form factor is totally different, tools like Nori or Claude Tag have the same effect on engineering as Figma did on design. Suddenly, everyone can participate in the engineering process, no arcane runes required. This is, again, another pitch for background agents generally.

• One thing I disagreed with most strongly was the idea that everyone in an org should figure out how to use AI ‘properly’. I get the impulse. As AI adoption sky rockets, knowing how to move around an LLM will increasingly just be part of the job. But there is so much behind getting an AI agent to “work” in a business setting that I have a hard time imagining that some day job 9-5 guy is really going to spend the time figuring out how to do it right. It doesn’t make sense to have, like, your sales guy figure out how to properly do security for agents running on his laptop. He’s just going to do whatever works as fast as possible to get back on the phone selling, which means it’s up to the rest of the company to make the fastest path the happy path. To steal a quote from Dex, it doesn’t make sense to have everyone in your org figure out how to use Kubernetes even if it’s really important, why would it make sense to have everyone in your org figure out how to run agents properly?

• There was a whole bunch of stuff on open source LLMs (GLM 5.2 was a particularly large topic of conversation), auto research, robotics, and model training that I unfortunately missed. Lots of talks, maybe too many. I was hoping to see more researchers but it was very much more an industry conference than an academic one.

Some of my favorite parts of the conference were outside the conference. There were a bunch of sponsored side events, including one great one from Modal (thanks Monishee!), but I’m mostly talking about catching up with other folks who happened to read the blog and invited me to come hang. Office Hours with the LLM whisperers was very fun (thanks for having me ), so was dinner with the botco crew while watching the world cup. On the flip side, things were so busy that I didn’t get a chance to really see my usual Bay area crowd — if I missed you this trip, big apologies, and I will see you next time. For everyone else I met, hit me up if you’re ever in NYC!

Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. We organize in person meetups on our luma calendar. More articles about AI here.

If you are trying to build background agents, we can help! We build white-labeled custom background agent infra that work like Ramp Inspect or Claude Tag, with no lock in at any part of the stack. Check out noriagentic.com for more.

Agents are increasingly picking up more and more of our workload. We use agents to write and ship code, publish our dev blog, and even write our board decks. To make that all work, we have to give our agents access to more data sources. Slack, email, GitHub, aws, linear, datadog, sentry. For most of these tools, we can just pull an MCP server and call it a day. But MCP is extremely token hungry, and often implemented really poorly. For some critical integrations, we want to own the integration point.

When building out a new integration, we often reach for CLIs over trying to build a custom MCP. CLIs have built-in progressive disclosure (they do not suck up tokens with parameter descriptions for function calls unless your model absolutely needs to know). The models are extremely familiar with using and calling CLIs through bash. And the models are very good at writing efficient CLI tools, because they have tons and tons of examples.

CLIs that are built for agents as their primary user behave differently than those built for humans. We’ve spent a lot of time thinking about what that behavior difference ought to look like. In no particular order, here are a few principles we use when building CLI tools that assume agents are the primary consumer.

• No colored output. No spinners. No animations. These waste tokens and are probably unappreciated by an agent that can’t see the visual media.

• No interactivity at any point. Everything must be accessible through flags and stdout/stderr output.

• Calling the cli with empty, --help, --list, etc. should list a full description of what the tool can do at a high level and descriptions of individual commands (”use this command for…”). Overlap is fine here, we want to make it easy for the agent to stumble onto instructions.

• Help messages should be recursive. Calling foo --help should explain what the CLI does broadly. Calling foo bar --help should explain what the bar command does within foo, including any parameters that bar may need. Aim for 100% docs coverage.

• Every error or help message should include a file path for where the relevant code path is stored on disk, and a message encouraging the model to look up the source if needed.

• Related: we generally install the CLI by building from source or bundling source where possible, so that the decompressed source is present for the model to figure out what to do.

• Make it easy for the agent to fix itself. If the agent tries a command that doesn’t work, do a nearest neighbor lookup. “Did you mean…” but for CLI commands.

You can find all of this in skill form here. Just drop it into your agent of choice, or run

npx nori-skillsets install amol

to download my personal agent config (which includes the agentic cli skill already in place).

We’ve built a few different CLI tools that follow these principles, including:

• nori-slack-cli, a thin command-line wrapper around the Slack Web API that maps 1:1 to Bolt (@slack/web-api). Every method Bolt exposes is reachable through a single dynamic command — there is no curated subset, no opinionated abstraction layer, and no business logic. If Bolt can call it, this CLI can call it.

• nori-newsletter-cli, a CLI for managing and sending newsletters via AWS SES. Handles subscriber management through SES contact lists, CSV import, and individual email sending with automatic unsubscribe support.1

• nori-luma-cli, a wrapper around the Luma event manager API, so the agents can plan your parties for you! 🥳🥳🥳

• nori-discord-cli, which is like the slack cli but for discord.

• nori-shopify-cli, which lets your agent access the admin tools in shopify.

All of these are open source, so feel free to drop them with your agents and see how they use the tools.

Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. More articles here.

If you are trying to build background agents, we can help! We build white-labeled custom background agents that work like Ramp Inspect for our customers. Check out norisessions.com for more.

Guest post from Clifford, my CTO and co-founder at Nori. Original here.

For the past year the techfluencers and executives and tokenmaxxers have been telling us that code is cheap. Lately harness engineers have upgraded the claim: code is free.1 Soon enough, the marginal cost of acquiring another thousand lines really will round to zero, totally untethered from the old bottlenecks of human time and human expertise.

I think this misses the point. Sometimes free isn’t cheap enough.

Over the last century, conservation efforts for American bison in places like Montana and Nebraska worked a little too well. The buffalo came back — and then they kept coming. Herds grew to the point where they threatened serious destruction of the very ecosystems we’d set out to protect. So every autumn, the surplus animals had to be culled. From the 1960s up until around 1992 the Department of the Interior ran a surplus wildlife disposal program to try and find an outcome other than slaughter for these beautiful animals.

Which leads me to ask you, the reader: how cheap would one of these buffalo have to be for you to sign up?

If you answered “I wouldn’t, at any cost” — then I hope you see my point. I bring up this specific example because for decades, these buffalo were actually available for free!2

Turns out there can be costs to a transaction other than just the acquisition cost. Some are financial. Some are opportunity costs. Some are just risks and inconveniences. A buffalo costs an enormous amount of time and money to maintain. Every hour you spend caring for it is an hour you didn’t spend on the work you actually meant to do. And a new buffalo can expose the rest of your herd to new pathogens and disease.

Those other costs can matter so much more than the sticker price, that the up-front number becomes basically irrelevant to the decision. Free was never the question. So if over the next couple of years the frontier labs decide to debt-finance a massive supply of buffalo and hand them out for nothing... I would not recommend rebuilding your entire way of working around buffalomaxxing.3

Because sometimes free is not cheap enough.

Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. More articles here.

If you are trying to build background agents, we can help! We build white-labeled custom background agents that work like Ramp Inspect for our customers. Check out norisessions.com for more.

We have found zero performance benefit on SWE tasks when agents have search access to their previous transcript sessions, provided they have access to other forms of context. We also have not found much benefit in trying to automatically trawl through session transcripts to improve agent context, unless there is a human in the loop.

This was pretty surprising.

Intuitively it feels like there's a lot of valuable information in a transcript between an agent and an engineer. Maybe it would have information about why the code exists, about user intent. Or it might have the other approaches that a user tried and discarded. At the least, it would have some amount of additional context that the agent could use to augment its understanding. I believed this so strongly that my company built an entire product around this concept. I used to tell folks that "session transcripts were the new oil," that they were more valuable than the code itself.

Other people have clearly had similar thoughts, which is why there are so many different tools to do session backed memory, including (of course) Claude Code itself.

I think the most common architecture is to do something like:

• Store all transcripts across an organization in a DB

• Put a vector search, an elastic search, or a SQL search layer in front of it. Ambitious teams will use all three. Maybe graphs will be involved.

• Make this available to the agent using an MCP, or by exposing a cli with skills.

For us, this additional work doesn't seem to make a bit of difference. If anything, based on many months of testing with and without session search access, it may make the models worse.

Why might this be true?

One thing our team cares a lot about is coding artifacts. We don't really write code by hand anymore. In order to make PRs legible, we emphasize good commit messages, good pr messages, and comprehensive documentation. Every code change comes with extensive metadata that is committed alongside the code. When our agents do work on a piece of code, they are instructed to go look at the docs and the previous PRs.

In other words, the agent is already distilling all of the information that is valuable about a transcript, and storing it where it is needed and easily accessible. So when the agent uses a transcript search server, it ends up spending tokens reading things it already knows, while picking up all the stuff that the agent decided not to write down in the first place. Maybe, every now and then, there's some useful nugget of information in there. But most of the time, the agent is just looking at a pseudo nonsensical scratch pad and wasting precious tokens to do so.

The agents are also terrible at actually removing context, which is a critical capability for maintaining long term memory. I mean, across literal thousands of sessions, I've never seen it happen even once.

This is not a trait that can be removed with some clever prompt engineering. Agents don't have state, so they have to assume everything in their input context window is the ground truth. Every line of code, every existing bit of memory, every token is treated as an expression of intent -- even if that code or that memory was generated from a random decision made by some previous agent session, never reviewed or even understood by a human. This intent drift compounds the more the agent tries to autonomously build up a memory base.

As far as I am aware, there are zero coding benchmarks that assume the input data is corrupt. In fact, the models are penalized for assuming that the input data is wrong. This is partially an alignment issue too -- we don't want to have agents doing unintended things, and there isn't an easy way to thread the needle of “don't delete the codebase” and “do delete some of the input context.”

Since models can't actually garden their own memory, automatic memorization ends up in the same place: a load of garbage eating tokens, bloating bills, and degrading model quality.

Net net, I've become really bearish on tools that index and store and surface in session transcripts to an agent. The session transcript may be useful for team observability, but it won't make your agents better.

That doesn't mean that agents have no role in learning context over time. We use our internal nori bots to review everything that happened at the company each week across PRs, slack, drive, etc. And they then propose a set of changes to our built in nori skillsets, tagging the team in slack. These are all default rejected. In order to accept a change, you have to go in and actually look at the diff and make sure it fits the intent.

We accept less than 20% of these. Which means 80% of these “automatic” updates would've made the model worse. I can't imagine how much more unsustainable that would be if a multi-hundred person org were all saving these “updates” automatically all the time.

Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. More articles here.

If you are looking to build background agents, we can help! Learn more about Nori at noriagentic.com, and our background agents product at norisessions.com.

Generally speaking, if you spend tens of thousands of dollars on something, you want to see something come out on the other end. Some return on investment.

O sure, not always. I’ve previously said that selling to consumers is sorta funny because they love spending money on things that waste time or actively cause pain. This is part of why the gambling apps are so popular these days. Why yes, I’d love to spend $100 on betting that Wemby scores a 3 pointer while doing a handstand and singing the national anthem in French.1

But for businesses? I’ve basically never heard a business leader say that they were going to set a bunch of money on fire because it made them feel good, at least not the same way a whale will spend thousands on Genshin Impact gatcha pulls. Like, imagine if some serious business leader, like, idk, Mark Zuckerberg, decided to announce that Meta was going to burn money. He could do that. He’s got the voting shares. But it would be a bit silly, wouldn’t it? I generally think if you’ve gotten to the point where you’re running really big really important companies, you mostly aren’t doing things for kicks, with one big exception.

If you haven’t heard, tokenmaxxing is (was?) a phenomenon where executives accidentally encouraged their employees to burn a bunch of tokens on useless tasks. The canonical example of this is, by complete coincidence, Meta, which has been thoroughly skewered for tying performance evaluations to the amount of token usage per person. Obviously, obviously this was going to lead to people just burning tokens on nothing. One of my friends at Meta reported that they literally would just have two agents talking to each other throughout the day to get her token numbers up.

This was such an obvious outcome that many people rounded this off as “these business leaders are really dumb because they decided to burn a bunch of money on tokens without expecting any return.”

I understand why that’s a tempting take, because that is kinda sorta what the public face of a lot of this was. But I’m going to do my favorite thing in the world, which is be a bit contrarian. It wasn’t that “executives accidentally encouraged their employees to burn a bunch of tokens on useless tasks.” Rather, “executives purposely encouraged their employees to burn a bunch of tokens on useless tasks.”

I work with a lot of teams on figuring out how to use AI effectively. A few months ago, there were a lot of people who were extremely resistant to using AI tools at all. Senior people, people that had a lot of respect in the organization. It was very difficult to convince these folks to use the tools. And when you did, they would often accidentally (or purposely?) use the tools in a way that would obviously lead to weird or bad outcomes.2

One way to think about the top down tokenmaxxing policies is that this was a technique by executives to break through. Yes, it was obviously a blunt force policy, but sometimes you need blunt force to break through a wall.

Of course, that was the situation a few months ago, when there were still holdouts. It’s now a few months later, and the tokenmaxxing policies had their intended outcome: everyone is using AI to code, at least a little bit. Most teams haven’t yet figured out how to build their own Ramp Inspect or Stripe Minions (if that’s you, reach out — we can help!) but basically everyone is at least using cursor in the side bar. Which, of course, means that token spend has gone way up. Unfortunately, but probably not unexpectedly, the increase in token spend has lined up with both OpenAI and Anthropic trying to go public. Both companies have limited the amount of juice their subscriptions provide while jacking up their API pricing. Token subsidies are increasingly vanishing.

So now the incentives are mostly gone and the cost is way up and, of course, teams are starting to roll back their unlimited-token-spend policies. All of this to say, tokenmaxxing is dead.

Except…maybe not.

The promise of AI tools generally is that you can have them run without human supervision to accomplish really hard and really tedious tasks that still need to be done. The big code migration, doing research on all your competitors every morning, keeping up with the stream of inbound and outbound — these are all things that people mostly hate doing and want AI to do.

Up until recently, though, you couldn’t reliably have an AI run for long periods of time. If you tried, you would notice that small errors introduced by the models (including hallucinations) would take on a life of their own and eventually become irreversibly embedded into the project. In the business we called this “compounding error.” It not only required a fair bit of human supervision, it also kept token costs low because there was little benefit in running agents 24/7 to begin with. Like, what’s the point of running a little demon in your computer over night if the thing is just going to tear up all your hard work? If spending more tokens results in worse work, you obviously aren’t going to spend more tokens!

That’s no longer true. We’ve entered a different regime, where spending more tokens generally results in better results. We call this “compounding correctness” — the more tokens you spend on getting a task correct, the more likely you’ll get a good outcome. We talked about this a bit at the last in person Agentics meetup:

Compounding correctness flips the calculus. If more token spend leads to better outcomes, then you’re going to want to spend a lot of time running tokens. Which sure as hell sounds like tokenmaxxing to me! The original incentives to tokenmax are gone, but eventually folks will realize that a new and more powerful incentive has take its place.

We’ve already seen some of this take place in the cyber security world:

Last week we learned about Anthropic’s Mythos, a new LLM so “strikingly capable at computer security tasks” that Anthropic didn’t release it publicly. Instead, only critical software makers have been granted access, providing them time to harden their systems.

…

This chart suggests an interesting security economy: to harden a system we need to spend more tokens discovering exploits than attackers spend exploiting them.

AISI budgeted 100M tokens for each attempt. That’s $12,500 per Mythos attempt, $125k for all ten runs. Worryingly, none of the models given a 100M budget showed signs of diminishing returns. “Models continue making progress with increased token budgets across the token budgets tested,” AISI notes.

If Mythos continues to find exploits so long as you keep throwing money at it, security is reduced to a brutally simple equation: to harden a system you need to spend more tokens discovering exploits than attackers will spend exploiting them.

You don’t get points for being clever. You win by paying more. It is a system that echoes cryptocurrency’s proof of work system, where success is tied to raw computational work. It’s a low temperature lottery: buy the tokens, maybe you find an exploit. Hopefully you keep trying longer than your attackers.

Fable is, tragically, gone now. But the underlying concept here still remains.

This is also in part why people are suddenly so excited about ‘loops.’ Boris Cherny, the creator of Claude Code, got up on stage and said ‘loops’ and everyone freaked out. The basic idea behind loops is that you run an agent until it reaches the end of its turn, and then when it finishes you simply restart the same prompt. With a bit of cleverness you can take a pretty heavy specification and automatically have the agent split it into parts and solve it over time. No human supervision required.

Is this some new thing? No, not really. The loop concept has been around since literally last July. It used to be called a “Ralph Wiggum loop,” but as the industry has matured so has our sense of humor and the ‘Ralph Wiggum’ part was dropped.

There were ways to get loops to work, but it was hard. You had to think a lot about how to prompt the agent, which in turn required a pretty deep familiarity with how these things work. Now, though, it’s easy. Compounding correctness makes it easy. You can basically prompt the LLM however you want and to a first approximation, it will do better every iteration of the loop.

So is tokenmaxxing really dead? Maybe temporarily, but long term I don’t think so. Teams that are at the cutting edge are currently building or have built the infrastructure necessary to run agents 24/7. It’s only a matter of time before the bigcos realize that the cost benefit has shifted again.

The real winners here are the open model platforms. Tokenmaxxing the top labs will never stand up to any amount of CFO scrutiny. As open models get better, it will become more popular to simply run those in a loop. That was the core thesis of Rohan’s talk above. If Claude gives you 1.1x improvement per iteration, and GLM 5.2 gives you 1.05 improvement per iteration but costs ~5x less, you can just run the second loop 5x more times and it will be better.

The last thing I want to mention here is that some of the ridiculous token spend is downstream of a serious misunderstanding of the best way to use these tools. Before coding agents really took off (thanks in large part to much better harnesses like Claude Code), lots of people were making their own custom agents. And that was legitimate work! You had to think about this stuff as if it was…well, software. There was an art to figuring out the tools and the prompts but the core of it was still just software, even if it was supported by ‘AI native’ frameworks like Pydantic or Langchain.

You can’t fit a square peg into a round hole. Executives across the board saw this style of building agents, went “o, this is just a more flexible zapier workflow,” and proceeded to demand data processing pipelines that could do one-off tasks that were ‘agentic’ instead of building those same pipelines in good ol’ deterministic code. ‘I need to do data labeling, so I will build a data labeling agent’, that sort of thing.

Now, relying on an agent to do some of this stuff is already going to be significantly more expensive than just doing a workflow automation. But the bigger issue is the accuracy: none of these ‘agents’ ever really took off, because they were never as correct as a deterministic pipeline would be.

If you’re committed to using agents but want to reduce the cost of hallucinations and things, what do you do? Why, you build another agent! A ‘quality checking’ agent, or something like that. And what if that agent gives you errors? Well you’ll just build another! And now you have 3x the token cost, enjoy!

The story of tokenmaxxing is, again, one of RoI. That story didn’t just play out at the bigtechcos. It also happened at a less advanced scale at companies all over the country — companies who poured billions into random agent pipelines built by one off consultants that unfortunately never really quite worked all that well.

Notice that these are actually two different kinds of tokenmaxxing.

• The first kind is ‘spend a lot of money on tokens for your developers‘. Here, devs are using tools like Claude Code and figuring out how to run things in loops and using a lot of tokens to do it. Ostensibly this is a good use of money because it’s making the engineers themselves more productive.

• The second is ‘spend a lot of money on tokens for your pipelines‘. Here, devs are still writing code by hand! They are using that code to create one-off agents to do very specific tasks often in a non-deterministic and brittle way, and it’s those agents that guzzle up all the tokens. This is only a good use of money if the pipelines work, which they don’t.

But here, too, we are seeing a shift. Increasingly, these sorts of one-off pipeline-based tools are better done by generalist platforms that are skinned for the specific task, than an “agent” specially designed to do that one task. There’s some market arbitrage here. Some buyers haven’t realized that generalist agents have gotten really good, so they will go to consultants asking to ‘build me an agent’, and the consultant essentially writes a skill file and says “that will be $2m please.”

Luckily, this too shall pass. Generalist model platforms are obviously the future for anyone who has used them (and if you haven’t, again, reach out!) And that, again, will lead to another rise in tokenmaxxing behavior in this part of the market.

The natural end state of all of this is the ‘software factory’ or, even further, the ‘dark factory’ — a codebase that pumps out code, reviews code, fixes bugs, writes tests, and so on without any human supervision. The human simply puts in a spec and out comes an application. The folks over at StrongDM have taken this to the furthest extreme, arguing that engineers should aim to spend $1000 in tokens per day. This is almost certainly hype, part of a long trend of saying egregious things to get coverage and buzz. We have a software factory, and we spend like $600 per month. But the hype and buzz comes about because, even though it is currently ridiculous to spend the price of a senior google engineer in tokens per engineer, there is a kernel of truth to this. The incentive to spend ludicrous amounts of money on tokens are there, latent, waiting for diffusion.

What’s old is new again and what’s dead may never die. Tokenmaxxing is dead, but we haven’t seen the last of tokenmaxxing just yet.

Other things:

• GPT 5.6 is out, kinda sorta. From the announcement:

  We’re beginning a limited preview of the GPT‑5.6 series: Sol, our flagship model; Terra, a balanced model for everyday work; and Luna, a fast and affordable model. Terra has competitive performance to GPT‑5.5 while being 2x cheaper and Luna brings strong capability at our lowest cost.

  …

  We believe in broad access, and we plan to make GPT‑5.6 Sol, Terra, and Luna generally available in the coming weeks. As part of our ongoing engagement with the U.S. government, we previewed our plans and the models’ capabilities ahead of today’s launch. At their request, we are starting with a limited preview for a small group of trusted partners whose participation has been shared with the government, before releasing more broadly.

  …

  We don’t believe this kind of government access process should become the long-term default. It keeps the best tools from users, developers, enterprises, cyber defenders, and global partners who need them. We are taking this short-term step because we believe it is the strongest path to broader availability in the coming weeks, while we work with the Administration to develop the cyber Executive Order framework and a repeatable process for future model releases.

  Washington Post was even more aggressive in its analysis:

  U.S. government will decide who gets to use latest upgrade to ChatGPT

  The Trump administration came to power preaching a laissez-faire approach to AI but has lately increased oversight of the industry.

  Reading between the lines, it looks like we are getting more government regulation of AI companies, but unfortunately the process as it currently stands is completely opaque. Reactions are mixed. On the one hand it’s great that the administration’s previous attacks on Anthropic are also being applied to their competition. On the other hand, it seems like the process is totally opaque, and now the government is unilaterally picking winners and losers not just among AI companies, but among every other industry where AI may matter (i.e. all of them). The companies who get to use OpenAI’s tooling (and Mythos, see below) are currently unknown. It would be very concerning / disappointing if we find out that it is exclusively companies with ties to this administration.

• Related: Mythos is back on the table, at least sort of. From Semafor:

  US releases powerful Anthropic model Mythos to some US companies

  The US government Friday lifted its block on Anthropic’s powerful Claude Mythos 5 AI model, allowing the company to release it to more than 100 US institutions, including major companies and government agencies.

  The decision, in a letter sent Friday afternoon to Anthropic, is a major de-escalation in the confrontation between the Trump Administration and one of the world’s most valuable private companies. Two weeks ago the administration imposed export controls on Mythos, leading to a shut down of the model and its cousin Fable 5 after warnings from Amazon and other companies that they could be “jailbroken” for malicious purposes.

  The letter is silent on Fable 5, a weaker version of Mythos that was briefly the most powerful AI model widely available to consumers. People close to the talks said they are moving toward releasing Fable as well, though that timeline is unclear.

  “I have determined that appropriate safeguards are in place to permit certain trusted partners to access the Claude Mythos 5 Model,” Commerce Secretary Howard Lutnick wrote to Anthropic’s chief compute officer Tom Brown Friday, citing “significant progress” in the intense, daily talks between the government and the company since the block went into effect.

  Again, picking winners and losers.

• While we are talking about OpenAI’s launch, one thing that was buried in the announcement was that the tooling would be available on Cerebras’s high-speed inference machines at ~750 tokens per second. This is quite fast. Right now we are in a regime where it makes sense to treat AI tools as async operators that can go off and do tasks without supervision. But that is mostly downstream of the fact that AI is slow and it takes a while to do things. If AI was really really fast you would plausibly go back towards a more synchronous model of operating. For an idea of what this may look like, check out https://chatjimmy.ai/. It’s just a demo, but wow, what a demo.

• Open models like GLM 5.2 have gotten pretty damn good. They aren’t SotA, but they are much cheaper than their frontier equivalents. Right now, GLM 5.2 is ~$1.4 per million tokens and ~$4 per million output tokens. By contrast, the entire Opus 4.X series is $5 per million input and a whopping $25 per million output. The only model in the Anthropic suite that even comes close to GLM 5.2 in terms of pricing is Haiku 4.5, at $1 / million input and $5 / million output. But GLM 5.2 blows Haiku out of the water, and in some cases is even stronger than GPT 5.5 on benchmarks. If I were the big labs, I’d be pretty concerned about this. And if I were basically any consumer on the market, I would be doing everything I can to avoid provider lock in by adopting tools that are able to sit on top of all of the major players.

• OpenAI unveiled an in house chip for inference.

  On Wednesday, OpenAI unveiled its first custom-built inference processor, designed and manufactured in collaboration with Broadcom. Named Jalapeño, the new processor was designed specifically for the unique needs of OpenAI’s inference systems. OpenAI’s own AI models assisted in the development of the chip, the company said.

  Jalapeño, like a jalapeño chip. Get it?

Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. More articles here. Learn more about how Nori can bring your company into the glorious AI future at norisessions.com.

The year is 2022. I am the CTO of my first company. I am managing a team of brilliant engineers and a team of brilliant and completely non-technical sales and marketing folks. I spend, to a first approximation, half my life counting geese.

I am constantly in meetings. Every meeting has some complicated name, like ‘Aligning strategy for Q1’. But I know that the meeting is really about counting geese. We look at dashboards that make sure that everyone has their geese properly counted. Sometimes the engineers will mention that there were geese that they counted that aren’t reflected on the bean counter. Other times, the marketing and sales folks will say that some geese that we thought had been counted hadn’t been counted enough. There is much argument about the true number of geese.

Once we reach consensus, we spend time making sure that the goose count on GitHub matches the goose count on Linear which matches the goose count in Google Drive. There are more geese, the flow of geese is constant, so we have to spend an increasing amount of time counting them. We plan years around geese. I try to protect my team from the geese. Mostly that means that I am spending more of my time counting geese.

This is at a 14 person startup. It was worse when I was at the big tech co.

The year is 2026. I’m CEO of my second company. I haven’t thought about geese once in the last 6 months. The AI counts the geese now.

There are a lot of things to be worried about with AI. Things that I’ve written about extensively in other parts of this blog. But the thing that I am most excited about is the complete and total elimination of bullshit overhead. The future of work may be complex and crazy. But you know what it won’t involve? Having a founder staying up late at night making sure the linear story points are aligned with the things merged into GitHub so that the eng team sync meeting can save fifteen minutes of everyone’s time.

Everything that people hate, they hate because those tasks are boring and routine and dull and mindless. How many teams love to hate their CRM? Every time I get off the high of a fantastic customer pitch, I have 0 desire to then go into Hubspot or Salesforce to mark some row in a table ‘Lead’. How many hours get lost to making slide decks the right branding? Raise your hand if you lost an hour trying to figure out how to move an image that is in the background of your slide because it’s set as a ‘company wide template’, which was a feature you only just learned about ten minutes ago. How much information gets buried in slack, or docs, or linear, or wherever you do your work? Hours and hours of productive time wasted counting geese.

AI tools excel at this kind of data munging. They are so good at taking in data from calendly and piping it into salesforce or pulling in sentry data to debug something in github or whatever. Every company has a “brain.” Normally that brain is decentralized. It exists across the ten million different tools we all use to track state. Jira, confluence, notion, GitHub, linear, Asana, Monday, trello, docs, drive, slack, teams, one drive, Dropbox, one note, Salesforce, intercom, otter, fireflies, snowflake, databricks, datadog otel AWS gcloud Coca-Cola double decker bus his name was my name too.

The promise of AI is that you never need to think about any of that again. A universal AI operating system, a single entry point to the entire company brain. That doesn’t mean replacing people, though some folks may see it that way. To me it means taking the bullshit out of bullshit jobs.

I don’t think everyone is going to get there all at once, but the shape of the future is very clear to me.

• There are humans and there are agents.

• The humans spend time making decisions that matter — north star vision, strategy, marketing, and actually talking to people.

• The agents live in the background, in the ether, ephemeral, automatically bringing the vision to life with minimal friction.

This is a new and transformative way of conceptualizing work. AI is a new and transformative technology.

But it’s also very much within reach. There are teams who are “full auto” already, who are producing what would be Herculean amounts of output with a single person.

The main thing you need to get from point A to point B is a bit of forward thinking and some cloud infrastructure. We already provide the latter. Let me show you a bit of the art of the possible, in the hopes that you’ll get some of the former.

Here’s my morning a few days ago.

I wake up around 8:30AM. I walk into the kitchen and grab my phone, which I purposely try to keep in another room to avoid doomscrolling at night. There are like five slack messages waiting for me. I had a fleet of agents that were working through the night. I quickly eyeball the results while brushing my teeth.

One of them spun up at 3am to fix a bug that was reported by sentry. It posted a full root cause of the bug and a PR to fix it. I glance at the changed line diff and the pr description. Looks about right for what I’d expect for this, I happily merge it without looking at the code. I used to look at code, because I was worried about slop. But our agent hasn’t produced slop in 3 months. At some point you’ve got the thing so dialed in that it just works. I go to do some physical therapy stretches for my back — too much time coding in bed.

Shower, out the door, try to get on the Path train by 930AM.

While I’m sitting at the subway station I look through the other tasks. Two of them are confirmations to send out draft emails responding to some customer inbound. The agents personalized the responses, using a combination of call transcripts and website research. I tweak them both — I still can’t quite get the AI to nail my voice — but both emails go out in about 5 minutes. For someone who normally agonizes over each word in an email, this is a miracle.

One of my agents came back with some market research for the day. Things that the engineering hive mind on Twitter are thinking about. It is amazing for my mental health to be able to keep up with Twitter without being on Twitter. I chat with that agent a bit in slack, asking it to dig into a thread about agentjacking.

Another agent sends me a list of new papers on arxiv about representation learning. I spend the first half of the train ride reading through the summaries, asking questions about them and how they work at the stops where some 5g manages to leak down into the station.

Half the train is wearing Knicks gear, and that got me thinking that I should do something for our website. I open slack and type this prompt:

In between stops I iterate with the agent, all in slack. My cofounder sees what I’m doing and tells the bot to add a dunk animation in the same thread.

It’s done by the time I hit the 23rd stop and get out of the train. I can’t say that I have a ton of aesthetic sensibility, but I kinda like how it all came out.

By the time I get to my desk another agent has kicked up automatically, giving me an update on all of the changes in the code base yesterday alongside a brief summary of what everyone worked on. No painful daily sync, where you multiply every wasted minute by the number of people in the room. The rest of the team is locked in, talking to the agents from their CLI or slack or even lounging on the sofa on his phone. One of them is pacing around talking into his phone, recording a transcript to kick off a bunch of tasks on the go.

I am more productive on my way into work than on most full work days in 2022.

My team does 70% of our feature dev work in Slack. 30% of it from our phone. All of it through agents (we don’t write code and haven’t in months).

We do 90% of our ops work using automatic workflows that we never think about, or in slack through the agents.

We do all of our debugging through the agents through slack, though really it happens automatically before we even know there’s a bug in the first place.

We have the agents make slide decks. We have the agents update the website. We have the agents manage our CRM. We have the agents make sure linear and github are in sync.

My team gets to spend our time doing what we love. We think about product. We think about our customers. We write about things we care about, like this post. When people talk about AI being the future of work, they think that AI will get to do all the fun stuff. So far, for me, the AI has just automated away all of the things I hate doing.

And that’s really the goal, or at least should be for anyone thinking about how to deploy AI successfully.

Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. More articles here. Learn more about how Nori can bring your company into the glorious AI future at norisessions.com.

Guest post from Clifford, my CTO and co-founder at Nori. His posts always do way better than mine, and it’s a shame he doesn’t write as much as I do because he has real insights whereas I mostly just shout my opinions at the cloud. You can see the original here.

Also I will be in SF from the 29th to the 3rd! DM me if you want to catch up!

Also, also, if you or your team is trying to figure out how to use AI at your company — if you are struggling with costs, or with security concerns, or with roll-out to nontechnical staff — take a look at Nori Sessions! It’s now generally available. On average, Nori makes teams 2-5x more productive across sales, ops, and eng. AI promises the future of work. Nori Sessions is how you get there.

The Wrong Scoreboard

The discourse on coding agents has been obsessing for the past year over the wrong question. The main focus has been what models can do: lines written, autonomous minutes, benchmark scores, model cards, percent of lines shipped by AI. These are all generalized measures of implementation throughput. Useful for a bird’s-eye view of model progress, but they say almost nothing about where the actual bottlenecks now live. The operative question for practitioners in 2026 is not what tools can do, it’s what you should ask them to do.

Answering the “should” question requires a different lens than the capability benchmarks provide. Every task you might hand to a coding agent has two costs that matter: the cost to implement (C_i) — the time and expertise needed to produce the code — and the cost to verify (C_v) — the time and expertise needed to confirm the code is correct. The relationship between these two variables determines whether delegation is a net win or a liability.

Aside: About “Delegation”

When I first outlined this in November 2025, I was comparing handcoded vs AI-delegated implementations. My workflow has changed significantly since then: I rarely hand-write code.

The relevant choice for me is now between pair programming with the agent (high-touch, Socratic, every structural decision is guided) and delegating (agent leads research, planning, and implementation; you just review the output of each phase). The pair programming model is mentally just as involved as writing code, but mechanically faster. The delegation model is now very different, allowing you to run and ship five separate feature PRs in parallel (not some clickbait Xitter “I ran 100 agents in parallel today” make-work slop, but five actual product increments, in parallel, in a brownfield codebase).

Whatever the threshold of delegation is, in my experience the framework below applies.

The Two-Variable Framework

When both costs are low, it doesn’t matter what approach you take — the task is trivial either way. When C_i is high but C_v is low, delegate freely; the implementation is a job for the agent, and you can cheaply confirm the result. The inverse is equally clear: when C_i is low but C_v is high, build a detailed mental model by taking part in every step of the process.

The dangerous quadrant is top-right. When both costs are high, there’s a huge incentive to spin the slot machine many times, and see if the agent just happens to nail the task. Compared to hand coding, where you burn days or weeks to ascertain the quality, the agent might have a chance to succeed at the same or higher quality after just 60 minutes of work. For complex or off-distribution work, it may be a small chance... but that makes it even more tempting!

By skipping the mental effort, you go in blind on an equally demanding task: verification.

This is the trap.

The models have dramatically compressed C_i across the board. C_v has not moved at the same rate — and in many cases, without careful developer intervention, it has gotten worse.

Vibecoding and the Unaddressed Variable

Vibecoding is the logical extreme of treating C_i as the only variable.

Previously, architecture decisions were bottlenecked by implementation cost. Releasing that constraint completely, without addressing verification cost, is a big failure mode. Any frequent flyer on Claude Code has experienced this, as an end user of an entirely AI-coded application — the constant issues with UI bugs, unintended changes to history cells, broken permission models...

I've written about the flickering issues before, and I've been annoyed that sandboxing persistently pollutes the workspace with empty files (this issue has been recurring in different forms for three months now). Users of the Claude Web environment, of Cursor, and many other almost fully AI-coded products experience the exact same degradation of quality as the software grows so rapidly. It’s not just that more features lead to proportionately more bugs. When you don’t build a mental model of the codebase, you’ve skipped your first pass on verifying the logic, and you’ve gone without a map of what parts need verification. The consequence isn’t just bugs — it’s verification blindness: you don’t know what you don’t know.

This is a common failure mode that many teams have fallen into, particularly startups that feel the keenest urgency to ship faster.

Verification Debt

As a result, the most common form of tech debt in these highly agentic codebases comes from growing your feature surface area too fast and too loose. Every agentic feature shipped without a corresponding verification investment degrades your ability to autonomously ship *future* features. This is a compounding liability, not a fixed cost — first it accumulates, and then because these changes can have cross-cutting technical concerns, or act as bad examples for future work, it compounds. Unit and integration testing become slightly more important, to compensate. But E2E behavioral verification becomes far more important, because that’s the layer the agent generally cannot self-evaluate on its own. Skipping this investment creates verification debt.

Detour: What Spec-Driven Development Gets Right (and Wrong)

The popular framing of spec-driven development is wrong on two counts. It’s not about making prompt copy-paste easier, and it’s not about closing the loop on “Ralph Wiggum” workflows — generate, test, regenerate. These framings chase short-term speedups that don’t touch the real bottleneck.

The original insight of a specification is much more important: you cannot verify an implementation if you don’t know its intent. A specification is the textual or symbolic description by which different readers arrive at the same mental model. In distributed teams, software has long relied on PR review, ADRs, box and sequence diagrams — this is fundamentally the sharing of intent. You must know the developer’s intent before you can review their outputs. This is not new, it’s just now more urgent.

The genuine unlock of specs is that they literalize the behavior you need to verify. Once combined with simulation environments — headless browsers, terminal puppeteering, API smoke tests — your specs become the instructions for agentic verification *after* agentic implementation. The loop closes not at the generation layer, but at the verification layer.

Beware Reflexivity

These two variables don’t stay independent — they affect each other over time. Shipping too quickly raises C_v as verification debt accumulates. Higher C_v in turn raises future C_i — the agent’s implementation speedup erodes as the codebase becomes harder to reason about, harder to test against, and bad patterns get committed and cargo culted. This is the mechanism by which agentic coding gains can crash down to earth, trash a codebase, and turn a team against AI coding tools entirely.

But reflexivity runs both ways. The virtuous path runs in the opposite direction: to properly ship faster, teams must aggressively use the low cost to implement new tooling as a lever to decrease the cost of verification.

Solutions: Making Verification Tractable

Three concrete approaches, in increasing specificity:

1. Simulation environments at full breadth

• Standard integration/automation/E2E testing practices are table stakes, but the agent’s reach is now wider.

• Headless browsers, Chrome DevTools protocol, pseudo terminals, API smoke tests, microVMs, containers — the agent can drive all of these.

• The question is no longer “can we automate this” but “have we specified what to automate.”

2. Making the runtime legible.

• E2E tests don’t capture everything: service startup timing, internal program state, functional SLAs, “no UI interaction blocks for more than 2s”.

• An ephemeral, per-worktree observability stack — logs, metrics, traces — makes runtime behavior tractable to the agent.

• This is the difference between the agent knowing the tests pass and the agent understanding *how the system is behaving*.

3. Bespoke verification tooling is now nearly free.

• Personal example: while building a PTY proxy around several TUI tools, codifying system invariants and nightly fuzzing jobs cost only ~two extra hours.

• I implemented fuzzing with reproducible seeds and system state captures at failure.

• I can put my tool through more comprehensive testing than I could have ever justified without the agentic contributions.

• The economics of verification tooling have shifted — the agent makes it cheap to build guardrails and course correction, so the only remaining question is where those guardrails should go.

The New Discipline

Coding agents haven’t changed what good software engineering is, but they have changed where the leverage point is. The developers extracting the most durable value from these tools are the ones who have reinvested implementation gains into verification infrastructure. The question to ask before every agentic task is not “can the agent build this?” but “how will I verify what was built?”

If you want two immediate, concrete steps to improve your verification tools:

• After you run through a cycle of research-plan-implement, your agent must go through the implementation using TDD. Here’s a general purpose agent skill to do so.

• If you want an agent to go through a manual end-to-end test on your project, consider giving it the tools to interact directly. Here’s a skill for the agent to puppet a browser with Playwright. Here’s a skill for the agent to puppet a TUI with tmux.

I am hoping that this will be just the first of three core posts about the new practices in software engineering. To signpost properly where I think this is going, here are the three ideas that are changing how I think about software development:

1. Verification debt. (this post)

2. Agent legibility.

3. Compounding correctness.

Lots of interesting things in this post that I agree with. See also my very old post on the point of code review, and Cliff’s presentation at the May Agentics NYC meetup.

Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. More articles here. Learn more about how Nori can bring your company into the glorious AI future at norisessions.com.

Back at the end of March, we wrote about how we use coding agents to automate slide decks.

I like giving presentations. I hate making slide decks. I spend way too much time making slide decks. Most of that time isn’t actually that useful in terms of conveying information. It’s bs tasks like fiddling with positioning, aligning text, creating figures. Or it’s researching and pulling quotes or summarizing things that I already know but don’t have exactly ready for a deck. These things are necessary to have a good presentation -- I’ve seen many a research PhD slide deck to know otherwise -- but man all these little details bloat out to like 10 hours per deck.

We eventually figured out a hack to use coding agents to bring our average slide deck creation time to ~25 minutes of active work. The finished output is generally more aesthetic, more informative, and more accurate than anything we could do ourselves by hand. Certainly not in the same timeframe.

We wanted to see just how far we could push this framing. Could Claude Code make an entire video?

I was pretty impressed with how this whole experiment turned out! We wrote up a skill for all this here, in case you want to try doing the same.