Notes from an AI Conference: What are the AI folks up to these days?
Thoughts and themes from AI Engineering World’s Fair
Note: we are hosting our second in person Agentics event (and our first bi-weekly) at The Brass Factory on July 22nd. We’ll have speakers, food, and hopefully some boardgames too. Sign up here: https://luma.com/agentics-qz0j
Do you ever look out at the beautiful Manhattan skyline, bagel in one hand, pizza slice in the other, and wonder “What are those guys who are doing the AI thing in the Bay up to these days?” They’re always saying all sorts of weird and crazy things on the bird app that no one else but the AI people use anymore.1 Probably worth checking in.
Now, cards on the table, I generally think of myself as one of those people who is doing the AI thing, on account of being the founder of an AI company and having been in ml research for years before this2 and also writing copiously on the subject of AI things. But I mostly avoid the bird app, and I don’t live in SF, so sometimes I do get some FOMO.
I’d heard about the AI Engineering conference a few times — a few friends at Google were previous sponsors and told me it was a good place to catch up with other folks who were thinking about coding agents on the cutting edge. Applied to give a talk, got in on the online track, and the organizers very kindly sent over a few free tickets by mistake.
Just got back from the Bay, FOMO eradicated. Here are my thoughts.
At a high level, the talks were split 85-15. 85% of the people there were enterprise folks who were trying to learn how to AI good. These people were interested in attending and giving talks about how to leverage more AI in their respective organizations. The other 15% of the people there were really on the bleeding edge, people who had plumbed the depths of lights off software factories and came back to tell of Lovecraftian horrors lying beneath (hi Dex!). These guys mostly were looking for less AI. I was expecting way more of camp two than camp one, and was somewhat surprised at the skew in the other direction. Even in this highly selected environment — the ~7000 people most likely to attend an “ai engineering” conference — the vast majority of people don’t really know how to use AI effectively.
Every month the AI world is onto something new. January was takeoff, February was ‘what is a skill’, March was ‘everyone has to use Claude Code all the time’, April was get the ops team on board, May was tokenmaxxing.
Apparently, June was the month of skill sprawl. There were a dozen talks that were variants of “how do we manage the 1000 skills that people have created in our organization?” These were inevitably packed to the brim, which should give a sense of where most of the market is. Over at Nori we solved this problem back in December (see: noriskillsets.com, a team wide package manager built on our open source local skills manager client).
Many of these ‘skill management’ talks landed flat for me. I think having that many skills is a ‘code smell’. In my experience the agents are very good at figuring things out from first principles. Having a skill that 1) will drift, 2) may not actually add anything new, 3) needs to be managed is more likely to create problems rather than solve them. Bluntly, I think many of the teams who are grappling with hundreds of skill files in their organizations are dealing with an employee base that is happily having the coding agents generate skills for all sorts of things without rigorously figuring out whether they are useful. This gives the veneer of “AI enabled” work without actually being so. I wrote about some of this here:
I was also disappointed at how many talks were about some form of self-learning that involved having agents read through transcripts and automatically create skills, so much so that I wrote about it. Anything that does not have a human in the loop for context management is going to quickly spiral out of control, which is why we end up with so many skills in the first place.
In my opinion, you cannot really rely on the agents to write good skills for you. The process needs to be intentional. There’s a particular format that we find works. We like to bundle sets of skills into specific groups instead of just having a big pile of them with a search server on top. Every prompt with an agent is like summoning an entity from the void. The particular skills that are bundled with the agent are a big part of eking out the highest quality results. Trying to slopify the agent brain will result in agents that produce slop. Don’t do it.
This was also the month of evals. Evals are something of an interesting beast; as with everything in AI, the term is wildly overloaded.
Some people (mostly researchers and model providers) use “evals” to mean “benchmarks that we can use to evaluate raw model performance, that the industry can hill-climb against to make models better overall.” SWE-Bench is an eval, and Humanity’s Last Exam is an eval, and ARC-AGI is an eval. These evals are artificial, designed to give researchers a metric to improve model training through simulated, verifiable tasks, and designed to give the industry a quick read on the relative capabilities of these models.
Other people (mostly folks in industry who are deploying these models) use “evals” to mean “regression tests that we can use to try and make production workflows as deterministic as possible.” Here, an eval is extremely task specific — things like “does the AI correctly label data” or “will it correctly respond to adversarial chat input in the customer service chat window?”
Even though the former use of the term is more relevant for folks on the cutting edge, I think the latter is what a lot of the larger enterprises are thinking about, and as a result were what much of the audience was thinking about.
It’s worth taking a step back and thinking about how AI is landing in different orgs. At tech companies and startups, folks primarily interact with AI as a developer tool. Cursor and, later, Claude Code are interfaces that a dev spends all day in. The average developer at these companies is, as a result fairly intimately aware of what AI can do, simply because they interact with it so much.
That is not how AI has landed in other industries like banks and retail. In organizations where engineering is not obviously a critical function — places where engineering is more a cost center than where the money gets made — the primary way AI gets used is as infrastructure. The most common shape is the ETL pipeline, a recurring process that Extracts data from some table, Transforms it somehow, and then Loads it into another table. These sorts of tasks are a dime a dozen at any large enterprise, and less tech-forward companies will pay a lot of money to external consultants to build these out. Traditionally, ETL pipelines are brittle and required a lot of fiddling with APIs and different fields and so on. LLMs, with their inherent flexibility, promised to massively simplify that effort.
New technology never gets adopted wholesale, especially among larger slow moving companies with a lot to lose. Instead, new technology comes in as a supplement or a replacement for something that already exists. That’s a large part of why AI landed in ETL world for so many companies — because that was the best fitting preexisting shape to slot it into. Unfortunately, if you’ve ever worked with ETLs, you’ll know that you can’t just, like, screw around with the data. These pipelines need to be consistent and auditable, and an AI agent is neither.
Which brings us back to evals.
In large tech companies, tokenmaxxing meant ‘give individual developers unlimited budgets for things like Claude Code’. Everywhere else, tokenmaxxing meant ‘let’s throw more tokens at these failing ETL pipelines to get them from 80% correct to 100% correct’. How do you do that with AI? Well, one naive way is to build another AI system that checks the output of the first. But what if that one fails? Well, you could build a third system that checks the output of the second.
You see the problem.
Now that tokenmaxxing is dead, some of these less tech enabled firms are limiting employees to ~$50-$100 token spend per month. At full unsubsidized price, that’s barely anything at all, but then again most of these teams haven’t exactly been using Claude Code or Codex anyway.
I mention all of this mostly because I find it interesting to see the gap between SF and everyone else. I’ve said in the past that SF is in a bubble, and that was very apparent during AIE. The evals track talks were all about researchers and labs trying to figure out better RL environments, but a lot of the people in the audience were like “that’s great but tell me how to make my ETLs not be bad.”
As an aside, I think there isn’t really a good way to make these things consistent / auditable, nor should you really try. The long tail is too long. Instead, have a human in the loop. AI that goes 95% of the way and has a human approve or not has, in my experience, been way more successful than any AI system that attempts to go 100% of the way.
The folks who are really on the cutting edge spent a lot of time talking about background agents / cloud agents, and the ten-million reasons why people should use them.
Now, I recognize that as the founder/ceo of a company that is selling a background agent stack, I am a bit biased here, but it seriously felt like a full half of the talks at AIE were shilling for our product. There were talks about why you need background agents for productivity, and talks about why you need background agents for security, and talks about why you need background agents for compliance. There were even talks about why you need background agents for AI enablement! I’d say a solid half of the security track, all of the sandbox track, and many many other talks (including keynotes) were about moving things from local machines to the cloud.
(This post is brought to you by Nori! When you think “background agent” or “cloud agent,” think Nori!)
If you are unfamiliar, the name basically tells you whatever you need to know: background agents / cloud agents are agents like Claude Code that run in the cloud, in the background.
Why might you want this? Well, when Claude Code is running locally on your computer, it is:
Stuck on your machine. If you close your laptop or walk away from it you cannot (easily) take the conversation with you on the go.
A massive security hole. Everything on your computer is fair game. There is a fundamental tension between usefulness and risk, and unfortunately besides a few very very stressed out CISOs people are not thinking about this trade off at all.
Unable to easily coordinate and collaborate. If you want to try and run multiple agents in the same code repository, you have to use tooling like git worktrees. If you run too many agents at once, they will happily eat all of your computer memory and kill the computer.
All of these problems are because the coding agents live on your computer, so…
Cloud agents neatly solve all three of these issues and a whole host of other things. The openclaw folks already showed how being able to access agents from anywhere was massively useful. Because they are centralized, you get massive leverage on security, observability, governance, and all those other lovely enterprise buzzwords. And cloud agents are a necessary step towards the ‘fully automated business’, because, by virtue of being not-on-your-computer, they can respond to events and timers and do things automatically.
It turns out this is an extremely powerful primitive to have. It’s a universal automator — you can literally fire off any event to this thing and have Claude respond, and Claude is a PhD level genius. We use background agents for everything from in-slack coding to company brain management to automatically responding to bug reports to onboarding new users.
These are still relatively new, but in ~6 months I think the market for these cloud runtimes is going to be really really hot.
The other big conversation topic among the teams on the cutting edge was about loops, software factories, and lights-off codebases.3 Or, well, that was ostensibly what people were talking about, but what they were actually talking about was something closer to “do we still need humans to do this stuff or are we all screwed?”
In some sense, a “loop” is a hack. LLMs suffer from context rot — the longer you talk to an LLM, the dumber it gets, until you hit a hard cap where you just can’t do anything with it anymore. It is a massive limitation, and a ton of “cutting edge” AI infrastructure is about solving this problem.
The loop is a very simple idea: what if you gave an LLM a big task and a scratchpad, and instead of having it try and finish the task in one shot, it takes on a small piece of the task and only does that small piece, and then writes out what it did. Then the thing loops to do it again. And again and again until it’s done.
while :; do cat PROMPT.md | claude-code ; donePeople have been talking about loops for a while. The “Ralph Wiggum loop” has been in the water supply since last year. Karpathy’s auto research work is slightly newer, AlphaEvolve is slightly older, but they’re all basically about the same thing, which is “can you exploit the fact that LLMs never tire to turn every problem into a brute force search problem?”
I’ve written about this before, back in October:
I started using agents effectively when I began to see code as a search problem. Agents are like our chess players above. At each turn in a conversation or agentic loop, they make a decision. That decision takes them to a new state in the search tree, and the process repeats. Agents can go any which way down the search tree. Sometimes they will start on the right path and then veer off track. Other times, they will be wrong from the start. If you are using a coding agent, your job is to help it narrow down the search tree.
…
As a software engineer using a coding agent, your goal is not to get the best single shot code output. Your goal is to simply get the best code, period. If coding is a search problem, a very easy way to improve your code is by running many searches at once. So do that!
This raises some interesting questions. Is…software engineering a brute force search problem? What about, say, running a company? Can you be like “fable make me a billion dollars” in a loop and just sit back drinking mimosas in Fiji until it succeeds?
Several startups have staked their business on the answer being yes. And Matt Levine has joked that this is obviously the right business model for the big labs:
It really is the greatest business plan in the history of capitalism: “We will create God and then ask it for money.” Perfect in its simplicity.
But back at AIE, the premier AI conference where everyone was basically obligated to be AI-pilled, the general consensus was “eh, not yet.”
The agents are good, and Fable is really good, but a 3 month old codebase is just a totally different beast than a 1 year old codebase, and both of those feel really really different when people are actually out there using your tooling and you have random backwards compatibility requirements that you have to remember (or you go to jail, or at least lose a big contract), o, and yea, memory is something that the agents are uniquely really terrible at. Most of the folks who were defending ‘loopsmaxxing’ were doing so somewhat aspirationally, acknowledging that the loop-based-tooling wasn’t all there right now but would get there eventually.
For what it’s worth, at Nori we are running a “hybrid” software factory codebase. The entire codebase is agent written. I am definitely not reading most of the PRs that are being shipped. But also, I have a pretty good mental model of how the code is structured. We have regular daily tasks that automatically look for tests to clean up or abstractions to define (or destroy), like little cleaner fish that are just swimming around looking for things to fix up. And roughly once a month, we all end up doing a big chunky refactor of some part of the codebase that has gotten too hairy, often because we’ll eyeball a PR and go ‘wait why is <small feature> so many LoC changed?’ and then dig in with an agent to figure out if there is an abstraction missing somewhere.
Isn’t that kinda just how normal swe works, but on steroids? You build stuff, and you build more stuff, and eventually you look at the stuff you built and refactor a few things to make it easier to build more stuff on top?
Some people argue that when using agents, no one really has any deep model of the codebase, the kind where you would know a priori what files and functions need to be changed to make a new feature work without regressions. Which is true.
But I think that is true of basically every codebase, modulo time and size. How much of Google’s massive codebase did anyone really have in their head? Jeff Dean is superhuman and even he probably only knows like 0.01% of that thing. Which is to say, it’s worth investing in things that make the codebase legible — it will probably make the models better too. And beyond that, any maximalist position of ‘only use agents never look at code’ to ‘never use agents only look at code’ is wrong.
I think the loops will get better. A good friend of mine at one of the companies that had early access to GPT 5.6 Sol was telling me about these crazy subagent harnesses, where Sol was coordinating a bunch of Opus agents which were coordinating a bunch of GLM agents. Seems really impressive. But it’s also worth mentioning that he was overall unimpressed by the models’ collective ability to do things without a human.
This all sorta brings me back to the evals thread from earlier. I think the right architecture is ‘automated artifact, human review’. Our definition of what an ‘artifact’ is will continue changing as the models get better, but you need that human in the loop somewhere, either for the initial prompt or the post hoc analysis.
A few other miscellaneous thoughts.
As usual, there was a lot of chatter about agents running other agents and agent swarms and things like this. Reading the tea leaves a bit, the most chaotic versions of this approach (“All the patrol agents Refinery, Witness, Deacon, Polecats — create wisp molecules for every single patrol or workflow run“) have somewhat failed to pass muster. I think the agent swarm thing was always somewhat linked to the loopsmaxxing tokenmaxxing software factory thing. The biggest problem with software factories is that agents are not very good at high level decision making, things like “what should I prioritize” and “what are the right trade-offs to make” and “what is the right way to break up a complex task into smaller interlocking useful pieces.” The agent swarm promised to solve these problems. I think the theory was something like “agents can handle low level decision making, so maybe all they need to handle high level decision making is a bunch of grunts to do the low level stuff and a metric ton of tokens.” Unfortunately, this framing is obviously flipping causality. I think a lot of the agent swarm hype has become more about orchestrating targeted subagents for things like research or code review, something I am / was already much more bullish about.
A lot of people waking up to the security risks of having these little imps running around on every computer in a company. Half the talks were about putting the fear of God into various CISOs.
“Multiplayer” was a surprisingly big topic of conversation, driven in part by Anthropic pushing their new Claude Tag slack bot in ~all of their talks. The basic idea is that when you have a lot of people collaborating with an agent at the same time, eg through slack, the experience of writing software — or even running a company — becomes fundamentally more collaborative. As someone who is selling a no lock-in version of Claude Tag, yea, I basically agree. Having a really good integrated AI setup is transformative, in part because it meaningfully changes everyone’s relationship to engineering. These tools actually remind me a lot of Figma. When Figma came out, it was obviously a great wireframing tool, but the real magic of the system was how it allowed everyone to participate in the design process because it was multiplayer by default. Even though the form factor is totally different, tools like Nori or Claude Tag have the same effect on engineering as Figma did on design. Suddenly, everyone can participate in the engineering process, no arcane runes required. This is, again, another pitch for background agents generally.
One thing I disagreed with most strongly was the idea that everyone in an org should figure out how to use AI ‘properly’. I get the impulse. As AI adoption sky rockets, knowing how to move around an LLM will increasingly just be part of the job. But there is so much behind getting an AI agent to “work” in a business setting that I have a hard time imagining that some day job 9-5 guy is really going to spend the time figuring out how to do it right. It doesn’t make sense to have, like, your sales guy figure out how to properly do security for agents running on his laptop. He’s just going to do whatever works as fast as possible to get back on the phone selling, which means it’s up to the rest of the company to make the fastest path the happy path. To steal a quote from Dex, it doesn’t make sense to have everyone in your org figure out how to use Kubernetes even if it’s really important, why would it make sense to have everyone in your org figure out how to run agents properly?
There was a whole bunch of stuff on open source LLMs (GLM 5.2 was a particularly large topic of conversation), auto research, robotics, and model training that I unfortunately missed. Lots of talks, maybe too many. I was hoping to see more researchers but it was very much more an industry conference than an academic one.
Some of my favorite parts of the conference were outside the conference. There were a bunch of sponsored side events, including one great one from Modal (thanks Monishee!), but I’m mostly talking about catching up with other folks who happened to read the blog and invited me to come hang. Office Hours with the LLM whisperers was very fun (thanks for having me deepfates), so was dinner with the botco crew while watching the world cup. On the flip side, things were so busy that I didn’t get a chance to really see my usual Bay area crowd — if I missed you this trip, big apologies, and I will see you next time. For everyone else I met, hit me up if you’re ever in NYC!
Agentics is the study of how to use and reason about agents. If you are an expert in coding agents, or interested in learning more about agents, join our community slack. We organize in person meetups on our luma calendar. More articles about AI here.
If you are trying to build background agents, we can help! We build white-labeled custom background agent infra that work like Ramp Inspect or Claude Tag, with no lock in at any part of the stack. Check out noriagentic.com for more.
except for some unsavory types but I won’t talk about them
I tell people I used to train models and they go “o you do traditional AI” like, no, traditional AI is gbdts and support vector machines, also that made me feel way older than anything my nieces and nephews could come up with.
named after lights off factories, which purposely do not have lights because no humans work there
pleasure to meet you! glad you came by