The Useful Part of OpenAI DevDay is the Assistants API

Sam really undersold the most interesting part of the whole conference. Also: musings on the AI startup market

I. OpenAI is undoubtedly at the forefront of the massive AI boom that we’ve seen in the last three years, and they are single handedly responsible for making the first killer app for AI that made ‘programming’ and task automation a reality for millions of people world wide. Those people…mostly don’t care about the dev day updates? Actually to be honest most of those people didn’t know dev day was even happening.

As the name might suggest, dev day was fantastic for people who already identify as developers — for that audience, OpenAI had a bag of goodies. My personal favorites were JSON Mode and improvements on the Function API — these things make my life easier by removing a lot of validation logic that I have to wrap around every single GPT-4 call.

But unless I’m wildly misunderstanding what’s happening behind the scenes here, both of these features are things that weren’t too hard to build. In fact, I built some version of ‘make sure this call always returns valid JSON’ and ‘please only use the following API’ previously, before OpenAI supported them.

So life as a developer working with actual code has gotten marginally better. This is good, in that this does lower the barrier to entry for more people to play with OpenAI as a blackbox of intelligence. But I suspect many AI developers who are plugged into the various open source communities (LAION, Eleuther) are already well aware of tooling to support LLM validation, as well as ways to get to other things that were announced, like multimodal support, DALLE-3 support, etc.

II. You’ll notice that I haven’t yet talked about the AI Assistants API.

Unlike the rest of the features that were announced, the AI Assistants API (and, as an extension, GPTs) stand out as features that are geared towards LLM-native programmers instead of traditional developers. These features aren’t really about making it easier to integrate OpenAI into Python, or whatever. Instead, they are about making it easier for devs to run all computation within the LLM environment. The LLM-native developer can program each LLM agent using prompts, done entirely in English (and through the GPTs configuration UI).

I…kinda hate how these APIs are being framed? These are massively important tools and they are being dramatically undersold.

Quoting from the blog post on the Assistants API:

This API is designed for flexibility; use cases range from a natural language-based data analysis app, a coding assistant, an AI-powered vacation planner, a voice-controlled DJ, a smart visual canvas

Jesus fucking Christ, Sam, these are not the interesting use cases of the Assistants API.

There’s been a lot of research and projects that show that multiple agents (using the term loosely) working together can do a better job solving higher order tasks than individual agents. Or, more concretely, a network of LLMs¹ is smarter than a single LLM. In some ways this should be intuitive, but it’s tricky to nail how the network should communicate and what each node should be prompted with.

That said, the LLM-native programmers I know are very aware of the value of having an LLM network — they are constantly asking ChatGPT to simulate multiple agents internally in order to get a wide range of actionable advice.

In my mind, the Assistants API is cool because it makes this pipeline much easier. You should be able to set up a network of assistants that can just talk to each other — each one responsible for a specific programmatic responsibility — that in aggregate can perform miracles. Like Voyager! And if your underlying models happen to improve, great, the network should perform better still.

Maybe we will in fact see a flood of one-off chat-bot applications. I kind of hope not? I think that chat bots are pretty frustrating for me, personally, and I’d hate if every single service site decided to put me through a chat API.

But I am psyched for the team that is able to use this API to create a generic high level problem solving agent, an abstraction of Voyager. Something that can take any task and generate sub-tasks, write code to solve those sub-tasks, store that code for later as a long term memory bank.

An army of AI agents masquerading as one. Recursively solving problems, recursively self improving, without any additional training.

That is the promise of the Assistants API.

I’m optimistic that there are a few other people out there thinking the way I am about this, because there is some research in this direction. But I want the LLM-native programmers to be the people leading the charge here, if only because there are 10x more of them than there are traditional developers, and their inability to program other languages means they have to evolve tricks and workarounds with the limitations of LLMs as compute engines. That’s not going to happen if

a. most of them don’t know about dev day to start

b. they get pitched a version of the tool that is all about vacation planners, or whatever.

III. Misc other thoughts.

People on a certain part of Twitter love to talk about how xyz startup is dead because OpenAI released some feature. And of course in the wake of Dev Day the usual suspects did the routine of pointing out YC companies that are now ‘dead’. Maybe at some point I’ll write a big post about information arbitrage, but for now I’ll give just a rough sketch.

• There are approximately everyone on the planet who could benefit from AI tools;

• The vast vast majority of them do not know that AI exists and may not even be aware of OpenAI;

• If you know about AI, you can sell literal magic to these people.

Every consumer-facing AI business right now is an execution play, not a tech play². The goal is ‘be the best sales team on the planet’ by understanding your niche and your customers way better than anyone else. The folks over at Jasper understand this very well. I’m certain their CEO doesn’t know anything about embeddings or how ML models work. Doesn’t matter. They make a killing because they know how to sell magic to people who want to buy magic, at a crazy markup.

OpenAI is unlikely to want to jump into every vertical in the world. It’s fine to be a thin wrapper over their API. And in that vein, I do think that OpenAI's announcements make it very easy for random people who aren't otherwise very technical to jump in and start making micro-businesses. Just, you know, be good at selling shit.

1. Note that these don’t need to literally be different LLMs. Often this is multiple prompts directed at a single LLM API, such as GPT4.

2. The joke is that every consumer facing business is always an execution play, but whatever