Today we're talking about LLM dev tools

If LLMs are a programming language, then we're going to need LLM devops

LLMs are Programming Languages

*Technically they are compilers for English, but I’m not here to quibble.

In my review of the Voyager paper (a presentation on the same concept, if that’s more your speed), I said:

A computer program is a set of instructions that a computer can execute. Computers understand bytecode. Bytecode is a small subset of commands that perform basic mathematical operations and manipulate the storage of numbers.Over time, people developed programming languages. These languages are generally easier to read than bytecode, but through a 'compilation' step, they can turn into bytecode
...
I think LLMs as we currently know them are already a higher level programming language. The LLM will take natural language (i.e. English), compile it into a Pytorch representation, which gets compiled into Python, which calls out to C, which (say it with me) compiles into assembly which compiles into bytecode. LLMs allow us to make English directly interpretable to a computer.

Later, in Part 2 of the same review, I said:

Going back to Part 1, programming languages are about making tasks easier. Each higher level programming language bridges the gap between how we think about a problem and how we describe it to a computer.

That gap has slowly closed. And with LLMs, I think that gap is almost 0. That in turn means the barrier to programming has dropped significantly. 

...
In a world where everyone can program and automate aspects of their lives, we should see a massive increase in diversity of tools, products, and (in the long run) efficiency. And of course, more efficient outcomes begets more investment in tools and products, creating a virtuous cycle.

So the kicker of tools like LLMs is that there should be way more people who are going to be 'programming' with large language models. Well, we’re coming up on the one year mark of ChatGPT, and when I look around I think we already do see countless people who use ChatGPT and other LLMs in interesting ways to accomplish and automate tasks — things that they would have previously required a developer to build. This random link says that there are 3x more Javascript developers than C developers. Maybe there are already 10x more 'LLM-code' developers, compared to Javascript developers?

The funny thing is that this entire growing labor market of developers doesn’t even identify with being developers. They are artists, creatives, executives, teachers, police…anything else, but not a developer. A bunch of them are scared of technology!

Which makes the LLM dev tool industry pretty fascinating to watch.

What are LLM dev tools?

Even though every programmer is essentially typing text onto a computer, they aren’t using some nice interface that looks like a Google Doc or a MS Word Doc. No, they’re using some nonsense like this:

Welcome to the world of dev tools — tools, for devs.

Dev tools make the life of a developer manageable. They help us deal with complexity and layers of abstraction so that we can focus on solving problems instead of munging with syntax.

Dev tools are everywhere.

• We have tools that help us write code faster — IDEs like VSCode, Sublime Text, and (my fave) Vim.

• We have tools that help us store the code, publicize it, and make it searchable for later — source control like Github and Gitlab.

• We have tools that help us install and manage other people’s code — package managers like pip or npm.

• We have tools to help us avoid bugs and improve collaboration by automatically running style checks and tests — CI tools like TravisCI, CircleCI, and a wide range of plugins and helpers like eslint or pytest.

• We have an infinite array of tools to help us understand how our code runs when it’s live — observability and metrics tooling like Google Analytics and DataDog.

The list goes on.

Folks who are programming in LLM world…aren’t using any of this. They’re using fucking HTML textboxes on websites and copy pasting prompt hacks in emails. Which means there is a huge, wide open opportunity for companies to sell dev tools to them. But how do you sell to devs who don’t think of themselves as devs?

So far, the answer is: you don’t!

I think basically every startup in this category that I have seen thus far has been more or less doing CI, versioning, and observability for enterprise. Which is interesting, I guess? But it's probably worth looking at how these dynamics played out in the developer market to get a sense of how they might play out in the 'new' LLM developer market.

• In the versioning world, there’s Github. Yes, Gitlab and Atlassian and Perforce and others exist. But this random link says Github has a 84% market share. No idea if that’s true, but it sure as hell feels true.

• In the CI world, there used to be a bunch of options, but I feel like Github Actions is eating a lot of people’s lunch. This other random link agrees, with Github Actions as the most popular CI option (Jenkins is apparently close behind, I’ve heard of it but can’t say I’ve ever used it).

• Observability apparently has a lot of competition, but anecdotally I feel it is pretty heavily dominated by DataDog and AWS CloudWatch — the latter really only because everyone uses AWS.

Meanwhile, open source options abound to fill in the gaps that are hard to monetize, like linters, test suites, and package managers. Even there, though, each language only really has one or two options — black/flake8, pytest, pip/conda for Python; eslint, jest, npm for Javascript, and so on.

So what can we learn from this in the LLM market?

My hunch is that there will be basically one or two winners in each category. Which means that most of the startups in this category are going to consolidate through acquisition, or getting out competed at the margins by someone scrappier (or better capitalized). Existing players may also just outright win — for example, DataDog claims to have generative AI monitoring — though they have to climb the uphill battle of rebranding for non-developer-developers. Either way, better pick winners soon! I don’t imagine the market having room for too many players.

A few misc other thoughts:

• Like I mentioned, it's funny to see how these companies are marketing. They have to essentially sell dev tools to people who do not identify as developers, who may not even realize they need these tools! Interesting to watch these companies dance around this, trying to avoid alienating both actual devs and this new category of developers -- sort of like watching a teacher explain to a student why they shouldn’t just write their code in notepad.

• It's also worth identifying places where existing developer tooling companies can easily subsume and meet the needs for this new category of LLM developers. Github maybe can't, because using Github requires an esoteric understanding of version control to use, and it’s branding is already very highly tied to ‘developers’; or maybe it can, if it manages to integrate a very very user friendly GUI for the LLM developer crowd.

• Devtools seem to come in waves. They take time to adopt, then they're everywhere. I think that wave pattern is much more true when discussing a market segment that doesn't even think of themselves as developers.

• The really exciting companies in this space, for me, are the ones that try to sell to developers as a consumer -- those companies are much more risky, but also the ones that may have a shot at being the next Github. Github in particular won by being essentially a form of social media vertical. It was free and allowed easy collaboration, which in turn led to it’s integration in every serious CS school curriculum and hobby project in the world. Github only started in 2008!