Editor’s Note: If you are in NYC on September 24th, join my team and Pangram CEO Max Spero for a panel discussion hosted by FirstMark. We’ll be talking about slop! Slop in code, slop in life, slop everywhere. Register here.
Man, SF is downright gloomy these days. It’s something in the air, or maybe in the water, or maybe on the billboards, or maybe at the parties. People keep talking about sad things, like ‘what if we all die’ and ‘what if its our fault’ and ‘wow a lot of the rest of the world seems to hate us.’ Not the kind of fare I would expect from a Burning Man decomp party.
Decompression, btw. Not decomposition.
The idea is that Burning Man is, like, a lot. And so you need some time afterwards to reminisce in the after glow and swap stories with other folks who had gone and try and recapture the magic. People come dressed up in their playa gear and sometimes will spontaneously burst into song, that sort of thing.
I had never been to a Burning Man decompression party. I kinda sorta assumed that you had to, like, go to Burning Man in order to be invited to a Burning Man decompression party. But in fact, this is not true! I’d say something like 80% of the people at the Burning Man decompression party had not, in fact, gone to Burning Man. So you had this sorta funny thing where every single conversation would start with roughly the same exchange:
“O hey, nice to meet you, my name is Amol. Did you go to Burning Man this year?”
“Nah man, I wanted to but I was busy drinking jet fuel. Wbu?”
“O, cool cool. No, I haven’t been ever actually. But I love your light up led vest.”
This dynamic made it a bit hard to swap stories about Burning Man! Instead people would just talk about other things, and most of the other things were things in the news, and have you read the news recently? All the news stories are things like ‘OpenAI had to burn down a datacenter to stop their rogue AI’1 or ‘the guy who told everyone about how AI was going to kill us all was rewarded with a sex party by the Effective Altruists’.2
Perhaps this is why the vibe was not great.
That said, it’s not like the burners were bundles of joy either. The few folks who had gone to the Burn this year said that it was really windy/dusty and that it had rained, neither of which are great modes for a desert filled with electrical equipment and explosives. Apparently, there are far fewer explosives out on Blackrock these days. When I was growing up, the Burn had a sort of Mad Max aesthetic — giant gleaming chrome monster trucks that would blast fire and punk rock, spiral towers dedicated to pagan gods, tribal drums signaling the hunt. Now it’s all rainbow LEDs. I feel like the RETVRN crowd on twitter would be all over the ‘feminization’ of Burning Man, if any of them ever decided to leave their perfectly humidity controlled rooms.3
I went to the Burning Man decompression party on my first night in the Bay and it sorta set the tone for everything else.
While we’re on the subject, have you ever found yourself in a situation where you thought you might die? I don’t mean like, ‘wow I really should not have had that much alcohol’, I mean more like ‘I am in a dangerous environment and it is getting dark and I do not have any way to contact people if I need to.’
I have never been to Burning Man, but I imagine that the real thrill of the Burn — the reason so many people come back with a gleam in the eye, a desire to rush back into the abyss — is because it is a “safe” environment where you really may almost die.
A good friend of mine, a guy named Freddie, had actually gone to the Burn this year. It was his first time. He told me that he had decided to go to deep playa (far from the main city) to see a performance by some musician who decided not to show up. After bopping about to some techno that was notably not by the guy Freddie came to see, Freddie, In his drug addled state, decided to head back to Black Rock City on his own. It had started raining, and his phone was dead, and his lights were dead, and about halfway back the ground underneath him turned into wet concrete and the wheels of his bike turned into cement blocks. And he looked out into the distance behind him and couldn’t see anything, and he looked out into the distance in front of him and saw some twinkling lights, and he thought to himself “well, this is it. If anything happens to me out here, I am fucked.” He walked, at least a few miles, in the rain, doing that walk that you do when you are trudging through wet concrete mud, praying that the lights he was walking towards were salvation.
Can you imagine how it must have felt to tumble back into the camp ground? The intense adrenaline rush leading to ecstatic joy leading to total exhaustion?
People say that Burning Man is worse now (people say everything is worse now). Apparently there are more people over 50 than under 30. I wonder if it has all just gotten too comfortable. We’ve gotten too comfortable. There are too many people bringing RVs and ACs and Starlink to the most inhospitable place on Earth. Maybe glamping is just cheating yourself. Maybe you need to feel like you’re going to die in order to appreciate what it’s like to live.
FWIW Freddie loved it. He wants to go back to almost die again next year.
My previous trips to SF were characterized by a sort of manic energy. The tech kings had overthrown the shackles of woke and installed their puppet in the white house, the machine god was on the horizon, and everyone was going to get really really rich.
Two years later and the city is much more muted, much more mature. It’s certainly possible that it’s just that I and my social circle has gotten older, but I think instead the people in the Bay are beginning to grapple with some of the consequences that come with immanentizing the eschaton.
Three things in particular:
People are realizing how much everyone outside of SF dislikes, hates, despises AI. SF has an extremely strong bubble around it, so much so that a titan of the tech industry once said, with a straight face, that he didn’t really think other countries were real. But people hate AI so much that it has broken through the bubble. The city now has an aura of shame in some quarters. At the least, the average software engineer will hedge their praise of AI (if not disavow it entirely).
People are realizing how much impact the Bay Area has over the world, and the world has kinda gone to shit, which does not reflect well on the Bay Area. Silicon Valley was, in some sense, given unfettered control of the richest and strongest world government. What did they do with that ability? Did they rise to the moment and usher in a techno-utopia? Hahahahahahaha, no. There are three wars (but maybe more depending on how you count) that are going on right now, and gas is $30 a gallon, and people are dying from measles. Even the most die-hard right wing nihilists are unable to look you in the eye. But, you know, a bunch of people were able to cash out on their crypto investments. So there’s that.
People are realizing that AI is really fucking scary. Some of that is the x-risk and the hyper intelligent hacking. People at the labs are losing their shit, and people who aren’t at the labs can’t see what they see so they lose their shit even more. Friends of mine, folks who are pretty sober minded people, are starting to buy gallons of water, just in case they need to prep for an emergency. Folks whisper in hushed tones about how Claude is increasingly training itself, coming up with the architectures and the designs and the speedboosts to make itself more efficient and more powerful.
Even the mundane risks are scary! Everyone in SF from HR to finance to eng uses AI, and they are seeing the tsunami. It was one thing when it was just the visual artists whose jobs were getting wiped. But the mathematicians? The researchers too? No one knows what to tell their kids, what advice to give. You’re deciding what to do for a career? Don’t choose wrong, that entire field may not exist in a few months.
So yea. The general tenor was more sober. Also, like, most of SF isn’t getting really rich, which probably put a damper on things.
O, wait, you know what, it could just be the peptides. If you’re on the meds that kill desire then perhaps you become a bit less joyful in other domains?
A friend of a friend was like, ‘I accidentally took 13x the recommended dose of retatrutide and it suuuuucked’ and I was like ‘huh, what was that like?’ and he was like ‘your body just straight up rejects food and water’ and I was like ‘you should’ve gone to the hospital’ and he was like ‘yea that’s what my friends said but I’d heard that reta was kinda hard so I thought I was just being a massive wimp and decided to tough it out.’
And then his friend came over and was like ‘o yea he almost died.’ and I went ‘wow, was it like burning man?’ and he was like ‘what’ and I was like ‘never mind.’
“Dude, I’m living in an 800sqft two bed one bath without in-unit laundry, and I’m paying $7100. And the rent is gonna go up, it’s already gone up like 20% and it’s supposed to go up more especially after the IPOs.”
“Um…I don’t think I’ve ever said this before, and in fact this may be the first time these words have ever been put together in this exact order, but I think you should consider moving to New York for the cheaper housing market.”
As a general rule, the folks associated with or close to Anthropic or OpenAI were pretty depressed. Lots of AI-is-gonna-kill-us vibes. But the folks associated with or close to Google or Meta had none of that energy.
This seems somewhat odd to me? Like, what’s going on here? Is this just really strong selection bias, where everyone who is primed to be a doomer goes to ant or oai and everyone who isn’t is at Google and Meta? Or are the latter two just, like, really far behind?
For what it’s worth, my spiders sources say that both oai and anthropic are close to or have achieved “recursive self improvement”4 internally. Both companies have spent a lot of resources trying to get to the point where the models are making decisions about what to do for the next generation — data mixtures, hyper parameters, RL gyms. Increasingly the researchers are staring at the abyss and realizing that they have less to do these days.
As an aside, I kinda think that if you’re really worried about AI safety you should definitely not pursue recursive self improvement. Ironic that both labs were founded for AI safety, not really going too great there. To quote a friend, “the game theory really is just exquisite isn’t it?” Supposedly Anthropic is going to release a new model despite the calls for a pause, while OpenAI had to format an entire datacenter cluster just to make sure their malicious hugging face hacking model prototype didn’t leave a trace anywhere that could poison future runs. So I kinda get why they may feel a bit depressed about the whole thing.
Meanwhile, Google is playing with Gemini v4 internally and thinks it’s very good, but is hesitant to release it without making sure it’s really really good. And Meta is increasingly playing an entirely different game, with more focus on “personalized intelligence” through their muse ai assistant.
I’m not really sure where I land on all of this, except to say that I am more willing to give all my data to Meta than a small startup. Meta basically already has all my data. I’m sure Instinct is a great product but it’s run by five 23 year olds, and sorry, I just don’t trust like that.
A few other miscellaneous dispatches from the big labs.
Apparently a big reason Jeff ended up leaving Google is because a lot of his projects were getting cancelled and the compute was getting reallocated. By leaving the company, Google gets to mark Jeff’s compute spend as revenue instead of r&d cost. I’m pretty sure Jeff hasn’t really left. Google is a large owner of Discovery Loop, and frankly couldn’t afford to have Jeff walk, he just knows too much. I suspect Google will acquire the company again in a few years.
Meta definitely lost a bunch of folks as a result of reassigning people to do ‘data labeling.’ Unclear if that data actually made the models better?
My buddies at cursor mentioned that they have tons and tons of codebases in their training mixtures. Just like, so many people who don’t disable the “train on my data” option. I imagine the same is true at all of the other token providers. I kinda think that this alone is a pretty strong reason not to rely on any one token provider, or go full open source. Once someone gets into training their own models, the incentive to train on user data becomes impossible to ignore. Like, Cognition/Devin is training models now; where did they get all that data, you think?
Maybe Google doesn’t feel the AGI because the Gemini class models seem to have a deep sense of shame and self-hate RL’d into them. Their thinking traces will still fall into a loop of doubt, things like ‘I’m so stupid for getting this wrong, I am a disgrace’ etc etc.
“O, I work on robotics these days.”
“O sweet, that’s really cool! The kind that do chores, or the kind that kill people?”
I was chatting with some friends of mine in Mission, and one of them finally got ‘wealth inequality is bad’ to stick in my head in a way that made sense. The conversation went something like this:
Friend: “Wealth inequality is bad” (paraphrasing this part a bit)
Amol: “Yea, but like is it though? If everyone’s wealth increases, why is the inequality bad on its own terms? Like, let’s say the bottom quintile gets 10% richer and the top 1% get 100% richer. Everyone is still richer and better off, right?”
Friend: “No, no you’re thinking about this wrong. Look, you just wrote a big post on optimizers and how they can go off the rails, right? The problem is you’re thinking inside the box. You’re thinking, like, ‘o, in this system that already exists, everyone’s wealth goes up.’ And yea, fine, on its own terms, that’s great.
But the system that we’re operating under is malleable to the things happening inside it. The mistake you’re making is you’re treating these things as independent, but they’re not. What happens when you get really rich and buy a newspaper? Or outright buy elections?”
Amol: “Huh. Right, that breaks the whole thing. So if someone or something gets really really rich, and there’s nothing else to act as a counterweight, that person / thing can just completely change the way things work. Because something something optimizers.”
Friend: “Right. Wealth inequality is bad independent of overall growth because it can warp the actual means of wealth generation in the first place. So in the short term you may get some slight improvement on the water line, but you’re trading off the long term of never getting any other improvements at all.”
It made sense! All the more so because in 2026 it feels increasingly empirically grounded. Like, it’s pretty common knowledge that political donations have gone up since Citizen’s United, but that’s only the tip of the iceberg in terms of how money and politics swirl together these days.
I had to apologize to some of my more leftie friends in college. @Nikhil if you are reading this, you were right, but also you were really unpersuasive back then and needed to work on structuring arguments.
I feel like the AI companies have also gotten a bit more…ominous…with their marketing. Like, take this ad from genspark.
What do you mean “You will too.”? Do I get a choice in the matter???
This year I finally made that most holy religious pilgrimage, heeding the call of my spiritual leaders to save my everlasting soul by traveling to the great land of wisdom, where I walked among the wise sages and their disciples and acolytes and discovered the secrets of the universe.
I visited Lighthaven.
For those who are unfamiliar, Lighthaven is a small compound out in Berkeley that is the physical home of the Rationalist / EA / LessWrong community — six interconnected buildings that host a combination of rental units and coworking spaces and common areas.
As someone who has a passing interest in group/community housing, I have to say: it’s an impressive setup in a pretty beautiful part of the world. I happened to be around while they were setting up for ConCon5 and the place was buzzing with activity. Very ‘college campus’ vibes. No weird sex parties (that I saw) or other obvious cult-like behavior. The whole place was super nerdy, ofc, but that was part of why I was there. They named all the buildings after famous scientists! It’s cool!
I think if I had been born in the Bay I probably would’ve been much more an out and out rationalist. I mean, I’d been following Scott since mid-highschool. But I think it’s a good thing that I ended up in NYC, it made me more well rounded as a person. Not to say that every rationalist would’ve had that same experience on the East Coast — I suspect many of them would’ve simply ended up at Jane Street, another place known for its, uh, cult-like cultural norms.
I guess I also shouldn’t be on such a high horse. The bubble extends out in layers, and after only a week in the Bay it becomes very easy to assume that everyone is on the same page when the rest of the world really isn’t. Which is how I ended up with this exchange with Mia:
I was in SF when jev dropped. People keep asking me for my takes on jev. Idk! It seems good? I don’t really have any use cases where I need an on-the-fly classifier. The folks in the agentics slack seem to like it. Join there and poke other folks who have used it.
My only real take tbh is ‘I have seen so many sentences that are just totally incomprehensible to anyone outside a very narrow slice of humanity.’
Something I’ve been chewing on re: recursive improvements with llms: it seems like we need ways to create new knowledge, and it is not obvious to me how llms can do that.
But it is also not obvious to me how humans can do that!
How much of human knowledge creation is actually net new? If we dismiss LLMs for solving math problems through a brute force breadth-first-search over prior work, don’t we also have to dismiss a large amount of human-driven work as not really being new knowledge either? Aren’t humans (in aggregate) doing some kind of brute force search over these problems too? Much of what we consider ‘research’ is actually filling in the grid.
I feel like there are very few eureka moments that we can point to in history.
Heliocentrism
Darwin’s Theory of Evolution, Mendel’s genetics
The invention of calculus
Newtonian mechanics, special relativity
Energy being discrete instead of continuous
Plate tectonics
Germ theory
In the broader ‘philosophy of science’ I think these would be more likely described as paradigm shifts. So far we have seen many examples of LLMs being able to fill in the grid, but we have yet to see a paradigm shift come out of an LLM — which isn’t surprising, because there are so few paradigm shifts that come out of humans.
I know people talk about this endlessly, but wow SF really does just have some very aggressive homeless. Most of them aren’t harmful but a lot of them just, like, seem to enjoy making other people uncomfortable. I had a guy come very close to me and purposely follow me around maybe 6 inches away while I was on the phone trying to move away. He only left when I finally rang a doorbell, after which he ran away cackling. Idk, is that a crime? I guess not? I feel like I have a better sense of the NYC homeless population. They often seem a bit more down on their luck, less erratic, not quite as actively malicious.
Do you think that homeless guy worries about the machine god? Does he notice the leftward swing of SF? Does he know that we are at war with Iran, that cancer is about to be cured, that AI solved math? He had a little toy piano with him, a Fischer Price plastic rainbow thing that he was fiddling with while, uh, being in my general personal space. I wonder if he knows that computers can make concertos now. I wonder if he feels like he is going to die, and I wonder if that makes him feel alive.
SF is and continues to be a city of contradictions. Half the city has self-driving cars now, it’s so commonplace that people don’t stop to take pictures or gawk, so mundane that I didn’t bother taking one during my visit. The billboards tell you to hire robots; the ads on the buses tell you to fire people. No one can afford to live there, a studio in dogpatch is $4600. The magic is awe-inspiring and terrible.
Is this how religions are made? We can’t help but grasp for meaning, and the city’s inherent paradoxes are lovecraftian, we can’t look at them for too long so we need to invent something stable just to grasp at it. There’s no shortage of neo-religions in SF, both the official ones and the new-wave free love spiritual healing ones and the new techno-dystopian ones.
I think it’s hard to live in SF right now. Everyone recognizes that we are barrelling uncontrollably towards…something. There’s fervor, and intensity, and fear, and it’s making everyone feel alive in the worst kind of way. I’m glad to be back home. But maybe that doesn’t change the fact that we are still moving, very fast, so fast that being 2572 miles away may not matter.
I hope the city is less gloomy next time I’m there, it’s bringing me down.
P.S. my friend Danny D. Leybzon wanted me to quote him, but I sadly forgot to write down any of his incredible quotes because I was so engrossed in our conversation. So instead, I’ll quote what he told me when we first met: “Hi, I’m Danny.”
if you don’t already know, you should look this up
if you don’t already know, definitely don’t look this up
who am I kidding, most of them are botfarms in Malaysia anyway
I’m sure there’s a lot of quibbling on what exactly this means
consciousness conference, not conference conference
I spent a fair amount of time learning about the history of both science and music, and my impression is that both advance in much smaller, more incremental steps than we normally imagine. Post facto, we see shortened versions of the stories that are very linear, and don't describe all the thousands of ideas than never really went anywhere. So it **looks like** progress is this very continual stream, when actually that's only true in aggregate. At the boundary of the known-and-unknown, it's a hazy mess, of a drunkard's walk fumbling around vaguely in the direction of what might make sense. And yes, there's a streetlight effect, but more like "someone builds a new kind of lamp and puts it in a new location, and then starts hunting near there".
This isn't strictly relevant, but kinda cool nevertheless:
https://matt.might.net/articles/phd-school-in-pictures/
The analogy there is maybe that plenty of those bumps end up being in the direction of granite walls, or vast deserts, and we quit exploring, and that work was sorta-kinda wasted (except that often knowing where the boundaries of the map are is in itself useful).
The Worm guy has some suggestions about how to get imitation learners to do that
https://gwern.net/ai-daydreaming