Tech Things: The Pope has Some Thoughts
OpenAI does Math. The Big Labs get into Consulting. SpaceX IPO, Mythos hype, Insta hacked, Chipotleai.
At this point I think everyone reading this blog probably knows that I have a long standing interest in AI. If you don’t, it’s probably your first time, and you should hit the button below.
Most people probably don’t know that I also have a long-standing interest in religious studies. I’m not particularly religious myself, mind you — I tend to swing between atheist and agnostic, in case it matters — but I have a lot of respect for religious contributions to philosophy, ethics, and culture. Reading religious texts is a great way to understand how entire civilizations organize themselves. And I also increasingly believe that some kind of faith is necessary to live a self-fulfilling life. Ethics is hard, personal responsibility for ethical choices is an immense burden. What a gift, to know what is moral or not based on the contents of a book or the teachings of an elder! It’s a shame that so many of my college-educated FAANG-employed friends never bothered opening the Quran or the Bible.
Anyway, this has been a particularly great week for me, because my two favorite hobbies randomly and unexpectedly intersected when the Pope released his encyclical on AI.1
Encyclicals have a long history, but you can think of them as letters to the public that outline the Catholic Church’s position on a subject. They’re not exactly the same as the word of God, but they are important entries into the long and storied Catholic philosophical tradition that includes Aquinas and Augustine, as well as Dante and Copernicus.
The encyclical on AI (officially called Magnifica Humanitas, or Magnificent Humanity) is a long document.2 It’s also, as you might expect, a nuanced document. If you were expecting a clear “AI is a force for good / evil”, you will be sorely disappointed. Leo has a fantastic grasp of the potential and risks of AI as a technology, and avoids the polarization that has plagued the discourse on basically everything.
One of the longest threads in the encyclical is on the question of what kind of technology AI will be, expressed through extended metaphor.
Will AI be like the Tower of Babel? Will it be an exclusionary project founded on the desire to surpass humanity? Will it be doomed to fail, to sweep away those who built it in a catastrophic fall rivaling the flood? Or will AI be like the walls of Jerusalem? Will it be a protective and defensive force that can shelter those who need it while encouraging the fulfillment of all mankind?
Despite being an encyclical about AI, the Pope is not particularly interested in AI as a technology. Technology does not have animating spirit. The ethical parameters of a piece of technology are defined by those who build it. What are their goals? What is their vision of the future?3 IMO, Magnifica Humanitas is less about AI and more about Silicon Valley.
The problem with AI is not AI. It is the transhumanism. Or the belief that money is a terminal indicator of worth. Or the belief that humans should be replaced by AI. It’s the culture, stupid.
92. In his Encyclical Laudato Si’, Pope Francis denounced the growing dominance of a technocratic paradigm [119] in our globalized world: the tendency to let the logic of efficiency, control and profit alone shape personal, social and economic decisions. This makes it clear that technology is not simply a tool. When it becomes the standard by which everything is judged, it begins to dictate what matters and what can be discarded, reducing creation to an object of exploitation and human beings to mere cogs in a system driven toward ever greater efficiency.
…
94. The danger of humanity becoming a victim of its own achievements was already clearly recognized by Saint Paul VI, who warned that “the most extraordinary scientific progress, the most astounding technical feats and the most amazing economic growth, unless accompanied by authentic moral and social progress, will in the long run go against man.” [121] For this reason, technological progress — valuable in itself — requires careful discernment of the anthropological vision that guides it and the ends it pursues. If technological development advances without a corresponding ethical and social progress, the result may be an increase in means without a growth in humanity: “having more” without “being more.” In such a scenario, there is a risk that individuals will be evaluated principally according to the outcomes they produce. [122]
…
115. In an attempt to shed light on the cultural assumptions accompanying the ongoing digital revolution, I would now like to turn our attention to certain currents of thought that interpret progress as surpassing the human condition, and which are often grouped under the labels of transhumanism and posthumanism. These perspectives form the ideological background present in some centers of technological power and occupy the collective imagination in a simplified form, especially in the media and on social networks. They tend to foster enthusiasm for new technologies through a futuristic vision of an “enhanced human being” or “human-machine hybrid.”
…
117. From the perspective of the Church’s Social Doctrine, the key issue is not the use of technology as such, but the vision that underlies it. If the human being is treated as something to be perfected or surpassed, it becomes easier to accept that some lives are less useful, less desirable or less worthy. In the name of progress, “necessary sacrifices” may begin to be justified, placing the burden on the most vulnerable in pursuit of a supposed optimization of the species. In this regard, the aforementioned warning of Saint Paul VI retains great foresight: indeed, scientific and technological advances, when detached from moral and social progress, end up turning against humanity. For this reason, a clear distinction must be made. It is one thing to integrate technology within a human-centered, relational vision; it is quite another to be guided by an outlook that devalues human limits and promises a purely technical form of “salvation.”
So is AI going to be the Walls of Jerusalem, or the Tower of Babel? Let’s ask any of the thousands of people in the Bay who believe they are literally “building God.”
One of my favorite quotes from the encyclical is about the ‘absolutizing of intelligence’:
In reality, elevating any single dimension of human existence to an absolute is always a mistake. Indeed, disorder does not arise only from scarcity; even unchecked growth can give rise to impoverishment. In an ecosystem, balance is disrupted when one species expands at the expense of others; in human life, something similar occurs when one faculty claims to be the measure of everything. Thus, intelligence, when absolutized, overshadows other essential dimensions of life, such as affection, the will, commitment and relationships.
Which really stands in stark contrast to various AI leaders who keep calling humans “meat computers”.
Musk:
We are all dumb meat computers compared to digital superintelligence.
One day, frontier AI research used to be done by meat computers in between eating, sleeping, having other fun, and synchronizing once in a while using sound wave interconnect in the ritual of "group meeting". That era is long gone.
The brain is very specialized, so are the AI models. But we’re not building a 20 watt meet computer. We’re building a 1,200,000,000 watt AI brain.
I think Leo is seeing and reacting to the same cultural trends I observed here (and to some extent here and here), but at a larger scale.
I want to reflect briefly on the surge of nihilistic discourse that seems to increasingly plague the tech community. This particular disease goes by many names — ‘black pill’, ‘stoicism’, ‘pragmatism’. But don’t let that confuse you. These are all forms of the same thing.
The core of this mindset is a logical fallacy:
The speaker has realized that ‘everything is made up’, that ‘there are no rules’, that ‘you can just do things’;
therefore having values, beliefs, or morals at all is misguided, naive, cringe, and uncool.
A classic example that exemplifies the mindset is something like “but it gets clicks/views/engagement.” Another one: “this is just how the world works.” A third: “it’s not my problem.”
The nihilism and pursuit of wealth at all costs doesn’t exist in a vacuum. It comes from the dark side of the entrepreneurial spirit, the same force that encourages creating gambling addiction or using rage bait marketing or blatantly violating laws to get a competitive edge. AI is a multiplier; in the hands of unscrupulous people, AI will be unscrupulous.
In some sense, Magnifica Humanitas is concerned about the sorts of things alignment researchers have been beating the drum about for ages. Chris Olah, an Anthropic founder and a leader in AI alignment, was invited to speak alongside the Pope for good reason. But the difference is that the researchers think they can solve the problem if they embed ethics into the technology. The Pope suggests that we embed ethics into the society that creates the technology instead.
Can AI Create New Knowledge
Two open questions, both extremely important.
First: is knowledge created or discovered? In some domains it seems obviously discovered — bacteria and viruses wreaked havoc well before germ theory could explain why, the arrangement of the stars and planets are agnostic to our telescopes, and the higgs boson continues to do…whatever it does regardless of our ability to observe it.
In others, it’s less clear. Mathematics is the study of axioms, a seemingly endless game of pushing a particular set of contrived rules as far as possible. Is the Pythagorean Theorem created or discovered? Any given proof of the Pythagorean Theorem is arguably invented. But also, in some sense, the relationship between sides of a triangle exists in the ether, waiting to be plucked out of the chaos by the first Greek guy with a ruler and compass. And proofs themselves are often just applications of existing knowledge in other places. Is that invention? Or do the laws of mathematics already exist, and mathematicians are merely filling in parts of the map?
Second: can LLMs create new knowledge? This seems to be the last remaining stronghold of the AI denialists. “Yes, fine, AI can create art and create music and write essays (though Sam Kriss may murder me for saying so) and build software out of nothing at all, but yawn wake me up when the AI can discover fundamental truths about the world.” Nevermind that most people cannot meet that standard, but whatever.
Notice that the second question is intimately tied to the first question. If creating new knowledge is an act of consciousness, a leap of ingenuity that cannot exist from any previous understanding of the world, then LLMs have not and arguably cannot create knowledge. But if creating new knowledge is an act of discovery, of interpolating between known points on a map and finding the points that are unexplored — that is, if creating new knowledge is merely a particularly exhaustive exercise in search — then of course LLMs can create new knowledge, it seems trivial to argue otherwise. LLMs are fantastic at search, and as I’ve written in the past, all problems are search problems.
In some ways, both of these questions are debates about semantics. What do you mean by knowledge? There’s an entire arm of philosophy (epistemology) dedicated to the question of what it means to know things! People have dedicated their lives to trying to understand this meta question! When Twitter user CiceroClone3524 smugly tweets “LLMS CAN’T CREATE KNOWLEDGE”4 without first providing a treatise on what knowledge even is, you should just treat it like the incoherent shitpost it is and move on.
This month, an LLM solved an open problem in discrete geometry.
For nearly 80 years, mathematicians have studied a deceptively simple question: if you place n points in the plane, how many pairs of points can be exactly distance 1 apart?
…
Today, we share a breakthrough on the unit distance problem. Since Erdős’s original work, the prevailing belief has been that the “square grid” constructions depicted further below were essentially optimal for maximizing the number of unit-distance pairs. An internal OpenAI model has disproved this longstanding conjecture, providing an infinite family of examples that yield a polynomial improvement.
…
Surprisingly, the key ingredients of the construction come from a very different part of mathematics known as algebraic number theory, which studies concepts like factorization in extensions of the integers known as algebraic number fields.
This was immediately met by cries of disdain. For example, Gary Marcus:
Clearly impressive. But as with so much else, it should be viewed with skepticism.
…
[Quoting Cal Newport] He noted that though the conjecture is old, those who have worked on it have largely shared Erdos’s original belief that it was true and therefore focused on trying to solve it. What the LLM-based tool did instead was to systematically apply and extend existing techniques in search of evidence that the conjecture was false.
…
[Quoting Tim Gowers] It has an encyclopedic knowledge of mathematics, and it does not have to worry nearly as much as we do about time management, so it is good at finding surprising connections.
And so on.
Without commenting on whether or not this is “new” knowledge,5 it’s worth pointing out just how much ground the AI denialists are ceding. Science is about building on the shoulders of giants. Virtually everything we know is derived from combining the ideas of our forefathers. Hell, almost every AI paper is about applying some idea from some other field to AI, and all the other AI papers are about applying some idea from AI to some other field. If a novel mathematical proof isn’t new knowledge, the term “knowledge” has ceased to refer to anything that corresponds to reality. At that point, just pack it up — humanity never had any hope anyway.
The Big Labs get into Consulting
I think “forward deployed engineer” is my favorite euphemism of the last decade or so. As far as I can tell, the only reason this term exists is because a significant portion of Silicon Valley will refuse to accept that they are consultants, but if you wrap their title with the term “engineer” suddenly everything is ok.
What does a forward deployed engineer do? Well, they are “deployed” to client teams in order to help those teams solve their business needs, often by coding. What does a consultant do? Well, if they work at say, Accenture, they are deployed to client teams in order to help those teams solve their business needs, often by coding. “No no, I’m not a consultant, I don’t work with slide decks and things. I’m beyond such mundane software as the Microsoft Office suite. I’m an engineer.” Uh huh. I mean I agree that management consulting probably uses slide decks, but my buddies at McKinsey write code too so IDK. Hell, with coding agents, everyone writes code these days, especially the consultants.
To be clear, I have nothing against consultants or consulting. It’s obviously necessary, especially when there’s a new technology that lots of people want to use but few people have expertise in. I just think it’s funny that the Valley had to invent a new word for it. Kinda in line with the whole “package an existing product in a new name and sell it as revolutionary,” though that is more often done to avoid regulatory scrutiny (looking at you, prediction markets / crypto / ride shares / whatever software landlords use to coordinate rental pricing increases).
Anyway, my point is that FDEs have been around for a while. If you are having trouble with the middle of your sales funnel — folks are maybe signing up for your product but not getting the full use of it, maybe not getting upsold effectively — sending out one of your employees to help them get set up is common sense.
This is why Microsoft hires Solution Engineers:
Drive technical sales by using technical demos, proofs of concept, and technical architecture accelerators to influence solution design and enable deployments.
Lead architecture sessions and technical workshops to accelerate Azure adoption.
Build trusted relationships with platforms leads to co-design secure, scalable solutions.
And why Oracle hires field engineers, and why Salesforce, Atlassian, Intel, Google, all have customer success engineers, and so on. Apparently IBM had “customer engineers” way back in 1942. The actual term “FDE” got popularized by Palantir, and since then I’ve seen Fortune 500s and Series A startups alike advertise for FDE roles. But, you know. It’s really just another name for consulting.
This is why I was somewhat surprised about the news that OpenAI and Anthropic were both starting consulting arms. Not that they were starting those arms, mind you, that makes tons of sense, see above. Rather, I was surprised by the news — the breathless takes from an entirely too credulous social/media that this was further evidence of all economic activity being subsumed by the big labs.
Guys. Guys. Come on. Get off the Internet for two minutes, please.
OpenAI is starting a consulting arm because people keep going “I want to AI more” and then OpenAI goes “here is your AI” and then people go “thanks but I have no idea what this is, also can you make it hook into my RBAC provider?”
AI tooling was built for consumers first, so it is easy to forget just how involved enterprise software is. “Just npm install -g claude” you fool. You absolute fool. The average enterprise user cannot just npm install -g claude. Hell, they can’t even install npm! They have to request approval to get access to the site that hosts an internal version of npm, then request approval to download npm, then set npm to point to an internal registry, then get approval to download an internal version of Claude off that registry, and then finally have that version of claude point to some internal enterprise exclusive Claude deploy. Someone has to maintain all of those security boundaries, someone has to explain how the AI works within those boundaries, and all of the people who know how to do that currently work at the big labs because AI has only existed in its current form for like 8 months. If you made wild predictions about how “Microsoft sending consultants to help enterprises onboard onto Windows” meant that MS was going to take over the world, you would’ve been right for about 5 years from 1995 to 2000 and then very very wrong for the next two decades.
That’s not to say that OpenAI and Anthropic and so on are bad products, nor even that they won’t benefit from the expansion of AI usage. Microsoft undoubtedly made more money in absolute terms as computer adoption skyrocketed, and the same is certainly true of the big AI labs. But also, there were clearly many billions to be made on the back of the computer boom, and on the back of the Internet boom, and on the back of the mobile boom. The AI boom is very much in its infancy, and, contrary to popular opinion, the existence of these deployment arms — which are essentially high labor, white glove services companies — just underscores how early we are.
Other things:
The Pope’s encyclical is a thoughtful and nuanced document, so of course most of the Internet spent most of the time debating whether or not the thing was written with AI. There’s good reason to believe parts of it were, or at least that some translations were done with AI. But also, this discussion completely misses the point. The Pope never says that using AI is bad! AI isn’t an all or nothing thing, and if used responsibly it is obviously extremely valuable. The problem with AI writing is AI slop; the Magnifica Humanitas is obviously not that.
SpaceX is IPO-ing. Matt Levine has the best take, as usual:
The deal, with SpaceX, is that Elon Musk runs it however he wants, and he does weird stuff, and you have to trust him, and if you don’t like it you can’t complain. When SpaceX acquired xAI a few months ago, did a special committee of independent directors approve the transaction? Did Musk recuse himself from negotiations? Was the price set by independent valuation experts using a rigorous process? Did outside shareholders sue to block the deal? Stop. Musk wanted SpaceX to buy xAI, so it did.
SpaceX claims to have a TAM of the entire global market and then some. This is obviously ridiculous, but if you’re investing in SpaceX you’re obviously along for the ride. The bigger concern, in my mind, is SpaceX getting index funds to change their rules. TBD on how that all works.
Mythos seems like it's beating the hype. From Anthropic: “Since then, we and our approximately 50 partners have used Claude Mythos Preview to find more than ten thousand high- or critical-severity vulnerabilities across the most systemically important software in the world. Progress on software security used to be limited by how quickly we could find new vulnerabilities. Now it’s limited by how quickly we can verify, disclose, and patch the large numbers of vulnerabilities found by AI.”
Instagram was hacked because of AI, but not the way you might think.
Step 01: Faking the Location & Initiating Support All the attacker needs to kick this off is your account username. Then, they hop on a VPN or proxy close to your city so Instagram’s security algorithms don’t suspect a thing. (You can quite easily get this from your public profile or “About” section or a hundred other ways.) Once it looks like the request is coming from the correct region, they tell the Meta support AI that the account is hacked and ask it to send the verification codes to an arbitrary email address they control.
Step 02: That’s It Really, that’s it. The first proper zero auth password reset I’ve seen in production. There appears to be no additional check as to whether the email being given is actually something the user has used before. Once the AI sends the security code to the attacker’s email, the attacker passes it right back to complete the verification. The platform hands over a fresh password reset link, granting full ownership to the attacker.“The first proper zero auth password reset” is hilarious. Reminds me of the good old days when you could talk to “customer support” and have it promise you a car that the company would then be forced to pay out.
Chipotle runs an AI model for customer service. Apparently it’s just a standard coding agent, so someone figured out how to hook into it for their coding work. “Free inference paid for by burritos.”
It’s not fully unexpected. When the Pope took the name Leo XIV, it was widely seen as a reference to Pope Leo XIII, who led the church during the industrial revolution. That Pope also wrote an encyclical (he wrote 86, but one famous one): “In his 1891 encyclical, Rerum Novarum, Pope Leo outlined the rights of workers to a fair wage, safe working conditions, and the formation of trade unions, while affirming the rights to property and free enterprise.” The encyclical on AI, Magnifica Humanitas, was released on the 135th anniversary of the older one, and explicitly references the Rerum Novarum.
But very readable. For folks who are less interested in the Catholic philosophy bits, I recommend reading the intro, and then most of Chapter 3.
“Ethical discernment cannot be limited to asking whether we are using a system for good or bad purposes; it must also examine how that system is designed and what vision of the human person and society is embedded in the data and models that guide it.” — 104
jk, they aren’t having this level of discussion on Twitter
My personal bias is that the response from Marcus et. al. is basically just cope.
good post as usual!!
you repeated this part twice
> “The first proper zero auth password reset” is hilarious. Reminds me of the good old days when you could talk to “customer support” and have it promise you a car that the company would then be forced to pay out.