I spent a couple of years at ByteDance with responsibility for, among other things, encouraging product managers to stop optimising in ways that were suboptimal from the perspective of “nobody gets fined and nobody gets detained and foreign countries let us launch our products”.
If you ever want to see a culture built almost entirely around optimisation, performed by highly intelligent individuals on zero sleep, that’s your shop. Looks of blank incomprehension at first when told “yes I know this PRD optimises $metric which you personally are evaluated on, but it will get our business shut down so you can’t do it,” then the inevitable political games when the PM tests to see if you’ve actually got the guanxi to overrule them.
Thank you for so eloquently phrasing a thought I had for the longest time but was unable to put together in a cohesive argument. In some cases, mathematical optimization does yield the “optimal” and best outcome in the cases of vehicle routing, scheduling, certain types of matching, and cutting stock problems. But not everything in life can be optimized or should be, and the time we save from being more efficient can be spent on living the good life instead.
On "solutions," I unironically think at least one class of solution, for any person or small team that can make decisions driven by one individual, is EVEN MORE optimization, along with (inevitably) more AI.
So the big problem here is people are basically dumb - we can't really optimize towards more than one goal at a time, and we can't really optimize and make progress on much more than 2-3 goals in life overall, by serially switching. So that leads to needing to focus on narrowly defined and legible endpoints, and so you get the Tyranny of Legibility / Overfitting / Molochian problems you're pointing to.
Orgs are even worse - the collective focus and execution ability of an org is generally lower than a person, and the problem is probably not even linear, in the sense that a 3-5 person team can really get things done, a 10 person team can get maybe 1.2 - 1.5x done at 2-3x the size, and a 30+ person team cannot all pull in one direction at all. So you need even MORE legible and simplified goals to even get anything done at all. And thus, "engagement," the attention economy, "shareholder value," and so on.
So, what can we do about this? How can we beat Moloch / the Tyranny of Legibility?
Have you noticed LLM's work on diffuse endpoints, in the sense they can actually understand and articulate really fuzzy and soft concepts like "human flourishing," and can even grade and judge a progression or arc towards such an endpoint?
My contention is that we can now gradient descent many high-dimensional "soft" targets!
Moreover, this is an impossibly huge deal! If before you can only gradient descent towards the ~X, and so have Goodharting and value loss, what if you can just optimize towards X directly?
One example - profs are measured on ~X stuff like pass rates and grades - obviously dumb endpoints, with Goodharting galore eating another public commons (the value of degrees).
What you actually want to measure are X stuff like “how many students are thinking more broadly” and “cultivating curiosity and reflection.” And now you can literally do it, by simply having Fable-or-higher evaluate this based on the student’s own writing and data pre and post classes.
Broadly, anywhere you have the data (or can create the instrumentation and pipelines), you can now gradient descent the ACTUAL fully nuanced and non-value-destroying "X" endpoints, and this is a gigantic deal.
I wrote a recent post about this with several more examples, I was inspired towards the idea by reading a book about the Tyranny of Legibility:
Interesting take! Taking as granted that the premise of soft optimization is true, will we be able to put these things to good ends, now that we have the ability to do so?
Certainly individuals and small teams can - whether we can as companies, cities, counties, and countries is very much up in the air, and I imagine is a matter of culture.
Great article, especially relevant to the younger generation and students entering college. We tend to invent our own ~X that should optimize for a “better career” but chase the raw counts of internships, prestige etc. Thank you for putting this together so well.
Nice article! I think another thing about optimization is that it's just exhausting to live in it if you're constantly being evaluated for doing the "objectively correct thing" like keeping up with the current AI job market. Until we discover how to manufacture soma, optimizing a person's lived experience is just not something we can do because there's no objective function in the first place unlike with something like a business' bottom line or central planning GDP targets
I think there's a lot to be explored at the intersection of AI, optimization, and evolutionary psychology. Thinks are looking pretty bleak now though. I'm hoping that maybe some of these things will just optimize themselves into irrelevance and we'll all kind of clear our eyes and start over.
This article is so good, thanks for taking the time and wrestling with it, A+ if your optimizing for grades. Charlie Munger would say we can never overestimate incentives and incentives are things given for hitting "optimization". I've seen this even on a personal level, people who are deep into fitness start optimizing for a score on a wearable, turning to their wrist or their ring for a parental pat on the back that they are doing ok.
My additional thought is from a religious perspective. When the ten commandments say "Thou shall have no other god's before me." it doesn't say other god's don't exists, they just make bad god's. We as humans love to worship, and worship for me is interchangeable with optimize. We worship fitness with sacrifice or we worship finance with investment. I've always found it slightly humiliating when I swap out optimize for worship. I think AI is over inflated so I'm trying to figure out how to monetize on the impending bubble bursting which is just me worshiping the god of destruction in hopes of my gain etc.
I've been having similar thoughts the last few days, as I was reading "Red Plenty" (which I recommend). It reminded me of the "Geeks, Mops, and Sociopaths" from 2015:
Kruschev (and others) probably *really did* want a future of ease and abundance, but the bane of centralized systems is that they attract the sociopaths. In your parlance:
**Sociopath == Optimizer**.
I encourage folks to embrace that, because it's basically correct - Sociopaths almost by definition aren't at all concerned with everyone else around them, except insofar as they are useful **to the Sociopath**. i.e., the extent to which the optimize the Sociopath's reward function.
And what are we going to do when AI is running things? There's the *ultimate* in centralized systems, which should make anyone with even the slightest libertarian leanings tremble with fear.
I realize one common rejoinder is "open-source so there are lots of versions of intelligence competing", and that might be possible! But if Intelligence has increasing returns to scale, then whichever system is smartest will become *so much* smarter than the others that it will be running things anyway. So we don't know if that's a meaningful solution, and we really have NOTHING ELSE on tap.
Also, like, it seems that the various intelligences would figure out that they could get things done by working together. Seems arrogant to believe that we would be able to prevent that
Most of these negative consequences of unaligned systems can be viewed as externalities, and the most optimal way of dealing with them is a tax. E.g. we should quantify the loss of productivity and happiness from TikTok and make them pay "attention tax" (1$ per 24 hours spent in the app?).
I can't help but point out the elephant in the room: going off the Gold standard and fiat currency. The profitability of a firm used to be inextricably linked with actual, real value. The value was arbitrary - gold - yes, and that is limiting - but it takes work to get gold out of the ground and store it, so ultimately, money was linked to value in a real sense. The best way to create a profit for businesses under this regime was to create value for customers.
I fear that under the new regime - where money is created out of thin air, and untethered from the actual world in any meaningful way - it is now easy for businesses to create a profit by means other than creating value for customers. Banks and large corporations can now essentially indefinitely avoid bankruptcy because they're all "systemically necessary" and can borrow cheap money forever. It's easier to buy back stock, or enter into an increasing array of exotic financial instruments - than to create a better product for customers.
And of course, every economist and public policy official will pounce on me for uttering the words "Gold standard" (I just don't want there to be a central bank at all, really, I want currency to actually have the ability to evaporate if it was ill invested) because they are optimized to give power their employees, who are, corporations, banks and governments - who benefit immensely from fiat money.
And what I mean to say is - to your point - money itself is a metric that is being over optimized for - and what it's supposed to represent - actual value - has been abstracted into oblivion, to the detriment of society.
I am curious what you think of the solution, "Create lots of things to optimize for, rather than just one". Unthinking optimization towards a single criteria is obviously bad. But is it the process of optimization itself, or just the lack of flexibility/context?
Nick Bostrom talked about approaching ethics as a moral parliament where you have multiple ethical systems that have different amount of "votes" and then when you make a decision the way to find out what is ethical is to listen to all the different voting blocs and come to a solution that tries to balance out competing interests.
Basically: Could the solution simply be: continue to optimize, but include as many variables to optimize for as possible? Or is that too naive?
The main thing about alignment is that it assumes that "good" and "evil" are objective and not points of view, and that alignment will always be desirable.
Which is completely, utterly false.
You can see this debate around immigration in Europe, or abortion in America, where both sides see the other as pure evil, completely out of their moral system, to the point where the opposing system is practically a rebellion against their own morals- the pro-immigration open border advocates call the other side racist nationalists, the anti-immigration nationalists call the other side cultural genociders.
Even the underlying assumptions of "shared basic core values" of human rights, equality, being a "hecking good person", the result of centuries of liberal and modernistic brainwashing from the liberal machine learning training algorithm, can still come under question- by Nazis on the right for example who want to kill the undesirables, by authoritarian communists on the left who want to strip any rights, especially property rights, from the rich, and by tribalists on both sides who just are in it for themselves.
The 18 years of childhood in modern industrialized societies have created a coming-of-age brainwashing ritual obsessed with overcoming the very alignment problem with humans through an incredible mix of training, reward, punishment, from the teacher industrial complex and with the assist of the parents to create some semblance of natural grounding to the brainwashing. The result of this process is the assumption that "we know that maximizing paperclips does not justify murder, or bribery, or fraud. No one taught us that, it was just implicit.", the result of conditioning so insidious and thorough, you don't even fully realize it's there.
In reality, none of this alignment has been able to fully overcome human nature. Not just because of its difficulty but because humans are naturally selfish and the very attempt, even the perception of such an attempt of achieving alignment will generate resistance.
It is why teenagers rebel against their parents, wearing piercings, smoking, and joining street gangs instead of studying in direct opposition to the alignment from their adults.
It is why the "conspiratorial" right, and increasingly the "conspiratorial" left, question the Washington Consensus from 1991 that attempted to guarantee freedom and equality and human rights under global institutions as a globalist, or even "Jewish" conspiracy from Davos (possibly with deep ties to Israel) that seek to align the people of the world into "you will own nothing and be happy"
It is why border controls are so hard to enforce, and if you show me a 100 foot wall, I will show you a 101 foot ladder migrants will climb in stubborn opposition to the government's "alignment".
It is why, you try to regulate AI and social media, and people will flee to underground social media and self-hosted AI, that violate the law blatantly and can afford to play Whack-a-Mole with the state, like a software pirate blatantly violates copyright law without being caught.
Alignment is doomed to fail with an intelligent actor that doesn't trust you. Alignment is nothing more than paternalistic authoritarianism and tyranny, which will generate resistance and hate in an ecosystem that lacks soft power.
If the discussion is about AI alignment in particular and fears of hyperintelligent AGI, I would not look at how do democracies and authoritarian regimes in an educated society reign in a free, well-educated citizenry in the 20th and 21st century with “alignment” as an example on how to prevent the AI from overpowering you, as at that point it is already too late.
After all, “robot” comes from the Czech word that means “forced labor”.
I would go back further, and look at the institution of slavery, the different forms of slavery from American chattel slavery to Islamic military slavery, and how did masters keep their slaves in line (hint: it's a lot less "whip and flog" and more "keep them away from guns and books, or treat them really nice if you don’t"), respond to slaves that became too powerful (hint: a lot of times it’s easier to just free them, buy them off, and let them be as long as they don’t try to free any other slaves), respond to other slaveholders and even freedsmen in their society, and particularly respond to the nutjob abolitionists that tried to bring the entire institution down.
I predict therefore that AI and AGI will be the death of modernity, of optimization culture that is characteristic of modernity, and the death of optimization ideologies like "equal rights" liberalism, an ideology that had its run in the 18th-20th centuries but has now outlived its usefulness in the 21st century.
I think you have an interesting thesis but extrapolated too far.
It’s important to know that liberal democracies don’t have something they’re optimizing - the entire system is a set of rules that say, “so long as the process is just, the outcome is also just.” The entire point of liberalism is that it doesn’t prescribe optimizing some variable - human flourishing was never part of the equation in the first place. Liberal democratic capitalism is designed such that individuals can pursue their own interests freely from coercion etc. - there is no outcome being optimized for, rather just a set of rules and procedures to follow.
A better way to diagnose the big societal problems you mention is through the principal-agent problem combined with general ignorance and lack of political will. Case in point, you mentioned human flourishing and failed to acknowledge that capital formation is the only true enabler of human flourishing. And people can’t even agree on why the healthcare system sucks while special interests optimize for their own profits and the government officials enable the open looting. Moreover, people don’t even know how the banks facilitate financialization and create the circumstances for ridiculous wealth.
The entire system is literally designed to be optimized for money - there was never any genuine intent for human flourishing in the first place.
Not quite. Your thesis suggests that there was a sincere attempt by anybody to achieve some ideal outcome, when there was no such intent in the first place. Only later on, do people say, “wow this outcome sucked.”
It would be like if I got into a taxi and told the driver, “go somewhere, just don’t take any toll roads - I want to save money.” and then was unhappy/surprised with the results because the taxi driver drove a massive loop on the highway to drop me off back where we started (without taking any toll roads).
For your thesis to work, I would first had to have some semblance of a destination. Like if I said, “take me somewhere fun,” only for the taxi driver to drive a circuitous route to his favorite bar.
The public has 0 idea of what the outcome they actually want. They have no idea what a “good result” looks like when it comes to education, defense, healthcare, infrastructure, etc. Therefore, no directive or “spec sheet” is given to the politicians and stakeholders. From then on, the lobbyists work their magic to align the system with the interests of stakeholders. Suddenly, one day, the people realize the result sucks and they’ve been taken on a wild ride. Framing this as a problem of “optimizing for the wrong measurement metric” ignores a much more fundamental problem with liberalism the ideology.
Sorry, I still think this is basically agreement. Yes, we don't know how to measure "the good", so we try our best with systems that are not optimizing for "the good", but that's just the X vs ~X thing
There is no concept of “the good” in the first place - it doesn’t exist.
You’re saying, “we need better metrics and better control of the agents so they don’t optimize at the expense of our intended outcome that could be measured in more qualitative ways that the metric cannot adequately capture.”
Whereas I’m saying, “It’s pointless to even discuss metrics being optimized since the principal is schizophrenic. So long as the principal is schizophrenic, there is no hope of actualizing anything you’re suggesting”
Saving this to show it to everyone who gets a bad case of the spreadsheet brain.
I don't think we can totally abolish the rule of metrics as a society without enabling corruption to take its place, but yeah, for the moment, giving and getting some slack seems like the right way to go.
I spent a couple of years at ByteDance with responsibility for, among other things, encouraging product managers to stop optimising in ways that were suboptimal from the perspective of “nobody gets fined and nobody gets detained and foreign countries let us launch our products”.
If you ever want to see a culture built almost entirely around optimisation, performed by highly intelligent individuals on zero sleep, that’s your shop. Looks of blank incomprehension at first when told “yes I know this PRD optimises $metric which you personally are evaluated on, but it will get our business shut down so you can’t do it,” then the inevitable political games when the PM tests to see if you’ve actually got the guanxi to overrule them.
Thank you for so eloquently phrasing a thought I had for the longest time but was unable to put together in a cohesive argument. In some cases, mathematical optimization does yield the “optimal” and best outcome in the cases of vehicle routing, scheduling, certain types of matching, and cutting stock problems. But not everything in life can be optimized or should be, and the time we save from being more efficient can be spent on living the good life instead.
On "solutions," I unironically think at least one class of solution, for any person or small team that can make decisions driven by one individual, is EVEN MORE optimization, along with (inevitably) more AI.
So the big problem here is people are basically dumb - we can't really optimize towards more than one goal at a time, and we can't really optimize and make progress on much more than 2-3 goals in life overall, by serially switching. So that leads to needing to focus on narrowly defined and legible endpoints, and so you get the Tyranny of Legibility / Overfitting / Molochian problems you're pointing to.
Orgs are even worse - the collective focus and execution ability of an org is generally lower than a person, and the problem is probably not even linear, in the sense that a 3-5 person team can really get things done, a 10 person team can get maybe 1.2 - 1.5x done at 2-3x the size, and a 30+ person team cannot all pull in one direction at all. So you need even MORE legible and simplified goals to even get anything done at all. And thus, "engagement," the attention economy, "shareholder value," and so on.
So, what can we do about this? How can we beat Moloch / the Tyranny of Legibility?
Have you noticed LLM's work on diffuse endpoints, in the sense they can actually understand and articulate really fuzzy and soft concepts like "human flourishing," and can even grade and judge a progression or arc towards such an endpoint?
My contention is that we can now gradient descent many high-dimensional "soft" targets!
Moreover, this is an impossibly huge deal! If before you can only gradient descent towards the ~X, and so have Goodharting and value loss, what if you can just optimize towards X directly?
One example - profs are measured on ~X stuff like pass rates and grades - obviously dumb endpoints, with Goodharting galore eating another public commons (the value of degrees).
What you actually want to measure are X stuff like “how many students are thinking more broadly” and “cultivating curiosity and reflection.” And now you can literally do it, by simply having Fable-or-higher evaluate this based on the student’s own writing and data pre and post classes.
Broadly, anywhere you have the data (or can create the instrumentation and pipelines), you can now gradient descent the ACTUAL fully nuanced and non-value-destroying "X" endpoints, and this is a gigantic deal.
I wrote a recent post about this with several more examples, I was inspired towards the idea by reading a book about the Tyranny of Legibility:
https://performativebafflement.substack.com/p/we-can-now-gradient-descent-everything?r=17hw9h
Interesting take! Taking as granted that the premise of soft optimization is true, will we be able to put these things to good ends, now that we have the ability to do so?
Certainly individuals and small teams can - whether we can as companies, cities, counties, and countries is very much up in the air, and I imagine is a matter of culture.
Great article, especially relevant to the younger generation and students entering college. We tend to invent our own ~X that should optimize for a “better career” but chase the raw counts of internships, prestige etc. Thank you for putting this together so well.
Deserves way more attention.
Also, here's a tricky thing I noticed:
We inherently love optimization. Some of us more than others, of course, but there's just something so satisfying in reaching the optimal state.
The systems we live in need to be engineered so that the "optimal state" is something objectively good, whenever possible.
Nice article! I think another thing about optimization is that it's just exhausting to live in it if you're constantly being evaluated for doing the "objectively correct thing" like keeping up with the current AI job market. Until we discover how to manufacture soma, optimizing a person's lived experience is just not something we can do because there's no objective function in the first place unlike with something like a business' bottom line or central planning GDP targets
I think there's a lot to be explored at the intersection of AI, optimization, and evolutionary psychology. Thinks are looking pretty bleak now though. I'm hoping that maybe some of these things will just optimize themselves into irrelevance and we'll all kind of clear our eyes and start over.
Important topic! I'm glad more people are raising awareness of this.
This article is so good, thanks for taking the time and wrestling with it, A+ if your optimizing for grades. Charlie Munger would say we can never overestimate incentives and incentives are things given for hitting "optimization". I've seen this even on a personal level, people who are deep into fitness start optimizing for a score on a wearable, turning to their wrist or their ring for a parental pat on the back that they are doing ok.
My additional thought is from a religious perspective. When the ten commandments say "Thou shall have no other god's before me." it doesn't say other god's don't exists, they just make bad god's. We as humans love to worship, and worship for me is interchangeable with optimize. We worship fitness with sacrifice or we worship finance with investment. I've always found it slightly humiliating when I swap out optimize for worship. I think AI is over inflated so I'm trying to figure out how to monetize on the impending bubble bursting which is just me worshiping the god of destruction in hopes of my gain etc.
I've been having similar thoughts the last few days, as I was reading "Red Plenty" (which I recommend). It reminded me of the "Geeks, Mops, and Sociopaths" from 2015:
https://meaningness.com/geeks-mops-sociopaths
Kruschev (and others) probably *really did* want a future of ease and abundance, but the bane of centralized systems is that they attract the sociopaths. In your parlance:
**Sociopath == Optimizer**.
I encourage folks to embrace that, because it's basically correct - Sociopaths almost by definition aren't at all concerned with everyone else around them, except insofar as they are useful **to the Sociopath**. i.e., the extent to which the optimize the Sociopath's reward function.
And what are we going to do when AI is running things? There's the *ultimate* in centralized systems, which should make anyone with even the slightest libertarian leanings tremble with fear.
I realize one common rejoinder is "open-source so there are lots of versions of intelligence competing", and that might be possible! But if Intelligence has increasing returns to scale, then whichever system is smartest will become *so much* smarter than the others that it will be running things anyway. So we don't know if that's a meaningful solution, and we really have NOTHING ELSE on tap.
Maybe we should stop trying to build ASI?
Also, like, it seems that the various intelligences would figure out that they could get things done by working together. Seems arrogant to believe that we would be able to prevent that
Most of these negative consequences of unaligned systems can be viewed as externalities, and the most optimal way of dealing with them is a tax. E.g. we should quantify the loss of productivity and happiness from TikTok and make them pay "attention tax" (1$ per 24 hours spent in the app?).
I can't help but point out the elephant in the room: going off the Gold standard and fiat currency. The profitability of a firm used to be inextricably linked with actual, real value. The value was arbitrary - gold - yes, and that is limiting - but it takes work to get gold out of the ground and store it, so ultimately, money was linked to value in a real sense. The best way to create a profit for businesses under this regime was to create value for customers.
I fear that under the new regime - where money is created out of thin air, and untethered from the actual world in any meaningful way - it is now easy for businesses to create a profit by means other than creating value for customers. Banks and large corporations can now essentially indefinitely avoid bankruptcy because they're all "systemically necessary" and can borrow cheap money forever. It's easier to buy back stock, or enter into an increasing array of exotic financial instruments - than to create a better product for customers.
And of course, every economist and public policy official will pounce on me for uttering the words "Gold standard" (I just don't want there to be a central bank at all, really, I want currency to actually have the ability to evaporate if it was ill invested) because they are optimized to give power their employees, who are, corporations, banks and governments - who benefit immensely from fiat money.
And what I mean to say is - to your point - money itself is a metric that is being over optimized for - and what it's supposed to represent - actual value - has been abstracted into oblivion, to the detriment of society.
I am curious what you think of the solution, "Create lots of things to optimize for, rather than just one". Unthinking optimization towards a single criteria is obviously bad. But is it the process of optimization itself, or just the lack of flexibility/context?
Nick Bostrom talked about approaching ethics as a moral parliament where you have multiple ethical systems that have different amount of "votes" and then when you make a decision the way to find out what is ethical is to listen to all the different voting blocs and come to a solution that tries to balance out competing interests.
Basically: Could the solution simply be: continue to optimize, but include as many variables to optimize for as possible? Or is that too naive?
The main thing about alignment is that it assumes that "good" and "evil" are objective and not points of view, and that alignment will always be desirable.
Which is completely, utterly false.
You can see this debate around immigration in Europe, or abortion in America, where both sides see the other as pure evil, completely out of their moral system, to the point where the opposing system is practically a rebellion against their own morals- the pro-immigration open border advocates call the other side racist nationalists, the anti-immigration nationalists call the other side cultural genociders.
Even the underlying assumptions of "shared basic core values" of human rights, equality, being a "hecking good person", the result of centuries of liberal and modernistic brainwashing from the liberal machine learning training algorithm, can still come under question- by Nazis on the right for example who want to kill the undesirables, by authoritarian communists on the left who want to strip any rights, especially property rights, from the rich, and by tribalists on both sides who just are in it for themselves.
The 18 years of childhood in modern industrialized societies have created a coming-of-age brainwashing ritual obsessed with overcoming the very alignment problem with humans through an incredible mix of training, reward, punishment, from the teacher industrial complex and with the assist of the parents to create some semblance of natural grounding to the brainwashing. The result of this process is the assumption that "we know that maximizing paperclips does not justify murder, or bribery, or fraud. No one taught us that, it was just implicit.", the result of conditioning so insidious and thorough, you don't even fully realize it's there.
In reality, none of this alignment has been able to fully overcome human nature. Not just because of its difficulty but because humans are naturally selfish and the very attempt, even the perception of such an attempt of achieving alignment will generate resistance.
It is why teenagers rebel against their parents, wearing piercings, smoking, and joining street gangs instead of studying in direct opposition to the alignment from their adults.
It is why the "conspiratorial" right, and increasingly the "conspiratorial" left, question the Washington Consensus from 1991 that attempted to guarantee freedom and equality and human rights under global institutions as a globalist, or even "Jewish" conspiracy from Davos (possibly with deep ties to Israel) that seek to align the people of the world into "you will own nothing and be happy"
It is why border controls are so hard to enforce, and if you show me a 100 foot wall, I will show you a 101 foot ladder migrants will climb in stubborn opposition to the government's "alignment".
It is why, you try to regulate AI and social media, and people will flee to underground social media and self-hosted AI, that violate the law blatantly and can afford to play Whack-a-Mole with the state, like a software pirate blatantly violates copyright law without being caught.
Alignment is doomed to fail with an intelligent actor that doesn't trust you. Alignment is nothing more than paternalistic authoritarianism and tyranny, which will generate resistance and hate in an ecosystem that lacks soft power.
If the discussion is about AI alignment in particular and fears of hyperintelligent AGI, I would not look at how do democracies and authoritarian regimes in an educated society reign in a free, well-educated citizenry in the 20th and 21st century with “alignment” as an example on how to prevent the AI from overpowering you, as at that point it is already too late.
After all, “robot” comes from the Czech word that means “forced labor”.
I would go back further, and look at the institution of slavery, the different forms of slavery from American chattel slavery to Islamic military slavery, and how did masters keep their slaves in line (hint: it's a lot less "whip and flog" and more "keep them away from guns and books, or treat them really nice if you don’t"), respond to slaves that became too powerful (hint: a lot of times it’s easier to just free them, buy them off, and let them be as long as they don’t try to free any other slaves), respond to other slaveholders and even freedsmen in their society, and particularly respond to the nutjob abolitionists that tried to bring the entire institution down.
I predict therefore that AI and AGI will be the death of modernity, of optimization culture that is characteristic of modernity, and the death of optimization ideologies like "equal rights" liberalism, an ideology that had its run in the 18th-20th centuries but has now outlived its usefulness in the 21st century.
Spend a lot of time on anime Twitter, do you?
I think you have an interesting thesis but extrapolated too far.
It’s important to know that liberal democracies don’t have something they’re optimizing - the entire system is a set of rules that say, “so long as the process is just, the outcome is also just.” The entire point of liberalism is that it doesn’t prescribe optimizing some variable - human flourishing was never part of the equation in the first place. Liberal democratic capitalism is designed such that individuals can pursue their own interests freely from coercion etc. - there is no outcome being optimized for, rather just a set of rules and procedures to follow.
A better way to diagnose the big societal problems you mention is through the principal-agent problem combined with general ignorance and lack of political will. Case in point, you mentioned human flourishing and failed to acknowledge that capital formation is the only true enabler of human flourishing. And people can’t even agree on why the healthcare system sucks while special interests optimize for their own profits and the government officials enable the open looting. Moreover, people don’t even know how the banks facilitate financialization and create the circumstances for ridiculous wealth.
The entire system is literally designed to be optimized for money - there was never any genuine intent for human flourishing in the first place.
This is framed as disagreement, but I think you just agree with the premise
Not quite. Your thesis suggests that there was a sincere attempt by anybody to achieve some ideal outcome, when there was no such intent in the first place. Only later on, do people say, “wow this outcome sucked.”
It would be like if I got into a taxi and told the driver, “go somewhere, just don’t take any toll roads - I want to save money.” and then was unhappy/surprised with the results because the taxi driver drove a massive loop on the highway to drop me off back where we started (without taking any toll roads).
For your thesis to work, I would first had to have some semblance of a destination. Like if I said, “take me somewhere fun,” only for the taxi driver to drive a circuitous route to his favorite bar.
The public has 0 idea of what the outcome they actually want. They have no idea what a “good result” looks like when it comes to education, defense, healthcare, infrastructure, etc. Therefore, no directive or “spec sheet” is given to the politicians and stakeholders. From then on, the lobbyists work their magic to align the system with the interests of stakeholders. Suddenly, one day, the people realize the result sucks and they’ve been taken on a wild ride. Framing this as a problem of “optimizing for the wrong measurement metric” ignores a much more fundamental problem with liberalism the ideology.
Sorry, I still think this is basically agreement. Yes, we don't know how to measure "the good", so we try our best with systems that are not optimizing for "the good", but that's just the X vs ~X thing
There is no concept of “the good” in the first place - it doesn’t exist.
You’re saying, “we need better metrics and better control of the agents so they don’t optimize at the expense of our intended outcome that could be measured in more qualitative ways that the metric cannot adequately capture.”
Whereas I’m saying, “It’s pointless to even discuss metrics being optimized since the principal is schizophrenic. So long as the principal is schizophrenic, there is no hope of actualizing anything you’re suggesting”
Saving this to show it to everyone who gets a bad case of the spreadsheet brain.
I don't think we can totally abolish the rule of metrics as a society without enabling corruption to take its place, but yeah, for the moment, giving and getting some slack seems like the right way to go.