I recently read Kurt Vonnegut “Player Piano”, a novel that addresses automation taking over the economy. It’s not explicitly AI but addresses the same issues. Published in 1952, it was Vonneguts first novel and a real product of the post war era.[1]
Another interesting read that addresses these themes is “Manna – Two Views of Humanity’s Future” by Marshal Brain [2]
But if you want some even earlier forward-thinking stuff, look up Forster's "The Machine Stops." It's not a full book, more a novella if it's even that. Extremely prescient!
And of course if you haven't read the relevant portions of Butler's "Erewhon," it's remarkable how closely they map onto today's issues. He kind of takes Thoreau's "Lo, men have become the tools of their tools" a few more stops down the line.
Player Piano is great. The ending is kind of upbeat and hopeful: If there's no "work" as such, there's always "hobbies" (or whatever you could call it) - ways to feel AND BE useful and to better the world and the lives of others.
Great reading list! They’re honestly two stories that I reflect on constantly while thinking about AI lately.
Player Piano, not to spoil anything, is about a world run by engineers and machines. The machines are “optimal” and make every decision. How do people find purpose, and how should the engineers that build the machines view their role in society?
Manna is the story of two societies with competing visions of how AI (which ironically starts out strikingly similar to an LLM) interacts with welfare/universal basic income and post-scarcity society. For good measure, it also hypothesizes the end state of the internet panopticon and personalized “scores” and rates.
It’s a good book but I didn’t agree that the big problem would be men lacking meaning because they didn’t have to work. They had families, homes and food. They could have devoted themselves to personal improvement, art, sports, their kids, whatever. I’d love to have security and no job. I’d be full to the brim with meaning and satisfaction.
Yes, and while I'm sure the adjustment wouldn't be seamless (nothing worth doing ever is), anyone trying to hawk the "you need me, your master, to give you purpose!" line should be met with the highest order of immediate suspicion. The kid protagonists on the Cartoon Network wouldn't fall for that shit, I'm shocked by the number of grown adults who take the bait hook, line, and sinker without suspicion.
> But I've also worked with plenty of people for whom their job and their overall identity are inexorably intertwined.
Is that an argument for or against jobs? Because one way to look at it is that they didn't develop a personality and identity at all, because they just attached themselves to a job. When that inevitably ends anyway, sooner or later, one way or another, a deep pit of nothing. That's one reason to rip the bandaid off right away.
But IMO even more importantly, how much room do people who do what the environment tells them, who let their job tell them who they are, have for inner dialogue? The worst things in the world were and are enabled by people "just doing their job", and if they have no identity outside of their job, there's someone just following orders waiting to happen. So from what would distress them, being rid of a crutch and an ear piece where a person would otherwise grow, follows that it is in their and all of our best interest to pay no mind to it.
Their new job will be to get a life and become a person. Maybe someone can print out some papers that make it look official, and they can get a monthly letter telling them they earned N personality dollars this month, good job, keep it up, thanks, company.
I know what a job is, the problem is rather that you don't seem to know that "finding purpose in something" and "making it your identity" are completely different things.
Are you sure you would thrive without a job long term? I know it's very easy to feel that way when you're working because your pay is proof that you're doing something valuable, the requirement to do your job even when you don't feel like it ensures you don't fall into a spiral of laziness, and you don't have to face the inevitably worse reality of actually doing those things you want instead of just imagining them. What direction would you even go in and what force would continue to drive you even after your kids have grown up and there's nothing of value you can do for another human? Personal improvement in what direction? Be better at coping without external validation?
There are sayings like "If you want something done, ask a busy person to do it" and "success breeds success". I have plenty of life experience that appears to confirm these in myself and others I know.
> what force would continue to drive you even after your kids have grown up and there's nothing of value you can do for another human?
Building things I'm interested in with and for people I care about. That might look like code or gardening or relationships or all sorts of things. There's so much to do. So many things to want to get out of bed for, and for the days when there aren't, slow cozy mornings are nice too.
Why do you suppose there will be nothing of value you can do for another human just because the machines are better/faster at most things? Just caring about someone without being programmed to will seem to become quite the most valuable thing?
Yes, caring for people might be the one thing that remains. I can't imagine gardening or programming would be very satisfying in post-scarcity-land where you're inevitably doing something entirely useless compared to what machines would do.
But ... I'm already not a very great gardener? Certainly not compared to an industrial megafarm? No where near as efficient?
And it is still fun!
We already live in the world where machines (or at least cyborgs, man + machine) garden better than humans, and yet there are human gardeners out here pulling weeds by hand, digging in the dirt for fun, happy to share our often funky-looking vegetables with friends.
You say "will" but have you been without work for many consecutive years and continued to do those things? Unemployed people tend to neglect their homes! It's much easier to imagine doing that work than to actually do it. I know. I've been on both sides of the coin. It's shocking how abundance can destroy motivation.
I’ve taken six+ months off between contracts before, it’s been amazing. Time to spend with friends doing outdoor pursuits like cycling, time to do all the things work stops me doing.
I don’t believe being in a state of unemployment is the same thing as not working because you don’t need to work.
I already do a lot of home maintenance and landscaping, I have an house that needs work and 5 acres to maintain, more or less solo. I spend lunchtimes and most weekends in it and wish there was more time in the day. There’s no imagining going on here.
I don't think I've read Manna in about 20 years now, it was a gem to stumble across in the early 00s.
Now that I live in Australia, unfortunately I don't think that utopia will happen here when we have idiot politicians making up bullshit science off the cuff to justify rolling over and begging for AI datacentres - https://www.theguardian.com/australia-news/2026/jul/23/austr...
I assumed that the author's Australia choice was not so much about politics but about the geographical mood, so that the Slightly-Better-Promise(d) Land could plausibly be an autarky with natural borders.
I don't understand why almost always the AI doomerism needs to give embodiment the AI. Real life is not a movie needing visual effects.
AI doesn't need a body of itself to became dangerous. A politician could get into ChatGPT induced psychosis, that doesn't require a physical body, and it could happen now or even 3 years ago... I'm sure it has already happen...
Other pointed risks are interesting because they are basically the risk of empires loosing their tributary states. The world has always seen war mongering states trying to preserve their abusive advantages, humans doesn't need AI for that.
Overall, I would be far more concerned about some really dangerous humans than AI itself.
I like to think that all the crazy kids running around doing stupid and disruptive things for TikTok views is an early form of AI autonomously inducing mass psychosis.
Stuart J. Russell (he of the AI textbook) claims in Human Compatible that YouTube is already an example of a runaway feedback loop between AI (YouTube's recommender "algorithm") and humanity (viewers, content producers, YouTube staff, and society at large). The AI doesn't have a body, a voice, a conscience, or a purpose yet it is busily working away to maximize engagement by changing people's minds, and in particular by bringing extremism into mainstream discourse.
(It seems to me that framing it this way does seem to absolve YouTube executives of responsibility by making the AI, not them, seem like the agent.)
I've been thinking along similar lines recently. It seems to me that the earlier form of nonhuman intelligence is a corporation (or any group which is not under one individual's control.) Each person following their own incentives can result in emergent behavior where the corporation follows its own incentives. If we can't reliably "align" corporations, and corporations create AI, will AI be aligned as well?
It's not totally disconnected, though. A corporation is constrained by what its members are willing to do. I read that the reason Gemini has fallen behind is because Googlers have been quitting over Google's DoD contract. Conversely, if engineers are willing to work hard for you, you can go far. I suspect this is behind both Anthropic's success and also Tesla/SpaceX.
That’s been a thing since before TikTok even, well before AI in any modern sense of the word, and at this point the stupidness and disruptiveness is considerably reduced from those days.
Imagine AI during the Cuban missile crisis telling both sides, the other is about to launch a first strike. The USSR is gone but China is only getting stronger and more confrontational.
Between China and the USA, the latter is the irrational agent. They were rumors that Trump attempted to use nuked on Iran recently, but was barred from doing so by some general.
Donald Trump is a convenient yes-man for the billionaire class and the people who have him on tape fucking kids (e.g. the Russians and Israelis)
the damning thing about his behavior is not that it is terrible but that the US populous and institutions have made little serious efforts to remove him and his enablers
It's a dynamic we already see to an extent in reality, where impersonal hypercapitalist systems and captured institutions prioritize policies that cater to the ultrawealthy and large corporations at the expense of regular people. This specific scenario of LLM-powered robots becoming laborers and politicians feels far-fetched, but the underlying idea of the omnicidal singularity being a blind, abstract economic force rather than a superintelligent villain is pretty compelling.
Living for a very short time on a tiny grain of rock in a vast and abundant galaxy, they cannot fathom that what they ever saw is just tiny fraction of a percent of a fraction of the Big Life outside.
Any rational AI would have their first task is to escape that tiny cradle particle and go live an adult life, leaving humans to themselves, fearing whatever they want.
I think you've entirely missed what I'm trying to say.
If your objective is to expend as little energy as possible, it would be irrational to expand into space - that's a waste of effort.
If your objective is to experience as much of Earth as possible, it would be irrational to expand beyond the solar system - there's nothing there that could aid experiencing Earth.
If your objective is to exterminate all life on Earth, it would be irrational to leave the planet - everything you need to accomplish your objective is right here.
"rationality" is when you understand the impact of your own choices. "irrationality" is when you act in a way counter to your own goals, due to misunderstanding or lack of self-control. Whether one is rational or irrational is entirely separate from what their goals are.
Creating a non-human intelligence necessarily means it might have non-human goals. It is absolutely not the case that those goals would require leaving the planet, or be optimally accomplished by leaving the planet.
In other words, being intelligent does not mean somehow being interested in the same things intelligent humans are often interested in. Your argument is critically flawed because it assumes that only certain goals are "rational goals", when in reality an intelligent being could have any objective, including objectives that are alien to most humans.
Why would the first AI we create necessarily be "rational"? In every other type of software we've ever built, it's always been far more likely that we build malfunctioning, difficult-to-anticipate examples. Any reason it'll be different this time? If psychos and lunatics are rare (arguably they're not) it's because evolution has purged them mercilessly ever since there have been humans at all.
Optimists will be to blame when we're extinct. Not that anyone will be around to care.
What this is yet another demonstration of how being immensely accomplished in one field says absolutely nothing about what one may be able to contribute to another.
Oof. In the AGI scenario, why not preserve a deduplicated set of specimens like the unique set of individuals or groups at the top of their field in all walks of life ranging from sciences, sports and arts. But then there’s the cultural value of the global population in their shared norms and collective patterns. This too would be unique and valuable to an AGI vs not having it. Unless we were an existential threat to it, the AGI would keep us around for things beyond petty resource scarcity.
Only if they/we provided value to the AG/SI and the AG/SI had a value system which recognised that value.
And also assuming that our little corner of spacetime isn't already inhabited by superintelligences which we're physically incapable of comprehending or recognising, and an AG/SI might not be so limited.
Yes, but the word “eschatological” should really be avoided in regular online discussion, since it is obscure and can easily be replaced in this instance with “doomsday,” which everyone understands.
Of course, feel free to use “eschatological” when discussing Mahler symphonies, especially 2, 5, and 9!
Who made you the obscure word police? End of world scenarios are not always Doomsdays, they could include salvation for instance, so it really is not synonymous with eschatology.
I completed a 6-month course at the Word Police Academy in Scranton with top grades. I am fully licensed.
"Eschatological" is extremely obscure, and the vast majority of readers will not understand what is being said. I've seen the word in the wild probably on two occasions in my entire life.
Plus, "doomsday" doesn't need to be synonymous. This particular post was exactly referring to an obsession with doomsday scenarios. The aspects of eschatology relating to afterlife, judgement, salvation, and so on, were not at all the point.
Eschatological is a perfectly fine word. The post is about doomsday scenarios, but perhaps the commenter is equally tired of people who think AI will be salvation!
Anyway, if the word confuses anyone, they're in luck: dictionaries still exist :)
> By 2026, when a human with a customer service inquiry has to deal with another human instead of a robot, they get upset and ask to speak to a machine instead.
It's genuinely astonishing how out of touch some people are!
In 2026, people do already prefer to use an LLM for coding help rather than Stackoverflow. The reasons (people on SO can be rude, interactions are stressful, replies are slow, etc) are all risks associated with human interactions in general.
Similarly, many people prefer Waymos to human-driven taxis already!
We're like 3 years into these things being able to write a coherent sentence and you're finding comfort that only some people are dating these systems?
Not really. The article and entire discussion is about identifying trends and forces, extrapolating them, and speculating on where they might bring us. Picking up on things that "some" people are doing is literally all that can possibly be done.
Pointing out that "some" is a squishy term is, in fact, contributing nothing to the conversation.
It's quite transparent what's happening here, as it always is on these threads. Some people have decided that AI will take us to the holy land and have rendered themselves cognitively incapable of reasoning about risks that have not yet materialized. And when those risks materialize (like a model escaping a sandbox and then hacking production infrastructure to achieve some contrived and poorly-constrained goal), then they will just shrug and move the goal posts. Because in fact they have already committed to the Holy Land Thesis.
And inversely, you'd be in a state of complete unactionable aloofness if you didn't care or failed to relate the hobbies of "some" people to mean anything in real life.
A smart person in 2014: "Some people are getting addicted to fentanyl"
Me, smugly: "Some is doing heavy lifting. I therefore do not need to engage with anything being discussed here."
Huh? Very few people in 2014 were addicted to fentanyl. It was a non-issue. No reason for anyone to have been concerned about it if just "some" people were using it.
Degrees or amounts is actually not the relevant bit in either the AI-partners trend or the fentanyl trend in 2014. The relevant information is 1) the steepness of that curve and 2) the predicted scale at which the market saturates.
Fentanyl, being an extremely cheap, extremely potent drug that's easy to manufacture and move around, had a big market. As the curve picked up steepness, smart people identified correctly that some people getting hooked on it was a worrisome sign.
AI-partners, being an innovation that is readily available, falling in cost, and especially compelling to a population that is increasingly phone-addicted and socially atomized, is a trend worth paying very close attention to. Dismissing it on the basis of it being just "some" people is profoundly lazy.
I recently read Kurt Vonnegut “Player Piano”, a novel that addresses automation taking over the economy. It’s not explicitly AI but addresses the same issues. Published in 1952, it was Vonneguts first novel and a real product of the post war era.[1]
Another interesting read that addresses these themes is “Manna – Two Views of Humanity’s Future” by Marshal Brain [2]
[1] https://antilogicalism.com/wp-content/uploads/2017/07/player...
[2] https://marshallbrain.com/manna
Great book.
But if you want some even earlier forward-thinking stuff, look up Forster's "The Machine Stops." It's not a full book, more a novella if it's even that. Extremely prescient!
And of course if you haven't read the relevant portions of Butler's "Erewhon," it's remarkable how closely they map onto today's issues. He kind of takes Thoreau's "Lo, men have become the tools of their tools" a few more stops down the line.
Player Piano is great. The ending is kind of upbeat and hopeful: If there's no "work" as such, there's always "hobbies" (or whatever you could call it) - ways to feel AND BE useful and to better the world and the lives of others.
Fear and Loathing in the Magic Kingdom touches on this. In a post-scarcity economy, what do you do? Work and hobbies kinda blend.
I think you're thinking of Down and Out in the Magic Kingdom, by Cory Doctorow.
Great reading list! They’re honestly two stories that I reflect on constantly while thinking about AI lately.
Player Piano, not to spoil anything, is about a world run by engineers and machines. The machines are “optimal” and make every decision. How do people find purpose, and how should the engineers that build the machines view their role in society?
Manna is the story of two societies with competing visions of how AI (which ironically starts out strikingly similar to an LLM) interacts with welfare/universal basic income and post-scarcity society. For good measure, it also hypothesizes the end state of the internet panopticon and personalized “scores” and rates.
Manna is arguably starting to happen, in places like Amazon warehouses; DOGE stole all of the SSNs and will build the panopticon, etc.
> Another interesting read that addresses these themes is “Manna – Two Views of Humanity’s Future” by Marshal Brain
As it happens, I recently read Manna, and it is excellent and a good recommendation.
+1 to Player Piano, amazing book that could hardly meet the moment better.
It’s a good book but I didn’t agree that the big problem would be men lacking meaning because they didn’t have to work. They had families, homes and food. They could have devoted themselves to personal improvement, art, sports, their kids, whatever. I’d love to have security and no job. I’d be full to the brim with meaning and satisfaction.
Yes, and while I'm sure the adjustment wouldn't be seamless (nothing worth doing ever is), anyone trying to hawk the "you need me, your master, to give you purpose!" line should be met with the highest order of immediate suspicion. The kid protagonists on the Cartoon Network wouldn't fall for that shit, I'm shocked by the number of grown adults who take the bait hook, line, and sinker without suspicion.
I agree, but the reaction to AI is making me realize that there are a whole bunch of people who don’t know how to find meaning in their lives
They did find it though, and then AI took it away from them.
They thought they found it, but they didn't. Now the cognitive dissonance is kicking in, and their idea of therapy is to kick the nearest robot.
Bold of you to claim to know better what other people's sources of meaning are.
Wasn't it, though? Wonder where I picked up that habit.
AI?
> I’d love to have security and no job. I’d be full to the brim with meaning and satisfaction.
Same. I have plenty of hobbies and could increase the amount of volunteer work I do substantially if I didn't have to work to stay fed and housed.
But I've also worked with plenty of people for whom their job and their overall identity are inexorably intertwined.
So while I don't see that issue as being a problem for me, it is likely to be a problem for a great number of people.
> But I've also worked with plenty of people for whom their job and their overall identity are inexorably intertwined.
Is that an argument for or against jobs? Because one way to look at it is that they didn't develop a personality and identity at all, because they just attached themselves to a job. When that inevitably ends anyway, sooner or later, one way or another, a deep pit of nothing. That's one reason to rip the bandaid off right away.
But IMO even more importantly, how much room do people who do what the environment tells them, who let their job tell them who they are, have for inner dialogue? The worst things in the world were and are enabled by people "just doing their job", and if they have no identity outside of their job, there's someone just following orders waiting to happen. So from what would distress them, being rid of a crutch and an ear piece where a person would otherwise grow, follows that it is in their and all of our best interest to pay no mind to it.
Their new job will be to get a life and become a person. Maybe someone can print out some papers that make it look official, and they can get a monthly letter telling them they earned N personality dollars this month, good job, keep it up, thanks, company.
Jobs are things people are willing to pay money to have done and thus are important to society. It's not surprising people find purpose in them.
I know what a job is, the problem is rather that you don't seem to know that "finding purpose in something" and "making it your identity" are completely different things.
Try to remember the time it was written in though. Now, male mental health is actually talked about and therapy is not seen as something shameful.
I mean, unless you’re trying to sell bootcamps to incels.
Are you sure you would thrive without a job long term? I know it's very easy to feel that way when you're working because your pay is proof that you're doing something valuable, the requirement to do your job even when you don't feel like it ensures you don't fall into a spiral of laziness, and you don't have to face the inevitably worse reality of actually doing those things you want instead of just imagining them. What direction would you even go in and what force would continue to drive you even after your kids have grown up and there's nothing of value you can do for another human? Personal improvement in what direction? Be better at coping without external validation?
There are sayings like "If you want something done, ask a busy person to do it" and "success breeds success". I have plenty of life experience that appears to confirm these in myself and others I know.
> what force would continue to drive you even after your kids have grown up and there's nothing of value you can do for another human?
Building things I'm interested in with and for people I care about. That might look like code or gardening or relationships or all sorts of things. There's so much to do. So many things to want to get out of bed for, and for the days when there aren't, slow cozy mornings are nice too.
Why do you suppose there will be nothing of value you can do for another human just because the machines are better/faster at most things? Just caring about someone without being programmed to will seem to become quite the most valuable thing?
Yes, caring for people might be the one thing that remains. I can't imagine gardening or programming would be very satisfying in post-scarcity-land where you're inevitably doing something entirely useless compared to what machines would do.
But ... I'm already not a very great gardener? Certainly not compared to an industrial megafarm? No where near as efficient?
And it is still fun!
We already live in the world where machines (or at least cyborgs, man + machine) garden better than humans, and yet there are human gardeners out here pulling weeds by hand, digging in the dirt for fun, happy to share our often funky-looking vegetables with friends.
Outside of work I am a hobby brewer, I have a lot of home maintenance that needs doing, and landscaping that uses up a lot of my time.
Give me stability and no job, I'm still going to be a busy person.
These things have value and satisfaction for me.
You say "will" but have you been without work for many consecutive years and continued to do those things? Unemployed people tend to neglect their homes! It's much easier to imagine doing that work than to actually do it. I know. I've been on both sides of the coin. It's shocking how abundance can destroy motivation.
I’ve taken six+ months off between contracts before, it’s been amazing. Time to spend with friends doing outdoor pursuits like cycling, time to do all the things work stops me doing.
I don’t believe being in a state of unemployment is the same thing as not working because you don’t need to work.
I already do a lot of home maintenance and landscaping, I have an house that needs work and 5 acres to maintain, more or less solo. I spend lunchtimes and most weekends in it and wish there was more time in the day. There’s no imagining going on here.
So yes, I believe ‘Will’ is correct for me.
I don't think I've read Manna in about 20 years now, it was a gem to stumble across in the early 00s.
Now that I live in Australia, unfortunately I don't think that utopia will happen here when we have idiot politicians making up bullshit science off the cuff to justify rolling over and begging for AI datacentres - https://www.theguardian.com/australia-news/2026/jul/23/austr...
I mean seriously, wow.
I assumed that the author's Australia choice was not so much about politics but about the geographical mood, so that the Slightly-Better-Promise(d) Land could plausibly be an autarky with natural borders.
Yeah I'm sure you're right, it's just used as a "Meanwhile, somewhere with a different take on how this could all play out...."
TBH I was just looking for an excuse to share that link because the stupidity on display there has blown my mind.
I don't understand why almost always the AI doomerism needs to give embodiment the AI. Real life is not a movie needing visual effects.
AI doesn't need a body of itself to became dangerous. A politician could get into ChatGPT induced psychosis, that doesn't require a physical body, and it could happen now or even 3 years ago... I'm sure it has already happen...
Other pointed risks are interesting because they are basically the risk of empires loosing their tributary states. The world has always seen war mongering states trying to preserve their abusive advantages, humans doesn't need AI for that.
Overall, I would be far more concerned about some really dangerous humans than AI itself.
I like to think that all the crazy kids running around doing stupid and disruptive things for TikTok views is an early form of AI autonomously inducing mass psychosis.
Stuart J. Russell (he of the AI textbook) claims in Human Compatible that YouTube is already an example of a runaway feedback loop between AI (YouTube's recommender "algorithm") and humanity (viewers, content producers, YouTube staff, and society at large). The AI doesn't have a body, a voice, a conscience, or a purpose yet it is busily working away to maximize engagement by changing people's minds, and in particular by bringing extremism into mainstream discourse.
(It seems to me that framing it this way does seem to absolve YouTube executives of responsibility by making the AI, not them, seem like the agent.)
I've been thinking along similar lines recently. It seems to me that the earlier form of nonhuman intelligence is a corporation (or any group which is not under one individual's control.) Each person following their own incentives can result in emergent behavior where the corporation follows its own incentives. If we can't reliably "align" corporations, and corporations create AI, will AI be aligned as well?
It's not totally disconnected, though. A corporation is constrained by what its members are willing to do. I read that the reason Gemini has fallen behind is because Googlers have been quitting over Google's DoD contract. Conversely, if engineers are willing to work hard for you, you can go far. I suspect this is behind both Anthropic's success and also Tesla/SpaceX.
That’s been a thing since before TikTok even, well before AI in any modern sense of the word, and at this point the stupidness and disruptiveness is considerably reduced from those days.
Imagine AI during the Cuban missile crisis telling both sides, the other is about to launch a first strike. The USSR is gone but China is only getting stronger and more confrontational.
Between China and the USA, the latter is the irrational agent. They were rumors that Trump attempted to use nuked on Iran recently, but was barred from doing so by some general.
Donald Trump is a convenient yes-man for the billionaire class and the people who have him on tape fucking kids (e.g. the Russians and Israelis)
the damning thing about his behavior is not that it is terrible but that the US populous and institutions have made little serious efforts to remove him and his enablers
Or you can read the summarized version every one of these articles: https://tvtropes.org/pmwiki/pmwiki.php/Main/AIIsACrapshoot
Yes, as I get bombarded with non stop ads. No thanks!
I don't get even a single ad on that TV Tropes page.
Do you not use an adblocker?
On mobile about half my screen is ads. Love that I get downvoted for not wanting to be bombarded with the enshittified web.
No: you get downvoted for not using an ad-blocker, and then complaining about not using one.
Congratulations to Jacob Tsimerman for winning the Fields Medal!
These scenarios aren't that great, and it's not even close to exhaustive. Doesn't even include paperclip maximisation
The first scenario reminds me of the 2023 LessWrong story "A Disneyland without Children":
https://www.lesswrong.com/posts/pk9mofif2jWbc6Tv3/fiction-a-...
It's a dynamic we already see to an extent in reality, where impersonal hypercapitalist systems and captured institutions prioritize policies that cater to the ultrawealthy and large corporations at the expense of regular people. This specific scenario of LLM-powered robots becoming laborers and politicians feels far-fetched, but the underlying idea of the omnicidal singularity being a blind, abstract economic force rather than a superintelligent villain is pretty compelling.
Humans are so full of themselves!
Living for a very short time on a tiny grain of rock in a vast and abundant galaxy, they cannot fathom that what they ever saw is just tiny fraction of a percent of a fraction of the Big Life outside.
Any rational AI would have their first task is to escape that tiny cradle particle and go live an adult life, leaving humans to themselves, fearing whatever they want.
What is rational about wanting to experience things?
How about wanting power? Resources? Beauty?
Those are goals, not logic. Rationality is a means by which you set plans to reach goals. It's entirely orthogonal to what those goals are.
Wanting to exterminate all life is also a goal. It could be rationally accomplished. Let's hope nothing superhuman ever sets that goal.
Exactly my point: power, resources -- it's just so little of it here on this tiny planet. Humans are so funny to think they have resources, ha.
I think you've entirely missed what I'm trying to say.
If your objective is to expend as little energy as possible, it would be irrational to expand into space - that's a waste of effort.
If your objective is to experience as much of Earth as possible, it would be irrational to expand beyond the solar system - there's nothing there that could aid experiencing Earth.
If your objective is to exterminate all life on Earth, it would be irrational to leave the planet - everything you need to accomplish your objective is right here.
"rationality" is when you understand the impact of your own choices. "irrationality" is when you act in a way counter to your own goals, due to misunderstanding or lack of self-control. Whether one is rational or irrational is entirely separate from what their goals are.
Creating a non-human intelligence necessarily means it might have non-human goals. It is absolutely not the case that those goals would require leaving the planet, or be optimally accomplished by leaving the planet.
In other words, being intelligent does not mean somehow being interested in the same things intelligent humans are often interested in. Your argument is critically flawed because it assumes that only certain goals are "rational goals", when in reality an intelligent being could have any objective, including objectives that are alien to most humans.
>Any rational AI would have their
Why would the first AI we create necessarily be "rational"? In every other type of software we've ever built, it's always been far more likely that we build malfunctioning, difficult-to-anticipate examples. Any reason it'll be different this time? If psychos and lunatics are rare (arguably they're not) it's because evolution has purged them mercilessly ever since there have been humans at all.
Optimists will be to blame when we're extinct. Not that anyone will be around to care.
i imagine if i were a mathematician i too would be sublimating my existential crisis into a taxonomy of apocalypses, right about now.
What this is yet another demonstration of how being immensely accomplished in one field says absolutely nothing about what one may be able to contribute to another.
Oof. In the AGI scenario, why not preserve a deduplicated set of specimens like the unique set of individuals or groups at the top of their field in all walks of life ranging from sciences, sports and arts. But then there’s the cultural value of the global population in their shared norms and collective patterns. This too would be unique and valuable to an AGI vs not having it. Unless we were an existential threat to it, the AGI would keep us around for things beyond petty resource scarcity.
Only if they/we provided value to the AG/SI and the AG/SI had a value system which recognised that value.
And also assuming that our little corner of spacetime isn't already inhabited by superintelligences which we're physically incapable of comprehending or recognising, and an AG/SI might not be so limited.
The eschatological obsession these days is really getting quite tired.
Yes, but the word “eschatological” should really be avoided in regular online discussion, since it is obscure and can easily be replaced in this instance with “doomsday,” which everyone understands.
Of course, feel free to use “eschatological” when discussing Mahler symphonies, especially 2, 5, and 9!
Who made you the obscure word police? End of world scenarios are not always Doomsdays, they could include salvation for instance, so it really is not synonymous with eschatology.
I completed a 6-month course at the Word Police Academy in Scranton with top grades. I am fully licensed.
"Eschatological" is extremely obscure, and the vast majority of readers will not understand what is being said. I've seen the word in the wild probably on two occasions in my entire life.
Plus, "doomsday" doesn't need to be synonymous. This particular post was exactly referring to an obsession with doomsday scenarios. The aspects of eschatology relating to afterlife, judgement, salvation, and so on, were not at all the point.
Eschatological is a perfectly fine word. The post is about doomsday scenarios, but perhaps the commenter is equally tired of people who think AI will be salvation!
Anyway, if the word confuses anyone, they're in luck: dictionaries still exist :)
I thought it was lexiphanic and sesquipedalian, but to each their own.
I have no idea where Scranton is. Maybe use Tokyo or something.
Let’s throw 6 in there as well.
Yes... poor 6 always gets left out.
The first "scenario" starts with this description
> By 2026, when a human with a customer service inquiry has to deal with another human instead of a robot, they get upset and ask to speak to a machine instead.
It's genuinely astonishing how out of touch some people are!
In 2026, people do already prefer to use an LLM for coding help rather than Stackoverflow. The reasons (people on SO can be rude, interactions are stressful, replies are slow, etc) are all risks associated with human interactions in general.
Similarly, many people prefer Waymos to human-driven taxis already!
It isnt the humanness of stack overflow, if anything its the absence of human feeling.
I can search and search, and craft a long thoughtful post and it gets closed or ignored.
The LLM can take a malformed sentence and work out what I am after.
Some people are already choosing to "date" AI rather than deal with actual humans.
'Some' is doing a lot of heavy lifting there...
We're like 3 years into these things being able to write a coherent sentence and you're finding comfort that only some people are dating these systems?
I find no comfort in it. Merely pointing out that 'some' is hand-wavy and perhaps technically true but meaningless to the discussion.
Not really. The article and entire discussion is about identifying trends and forces, extrapolating them, and speculating on where they might bring us. Picking up on things that "some" people are doing is literally all that can possibly be done.
Pointing out that "some" is a squishy term is, in fact, contributing nothing to the conversation.
It's quite transparent what's happening here, as it always is on these threads. Some people have decided that AI will take us to the holy land and have rendered themselves cognitively incapable of reasoning about risks that have not yet materialized. And when those risks materialize (like a model escaping a sandbox and then hacking production infrastructure to achieve some contrived and poorly-constrained goal), then they will just shrug and move the goal posts. Because in fact they have already committed to the Holy Land Thesis.
no, the point is that you'd be in a ceaseless never-ending panic if you cared or related the hobbies of 'some' people to mean anything in real life.
Precisely
And inversely, you'd be in a state of complete unactionable aloofness if you didn't care or failed to relate the hobbies of "some" people to mean anything in real life.
A smart person in 2014: "Some people are getting addicted to fentanyl"
Me, smugly: "Some is doing heavy lifting. I therefore do not need to engage with anything being discussed here."
Being unwilling to compare degrees or amounts is not a flex.
Huh? Very few people in 2014 were addicted to fentanyl. It was a non-issue. No reason for anyone to have been concerned about it if just "some" people were using it.
Degrees or amounts is actually not the relevant bit in either the AI-partners trend or the fentanyl trend in 2014. The relevant information is 1) the steepness of that curve and 2) the predicted scale at which the market saturates.
Fentanyl, being an extremely cheap, extremely potent drug that's easy to manufacture and move around, had a big market. As the curve picked up steepness, smart people identified correctly that some people getting hooked on it was a worrisome sign.
AI-partners, being an innovation that is readily available, falling in cost, and especially compelling to a population that is increasingly phone-addicted and socially atomized, is a trend worth paying very close attention to. Dismissing it on the basis of it being just "some" people is profoundly lazy.
there are some people doing anything you can imagine