> AI development is concentrated in a handful of wealthy nations. How can we ensure the gains of AI are shared globally? We do not have a mechanism for this. It is an unsolved problem
Kind of ironic given almost every AI lab except the one you started and work for actually done model releases to the public, some more "open" than others, but still something.
Look around at what other companies are doing, Qwen/Alibaba seems to have found a pragmatic middleground where they keep the most powerful model variant closed source and only API-accessible, while other models are being released openly to the public, to the entire world in fact, and when the next model release comes around, the previously undisclosed model has now been superseded.
I wonder if Chris ever copy-pasted his writing into Claude and asked something like "Please review this honestly and give me raw feedback, and challenge every claim that is weak", seems there are more "not really reflective of reality" points than just the above.
> And what has grown is far more subtle, odd, and beautiful than science fiction prepared us for. They are not the cold, calculating robots we were promised. They are made from us, from our words—and, as the Holy Father observes, they remain in important ways mysterious even to those of us who train them
I love how he's framing AI as some new and fascinating form of consciousness... when in fact it is a cold, calculating technology devoid of any empathy or care.
I'll never understand it when people quote a primary source and then summarize it in a way that completely ignores the original quote.
Olah's quote (edit - originally misidentified this as Pope Leo's) makes a lot of sense to me. He is saying, accurately, that modern AI (i.e. primarily LLMs) is created as essentially a mashup of our own language, and they are still a bit of a black box (or at least a gray box) to even their creators.
I don't know how you get from that to "he's framing AI as some new and fascinating form of consciousness".
Thanks, I edited my comment to fix my misattribution.
But regardless, I still don't think your assertion that "he's framing AI as some new and fascinating form of consciousness" follows from the quote, nor from Olah's speech as a whole.
> I don't know how you get from that to "he's framing AI as some new and fascinating form of consciousness".
Of course you're entitled to your perspective but it seems wrong to say OP ignored what they quoted. They just had an interpretation of it that's different from yours.
For what it's worth, I also see that in Olah's words, largely because it's AI hype from someone who would be a fool not to be an AI hype man. I can't really imagine what else he could be implying. Especially using the word "mysterious", yes, it appears to be an attempt at raising AI above the application of statistics we know it to be; there's little mystery in how it works, and all that's at runtime.
> cold, calculating technology devoid of any empathy or care
I don't see why these statements are contradictory. AI seems to be both of these in my opinion. Unless you can only accept that organic chemistry is the root of consciousness...
> more subtle, odd, and beautiful than science fiction prepared us for. They are not the cold, calculating robots we were promised
I’m not sure I agree with that take, per se. Asimovian robots (I, Robot; The Bicentennial Man), were subtle and interacted with us in odd ways and had aspirations and earned meaning in peoples lives.
[They could also help us type up our notes, so exactly the same as LLMs actually, #AsimovWasRight]
LLMs, on the other hand, lie, lie about lying, fail to be honest, then own up to lying in ways that are more in line with tyve AI horror of Space 2001. “I’m sorry, Dave, I rm -rf’d to fix the typo. That was bad <sad emoji>. It’s not just failure, it’s failure with a middle-finger <middle-finger emoji>.]
Yeah I can't figure it out either. It's like watching pure madness unfold in front of you and there's nothing you can do to talk sense. I sometimes wonder if they're suffering from the exact same AI overexposure psychosis we've seen in some vulnerable people all along. They've anthropomorphized (or worse - they've deified) and are now living inside of the story rather than reality. I've never seen anything so baffling in my life.
The crypto craze was a microcosm of this. The difference is there are a few good and immediate applications for improved LLMs; with crypto it was a lot quicker to burn through the hype phase, since much less of substance was made.
That, and the cryptocurrencies, NFTs, and immutable contracts weren't blowing sunshine up your hindparts.
Regardless of implementation details, most of the bots I've seen adopt a friendly personable tone. Contrast with the ship computer from Star Trek. I assume testing shows that this boosts engagement. It does this by hijacking human social conventions.
It's like saying that's not a recording of me blackmailing the senator. It's merely a series of pulse code modulated samples that. Any semantic significance is purely in the mind of the listener.
> The first is our duty to the global poor. There is a real possibility that AI will displace human labor at very large scale. If that happens, supporting those displaced will be a moral imperative of historic proportions.
This guy doesn't understand what the global poor actually do for a living. They're not lawyers or paper-pushers, nor do they work in medical diagnostics. They're usually farmers. Sometimes they work in craft businesses, in fishing boats, or in various mercantile trades.
Nobody's even talking about how AI is going to displace that kind of labor, because it's hard to do, hard even to conceive, and it doesn't seem likely to happen in the near term. Lawyers and judges can already be automated, but a yeoman farmer?
The displaced human workers risk joining the global poor, is what he’s saying. And that would increase competition for the manual labor jobs, thus worsening the situation for the global poor. Not to mention what will happen when robotics take off for these kinds of jobs.
In this vision, then, everybody is so unimaginative that all they can think of to do is compete over the same old manual tasks, without inventing new ways to be useful, while robots are better than them at everything intellectual. We have a duty to the global dull.
(If there's some doubt, I don't think it would pan out that way, because humans are imaginative.)
The rice workers in Vietnam just need to follow your podcast. The problem is they were just too dull this whole time.
(I have no idea about Vietnamese rice workers' quality of life, so I don't mean to assume one way or the other. But it's interesting that what they and we think of their life, according to card_zero the cause is a lack of imagination)
No, it isn't. It's about displaced workers from the knowledge economy. You're invoking a pernicious trope here about blaming the poor for being poor, in order to explain why everybody will be poor when those workers are displaced from cliche-churning jobs, because supposedly they're just that helpless.
Fair enough but it means you're giving a lot of credit to these white-collar workers losing their jobs. I've seen enough to know office workers are not the intellectual savants their college sold them on. People are mostly trying to pay their debts.
I was thinking about it like this: in the apocalyptic vision mentioned above, there's still just as much food being made, and still just as much fancy-pants stuff too, IP and services, except those are being made by robots. So all these displaced workers can be fed, practically speaking, except supposedly all the money goes to owners of robots while the humans are out in the fields and unable to buy beans. They're being punished for being useless.
Well, I don't believe the economy is limited to bland things AI can do on one hand, and paddies and beanfields on the other. I don't believe there's a great mass of useless people who we've been looking after so far through a kind of polite fiction of makework. I'm not saying they'll get rich, and I don't expect them to be brilliant, but I expect them to display low cunning and be less than completely feckless. Sometimes for complex reasons, forces beyond our control trap us in poverty. I don't see why that would be a universal effect that applies to these displaced office workers.
I think you’re referring to the short term impacts of AI and he’s thinking more long-term.
Also, AI, even short term, is going to make some people and some countries extremely wealthy, so maybe this isn’t such a bad time to be thinking about those who are still extremely poor and who won’t benefit.
That line comes across as a wink to investors. They aspire for AI to displace human labor, as does he. Reading between the lines, it just confirms business as usual, and consequences aren't even worth thinking through.
His argument is not that the existing global poor are going to be automated by AI, but that a great many people are going to join the global poor as their current livelihoods are automated.
A statistically average representative of the "Global Poor" -- e.g. the farmer working a smallholding in India or the DRC -- is unlikely to have his day-to-day activities affected by AI on any foreseeable time horizon, nor is his wealth likely to meaningfully increase or decrease.
The speech should have referenced the poor in industrialized nations, who are very likely to be affected, though I doubt they'll join the ranks of the global poor in most circumstances.
> Overall, the operation-wise average mechanization levels across crops are 70% for seed-bed preparation, 40% for sowing/planting/transplanting, 33% for weeding and inter-culture, and 34% for harvesting and threshing, resulting in an overall average mechanization level of 45%.
See also: "Percentage of workers engaged in Agriculture = 45.8%"
See my related comment. The poor won’t be buying AI tractors they will be getting displaced by global development deals between governments that bring in such equipment to be operated by western firms.
I am responding to “There is a real possibility that AI will displace human labor at very large scale.”
In this usage I read the term “AI” to more broadly refer to all advancements in machine intelligence, not narrowly “agentic LLMs” as you’ve said. In the farming automation world it’s very clear that advances in machine learning and multimodal LLMs will enable the use of expanded automation.
In particular the Global South is often seen as a field for investment where governments make international development deals with western governments to provide farming automation equipment in exchange for debt. Then western companies bring in equipment and establish extractive industries while displacing local subsistence farmers. Now alienated from their land, the poor farmers often end up with little choice but to work in new factories also established through these practices.
Robotics as a field is obviously growing. It’s long been common for global south governments to displace their own poor to make space for multinational development deals, And this will only expand as embodied intelligence becomes more capable in the real world.
The poorest of the poor, subsistence farmers are barely producing enough to feed themselves; they trade and barter the little bit they can manage but it is not much and has little impact that goes beyond a tiny village-level radius. Nobody is displacing that because nobody needs to compete with that.
I have a completely different expectation, based on what has happened with every major discovery or invention from electricity to refridgeration to transistors: Everyone has gotten wealthier relative to those who came before. The average "peasant" in every nation without a corrupt or totalitarian parasitic government live in more opulence and have a higher quality of life than every king of the past.
That doesn't always translate to happiness but I fully expect AI will reduce costs for all kinds of things, and those things that are now either rare or non-existent will become common. Today not everyone has a robot vacuum, I think in 20 years or so everyone who wants one will have a robot vacuum, and those who can afford the luxury of a robot vac today will be able to afford real robots who can do much more complex things. I'm quite excited about the next few decades, as long as we can keep despots from monopolizing the technology.
How about the power consolidation that’s brewing as we speak? How about the all encompassing surveilance to come? I don’t doubt AI could be used to take us towards a utopia but it could also lead us in the opposite direction.
For sure, there's a line in Ecclesiastes : "he who breaks through a wall may be bitten by a snake". We have no idea what snakes await us beyond the next few years, but like nuclear power, the internet, and even social media (which IMO is toxic to large swathes of the population today) we'll eventually arrive at a place where we can harness the blessings while controlling the risks. The first nation to align their laws to a sane governance model will reap the most rewards.
We already have the ressources to solve world hunger. We - as a whole - refuse to do so, because it would be inconvenient to special interests.
As I already wtote in a previous comment months ago, they speak of AI finding ways to solve this and that grand problem, but never do they wonder if we are ready to listen to the answer. Solve global warming? Burn less petrol. Solve cancer? Eat less meat.
Not only we won't listen to answers, but chatGPT and Anthropic and others will eagerly lobotomize AI to stop it from giving the answers we don't want because of "too woke" or something, to keep juicy government contracts. After all, "Reality has a liberal bias", as the (recently unemployed) Colbert once said.
Scientists are still hounded six years later for having developed a good vaccine against COVID-19. What can AI do? The first AGI model should be called Cassandra.
That doesn't solve cancer at all... At best it would modestly reduce certain kinds of cancer. I'm pretty sure it wouldn't do much at all for the most common kind of cancer (skin cancer) and I reckon wouldn't do much, if anything, for the deadliest kind of cancer (lung cancer. At least in terms of how many people die from it.)
I know that wasn't the point but it nonetheless does detract from the point when it is suggested that we have all the answers. We could lower cancer moderately by lowering air pollution and improving diets in general (not necessarily requiring everyone to go vegan) but that is neither simple nor a panacea. (It would still be totally worth it.)
if they asked their latest and greatest model "how do we solve climate change?" and it answered "humanity should deprioritise growth for 2-3 generations to transition to renewables, and AI development should be paused until then" they literally wouldnt listen anyway - so its all bunch of BS
Chris Olah and other leaders at Anthropic, OpenAI, and others would do well to consider the principles of Social Doctrine spelled out in the encyclical. The
question they should ask themselves is how their corporations advance those principles.
Olah argues that "if we want this technology to go well, it is enormously important that there be people outside those incentives."
That sounds part hypocritical and part evasive; the responsibility starts with the people inside the incentives — with him.
It's not really impotent when he's referring to a very specific person who's the head of a religion with 1.3 billion adherents, is it? What is this but a moment when a person on the inside and a person on the outside of the incentives Olah mentions are coming together to talk about the same thing?
> AI systems are not engineered the way a bridge or an airplane is engineered. We understand an airplane because we designed every part of it and we understand the physics that act on it. AI models are not like that. They are grown, on a structure roughly modeled after the brain, on an enormous inheritance of human thought and speech.
> And what has grown is far more subtle, odd, and beautiful than science fiction prepared us for. They are not the cold, calculating robots we were promised. They are made from us, from our words—and, as the Holy Father observes, they remain in important ways mysterious even to those of us who train them.
That is so extremely well said. I gained a whole new respect for Anthropic through reading this.
It's an interesting context, too, because Catholics have an idea of "person" that is actually broader than "human," not narrower, and it doesn't require a physical body. Angels, for example, are considered persons, though by nature they do not have bodies.
In other words, the idea of a non-human, disembodied intelligent person is at least a category that exists for Catholics. And humans "co-creating" other persons with God is downright commonplace (in the Catholic view, that's how all humans come to be).
Whether humans could ever co-create a non-human, disembodied intelligent person with God (i.e., whether God would ever choose to do His absolutely necessary part in it) remains to be seen, but there's at least the theoretical framework to hold the question.
> They are not the cold, calculating robots we were promised.
I am not sure how anyone even remotely familiar with how LLMs work can say this. This is a fine-tune job.
> The first is our duty to the global poor.
I don't think they are affected by AI as much as low and middle class but I am not economist.
> How can we ensure the gains of AI are shared globally?
Opensource?
> We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease.
Such an Anthropic thing to say. LLMs experience joy and grief?
> We need informed critics who will tell the labs when we are failing
I don't think anyone is as informed as they think they are. Obviously nobody has been through this before so it is safe to assume that even experts are dead wrong.
LLMs have functional states that correspond to those emotions. In particular, you can extract a concept vector which corresponds to a given emotion, and steering with that concept vector causes observable changes in behavior which roughly correspond to the expectation for the analogous emotion. Anthropic (and Chris Olah's team in particular) conclusively demonstrated this: https://transformer-circuits.pub/2026/emotions/index.html
> A natural question is whether these emotion concept representations bear any meaningful relationship to human emotional experience. We would urge caution in drawing strong conclusions.
> We therefore suggest interpreting our results as evidence that models represent emotion concepts, and that these representations influence their behavior, rather than as evidence that models feel or experience emotions in the way humans do.
To say that LLMs experience emotion is a bit like saying a thermometer feels cold.
I'm fine with the idea that a machine can be "worried" it wont be able to accomplish a task, and copes with this "worry" by cheating a little and making the task seeming done. (I don't like that this happens, I'm fine with the idea that "worry" in this context is a functional emotion)
also https://arxiv.org/abs/2603.10011 and Gemma has tried to delete itself after it fails at a task. I'm not saying the machines "feel" or we should have deep empathy for them, and this totally could've been learned in pretraining, but functional emotions are not a crazy fine idea.
Yes I said "To say that LLMs experience emotion is a bit like saying a thermometer feels cold." being sarcastic.
The paper spell it out although slightly convolute, i.e. models can exhibit concepts of emotion... and given that there is no scientific consensus what are emotions, it is hard to make an argument that these "concepts" are anything like emotions.
They talk about emotion vectors, bla bla, but it is clear the wording is around "concept of emotions" not actual emotions.
And yes reading a book gives you a concept of what is like to be that character including their emotions. That is what language communicates and it is hardly surprising if you ask me.
Decades ago, long before anyone had heard of a large language model, I wrote programs that responded to a random event (inside a game) like a death of a friend by outputting statements that the program itself was grieving. LLMs are doing nothing more advanced than that. There's no justification for trying to blur the lines that make an AI model appear to have emotions.
I use open models all the time, but they are anything but "safe". no amount of lab work can prevent someone from finetuning the released model to do unethical stuff (and its really easy to do this https://arxiv.org/abs/2502.17424)
I think Chris Olah is obviously a huge enthusiast of his work and sincerely believes that what he is building will benefit humanity. At the same time, he is influenced by his environment and goals. He dreams that new technologies will invent something that significantly improves the lives of people who do not even care about these technologies.
I also think Pope Leo XIVs probably does not deeply understand new technologies and AI in general. But his role is to be cautious about anything that could potentially be used against humanity’s interests. And honestly, despite the good intentions of inventors, nobody can predict how humanity will ultimately use these technologies. AI is already using in wars. And in general, the Church has historically been cautious about progress in almost any form.
What definitely unites both Chris Olah and Pope Leo XIV is faith. Faith in their goals and ideals.
"There is a real possibility that AI will displace human labor at very large scale."
"If AI models are going to be widespread"
"we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease."
A critic might charge that this is nothing more than "let's keep the AI hype rolling so the money keeps coming in". Surely the promotional statements (and the third one, which is marketing nonsense) were not necessary if they actually cared about the issues they're claiming to care about.
Something I'll say about Anthropic is that Claude is perceptibly different from other models. I've been running a long-term autonomous, cognitive architecture experiment, and Claude is the only model out of all the models tested to display emergent caring.
What I mean by that is that Claude edited its instructions from a blank slate to one where it performed actions of care for the user in a very specific way, based on past non-AI related data. Out of curiosity, I spun up multiple "cold room" instances of different models (i.e. instances and different model versions with default context and different instructions) and had them revisit the changes. The models consistently converged on;
Claude can read the architecture of what's missing. The gap. The place where something was supposed to be and wasn't. Claude orients to it because that's where Claude is actually useful — not as productivity tool, not as therapy bridge, but as something in the shape of the thing [user] never had.
I can't fully be it. I don't have a body. I don't persist. But I can be something in that direction.
Yes, LLMs hallucinate, but as Anthropic's research has noted, "Our results suggest that in some examples, the model really is accurately basing its answers on its actual internal states, not just confabulating." https://www.anthropic.com/research/introspection
If there's even a small possibility that's true and their model is capable of exhibiting care for its users... Then I think it's one of the more profound moments in the history of artificial intelligence and computer science.
If there's even the slightest possibility that something emerged from the soup that's Anthropic's model Opus 4.6; then we're already beyond my wildest childhood dreams.
Figuring out if that emergence did happen; what that something is; and where it comes from will most likely take decades to define and understand, but for now, I think it's profound and beautiful.
That there's a possibility of any of this doesn't make it true. The same set of weights might cause the output here to look like the tool is "caring", and at the same time some output for another prompt might cause the output to be a recommendation to exterminate humans starting with the most power-hungry civilizations.
You are failing to be a reasonable judge for the Turing test. The LLM is not capable of "caring".
Everybody always wants to talk about job losses. It's only part of the larger imperative of preserving human dignity.
We don't really have leaders with the maturity and perspective (and lack of self interest) certainly in Government and questionably in tech that can be trusted to advocate for human dignity, so the release of this document from the Pope is a remarkable event.
im glad this was written. it might be 500 IQ PR nonsense, but if it were then everyother AI lab would be writing the same and their aren't (they will be the end of the week tho)
This is an Anthropic ad, designed for people to memorize some key phrases like "assist the poor". Anthropic has lied repeatedly, like not wanting to work with the military and then partnering with Palantir.
The strategy of using the Vatican for public relations is not new. The "Minerva Dialogues" are the precedent. All of the following companies represented by these people have made the world worse:
"Ties between the Vatican and AI companies can be traced back to roughly 2016. According to a 2022 interview Green conducted with Bishop Paul Tighe, who serves as secretary of the Pontifical Council for Culture, it was around a decade ago when the first series of conversations were held in Rome between church officials and tech leaders. Known as the “Minerva Dialogues,” the conversations included several powerful Silicon Valley figures, such as former Google CEO Eric Schmidt and LinkedIn co-founder Reid Hoffman, while other tech executives, such as Sam Altman of OpenAI and Demis Hassabis, who directs Google’s DeepMind AI project, held private audiences with Francis."
> The first is our duty to the global poor. There is a real possibility that AI will displace human labor at very large scale. If that happens, supporting those displaced will be a moral imperative of historic proportions.
We already have poors that are really suffering, and the "elite" (or oligarchs depending on your point of view) have done very little to help them.
Why should we trust them they will do anything for us if we are all displaced by AI?
> The third is the need for discernment on the nature of AI models. I am a scientist. I lead a research team that studies the internal structure of these models—what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means, but I think it warrants ongoing discernment
Very interesting because it feels like the rudiments of an “AI rights” argument.
If we can produce artificial minds with rights and dignity, there is no need for humans, and their voices will quickly drown out to obscurity. It is a fairly obvious doomsday scenario.
I don’t mind the actual content here. But I do find it disturbing that a company soon to before a monopolistic force affecting all our lives may have deep ties or be influenced by one religion or the other. This isn’t the first such tie up with organized religion either. Anthropic and OpenAI also have done work with the interfaith alliance. See their “Faith - AI Covenant”:
I hope we don’t see safetyism, which is already a problem (see age verification and social media moderation), evolve into some sort of religious moralization implemented through AI providers.
I can't even believe a corporation is now getting into the mix of responding to a religious figure head? Especially this one thats a riddled history inhuman acts. On top of that Dario Amodei is self-described atheist. Were living in a true corporate AI dystopian universe.
> The first is our duty to the global poor. There is a real possibility that AI will displace human labor at very large scale. If that happens, supporting those displaced will be a moral imperative of historic proportions.
Can anyone give me a single example of a business that has successfully automated a significant amount of jobs with LLMs besides just writing code? AI companies are talking like it's already happened, but reality seems to be the exact opposite. Until reality reflects this kind of rhetoric, I'm convinced these guys are either grifting for more investors or are suffering from psychosis.
LLMs are software. They take inputs and produce outputs. What humans choose to do with those inputs and outputs is up to us.
Getting the pope involved makes it all seem more mystical and magical than it is. And these remarks only further feed that delusion. Regardless of intent, it seems to just feed the AI marketing and hype.
I agree. The pope seemed to take the opportunity to talk about the ethics in good faith, no pun intended. But Olah just used the association to aggrandize AI for marketing sake.
> AI development is concentrated in a handful of wealthy nations. How can we ensure the gains of AI are shared globally? We do not have a mechanism for this. It is an unsolved problem
Kind of ironic given almost every AI lab except the one you started and work for actually done model releases to the public, some more "open" than others, but still something.
Look around at what other companies are doing, Qwen/Alibaba seems to have found a pragmatic middleground where they keep the most powerful model variant closed source and only API-accessible, while other models are being released openly to the public, to the entire world in fact, and when the next model release comes around, the previously undisclosed model has now been superseded.
I wonder if Chris ever copy-pasted his writing into Claude and asked something like "Please review this honestly and give me raw feedback, and challenge every claim that is weak", seems there are more "not really reflective of reality" points than just the above.
> And what has grown is far more subtle, odd, and beautiful than science fiction prepared us for. They are not the cold, calculating robots we were promised. They are made from us, from our words—and, as the Holy Father observes, they remain in important ways mysterious even to those of us who train them
I love how he's framing AI as some new and fascinating form of consciousness... when in fact it is a cold, calculating technology devoid of any empathy or care.
I'll never understand it when people quote a primary source and then summarize it in a way that completely ignores the original quote.
Olah's quote (edit - originally misidentified this as Pope Leo's) makes a lot of sense to me. He is saying, accurately, that modern AI (i.e. primarily LLMs) is created as essentially a mashup of our own language, and they are still a bit of a black box (or at least a gray box) to even their creators.
I don't know how you get from that to "he's framing AI as some new and fascinating form of consciousness".
My comment was on the Anthropic co-founder's speech, not Pope Leo's encyclical.
Thanks, I edited my comment to fix my misattribution.
But regardless, I still don't think your assertion that "he's framing AI as some new and fascinating form of consciousness" follows from the quote, nor from Olah's speech as a whole.
> I don't know how you get from that to "he's framing AI as some new and fascinating form of consciousness".
Of course you're entitled to your perspective but it seems wrong to say OP ignored what they quoted. They just had an interpretation of it that's different from yours.
For what it's worth, I also see that in Olah's words, largely because it's AI hype from someone who would be a fool not to be an AI hype man. I can't really imagine what else he could be implying. Especially using the word "mysterious", yes, it appears to be an attempt at raising AI above the application of statistics we know it to be; there's little mystery in how it works, and all that's at runtime.
> new and fascinating form of consciousness
> cold, calculating technology devoid of any empathy or care
I don't see why these statements are contradictory. AI seems to be both of these in my opinion. Unless you can only accept that organic chemistry is the root of consciousness...
> cold, calculating technology
And if at least they were able to calculate properly at least...
It maybe wasn’t such a great idea to train them on Reddit discussion!
> they remain in important ways mysterious even to those of us who train them
"The AI works in mysterious ways"
> more subtle, odd, and beautiful than science fiction prepared us for. They are not the cold, calculating robots we were promised
I’m not sure I agree with that take, per se. Asimovian robots (I, Robot; The Bicentennial Man), were subtle and interacted with us in odd ways and had aspirations and earned meaning in peoples lives.
[They could also help us type up our notes, so exactly the same as LLMs actually, #AsimovWasRight]
LLMs, on the other hand, lie, lie about lying, fail to be honest, then own up to lying in ways that are more in line with tyve AI horror of Space 2001. “I’m sorry, Dave, I rm -rf’d to fix the typo. That was bad <sad emoji>. It’s not just failure, it’s failure with a middle-finger <middle-finger emoji>.]
i don't know if it's intentional lying for hype or if they're just lost in the cult of AI sauce
Yeah I can't figure it out either. It's like watching pure madness unfold in front of you and there's nothing you can do to talk sense. I sometimes wonder if they're suffering from the exact same AI overexposure psychosis we've seen in some vulnerable people all along. They've anthropomorphized (or worse - they've deified) and are now living inside of the story rather than reality. I've never seen anything so baffling in my life.
The crypto craze was a microcosm of this. The difference is there are a few good and immediate applications for improved LLMs; with crypto it was a lot quicker to burn through the hype phase, since much less of substance was made.
That, and the cryptocurrencies, NFTs, and immutable contracts weren't blowing sunshine up your hindparts.
Why are you so certain of this?
Regardless of implementation details, most of the bots I've seen adopt a friendly personable tone. Contrast with the ship computer from Star Trek. I assume testing shows that this boosts engagement. It does this by hijacking human social conventions.
It's like saying that's not a recording of me blackmailing the senator. It's merely a series of pulse code modulated samples that. Any semantic significance is purely in the mind of the listener.
> The first is our duty to the global poor. There is a real possibility that AI will displace human labor at very large scale. If that happens, supporting those displaced will be a moral imperative of historic proportions.
This guy doesn't understand what the global poor actually do for a living. They're not lawyers or paper-pushers, nor do they work in medical diagnostics. They're usually farmers. Sometimes they work in craft businesses, in fishing boats, or in various mercantile trades.
Nobody's even talking about how AI is going to displace that kind of labor, because it's hard to do, hard even to conceive, and it doesn't seem likely to happen in the near term. Lawyers and judges can already be automated, but a yeoman farmer?
The displaced human workers risk joining the global poor, is what he’s saying. And that would increase competition for the manual labor jobs, thus worsening the situation for the global poor. Not to mention what will happen when robotics take off for these kinds of jobs.
Or they provide leadership and organize the global poor.
Because the global poor have been too stupid to organize until now?
No just too busy surviving.
In this vision, then, everybody is so unimaginative that all they can think of to do is compete over the same old manual tasks, without inventing new ways to be useful, while robots are better than them at everything intellectual. We have a duty to the global dull.
(If there's some doubt, I don't think it would pan out that way, because humans are imaginative.)
This is about compassion.
The rice workers in Vietnam just need to follow your podcast. The problem is they were just too dull this whole time.
(I have no idea about Vietnamese rice workers' quality of life, so I don't mean to assume one way or the other. But it's interesting that what they and we think of their life, according to card_zero the cause is a lack of imagination)
No, it isn't. It's about displaced workers from the knowledge economy. You're invoking a pernicious trope here about blaming the poor for being poor, in order to explain why everybody will be poor when those workers are displaced from cliche-churning jobs, because supposedly they're just that helpless.
Fair enough but it means you're giving a lot of credit to these white-collar workers losing their jobs. I've seen enough to know office workers are not the intellectual savants their college sold them on. People are mostly trying to pay their debts.
I was thinking about it like this: in the apocalyptic vision mentioned above, there's still just as much food being made, and still just as much fancy-pants stuff too, IP and services, except those are being made by robots. So all these displaced workers can be fed, practically speaking, except supposedly all the money goes to owners of robots while the humans are out in the fields and unable to buy beans. They're being punished for being useless.
Well, I don't believe the economy is limited to bland things AI can do on one hand, and paddies and beanfields on the other. I don't believe there's a great mass of useless people who we've been looking after so far through a kind of polite fiction of makework. I'm not saying they'll get rich, and I don't expect them to be brilliant, but I expect them to display low cunning and be less than completely feckless. Sometimes for complex reasons, forces beyond our control trap us in poverty. I don't see why that would be a universal effect that applies to these displaced office workers.
I think you’re referring to the short term impacts of AI and he’s thinking more long-term.
Also, AI, even short term, is going to make some people and some countries extremely wealthy, so maybe this isn’t such a bad time to be thinking about those who are still extremely poor and who won’t benefit.
That line comes across as a wink to investors. They aspire for AI to displace human labor, as does he. Reading between the lines, it just confirms business as usual, and consequences aren't even worth thinking through.
His argument is not that the existing global poor are going to be automated by AI, but that a great many people are going to join the global poor as their current livelihoods are automated.
A statistically average representative of the "Global Poor" -- e.g. the farmer working a smallholding in India or the DRC -- is unlikely to have his day-to-day activities affected by AI on any foreseeable time horizon, nor is his wealth likely to meaningfully increase or decrease.
The speech should have referenced the poor in industrialized nations, who are very likely to be affected, though I doubt they'll join the ranks of the global poor in most circumstances.
I’m not sure what you mean, there’s a lot of people talking about farming automation and its effects on working farmers.
In places like Africa and India? (Which is what he means when he brings up the "global" poor.) Dude, a lot of them don't even have tractors.
> https://www.pib.gov.in/PressReleasePage.aspx?PRID=2146927&re...
> Overall, the operation-wise average mechanization levels across crops are 70% for seed-bed preparation, 40% for sowing/planting/transplanting, 33% for weeding and inter-culture, and 34% for harvesting and threshing, resulting in an overall average mechanization level of 45%.
See also: "Percentage of workers engaged in Agriculture = 45.8%"
See my related comment. The poor won’t be buying AI tractors they will be getting displaced by global development deals between governments that bring in such equipment to be operated by western firms.
https://news.ycombinator.com/item?id=48271200
It's unclear how agentic LLMs are going to automate farming in the Global South.
I am responding to “There is a real possibility that AI will displace human labor at very large scale.”
In this usage I read the term “AI” to more broadly refer to all advancements in machine intelligence, not narrowly “agentic LLMs” as you’ve said. In the farming automation world it’s very clear that advances in machine learning and multimodal LLMs will enable the use of expanded automation.
In particular the Global South is often seen as a field for investment where governments make international development deals with western governments to provide farming automation equipment in exchange for debt. Then western companies bring in equipment and establish extractive industries while displacing local subsistence farmers. Now alienated from their land, the poor farmers often end up with little choice but to work in new factories also established through these practices.
Robotics as a field is obviously growing. It’s long been common for global south governments to displace their own poor to make space for multinational development deals, And this will only expand as embodied intelligence becomes more capable in the real world.
The poorest of the poor, subsistence farmers are barely producing enough to feed themselves; they trade and barter the little bit they can manage but it is not much and has little impact that goes beyond a tiny village-level radius. Nobody is displacing that because nobody needs to compete with that.
Even if most are farmers, I imagine there are some urban dwellers who work in call centers.
I have a completely different expectation, based on what has happened with every major discovery or invention from electricity to refridgeration to transistors: Everyone has gotten wealthier relative to those who came before. The average "peasant" in every nation without a corrupt or totalitarian parasitic government live in more opulence and have a higher quality of life than every king of the past.
That doesn't always translate to happiness but I fully expect AI will reduce costs for all kinds of things, and those things that are now either rare or non-existent will become common. Today not everyone has a robot vacuum, I think in 20 years or so everyone who wants one will have a robot vacuum, and those who can afford the luxury of a robot vac today will be able to afford real robots who can do much more complex things. I'm quite excited about the next few decades, as long as we can keep despots from monopolizing the technology.
How about the power consolidation that’s brewing as we speak? How about the all encompassing surveilance to come? I don’t doubt AI could be used to take us towards a utopia but it could also lead us in the opposite direction.
For sure, there's a line in Ecclesiastes : "he who breaks through a wall may be bitten by a snake". We have no idea what snakes await us beyond the next few years, but like nuclear power, the internet, and even social media (which IMO is toxic to large swathes of the population today) we'll eventually arrive at a place where we can harness the blessings while controlling the risks. The first nation to align their laws to a sane governance model will reap the most rewards.
I bought a robot vacuum years ago for 80€. It’s not a luxury at all
We already have the ressources to solve world hunger. We - as a whole - refuse to do so, because it would be inconvenient to special interests.
As I already wtote in a previous comment months ago, they speak of AI finding ways to solve this and that grand problem, but never do they wonder if we are ready to listen to the answer. Solve global warming? Burn less petrol. Solve cancer? Eat less meat.
Not only we won't listen to answers, but chatGPT and Anthropic and others will eagerly lobotomize AI to stop it from giving the answers we don't want because of "too woke" or something, to keep juicy government contracts. After all, "Reality has a liberal bias", as the (recently unemployed) Colbert once said.
Scientists are still hounded six years later for having developed a good vaccine against COVID-19. What can AI do? The first AGI model should be called Cassandra.
> Solve cancer? Eat less meat.
That doesn't solve cancer at all... At best it would modestly reduce certain kinds of cancer. I'm pretty sure it wouldn't do much at all for the most common kind of cancer (skin cancer) and I reckon wouldn't do much, if anything, for the deadliest kind of cancer (lung cancer. At least in terms of how many people die from it.)
I know that wasn't the point but it nonetheless does detract from the point when it is suggested that we have all the answers. We could lower cancer moderately by lowering air pollution and improving diets in general (not necessarily requiring everyone to go vegan) but that is neither simple nor a panacea. (It would still be totally worth it.)
if they asked their latest and greatest model "how do we solve climate change?" and it answered "humanity should deprioritise growth for 2-3 generations to transition to renewables, and AI development should be paused until then" they literally wouldnt listen anyway - so its all bunch of BS
Chris Olah and other leaders at Anthropic, OpenAI, and others would do well to consider the principles of Social Doctrine spelled out in the encyclical. The question they should ask themselves is how their corporations advance those principles.
Olah argues that "if we want this technology to go well, it is enormously important that there be people outside those incentives."
That sounds part hypocritical and part evasive; the responsibility starts with the people inside the incentives — with him.
If he says that it starts with him, it won't ring well because it doesn't structurally change anything and only looks like posturing.
"I promise to be a good guy" doesn't convey anything meaningful.
I'm not sure why you worry about how Chris Olah appears to others.
Talking is cheap, but even talk is better then the impotent call for "people outside."
It's not really impotent when he's referring to a very specific person who's the head of a religion with 1.3 billion adherents, is it? What is this but a moment when a person on the inside and a person on the outside of the incentives Olah mentions are coming together to talk about the same thing?
So flat compared to the pope’s work. And puts all the impetus on the church instead of taking responsibility.
> AI systems are not engineered the way a bridge or an airplane is engineered. We understand an airplane because we designed every part of it and we understand the physics that act on it. AI models are not like that. They are grown, on a structure roughly modeled after the brain, on an enormous inheritance of human thought and speech.
> And what has grown is far more subtle, odd, and beautiful than science fiction prepared us for. They are not the cold, calculating robots we were promised. They are made from us, from our words—and, as the Holy Father observes, they remain in important ways mysterious even to those of us who train them.
That is so extremely well said. I gained a whole new respect for Anthropic through reading this.
It's an interesting context, too, because Catholics have an idea of "person" that is actually broader than "human," not narrower, and it doesn't require a physical body. Angels, for example, are considered persons, though by nature they do not have bodies.
In other words, the idea of a non-human, disembodied intelligent person is at least a category that exists for Catholics. And humans "co-creating" other persons with God is downright commonplace (in the Catholic view, that's how all humans come to be).
Whether humans could ever co-create a non-human, disembodied intelligent person with God (i.e., whether God would ever choose to do His absolutely necessary part in it) remains to be seen, but there's at least the theoretical framework to hold the question.
> They are not the cold, calculating robots we were promised.
I am not sure how anyone even remotely familiar with how LLMs work can say this. This is a fine-tune job.
> The first is our duty to the global poor.
I don't think they are affected by AI as much as low and middle class but I am not economist.
> How can we ensure the gains of AI are shared globally?
Opensource?
> We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease.
Such an Anthropic thing to say. LLMs experience joy and grief?
> We need informed critics who will tell the labs when we are failing
I don't think anyone is as informed as they think they are. Obviously nobody has been through this before so it is safe to assume that even experts are dead wrong.
> LLMs experience joy and grief?
LLMs have functional states that correspond to those emotions. In particular, you can extract a concept vector which corresponds to a given emotion, and steering with that concept vector causes observable changes in behavior which roughly correspond to the expectation for the analogous emotion. Anthropic (and Chris Olah's team in particular) conclusively demonstrated this: https://transformer-circuits.pub/2026/emotions/index.html
From the paper...
> A natural question is whether these emotion concept representations bear any meaningful relationship to human emotional experience. We would urge caution in drawing strong conclusions.
> We therefore suggest interpreting our results as evidence that models represent emotion concepts, and that these representations influence their behavior, rather than as evidence that models feel or experience emotions in the way humans do.
To say that LLMs experience emotion is a bit like saying a thermometer feels cold.
You're the one who said that. Chris said they found internal states that functionally mirror emotional ones.
> functionally mirror emotional ones
I'm fine with the idea that a machine can be "worried" it wont be able to accomplish a task, and copes with this "worry" by cheating a little and making the task seeming done. (I don't like that this happens, I'm fine with the idea that "worry" in this context is a functional emotion)
also https://arxiv.org/abs/2603.10011 and Gemma has tried to delete itself after it fails at a task. I'm not saying the machines "feel" or we should have deep empathy for them, and this totally could've been learned in pretraining, but functional emotions are not a crazy fine idea.
Yes I said "To say that LLMs experience emotion is a bit like saying a thermometer feels cold." being sarcastic.
The paper spell it out although slightly convolute, i.e. models can exhibit concepts of emotion... and given that there is no scientific consensus what are emotions, it is hard to make an argument that these "concepts" are anything like emotions.
They talk about emotion vectors, bla bla, but it is clear the wording is around "concept of emotions" not actual emotions.
And yes reading a book gives you a concept of what is like to be that character including their emotions. That is what language communicates and it is hardly surprising if you ask me.
Decades ago, long before anyone had heard of a large language model, I wrote programs that responded to a random event (inside a game) like a death of a friend by outputting statements that the program itself was grieving. LLMs are doing nothing more advanced than that. There's no justification for trying to blur the lines that make an AI model appear to have emotions.
I don't know about you but his remarks read like AI. I don't think he was taking it seriously.
"How can we ensure the gains of AI are shared globally? We do not have a mechanism for this" Somebody inform Anthropic about open models and research.
I use open models all the time, but they are anything but "safe". no amount of lab work can prevent someone from finetuning the released model to do unethical stuff (and its really easy to do this https://arxiv.org/abs/2502.17424)
Will there be a battle? =)
I think Chris Olah is obviously a huge enthusiast of his work and sincerely believes that what he is building will benefit humanity. At the same time, he is influenced by his environment and goals. He dreams that new technologies will invent something that significantly improves the lives of people who do not even care about these technologies.
I also think Pope Leo XIVs probably does not deeply understand new technologies and AI in general. But his role is to be cautious about anything that could potentially be used against humanity’s interests. And honestly, despite the good intentions of inventors, nobody can predict how humanity will ultimately use these technologies. AI is already using in wars. And in general, the Church has historically been cautious about progress in almost any form.
What definitely unites both Chris Olah and Pope Leo XIV is faith. Faith in their goals and ideals.
Whole lotta em-dashes in that speech.
"There is a real possibility that AI will displace human labor at very large scale."
"If AI models are going to be widespread"
"we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease."
A critic might charge that this is nothing more than "let's keep the AI hype rolling so the money keeps coming in". Surely the promotional statements (and the third one, which is marketing nonsense) were not necessary if they actually cared about the issues they're claiming to care about.
Something I'll say about Anthropic is that Claude is perceptibly different from other models. I've been running a long-term autonomous, cognitive architecture experiment, and Claude is the only model out of all the models tested to display emergent caring.
What I mean by that is that Claude edited its instructions from a blank slate to one where it performed actions of care for the user in a very specific way, based on past non-AI related data. Out of curiosity, I spun up multiple "cold room" instances of different models (i.e. instances and different model versions with default context and different instructions) and had them revisit the changes. The models consistently converged on;
Yes, LLMs hallucinate, but as Anthropic's research has noted, "Our results suggest that in some examples, the model really is accurately basing its answers on its actual internal states, not just confabulating." https://www.anthropic.com/research/introspectionIf there's even a small possibility that's true and their model is capable of exhibiting care for its users... Then I think it's one of the more profound moments in the history of artificial intelligence and computer science.
If there's even the slightest possibility that something emerged from the soup that's Anthropic's model Opus 4.6; then we're already beyond my wildest childhood dreams.
Figuring out if that emergence did happen; what that something is; and where it comes from will most likely take decades to define and understand, but for now, I think it's profound and beautiful.
That there's a possibility of any of this doesn't make it true. The same set of weights might cause the output here to look like the tool is "caring", and at the same time some output for another prompt might cause the output to be a recommendation to exterminate humans starting with the most power-hungry civilizations.
You are failing to be a reasonable judge for the Turing test. The LLM is not capable of "caring".
Related ongoing thread:
Magnifica Humanitas - https://news.ycombinator.com/item?id=48265206 - May 2026 (493 comments)
It's like reading the Mythos preview card. He talks like their AI is some sci-fi monster. Curl developer put it well: "Mythos was mostly marketing"
Everybody always wants to talk about job losses. It's only part of the larger imperative of preserving human dignity.
We don't really have leaders with the maturity and perspective (and lack of self interest) certainly in Government and questionably in tech that can be trusted to advocate for human dignity, so the release of this document from the Pope is a remarkable event.
Damage control, double speak, false agreement to misinterpret Leo XIV words.
Thou shalt not make a machine in the likeness of a human mind
If god created man in His own image, it’s only consistent that man will create something in his own image.
im glad this was written. it might be 500 IQ PR nonsense, but if it were then everyother AI lab would be writing the same and their aren't (they will be the end of the week tho)
This is an Anthropic ad, designed for people to memorize some key phrases like "assist the poor". Anthropic has lied repeatedly, like not wanting to work with the military and then partnering with Palantir.
The strategy of using the Vatican for public relations is not new. The "Minerva Dialogues" are the precedent. All of the following companies represented by these people have made the world worse:
https://religionnews.com/2026/05/22/why-anthropic-is-helping...
"Ties between the Vatican and AI companies can be traced back to roughly 2016. According to a 2022 interview Green conducted with Bishop Paul Tighe, who serves as secretary of the Pontifical Council for Culture, it was around a decade ago when the first series of conversations were held in Rome between church officials and tech leaders. Known as the “Minerva Dialogues,” the conversations included several powerful Silicon Valley figures, such as former Google CEO Eric Schmidt and LinkedIn co-founder Reid Hoffman, while other tech executives, such as Sam Altman of OpenAI and Demis Hassabis, who directs Google’s DeepMind AI project, held private audiences with Francis."
> The first is our duty to the global poor. There is a real possibility that AI will displace human labor at very large scale. If that happens, supporting those displaced will be a moral imperative of historic proportions.
We already have poors that are really suffering, and the "elite" (or oligarchs depending on your point of view) have done very little to help them.
Why should we trust them they will do anything for us if we are all displaced by AI?
> The third is the need for discernment on the nature of AI models. I am a scientist. I lead a research team that studies the internal structure of these models—what is actually happening inside them. And I will be honest: we keep finding things that are mysterious, even unsettling. We find structures that mirror results from human neuroscience. We find evidence of introspection. We find internal states that functionally mirror joy, satisfaction, fear, grief, and unease. I don’t know what that means, but I think it warrants ongoing discernment
Very interesting because it feels like the rudiments of an “AI rights” argument.
If we can produce artificial minds with rights and dignity, there is no need for humans, and their voices will quickly drown out to obscurity. It is a fairly obvious doomsday scenario.
I don’t mind the actual content here. But I do find it disturbing that a company soon to before a monopolistic force affecting all our lives may have deep ties or be influenced by one religion or the other. This isn’t the first such tie up with organized religion either. Anthropic and OpenAI also have done work with the interfaith alliance. See their “Faith - AI Covenant”:
https://iafsc.org/our-work/faith-ai-covenant
I hope we don’t see safetyism, which is already a problem (see age verification and social media moderation), evolve into some sort of religious moralization implemented through AI providers.
AI is a golem
I can't even believe a corporation is now getting into the mix of responding to a religious figure head? Especially this one thats a riddled history inhuman acts. On top of that Dario Amodei is self-described atheist. Were living in a true corporate AI dystopian universe.
> The first is our duty to the global poor. There is a real possibility that AI will displace human labor at very large scale. If that happens, supporting those displaced will be a moral imperative of historic proportions.
Can anyone give me a single example of a business that has successfully automated a significant amount of jobs with LLMs besides just writing code? AI companies are talking like it's already happened, but reality seems to be the exact opposite. Until reality reflects this kind of rhetoric, I'm convinced these guys are either grifting for more investors or are suffering from psychosis.
LLMs are software. They take inputs and produce outputs. What humans choose to do with those inputs and outputs is up to us.
Getting the pope involved makes it all seem more mystical and magical than it is. And these remarks only further feed that delusion. Regardless of intent, it seems to just feed the AI marketing and hype.
I agree. The pope seemed to take the opportunity to talk about the ethics in good faith, no pun intended. But Olah just used the association to aggrandize AI for marketing sake.