> in part because Google fired two of the authors, Timnit Gebru
I remember being angry about this situation when I first saw it on social media, until I read the details: This person submitted a list of demands to her employer and said that if they weren’t met, she quit. Google wasn’t going to meet her demands so they considered it acceptance of her resignation. There has been a movement trying to debate whether it was a firing or resignation ever since.
The original paper they published gets recirculated every year or two as some landmark history of AI safety, but as other commenters have noted it wasn’t really a great paper nor was it groundbreaking at the time. If not for the controversy surrounding the resignation/firing (depending on your POV), I don’t think it would have been notable.
True but also... she wasn't a software engineer putting code in production nor a researcher working no the fundamentals of machine learning negotiating a raise.
She was part of the "Ethical Artificial Intelligence Team" of what was then, and still is now, one of the corporations World wide spending the largest amount of resources precisely on using AI commercially.
I'm saying the paper itself wasn't a bombshell or even that noteworthy. The reason it got PR and continues to come up was because the authors manufactured this self-inflicted drama around it, not because it was leaking secret revelations that harmed the company.
I think part of it is she had excellent PR skills and a dedicated fan base. I was at Google when she quit, working in ML, and hadn't heard of her until the story broke. I remember there were a large number of Memegen posts about it, but no one I spoke with knew about her, so I assumed it was brigading.
I think she's since since lost a lot of her allure, especially when she didn't change her mind when the facts about the AI water usage changed 1000x
What exactly are the facts about AI water usage? I have trouble separating hysteria from reality but most of what I see still claims water usage is enormous
The hysteria around water usage rests on people not knowing the scale of industrial civilization. First thing to do is compare any estimate of data center water usage with the water usage of almond farming. Or, if you want to focus on individual consumer choices, the water footprint of eating a hamburger.
Breaking it down this way is a great way to minimize the numbers so that it appears reasonable.
See? Middle-Eastern investors are growing alfalfa in the western desert using legal allotments of water! That is so much worse than what we’re doing! Go after them!
They can both be using an egregious amount of water for silly purposes.
The other part of the water debate is also the pollution different systems create. Many data centres went in with the promise of closed-loop systems but changed half-way through construction and couldn’t be stopped.
I think it’s more complicated than, “they’re wrong, it’s just hype.”
Putting things into perspective is not minimising the problem. We literally have to do this to prioritise where our efforts can be useful.
Your argument makes sense if ai datacenters were using something close to like alfafa farming but the difference between them is soo massive it does not make sense.
Reducing pollution is a much better problem to fight for
Nice false dichotomy you got there there, might as well calculate the entire water usage for a a single GPU in the supply chain too. Something tells me one is extremely worse than the other when you account for all the water that's used in a single supply chain for high end electronics, but if you want to plop the measuring stick where ever along the whole pony show that makes you look better people will notice.
Also to compare growing food with the totally optional, not useful in the slightest, LLMs that somehow demand local populaces bend to their will for reasons that never seem to benefit them is just bonkers level of self-blinding when it comes to populations absolutely despising big tech, big tech leadership, and big tech practices.
This mania might finally cause the software industry to become a highly regulated with licenses similar to that of other engineering disciplines due to amount of optional destruction they have decided to unleash upon on the planet in such a short time frame.
What is interesting about it for me is: why would someone working on AI ethics choose to work for Google at all?
Did she really think Google cares about ethics? Such positions seem purely performative, we all know that ethics go out the window first to make room for more profits.
I missed that thread when it came out, it's really a wild read. The difference between how people describe her vs how she's normally portrayed in the media is really startling
Her demands included wanting to know the identities of anyone who wanted to comment on her paper, after she had a history of going after people publicly. That's enough right there, nobody should tolerate toxic behavior regardless of whether you agree with the politics.
Meanwhile, the paper has 2 points of criticism towards AI. 1 is a bunch of carbon consumption complaints assuming NVIDIA cards with coal-fired power, while a lot of effort at contemporary Google went towards getting TPUs running on green power. I suspect this was what people wanted to object to, a lot of effort went into those green power projects and she was just denying it. The complaint seems prophetic now but it was not true about Google then.
The other criticism was about which language the LLMs use, they average the input data of normal humans instead of talking the way the paper author thinks they should talk. The phrase "women doctors" is called out as problematic. I'm less inclined to think people objected strongly to this given the zeitgeist at the time, it was probably people who worked on the green energy projects and were pissed off that their contributions were ignored, but still, nobody elected her Queen of English, she can have her opinions but she's not a victim for not having them adopted by everyone.
I think this is completely misleading. If your employer asks to redact your paper because it ignores relevant research, naturally you want to know what research they are talking about and on what grounds, and also which researches reviewed the paper (probably assuming there was none, review process was used as an excuse to silence criticism, G has done it many times).
(BTW, quite bold to say input data from Reddit and 4Chan is how “normal” people speak. There is a lot of language in the training data of any model you really do not wish your application to use ever.)
This is not why she was fired and wouldn't have been a plausible reason in 2020 when it happened.
>Timnit responded with an email requiring that a number of conditions be met in order for her to continue working at Google, including revealing the identities of every person who Megan and I had spoken to and consulted as part of the review of the paper and the exact feedback. Timnit wrote that if we didn’t meet these demands, she would leave Google and work on an end date. We accept and respect her decision to resign from Google.
This is Google's side of it; I think the following is a fair piece of primary-source journalism if you want to go deeper:
> With the octopus thought experiment, I initially had told the story in terms of a dolphin, because dolphins clearly are intelligent animals. My co-author on that paper, Alexander Koller, said it should be an octopus, because first of all, the environment that octopuses live in is much more distinct from where people live. It makes the metaphor more vivid, that the octopus is just feeling these pulses in the cable and has no way to look at what the people are looking at.
It's such a tragedy that they're also extremely solitary animals and die shortly after reproducing the first (and only) time.
Almost all other particularly intelligent animals seem to be gregarious, and it's easy to conclude that a social lifestyle tends to select for more intelligence, a sophisticated theory of mind, and so on (I like to think that that's exactly what was responsible for a runaway intelligence explosion in humans). But in the case of cephalopods, there's something else that has been applying selection pressure towards exceptional intelligence.
It's also a bit of a chicken-and-egg problem: if they were raised by their parents like all other more intelligent animals, they wouldn't need to be as intelligent as they are in order to be able to relearn "octopus behaviour" without help from other members of the species.
In Stefan Wul's SF novel "Nyourk", octopi evolve to become the Earth's dominant species, which was quite prescient back in 1957, when almost nobody knew octopi to be intelligent. :)
That's a bit exaggerated - monkeys can't fly, but both dolphins and octopuses can swim. I'm aware octopuses prefer to stay at the bottom, while dolphins have to come to the surface regularly to breathe, but for me it's still the same environment...
The continued use of animal metaphors is doing them a great disservice. Esp as we learn more about animal cognition, on first look, it smacks of human exceptionalism that has littered the historic scientific consensus.
Now if they had said, "Imagine your average American ..." (/s)
Bender's paper had this to say about stochastic parrots:
"Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."
This was not even a correct criticism in 2021. She is right that, at the time, the pretraining -- where it learns to predict missing words in pre-existing corpuses of text -- is basically a stochastic parrot.
But nowhere in her paper does the term "reinforcement" come up. At the time, this was done mainly through RLHF (reinforcement learning from human feedback) - after the initial training is done, you then tune the model's responses based on human grading. Humans imbue their own meanings into the parameter weights through their judgment.
At this point, they aren't really stochastic parrots anymore, because parameter weights have been shaped beyond the text corpus. It's not purely probabilistic in the sense of using the probabilities of the underlying text sequences. (It still is probabilistic in its output, but that is a pointless claim, because all events in the universe are also probabilistic; it is not enough to merely claim that probability is involved in some way in the outputs.)
RLHF was already in use prior to the paper, and was written about by Christiano in 2017 "Deep reinforcement learning from human preferences," so it's surprising that Bender apparently didn't know about this well-known paper.
RLHF was also, of course, a precursor to a more advanced form of parameter shaping - reinforcement learning with verified rewards, or RLVF, which has driven a lot of the gains in verifiable domains lately. That was not done in 2021 when she wrote the paper. But if you knew about RLHF -- and knew how Alpha Zero worked, with training neural nets on game rollouts -- you could squint and see that it might be useful for language models.
So after being proven to not only having a limited understanding of the field at the time, but also not being able to forecast the field, she's now walking back what she meant by "stochastic parrot," I assume because she believes readers will not read what she wrote. But despite the protests, her original claim was that it is a parrot because the text has no meaning -- a direct quote from the paper, which only really makes sense if training stops at the pretrain.
I paid a bit of attention to this paper and the phrase 'stochastic parrots' when it came out and i thought this was worth saying and doing at that time. their suggestions about financial and environmental costs are worth studying, their concern about carefully evaluating datasets to feed to the model rather than feeding the entire internet is fully justified. so - to everyone saying this was a bad paper; if you have actually read the paper then please list a few criticisms. all i have seen is "oh this wasn't that good of a paper" or "can't believe how bad this paper was".
My main criticism of the paper is that it says LLMs work "haphazardly", using probabilistic information. That is a hypothesis, but it is stated as a known fact, a fundamental limitation.
It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways!
Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.
not sure your criticism makes sense though - they did this pre chatgpt. they are talking about the language models of that time. they did not make predictions about the future.
Whether Bender intended it or not, the term has an inherently pejorative sense. "Parroting" is not really indicative of what modern LLMs do. However, when most people bring it up as a criticism of "AI in general" in 2026, they're using it as a pointer to all of the social/environmental criticisms, rather than the technological capabilities.
We're all going to stay fixated on what exactly a "parrot" is, but the paper doesn't rest on the definition of tropical birds: it advances a specific and detailed hypothesis about how language models don't operate with communicative intent, and in fact the meaning humans obtain from LLM output is a cognitive pattern matching illusion. Those claims have not held up at all.
At this point, I think the authors are really counting on people not to read their paper.
There's plenty of beautiful math there, but the relationship to what our neocortex does is pretty distant. Individual biological neurons can do fairly complicated things, including compute 10-bit parity functions (you would normally need a 3-layer MLP with a bunch of digital neurons to do this). And they don't seem to use backpropagation for learning.
Could you provide more detail? My understanding is that the neocortex is predominantly focused on forwards simulation, which seems distinct to how transformers operate.
That's fundamental to how anything that compresses/understands the world has to work, in the Kolmogeravian sense. That's why people denigrate LLMs as being just "next token predictors" - they're not wrong, but they're missing the point.
Because to do that kind of prediction out in the world you have to build up an accurate model of reality - a model that includes yourself! Which is why we and LLMs are self aware.
For the "how", it's been known for some time that LLMs operate on a Reimannian manifold - the semantic manifold - and that's a good place to start if you want to learn how they actually work; how a Reimannian manifold (plus some extra structure on top) can represent natural language in a form you can do work with is the part I find particularly beautiful. At a high level, the neocortex and LLMs appear to compute on the manifold in basically the same way - though a lot of the details are different; both are more sophisticated in some areas and less in others.
I think stochastic parroting is really a very accurate description of what they do (if underserving of the overall usefulness of LLMs). As long as you consider they are parroting from the whole of human intelligence. Its just that as they have gotten more sophisticated, the amount of gates, guardrails, and tertiary tools add variety. Trace any LLM hallucination back to provenance and you begin to see how the stochastic parrot works.
How can you articulate a criticism without a repetition of your criticism becoming "inherently pejorative"?
You can't. That's just the way the news goes.
Essentially, what you are saying is that because some people somewhere frame a statement as pejorative, the statement itself is inherently pejorative. By that logic, every criticism ever articulated is inherently an insult.
Really? Are you under the impression that parrots are able to synthesize their input and create entirely new, useful outputs which they have never heard before?
Because they don't just parrot, they interpolate, which is why they have such varied abilities. You can't explain the range of behaviours they have with just parroting, and once you accept that, why shouldn't this qualify as some form of intelligence?
There’s a lot of confusion here on what parrots exactly are capable of doing and assumptions how smart they are. I also don’t quite see what you mean by behaviour in respect to LLM’s, possibly “agentic” tool usage? Because I’m fairly certain you can explain their behaviour as “stochastic parroting” combined with very strong program, ie. harness, to interact with other systems. And perhaps this all is “some kind of intelligence”, but then you have to be very careful with what you mean by intelligence. It becomes a terribly slippery surface described like this and you see, who knows where this gradient descent takes us if we are not careful.
So personally I would still rather undersell it. But parrot or not, it is still a terribly useful little bird indeed.
Personally, I've always read that paper as a political criticism of industry and industrialized research and capitalism. After decades in academic (and industrialized research) I've learned that smart people can write convincing takedowns of things they hate- and those takedowns, due to being well written, often punch above their weight in terms of impact on the community.
I think this paper would have been best split off from the conjoined criticism of environmental effects (which could have been its own paper, but not one published by Google, since their leadership's fundamental beliefs disagree with the paper's environmental impact premise. And the remaining part on text models could have been a bit more focused on the technical issues associated with statistical text processing and meaning, rather than criticism of the power structure that is loosely associated with the current AI push.
Her language consistently defines LLMs in negative terms like “synthetic text extruder” but she claims she’s not trying to denigrate it. What’s missing for me are similar terms from her about how humans create sentences and thoughts. Judging by the state of the internet humans are quite capable of making shit up to argue their point (see latest Fox News apology). She talks about sycophantic AI but give me a car battery and some cables and I can train a sycophantic human (no I can’t but there are people who can). She’s pretty much a walking counter argument for her own claims.
I'm sorry but I do tend to feel like this muddies up the discussion on "what this technology really is".
I think "artificial" is actually a pretty good term to describe the output of the models. That output does appear to resemble at least some definition of the word "intelligence" - there is some ability there to do cognition over information that's been provided to them in-context.
What is it to understand, then? If they can work in complex domains and produce coherent output, it would seem to necessitate at least some definition of "understanding" of the corpus, even if that understanding is unlike how a human's brain would understand it.
What else should we call them then? They model language and information in ways that allow them to manipulate it on the fly. They do so 'unnaturally' from a human's point of reference.
I legitimately can't come up with a better term than 'artifical intelligence' -- not to be confused with artificial consciousness, which I don't think exists (yet).
"Virtual intelligence" is better. Transformer ANNs are dramatically dumber than cockroaches and it doesn't make sense to describe such a system as being artificially intelligent, for the same reason it doesn't make sense to describe Half-Life: Alyx as an "artificial reality." An artificial reality implies some sort of scientific fidelity to actual reality. A virtual reality just has to be temporarily convincing. Likewise transformer LLMs have essentially zero actual intelligence - e.g. SOTA "reasoning" models still seem much worse at small-integer quantitative reasoning than almost all vertebrates. But LLMs have an enormous amount of formal subject matter knowledge and inexhaustible stamina at solving tedious O(n) problems. So for many purposes they are an adequate virtual intelligence. At least temporarily.
My source is "none of us have ever seen a robot that can navigate unfamiliar 3D spaces as well as a cockroach." If transformers were capable of the job we would have seen a smart robot by now. But all of our robots are truly mindless compared to the simplest insects.
I will change my mind if someone demonstrates such a robot. Absent this demonstration, cockroach-level AI is still an unsolved problem. Given how ignorant and arrogant and wealthy AI researchers are, it will remain unsolved. I don't think anyone alive today will live to see a robot smarter than an ant.
Large language model is the term for what most people call "artificial intelligence". Bender's point is that labeling everything as artificial intelligence makes it more difficult to get funded or to regulate the technology. It's like walking into a car and being enamored by all the technology and saying "It's all computer". Yes, it's computers but that is not an accurate description of the many technologies inside that car.
OpenAI offered ChatGPT to the world. A large, monied cross-section of the world had yet to throw its capital behind the Large Language Model technology that made the ChatBot possible. While it is fair to see AI development now as a global imposition, OpenAI did not have the agency as a 2022 startup to impose on the scale we see now.
I asked Mistral, and it guestimated that Altman, Thiel, Musk, and Hoffman had like $20.3B together when they founded it. Sound to me that the founding of OpenAI was exactly the point when the monied world threw its dollars behind AI.
Doesn't really matter that much what they're called as long as they're useful, and LLMs (particularly when harnessed) are already ridiculously useful. But it also begs the question: are stochastic parrots useful?
Yes, they are. Likely due to a deep relationship between math and physics, statistical modelling of complex natural phenomena has repeatedly been shown to be the most effective approach. This is true of LLMs, but also of many stochastic (and other) systems.
This all sounds like a lot of backpedaling and “well actually” kind of stuff.
“Stochastic parrot got picked up and interpreted by other people as a minimization or an insult. It was not meant that way. Other people might be using it that way but that’s not how I intended it”.
Yeah that’s because it was chosen to be an insulting phrase.. Parroting is only ever used as a pejorative phrase. But sure, everyone else mindlessly parroting this line is the problem here.
This paper was always lousy, but it has really not aged well. We are living in a world when where an LLM has solved an Erdos problem. In a world where LLMs produce novel results that rival human thinking any conceptual reduction of an LLM is going to start inviting some unpleasant comparisons with human thinking.
Yes, and I don't understand how people like this paper authors mostly disregard all these achievements. It is obvious now that our common definition of "understanding" either is flawed, or at least needs redefining and precisioning.
Anyone who has spent time with parrots would realise that they can understand the meaning of speech without knowing what the words mean. Then somehow the meaning of word parts, and then you will find them making new words out of other words. Very clever indeed.
So stochastic parrots could indeed be a good description of LLMs. But I think that she meant it as a diminishing term (against the technology) which is pointless. Probably more of a reaction against SV tech bros than more nuanced interpretations.
Five years on, which term do we see as less accurate to describe LLMs? Artificial Intelligence or Stochastic Parrot? I guess it's still an open debate.
Statistical models have repeatedly shown themselves to be the most productive research method for working with complex human-based systems (and in the larger study of natural phenomena). It remains unclear whether there is any short term path for symbolic methods to catch up and exceed the capabilities of current/near-future statistical systems.
To me the real question begins only once we have a clear example of a non-trivial scientific discovery that is implicit (IE, not an obvious outcome of reading the literature and talking to the experts) and experimentally verifiable. Once that happens- especially if it is a reproducible process (IE, more discoveries) and it's significant (IE, impacts human life and mind in some profound way)- then the onus very much lies on Bender and her coauthors to explain whether we need more than a sufficiently advanced stochastic parrot.
There seems to be some confusion between "we can" and "we should" in your comment. Bender (and others) are not discussing the capabilities, but rather (a) the fundamental mechanism(s) (b) the advisability and desirability of deploying systems that use these mechanisms.
There's no statement one way or another about should in my comment; and, for what it's worth, my ideal would be an immediate global pause in AI research and development.
But the different terms imply different mental models of what LLMs are and can do. If you take two people, one who thinks of them as "artificial intelligence" and one as "stochastic parrots" (with all the implicit context and connotations of the individual words composing them), what mental model would have led to better predictions of LLMs' future circa 2020?
The "stochastic parrots" phrase is very dangerous in that frame. People read far more into what capabilities it implies are (im)possible than the narrow technical description the authors originally argued for. If all they are is spicy autocomplete or pastiche plagiarizers, there's nothing serious to worry about. And when an opposition gets stuck in a trough that mindlessly dismisses their future capabilities out of hand because of a bad mental model, it renders them ineffective at preventing the worst outcomes.
>Which frame inspires a more productive research program?
This question depends on how you define research productivity. There is close to two hundred AI papers published every weekday. Most of them are about GenAI. Most don't seem to be all thay good. The progress in actual model improvement had mostly stalled. If you interact with the latest "raw" models they display all of the issues we've seen in GPT-3.5, just at a smaller rate. The "amazing gamechanger breakthroughs" I read about on social media every week do not seem to lead anywhere. It's all kind of boring, really.
The new "hotness" in AI is clearly building more and more elaborate harnesses. This is not at all the direction AI boosters have predicted couple years ago.
Personally, I think the "stochastic parrot" mental model is far more useful for science, because it primes people for proper testing, skepticism and researching alternatives. If you want useful AI, you want people working on it being skeptical, not credulous.
> Which has better predicted the trajectory of capabilities over the past five years?
By that standard, parrots, and it's not even close. The framing of intelligence led to an enormous number of predictions that simply haven't been realised: an end to all white collar work, UBI, a total revolution in society, a literal robot god.
People are so desperate to view 'stochastic parrots' as dismissive that they misread the original argument while quickly ignoring all the failed predictions about how AI was going to overturn, save, and destroy everything.
The latter is definitely more colorful, and reflects a parrot's tendency to glom on to patterns. "Not X, but Y" being one of the more infamous ones.
Once in frustration I called a certain frontier model "Sam Altman's Tin Bird" to another agent with memory, and ever since then that other agent refers to ChatGPT as "the tin bird". Definitely a RAG artifact more than an attractor in that case, but I found it amusing.
I think "(intelligent) language understander" is an apt term. It contains within it the fact that these models are mainly trained on text, and "understand" it beyond a simple token-by-token level (i.e. their latent space maps to more and more complex concepts).
It also separates them from "world understanders" since any understanding they might have about the world comes from text (or images if we include multimodal models). They do not gather experience, memories or other "qualia" that many people (me included) would probably include in a definition of human experience/intelligence.
(fwiw i think artificial intelligence is a good, broad term, but it is both too broad to describe the current sota, and too loaded nowadays to be using in nuanced discussions)
This is a false dichotomy. Artificial Intelligence is more of a marketing term type of Hi-Fi or High Definition, ie. being a “suitcase word”[1], ie. it packs various different meanings and phenomena together to the point that without explication one cannot know what we are even talking about. Content recommendation system and LLM are completely different things.
What professor Bender is trying to explain here is that they were trying to describe how the LLM’s actually operate, to which point stochastic parrots is a fairly decent term. It is only disparaging if you know absolutely nothing how LLM’s work or you have some strange affixation to chatbots and believing they are far more capable than they actually are.
Its less of open debate would say, and although superposition [1] is interesting, as a way to explain power of some effects, it is clear they are right now closer to Stochastic Parrots than AGI.
Why do I say that? Because you can trivially beat most guardrails, simply by encoding your prompt in base64 for example. :-) Just word matching...no real understanding.
Nearly all (99%+) people who use this phrase are anti-AI and just looking to show off how much they dislike AI and how clever they can be in insulting it.
So it's a great phrase because in just about every case I can ignore what someone says afterwards.
At least "glorified autocomplete" is technically accurate, even if vastly underestimating the capability of LLMs. It's just trying to make something very impressive sound trivial.
From an external standpoint, talking to another human, it's like the other human says one word and then says the next word. That's just how language works. Humans look like "glorified autocomplete" from this perspective.
I mean, looking at the time evolution of the state of the universe, one could say that all of physics and creation is "glorified autocomplete" to posit a next state of the universe given current and past state.
I dunno, man, I looked at that text and I see one word after another.
Obviously language and the connection to human thought is more subtle than this; I think we all have a rich inner life. Just from an external perspective we can't observe it; all we can see is the token/phoneme stream. I'm just saying that it's a mistake to try to criticize LLMs on this basis because it's hard to see how the same criticism would not apply to any system (like humans) that generate language.
If you want to see words form a shape I could point you towards concrete poetry, but I guess there is no point. Joyce wrote Finnegan’s Wake for 17 years and although superficially it seems complete gibberish, trodding through it you find meaning to words that are in no dictionary, sentence structures alien to English, etc. but still you are able to understand it, and perhaps some way the mind that produced it. So I disagree with you, we can observe each other’s inner life. It is always unexpected, strange, exciting, but always rooted to our shared experience or what it like being a very big and confused ape.
LLM’s are usually unexpected only when they malfunction and sprout same letter again and again etc - hardly a literary masterpiece. They make very easily recognisable patterns that we can use as helpful tools, but in the end they are devoid of any meaning apart from what we give them. Of course one could say same about art and all language, but I think there still is the fact that we apes somehow recognise each other. And besides, we do know the internal functions that drive the parroting. It is admittedly bit tricky, but in no way as magical as people purport it to be.
afaik before the final sampling, every "next" token has a probability, so theoretically it could select the 10 most likely tokens (based on some kind of sampling algorithm), but you'd end up with exponentially many output-sequences, so nobody does that.
I think the point the poster above was making is that it doesn't predict a phrase or anything like that - just the single next token. So all 10 or 1000 or whatever number of tokens you want are each individually candidates for the single next token, not a sequence of 10 or 100 next tokens. If you wanted to create multiple possible seuqneces, you'd then feed each of the 10 tokens to the network in the initial state, and extract the next token (or 10 next tokens) from that one, than revert back and feed another single one of the 10 tokens, etc.
it annoys me how eager people are to hurl the word stochastic as pejorative. Statistics are a great tool for gleaning information from stochastic processes; statistics don't contribute randomness. Random sampling is necessary in order not to bias a sample, it's not used to contribute randomness to the sample but to preserve/measure the underlying distribution. (not meant to imply that training is random sampling)
It's a pejorative only because determinism is what makes computers useful in the first place. You get a consistent result, every single time, unlike if you have a human in the loop. Because LLMs are stochastic, they have removed the thing that makes computers useful to us, thus it's a pejorative.
The term is not very useful since most humans are stochastic parrots... At least most of the time.
Not suggesting that I don't say stuff on autopilot sometimes but for many people, it's their only mode of operation. They never actually think about anything from first principles. Their whole approach to language is just chaining catchphrases together. It's how a toddler thinks; it seems like many people never moved past that stage of development.
It sometimes feels same as with the models, especially in corporate:
- Lots of Haiku around, many mistakes unless process is very clear
- Some Sonnets, still do mistakes but can adapt
- Some Opus, able to improvise and think outside the box.
But even the Human Opus/Mythos are hilariously wrong sometimes.
Conversely, that the most prominent proponents of LLMs call them artificial intelligence and then treat them like slaves they're free to abuse ought to be horrifying.
The "parrot" part of "stochastic parrot" is quite an ambiguous choice. Taken literally, it's referencing an animal that's actually quite intelligent and capable of complex, novel tasks but has no way to connect those to human language. How I've always read this though is the more literary meaning of "parrot" as "a thing that repeats words with no context". Perhaps "stochastic photocopier" would be a clearer metaphor.
Humans are not stochastic parrots. You are 100% wrong about toddlers. This was clearly explained by St. Augustine 1500 years ago:
Did I not, then, as I grew out of infancy, come next to boyhood, or rather did it not come to me and succeed my infancy? My infancy did not go away (for where would it go?). It was simply no longer present; and I was no longer an infant who could not speak, but now a chattering boy. I remember this, and I have since observed how I learned to speak. My elders did not teach me words by rote, as they taught me my letters afterward. But I myself, when I was unable to communicate all I wished to say to whomever I wished by means of whimperings and grunts and various gestures of my limbs (which I used to reinforce my demands), I myself repeated the sounds already stored in my memory by the mind which thou, O my God, hadst given me. When they called some thing by name and pointed it out while they spoke, I saw it and realized that the thing they wished to indicate was called by the name they then uttered. And what they meant was made plain by the gestures of their bodies, by a kind of natural language, common to all nations, which expresses itself through changes of countenance, glances of the eye, gestures and intonations which indicate a disposition and attitude--either to seek or to possess, to reject or to avoid. So it was that by frequently hearing words, in different phrases, I gradually identified the objects which the words stood for and, having formed my mouth to repeat these signs, I was thereby able to express my will. Thus I exchanged with those about me the verbal signs by which we express our wishes and advanced deeper into the stormy fellowship of human life, depending all the while upon the authority of my parents and the behest of my elders.
Humans learn language opportunistically. Toddlers start with a powerful "superchimpanzee" understanding of the real world, and use that to learn words in order to satisfy their needs and desires. Statistical frequency is incidental to what words a toddler learns: what matters is the real-world context. Also note how important it is that infants instinctively understand nonverbal communication.
The most depressing thing about the 2020s AI summer is watching ignorant tech workers use the success of LLMs to launder their own ignorant misanthropy. Your views are many many centuries out of date.
I’m absolutely positive St. Augustine wasn’t some prodigious person who was able to self-introspect his early language learning and break past infant amnesia. If we’d have any means on verifying, my bet would be on that he just made this all up, post-hoc rationalizing all of that.
He can remember some sensory and pre-language mental states (desire to achieve some other state, like change of body position, or experience a taste, or a different view), but all that “whimpering and grunts and various gestures of my limbs” rationalized as purposeful communication is pure fiction. At least to my understanding, of course.
> Also note how important it is that infants instinctively understand nonverbal communication.
What does “understand” here means, and what infants exactly?
From what we know about human development, before somewhere between 2 and 9 months of age infants are unable to purposefully communicate in principle. No recipient in mind: they don’t track gaze, they don’t point at objects, they don’t share affect. Purely reflexively they can cry when they hear others crying (just like how we yawn with others), but that’s about it - just a resonance, not communication.
Obviously, comparing two different machineries suggests us they’re different (duh!) Humans aren’t LLMs, and language acquisition happens significantly different between two. But stochastic parrot is not an animal, it’s a basic principle, and that principle may possibly hold true to all life with associative learning capabilities.
FWIW nothing in this comment refutes any claims made in the comment it replies to. It's probably not the worst thing in the world for humans to start being a little more humble about themselves and their capabilities. Anthropocentrism has been a fucking disaster.
This is a complete misunderstanding of how even idiots function in the real world. There is a lot of thinking that goes into living a human (or even animal) life that models are nowhere near ready to model yet. Even ignoring the physical interaction side, the way any human sets and achieves long term goals (such as getting and maintaining a job), interacting with the huge amount of systems present in day to day life, and learning new tools along the way for decades is far beyond the current abilities of these models - even if they handily beat 90-100% of humanity on some tasks normally considered much harder.
> It argued that large language models (LLMs) generate text by statistically predicting likely sequences of words rather than understanding what they are saying—a process the authors captured with the metaphor of a “stochastic parrot,” a system that repeats patterns without comprehension.
I don't understand what we're setting the record straight on. This is the core point of dispute, and the author just blazes past it to focus on other things. I'm glad to hear "stochastic parrot" isn't intended as an insult, and I agree that it's not right to think of LLMs as a box with a little homunculus inside replying to you. But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
I think it’s pretty clear that they are repeating without “comprehension” - both mechanistically (as in there is no facility for comprehension in their formulation) and in the ways they fail. The standard rs in strawberry, should I walk or drive to the car wash, etc all play on the fact that they don’t have any real world model or thoughts against which they can judge their output, as do many of the jailbreaks which basically play on the fact that the model has memorized patterns.
There are people who argue semantics, that we can call the pattern matching that LLMs do “understanding”, or the moronic “how do we know that’s isn’t all we do” but for the normal use of comprehension, LLMs at a fundamental level don’t.
> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
So this seems obvious to you, and yet to many others, it is equally obvious that LLMs can/could do the things they routinely do without any meaningful sense of "understanding".
I think it's a mistake to disentangle their abilities from understanding. Just swallow the pill that they have some form of understanding, even if it slightly differs from ours. I really don't see the problem.
I prefer to work the other way around. That is, accept that a lot of human speech (and text) is generated via similar mechanisms to the ones that drive LLMs, but note that there is another kind of behavior - reasoning - which seems to be distinct.
> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
Is it possible you're making the following error described in the article?
> The fact that these systems are designed to mimic the way we use language makes it very easy for people to mistake them for other people.
Clearly you don't believe it's actually a person ("it's not right to think of LLMs as a box with a little homunculus inside replying to you"), but you do believe it's doing something a little bit magical. Is it possible because the interface is linguistic, and every other thing in your world that communicates with language is intelligent, that you're projecting something that just isn't there onto the situation?
I'm sorry if this line of questioning is a little invasive. But this is literally the "danger" the original paper talks about, and it seems an awful lot like you've fallen for it.
I'm not offended by the line of questioning! But I don't really follow it. I don't and IIUC Bender doesn't use "understanding" to refer to any kind of magical property. Understanding is the capability of using words as consistent handles to things in the exterior world which the language is describing. And this is something LLMs can clearly do. I just went to ChatGPT and asked this question, which is almost surely not in its training data:
> What would happen if I walked to the top of a skyscraper with a soda can full of Maraschino cherries and let them go?
And its answer (https://chatgpt.com/s/t_6a4bd9ffa5708191901bb6d43c89f43b) clearly demonstrates understanding. It knew that this is a dangerous thing I should not do in real life, and that my question is ambiguous about whether I intend to drop the can, and that this might be intended as a physics problem rather than a real life scenario.
> And this is something LLMs can clearly do.
...
> It knew that this is a dangerous thing I should not do in real life
From the ChatGPT response you linked, all I see for sure is some matches on the following patterns:
drop $thing from skyscraper --> bad behavior
drop $thing --> physics
can of $stuff --> contents in/out of can
Then there are some sentences of likely characters following those patterns. You don't need anything more than a basic cartoon-level understanding of how an LLM works to explain this output. I see no evidence of reasoning or understanding here, or any theory of "real life".
It also does an incredibly poor job of answering your question. It makes no attempt to explain what might actually happen. If it has been trained on the entire corpus of medical science, and it is indeed intelligent, then surely it can reference ballistics studies and give you a very detailed and thorough theory of what--exactly--injuries you might expect from a 12oz can being dropped from the height of a skyscraper. Calculating the terminal velocity and therefore the momentum of the can is trivial. Characterizing the physics of the impact on various parts of a human body is trivial. If it actually understood your request why didn't it just answer the question?
It's a rhetorical question. LLMs do not "understand". It is completely outside their capability. "Understanding" is something we impose upon their output (to loosely quote TFA). [edit] I think the most powerful evidence for a lack of any understanding whatsoever is all the stuff about the cherries being in or out of the can. Yes, cans contain things. That is not a profound observation, nor is it at all relevant to the question. If you drop an empty can off a skyscraper nothing meaningful will happen. And, no, probably dumping all the cherries out won't hurt anyone or cause a slipping hazard... It's also not particularly relevant to point out that dropping things off skyscrapers is bad behavior. But that's more forgivable from a CYA standpoint.
I believe you are projecting something that is not there onto a completely mindless stochastic process.
But it shouldn't even be contentious like that. It's not a fundamental mystery how these things work. It is for the most part not a valid target for the kind of speculation you seem to want to do about it.
It's not like you can be agnostic, or measured about this. It's like someone explaining a car to you, saying, "look here is where you put the fuel, here is where it ignites, where the axels are turned..." And you, trying to be measured, are like "hm well yes of course that all is clearly important, but there is clearly just a bit of magic here somewhere, between all the different 'parts'."
The "magic here somewhere" in the car is in the design that reference aspects of animal anatomy (facial features, stance) and in the millions of dollars of advertising that prime the public with expectations about how they'll feel driving it, or how to see other people in the car. There's a direct connection there to packaging LLMs as chatbots, it gives them a recognizable shape and behavior that a lot of people interpret as consciousness and personality.
This is a facile point. Lisp expert systems transparently don't understand the meaning of any symbols they process, yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability. The fact that LLMs are less transparent than Lisp expert systems (and easier to program) is extremely bad evidence that they understand language. Especially given that AFAICT Opus does not properly understand concepts like "four."
> yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability
Where can I access such a Lisp expert system?
If I cannot because they don't exist: then they cannot do the same things an LLM can do. And of course one can assert anything and everything about what a non-existing thing could do.
> Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
I think this metaphor is so strained as to not be useful. I think key here is that the authors say "without any reference to meaning", which is a heavily loaded term, that does definitely apply to parrots, but does not apply when you apply it to immense bodies of text.
Namely that language embeds meaning in language. A sentence being written by a human (as a starting point) is designed to have consistent meaning. While it is possible to write syntactically correct meaningless text, that is not what most of human language has done; the meaning cannot be removed from the text.
This I think is clarifying, from the same paragraph in the text:
> ... the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.
That's just facially incorrect. The training data is entirely about sharing thoughts with a listener. Else why is the text being written?
What I have been doing in many places—the octopus thought experiment, stochastic parrots, the phrase “synthetic text-extruding machines”—it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do
> Meanwhile, O, a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially, but is very good at detecting statistical patterns. Over time, O learns to predict with great accuracy how B will respond to each of A’s utterances. O also observes that certain words tend to occur in similar contexts, and perhaps learns to generalize across lexical patterns by hypothesizing that they can be used somewhat interchangeably. Nonetheless, Ohas never observed these objects, and thus would not be able to pick out the referent of a word when presented with a set of (physical) alternatives.
This seems kind of obviously wrong at least in the context of coding agents. These models get trained on actual output of the previous version of the model doing its job, often "IRL" on a real computer/project. It's like O is in the conversation for years now and learning from his own interactions between A <-> O <-> B, where A is the human and B is the computer.
The idea O ontologically has never "observed" "these objects" or referents is philosophically strained. Have I observed the moon, or a finger pointing at the moon? Have I observed `sed` more than Fable?
I think this is the most measured take I've seen from Bender, and I think it summarizes her only compelling point well (technologies should be referred to specifically rather than generally as AI, and that referring to everything as AI is not useful and helps hype the technology in a way that benefits those selling it).
In her previous interviews, I've found her assertion that LLMs aren't useful and will never be good at anything totally uncompelling. Also laughed at this quote as she's been pretty harsh IMO on "the people who like the systems".
> it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do, which is not the same thing as insulting the systems or insulting the people who like the systems.
After having used LLMs for some time now, I don't agree with the concept they are just token generators, unless you think that's all humans are too. The way we test in most schools is just picking the right token. We also give them unique problems that they never saw in their training, which is the nature of programming. I realize they are probabilistic token generator models, but I find it harder and harder to accept that somehow there isn't something more going on. I'm not sure whether they are intelligent or not, but for the most part token generation is how you get degrees too. The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.
Here's the thing: most things people do does not involve tokens of any kind. It is, in fact, stuff that not easily describable. For example, it's trivial for a person to walk, but they cannot verbally describe what muscles they're activating in what order to make that happen.
Cognitive skills such as tool use and complex navigation predate language as well. That means there's a core of reasoning in humans that doesn't depend on "tokens" or "language" of any kind. Language is a tool for communication and forming complex human societies, but it's not cognition.
> The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.
Well a parrot does perform complex reasoning on novel situations all the time. It just doesn't have the wiring to connect that to "tokenized" human language. I suspect LLMs have the opposite problem, where they exist in the domain of their "tokens" and have no way to connect these to truly novel situations that have no existing words to describe them.
> in part because Google fired two of the authors, Timnit Gebru
I remember being angry about this situation when I first saw it on social media, until I read the details: This person submitted a list of demands to her employer and said that if they weren’t met, she quit. Google wasn’t going to meet her demands so they considered it acceptance of her resignation. There has been a movement trying to debate whether it was a firing or resignation ever since.
The original paper they published gets recirculated every year or two as some landmark history of AI safety, but as other commenters have noted it wasn’t really a great paper nor was it groundbreaking at the time. If not for the controversy surrounding the resignation/firing (depending on your POV), I don’t think it would have been notable.
True but also... she wasn't a software engineer putting code in production nor a researcher working no the fundamentals of machine learning negotiating a raise.
She was part of the "Ethical Artificial Intelligence Team" of what was then, and still is now, one of the corporations World wide spending the largest amount of resources precisely on using AI commercially.
I'm saying the paper itself wasn't a bombshell or even that noteworthy. The reason it got PR and continues to come up was because the authors manufactured this self-inflicted drama around it, not because it was leaking secret revelations that harmed the company.
I think part of it is she had excellent PR skills and a dedicated fan base. I was at Google when she quit, working in ML, and hadn't heard of her until the story broke. I remember there were a large number of Memegen posts about it, but no one I spoke with knew about her, so I assumed it was brigading.
I think she's since since lost a lot of her allure, especially when she didn't change her mind when the facts about the AI water usage changed 1000x
What exactly are the facts about AI water usage? I have trouble separating hysteria from reality but most of what I see still claims water usage is enormous
I like this summary https://www.andymasley.com/writing/the-ai-water-issue-is-fak...
And this is a reply to her comment about water usage where it becomes clear she's not arguing in good faith. https://x.com/AndyMasley/status/1990498830131888173
The hysteria around water usage rests on people not knowing the scale of industrial civilization. First thing to do is compare any estimate of data center water usage with the water usage of almond farming. Or, if you want to focus on individual consumer choices, the water footprint of eating a hamburger.
Or even better: the footprint of doing something like farming corn for ethanol
Breaking it down this way is a great way to minimize the numbers so that it appears reasonable.
See? Middle-Eastern investors are growing alfalfa in the western desert using legal allotments of water! That is so much worse than what we’re doing! Go after them!
They can both be using an egregious amount of water for silly purposes.
The other part of the water debate is also the pollution different systems create. Many data centres went in with the promise of closed-loop systems but changed half-way through construction and couldn’t be stopped.
I think it’s more complicated than, “they’re wrong, it’s just hype.”
Putting things into perspective is not minimising the problem. We literally have to do this to prioritise where our efforts can be useful.
Your argument makes sense if ai datacenters were using something close to like alfafa farming but the difference between them is soo massive it does not make sense.
Reducing pollution is a much better problem to fight for
Nice false dichotomy you got there there, might as well calculate the entire water usage for a a single GPU in the supply chain too. Something tells me one is extremely worse than the other when you account for all the water that's used in a single supply chain for high end electronics, but if you want to plop the measuring stick where ever along the whole pony show that makes you look better people will notice.
Also to compare growing food with the totally optional, not useful in the slightest, LLMs that somehow demand local populaces bend to their will for reasons that never seem to benefit them is just bonkers level of self-blinding when it comes to populations absolutely despising big tech, big tech leadership, and big tech practices.
This mania might finally cause the software industry to become a highly regulated with licenses similar to that of other engineering disciplines due to amount of optional destruction they have decided to unleash upon on the planet in such a short time frame.
This is a useful piece on that: https://andymasley.substack.com/p/the-ai-water-issue-is-fake
What is interesting about it for me is: why would someone working on AI ethics choose to work for Google at all?
Did she really think Google cares about ethics? Such positions seem purely performative, we all know that ethics go out the window first to make room for more profits.
https://news.ycombinator.com/item?id=25324263
She was well known to be toxic and extremely exploitive of victim privilege
I missed that thread when it came out, it's really a wild read. The difference between how people describe her vs how she's normally portrayed in the media is really startling
Her demands included wanting to know the identities of anyone who wanted to comment on her paper, after she had a history of going after people publicly. That's enough right there, nobody should tolerate toxic behavior regardless of whether you agree with the politics.
Meanwhile, the paper has 2 points of criticism towards AI. 1 is a bunch of carbon consumption complaints assuming NVIDIA cards with coal-fired power, while a lot of effort at contemporary Google went towards getting TPUs running on green power. I suspect this was what people wanted to object to, a lot of effort went into those green power projects and she was just denying it. The complaint seems prophetic now but it was not true about Google then.
The other criticism was about which language the LLMs use, they average the input data of normal humans instead of talking the way the paper author thinks they should talk. The phrase "women doctors" is called out as problematic. I'm less inclined to think people objected strongly to this given the zeitgeist at the time, it was probably people who worked on the green energy projects and were pissed off that their contributions were ignored, but still, nobody elected her Queen of English, she can have her opinions but she's not a victim for not having them adopted by everyone.
I think this is completely misleading. If your employer asks to redact your paper because it ignores relevant research, naturally you want to know what research they are talking about and on what grounds, and also which researches reviewed the paper (probably assuming there was none, review process was used as an excuse to silence criticism, G has done it many times).
(BTW, quite bold to say input data from Reddit and 4Chan is how “normal” people speak. There is a lot of language in the training data of any model you really do not wish your application to use ever.)
Pressuring an employee to add unethical behavior or specific religious practices to their job description is constructive termination.
I'd say what's under debate is whether uncritical LLM adoption is mainly unethical or mainly religious.
This is not why she was fired and wouldn't have been a plausible reason in 2020 when it happened.
>Timnit responded with an email requiring that a number of conditions be met in order for her to continue working at Google, including revealing the identities of every person who Megan and I had spoken to and consulted as part of the review of the paper and the exact feedback. Timnit wrote that if we didn’t meet these demands, she would leave Google and work on an end date. We accept and respect her decision to resign from Google.
This is Google's side of it; I think the following is a fair piece of primary-source journalism if you want to go deeper:
https://www.platformer.news/the-withering-email-that-got-an-...
I don't see why it's those are the only two options, nor why they are even mutually exclusive.
Here is what Jeff Dean said about the firing at the time: https://docs.google.com/document/d/1f2kYWDXwhzYnq8ebVtuk9CqQ...
> resignation
I appeciate short letters like this that get straight to the point...
> With the octopus thought experiment, I initially had told the story in terms of a dolphin, because dolphins clearly are intelligent animals. My co-author on that paper, Alexander Koller, said it should be an octopus, because first of all, the environment that octopuses live in is much more distinct from where people live. It makes the metaphor more vivid, that the octopus is just feeling these pulses in the cable and has no way to look at what the people are looking at.
On a completely tangential sidenote, octopusses are actually very very intelligent: https://www.nhm.ac.uk/discover/octopuses-keep-surprising-us-...
It's such a tragedy that they're also extremely solitary animals and die shortly after reproducing the first (and only) time.
Almost all other particularly intelligent animals seem to be gregarious, and it's easy to conclude that a social lifestyle tends to select for more intelligence, a sophisticated theory of mind, and so on (I like to think that that's exactly what was responsible for a runaway intelligence explosion in humans). But in the case of cephalopods, there's something else that has been applying selection pressure towards exceptional intelligence.
I agree, which is why I think this species might be the start of something amazing: https://en.wikipedia.org/wiki/Larger_Pacific_striped_octopus
I've never heard of these before, that's fascinating, thank you!
It's also a bit of a chicken-and-egg problem: if they were raised by their parents like all other more intelligent animals, they wouldn't need to be as intelligent as they are in order to be able to relearn "octopus behaviour" without help from other members of the species.
In Stefan Wul's SF novel "Nyourk", octopi evolve to become the Earth's dominant species, which was quite prescient back in 1957, when almost nobody knew octopi to be intelligent. :)
Also, last time I checked, the environment where octopuses live is actually the exact same environment where dolphins live?
Well in a sense that monkeys and Great Condor inhabit the exact same environment.
That's a bit exaggerated - monkeys can't fly, but both dolphins and octopuses can swim. I'm aware octopuses prefer to stay at the bottom, while dolphins have to come to the surface regularly to breathe, but for me it's still the same environment...
The continued use of animal metaphors is doing them a great disservice. Esp as we learn more about animal cognition, on first look, it smacks of human exceptionalism that has littered the historic scientific consensus.
Now if they had said, "Imagine your average American ..." (/s)
Bender's paper had this to say about stochastic parrots:
"Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot."
This was not even a correct criticism in 2021. She is right that, at the time, the pretraining -- where it learns to predict missing words in pre-existing corpuses of text -- is basically a stochastic parrot.
But nowhere in her paper does the term "reinforcement" come up. At the time, this was done mainly through RLHF (reinforcement learning from human feedback) - after the initial training is done, you then tune the model's responses based on human grading. Humans imbue their own meanings into the parameter weights through their judgment.
At this point, they aren't really stochastic parrots anymore, because parameter weights have been shaped beyond the text corpus. It's not purely probabilistic in the sense of using the probabilities of the underlying text sequences. (It still is probabilistic in its output, but that is a pointless claim, because all events in the universe are also probabilistic; it is not enough to merely claim that probability is involved in some way in the outputs.)
RLHF was already in use prior to the paper, and was written about by Christiano in 2017 "Deep reinforcement learning from human preferences," so it's surprising that Bender apparently didn't know about this well-known paper.
RLHF was also, of course, a precursor to a more advanced form of parameter shaping - reinforcement learning with verified rewards, or RLVF, which has driven a lot of the gains in verifiable domains lately. That was not done in 2021 when she wrote the paper. But if you knew about RLHF -- and knew how Alpha Zero worked, with training neural nets on game rollouts -- you could squint and see that it might be useful for language models.
So after being proven to not only having a limited understanding of the field at the time, but also not being able to forecast the field, she's now walking back what she meant by "stochastic parrot," I assume because she believes readers will not read what she wrote. But despite the protests, her original claim was that it is a parrot because the text has no meaning -- a direct quote from the paper, which only really makes sense if training stops at the pretrain.
I paid a bit of attention to this paper and the phrase 'stochastic parrots' when it came out and i thought this was worth saying and doing at that time. their suggestions about financial and environmental costs are worth studying, their concern about carefully evaluating datasets to feed to the model rather than feeding the entire internet is fully justified. so - to everyone saying this was a bad paper; if you have actually read the paper then please list a few criticisms. all i have seen is "oh this wasn't that good of a paper" or "can't believe how bad this paper was".
It is a good blog, not a good paper.
My main criticism of the paper is that it says LLMs work "haphazardly", using probabilistic information. That is a hypothesis, but it is stated as a known fact, a fundamental limitation.
It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways!
Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.
not sure your criticism makes sense though - they did this pre chatgpt. they are talking about the language models of that time. they did not make predictions about the future.
Those costs have to be compared to the way things are currently done without AI.
They never are. Ever.
I don’t see a problem with the “stochastic parrot” label. It just turns out stochastic parrots are incredibly useful.
At a minimum it’s probably more accurate than “AI”.
Whether Bender intended it or not, the term has an inherently pejorative sense. "Parroting" is not really indicative of what modern LLMs do. However, when most people bring it up as a criticism of "AI in general" in 2026, they're using it as a pointer to all of the social/environmental criticisms, rather than the technological capabilities.
We're all going to stay fixated on what exactly a "parrot" is, but the paper doesn't rest on the definition of tropical birds: it advances a specific and detailed hypothesis about how language models don't operate with communicative intent, and in fact the meaning humans obtain from LLM output is a cognitive pattern matching illusion. Those claims have not held up at all.
At this point, I think the authors are really counting on people not to read their paper.
Yeah, there's some beautiful math underlying what LLMs are doing, and it's the same math our neocortex runs on.
There's plenty of beautiful math there, but the relationship to what our neocortex does is pretty distant. Individual biological neurons can do fairly complicated things, including compute 10-bit parity functions (you would normally need a 3-layer MLP with a bunch of digital neurons to do this). And they don't seem to use backpropagation for learning.
Could you provide more detail? My understanding is that the neocortex is predominantly focused on forwards simulation, which seems distinct to how transformers operate.
That's fundamental to how anything that compresses/understands the world has to work, in the Kolmogeravian sense. That's why people denigrate LLMs as being just "next token predictors" - they're not wrong, but they're missing the point.
Because to do that kind of prediction out in the world you have to build up an accurate model of reality - a model that includes yourself! Which is why we and LLMs are self aware.
For the "how", it's been known for some time that LLMs operate on a Reimannian manifold - the semantic manifold - and that's a good place to start if you want to learn how they actually work; how a Reimannian manifold (plus some extra structure on top) can represent natural language in a form you can do work with is the part I find particularly beautiful. At a high level, the neocortex and LLMs appear to compute on the manifold in basically the same way - though a lot of the details are different; both are more sophisticated in some areas and less in others.
I think stochastic parroting is really a very accurate description of what they do (if underserving of the overall usefulness of LLMs). As long as you consider they are parroting from the whole of human intelligence. Its just that as they have gotten more sophisticated, the amount of gates, guardrails, and tertiary tools add variety. Trace any LLM hallucination back to provenance and you begin to see how the stochastic parrot works.
How can you articulate a criticism without a repetition of your criticism becoming "inherently pejorative"?
You can't. That's just the way the news goes.
Essentially, what you are saying is that because some people somewhere frame a statement as pejorative, the statement itself is inherently pejorative. By that logic, every criticism ever articulated is inherently an insult.
Why is it not indicative of what LLM’s do?
Really? Are you under the impression that parrots are able to synthesize their input and create entirely new, useful outputs which they have never heard before?
The "stochastic" is the key modifier to "parrot".
> "Stochastic" means having a random pattern or variable that can be analyzed statistically but not predicted precisely
Yes, I am: https://en.wikipedia.org/wiki/Alex_(parrot)
Because they don't just parrot, they interpolate, which is why they have such varied abilities. You can't explain the range of behaviours they have with just parroting, and once you accept that, why shouldn't this qualify as some form of intelligence?
There’s a lot of confusion here on what parrots exactly are capable of doing and assumptions how smart they are. I also don’t quite see what you mean by behaviour in respect to LLM’s, possibly “agentic” tool usage? Because I’m fairly certain you can explain their behaviour as “stochastic parroting” combined with very strong program, ie. harness, to interact with other systems. And perhaps this all is “some kind of intelligence”, but then you have to be very careful with what you mean by intelligence. It becomes a terribly slippery surface described like this and you see, who knows where this gradient descent takes us if we are not careful.
So personally I would still rather undersell it. But parrot or not, it is still a terribly useful little bird indeed.
Personally, I've always read that paper as a political criticism of industry and industrialized research and capitalism. After decades in academic (and industrialized research) I've learned that smart people can write convincing takedowns of things they hate- and those takedowns, due to being well written, often punch above their weight in terms of impact on the community.
I think this paper would have been best split off from the conjoined criticism of environmental effects (which could have been its own paper, but not one published by Google, since their leadership's fundamental beliefs disagree with the paper's environmental impact premise. And the remaining part on text models could have been a bit more focused on the technical issues associated with statistical text processing and meaning, rather than criticism of the power structure that is loosely associated with the current AI push.
Her language consistently defines LLMs in negative terms like “synthetic text extruder” but she claims she’s not trying to denigrate it. What’s missing for me are similar terms from her about how humans create sentences and thoughts. Judging by the state of the internet humans are quite capable of making shit up to argue their point (see latest Fox News apology). She talks about sycophantic AI but give me a car battery and some cables and I can train a sycophantic human (no I can’t but there are people who can). She’s pretty much a walking counter argument for her own claims.
I'm sorry but I do tend to feel like this muddies up the discussion on "what this technology really is".
I think "artificial" is actually a pretty good term to describe the output of the models. That output does appear to resemble at least some definition of the word "intelligence" - there is some ability there to do cognition over information that's been provided to them in-context.
What is it to understand, then? If they can work in complex domains and produce coherent output, it would seem to necessitate at least some definition of "understanding" of the corpus, even if that understanding is unlike how a human's brain would understand it.
What else should we call them then? They model language and information in ways that allow them to manipulate it on the fly. They do so 'unnaturally' from a human's point of reference.
I legitimately can't come up with a better term than 'artifical intelligence' -- not to be confused with artificial consciousness, which I don't think exists (yet).
"Virtual intelligence" is better. Transformer ANNs are dramatically dumber than cockroaches and it doesn't make sense to describe such a system as being artificially intelligent, for the same reason it doesn't make sense to describe Half-Life: Alyx as an "artificial reality." An artificial reality implies some sort of scientific fidelity to actual reality. A virtual reality just has to be temporarily convincing. Likewise transformer LLMs have essentially zero actual intelligence - e.g. SOTA "reasoning" models still seem much worse at small-integer quantitative reasoning than almost all vertebrates. But LLMs have an enormous amount of formal subject matter knowledge and inexhaustible stamina at solving tedious O(n) problems. So for many purposes they are an adequate virtual intelligence. At least temporarily.
> Transformer ANNs are dramatically dumber than cockroaches
Source?
My source is "none of us have ever seen a robot that can navigate unfamiliar 3D spaces as well as a cockroach." If transformers were capable of the job we would have seen a smart robot by now. But all of our robots are truly mindless compared to the simplest insects.
I will change my mind if someone demonstrates such a robot. Absent this demonstration, cockroach-level AI is still an unsolved problem. Given how ignorant and arrogant and wealthy AI researchers are, it will remain unsolved. I don't think anyone alive today will live to see a robot smarter than an ant.
We might as well say that none of us have ever seen a cockroach that can solve Erdos problems as well as an LLM. Or write code, or stories, etc.
Large language model is the term for what most people call "artificial intelligence". Bender's point is that labeling everything as artificial intelligence makes it more difficult to get funded or to regulate the technology. It's like walking into a car and being enamored by all the technology and saying "It's all computer". Yes, it's computers but that is not an accurate description of the many technologies inside that car.
> when OpenAI imposed ChatGPT on the world...
OpenAI offered ChatGPT to the world. A large, monied cross-section of the world had yet to throw its capital behind the Large Language Model technology that made the ChatBot possible. While it is fair to see AI development now as a global imposition, OpenAI did not have the agency as a 2022 startup to impose on the scale we see now.
> A large, monied cross-section of the world
I asked Mistral, and it guestimated that Altman, Thiel, Musk, and Hoffman had like $20.3B together when they founded it. Sound to me that the founding of OpenAI was exactly the point when the monied world threw its dollars behind AI.
I agree with a lot of her points but that word really is revealing of her thoughts about OpenAI.
Doesn't really matter that much what they're called as long as they're useful, and LLMs (particularly when harnessed) are already ridiculously useful. But it also begs the question: are stochastic parrots useful?
Yes, they are. Likely due to a deep relationship between math and physics, statistical modelling of complex natural phenomena has repeatedly been shown to be the most effective approach. This is true of LLMs, but also of many stochastic (and other) systems.
This all sounds like a lot of backpedaling and “well actually” kind of stuff.
“Stochastic parrot got picked up and interpreted by other people as a minimization or an insult. It was not meant that way. Other people might be using it that way but that’s not how I intended it”.
Yeah that’s because it was chosen to be an insulting phrase.. Parroting is only ever used as a pejorative phrase. But sure, everyone else mindlessly parroting this line is the problem here.
This paper was always lousy, but it has really not aged well. We are living in a world when where an LLM has solved an Erdos problem. In a world where LLMs produce novel results that rival human thinking any conceptual reduction of an LLM is going to start inviting some unpleasant comparisons with human thinking.
Yes, and I don't understand how people like this paper authors mostly disregard all these achievements. It is obvious now that our common definition of "understanding" either is flawed, or at least needs redefining and precisioning.
I respect you and parrots, please don’t use parrots as an insult.
Anyone who has spent time with parrots would realise that they can understand the meaning of speech without knowing what the words mean. Then somehow the meaning of word parts, and then you will find them making new words out of other words. Very clever indeed.
So stochastic parrots could indeed be a good description of LLMs. But I think that she meant it as a diminishing term (against the technology) which is pointless. Probably more of a reaction against SV tech bros than more nuanced interpretations.
Five years on, which term do we see as less accurate to describe LLMs? Artificial Intelligence or Stochastic Parrot? I guess it's still an open debate.
Which frame inspires a more productive research program? Which has better predicted the trajectory of capabilities over the past five years?
Statistical models have repeatedly shown themselves to be the most productive research method for working with complex human-based systems (and in the larger study of natural phenomena). It remains unclear whether there is any short term path for symbolic methods to catch up and exceed the capabilities of current/near-future statistical systems.
To me the real question begins only once we have a clear example of a non-trivial scientific discovery that is implicit (IE, not an obvious outcome of reading the literature and talking to the experts) and experimentally verifiable. Once that happens- especially if it is a reproducible process (IE, more discoveries) and it's significant (IE, impacts human life and mind in some profound way)- then the onus very much lies on Bender and her coauthors to explain whether we need more than a sufficiently advanced stochastic parrot.
There seems to be some confusion between "we can" and "we should" in your comment. Bender (and others) are not discussing the capabilities, but rather (a) the fundamental mechanism(s) (b) the advisability and desirability of deploying systems that use these mechanisms.
There's no statement one way or another about should in my comment; and, for what it's worth, my ideal would be an immediate global pause in AI research and development.
But the different terms imply different mental models of what LLMs are and can do. If you take two people, one who thinks of them as "artificial intelligence" and one as "stochastic parrots" (with all the implicit context and connotations of the individual words composing them), what mental model would have led to better predictions of LLMs' future circa 2020?
The "stochastic parrots" phrase is very dangerous in that frame. People read far more into what capabilities it implies are (im)possible than the narrow technical description the authors originally argued for. If all they are is spicy autocomplete or pastiche plagiarizers, there's nothing serious to worry about. And when an opposition gets stuck in a trough that mindlessly dismisses their future capabilities out of hand because of a bad mental model, it renders them ineffective at preventing the worst outcomes.
>Which frame inspires a more productive research program?
This question depends on how you define research productivity. There is close to two hundred AI papers published every weekday. Most of them are about GenAI. Most don't seem to be all thay good. The progress in actual model improvement had mostly stalled. If you interact with the latest "raw" models they display all of the issues we've seen in GPT-3.5, just at a smaller rate. The "amazing gamechanger breakthroughs" I read about on social media every week do not seem to lead anywhere. It's all kind of boring, really.
The new "hotness" in AI is clearly building more and more elaborate harnesses. This is not at all the direction AI boosters have predicted couple years ago.
Personally, I think the "stochastic parrot" mental model is far more useful for science, because it primes people for proper testing, skepticism and researching alternatives. If you want useful AI, you want people working on it being skeptical, not credulous.
> Which has better predicted the trajectory of capabilities over the past five years?
By that standard, parrots, and it's not even close. The framing of intelligence led to an enormous number of predictions that simply haven't been realised: an end to all white collar work, UBI, a total revolution in society, a literal robot god.
People are so desperate to view 'stochastic parrots' as dismissive that they misread the original argument while quickly ignoring all the failed predictions about how AI was going to overturn, save, and destroy everything.
The latter is definitely more colorful, and reflects a parrot's tendency to glom on to patterns. "Not X, but Y" being one of the more infamous ones.
Once in frustration I called a certain frontier model "Sam Altman's Tin Bird" to another agent with memory, and ever since then that other agent refers to ChatGPT as "the tin bird". Definitely a RAG artifact more than an attractor in that case, but I found it amusing.
Though, I would point out that where people fall on that seems to correlate very highly with their ability to explain how an attention head works.
Which direction is the correlation?
I don’t think this phrase means what people assume when it’s applied to post trained instruct models - which did not exist when the paper was written.
After RL it is not predicting based on samples of the original corpus - but is also chasing a reward function that does require other features.
There has been a lot of subsequent research that really calls many of the statements in this article into question.
Explain it to me
I think "(intelligent) language understander" is an apt term. It contains within it the fact that these models are mainly trained on text, and "understand" it beyond a simple token-by-token level (i.e. their latent space maps to more and more complex concepts).
It also separates them from "world understanders" since any understanding they might have about the world comes from text (or images if we include multimodal models). They do not gather experience, memories or other "qualia" that many people (me included) would probably include in a definition of human experience/intelligence.
(fwiw i think artificial intelligence is a good, broad term, but it is both too broad to describe the current sota, and too loaded nowadays to be using in nuanced discussions)
Understand is a pretty imprecise term. What does it mean for a computer to understand? Does an H264 decoder understand Eraserhead.mkv?
This is a false dichotomy. Artificial Intelligence is more of a marketing term type of Hi-Fi or High Definition, ie. being a “suitcase word”[1], ie. it packs various different meanings and phenomena together to the point that without explication one cannot know what we are even talking about. Content recommendation system and LLM are completely different things.
What professor Bender is trying to explain here is that they were trying to describe how the LLM’s actually operate, to which point stochastic parrots is a fairly decent term. It is only disparaging if you know absolutely nothing how LLM’s work or you have some strange affixation to chatbots and believing they are far more capable than they actually are.
[1] Coined by Marvin Minsky: https://www.thekurzweillibrary.com/consciousness-is-a-big-su...
What's wrong with "large language model"?
Seems like a lot of people are upset about other people calling both apples and oranges “fruit”.
Its less of open debate would say, and although superposition [1] is interesting, as a way to explain power of some effects, it is clear they are right now closer to Stochastic Parrots than AGI.
Why do I say that? Because you can trivially beat most guardrails, simply by encoding your prompt in base64 for example. :-) Just word matching...no real understanding.
[1] https://chrisclay.substack.com/p/what-is-superposition-in-ne...
Spicy autocomplete
> Stochastic Parrot
Nearly all (99%+) people who use this phrase are anti-AI and just looking to show off how much they dislike AI and how clever they can be in insulting it.
So it's a great phrase because in just about every case I can ignore what someone says afterwards.
Similar to "glorified autocomplete."
At least "glorified autocomplete" is technically accurate, even if vastly underestimating the capability of LLMs. It's just trying to make something very impressive sound trivial.
From an external standpoint, talking to another human, it's like the other human says one word and then says the next word. That's just how language works. Humans look like "glorified autocomplete" from this perspective.
I mean, looking at the time evolution of the state of the universe, one could say that all of physics and creation is "glorified autocomplete" to posit a next state of the universe given current and past state.
That’s not how language works https://www.telelib.com/authors/J/JoyceJames/prose/finnegans...
I dunno, man, I looked at that text and I see one word after another.
Obviously language and the connection to human thought is more subtle than this; I think we all have a rich inner life. Just from an external perspective we can't observe it; all we can see is the token/phoneme stream. I'm just saying that it's a mistake to try to criticize LLMs on this basis because it's hard to see how the same criticism would not apply to any system (like humans) that generate language.
If you want to see words form a shape I could point you towards concrete poetry, but I guess there is no point. Joyce wrote Finnegan’s Wake for 17 years and although superficially it seems complete gibberish, trodding through it you find meaning to words that are in no dictionary, sentence structures alien to English, etc. but still you are able to understand it, and perhaps some way the mind that produced it. So I disagree with you, we can observe each other’s inner life. It is always unexpected, strange, exciting, but always rooted to our shared experience or what it like being a very big and confused ape.
LLM’s are usually unexpected only when they malfunction and sprout same letter again and again etc - hardly a literary masterpiece. They make very easily recognisable patterns that we can use as helpful tools, but in the end they are devoid of any meaning apart from what we give them. Of course one could say same about art and all language, but I think there still is the fact that we apes somehow recognise each other. And besides, we do know the internal functions that drive the parroting. It is admittedly bit tricky, but in no way as magical as people purport it to be.
> one could say that all of physics and creation is "glorified autocomplete"
Exhibit A.
Pattern matching machines seems more appropriate.
For humans?
LLMs do not match patterns. They predict one statistically most likely token (only one!) given a history of some N previously known tokens.
Is that prediction not based on matching previous patterns, whose frequencies are more or less encoded in the weights?
you're really reaching for no apparent reason. Just move on from pattern matching machines it's not a good mental model for LLMs
> statistically most likely
Isn't that pattern matching essentially?
afaik before the final sampling, every "next" token has a probability, so theoretically it could select the 10 most likely tokens (based on some kind of sampling algorithm), but you'd end up with exponentially many output-sequences, so nobody does that.
I think the point the poster above was making is that it doesn't predict a phrase or anything like that - just the single next token. So all 10 or 1000 or whatever number of tokens you want are each individually candidates for the single next token, not a sequence of 10 or 100 next tokens. If you wanted to create multiple possible seuqneces, you'd then feed each of the 10 tokens to the network in the initial state, and extract the next token (or 10 next tokens) from that one, than revert back and feed another single one of the 10 tokens, etc.
it annoys me how eager people are to hurl the word stochastic as pejorative. Statistics are a great tool for gleaning information from stochastic processes; statistics don't contribute randomness. Random sampling is necessary in order not to bias a sample, it's not used to contribute randomness to the sample but to preserve/measure the underlying distribution. (not meant to imply that training is random sampling)
It's a pejorative only because determinism is what makes computers useful in the first place. You get a consistent result, every single time, unlike if you have a human in the loop. Because LLMs are stochastic, they have removed the thing that makes computers useful to us, thus it's a pejorative.
It turns out that determinism isn't what makes computers useful in the first place.
1. Determinism is a very small subset of what makes computers useful. Non-determinism like stochasticity is literally everywhere, like random seeds.
2. LLMs are detemrinistic. They have a parameter to tune how stochastic they are.
The term is not very useful since most humans are stochastic parrots... At least most of the time.
Not suggesting that I don't say stuff on autopilot sometimes but for many people, it's their only mode of operation. They never actually think about anything from first principles. Their whole approach to language is just chaining catchphrases together. It's how a toddler thinks; it seems like many people never moved past that stage of development.
It sometimes feels same as with the models, especially in corporate:
- Lots of Haiku around, many mistakes unless process is very clear - Some Sonnets, still do mistakes but can adapt - Some Opus, able to improvise and think outside the box.
But even the Human Opus/Mythos are hilariously wrong sometimes.
i think it actually makes sense, an LLM just imitates human communication, which happens to be useful from time to time.
Conversely, that the most prominent proponents of LLMs call them artificial intelligence and then treat them like slaves they're free to abuse ought to be horrifying.
Nothing in that term implies sentience.
> most humans are stochastic parrots
There's a lot more happening behind the scenes when a human repeats phrases than what's happening in an LLM.
Sociological phenomenon. The desire to be liked, successful, or popular. The feeling that those phrases brings up.
LLMs are not experiencing any of that. As far as we know, neither is a parrot.
Parrots certainly do experience social needs
The "parrot" part of "stochastic parrot" is quite an ambiguous choice. Taken literally, it's referencing an animal that's actually quite intelligent and capable of complex, novel tasks but has no way to connect those to human language. How I've always read this though is the more literary meaning of "parrot" as "a thing that repeats words with no context". Perhaps "stochastic photocopier" would be a clearer metaphor.
Or perhaps tape recorder would reflect the linearity of the input/output strings
Humans are not stochastic parrots. You are 100% wrong about toddlers. This was clearly explained by St. Augustine 1500 years ago:
[https://faculty.georgetown.edu/jod/augustine/conf.pdf]Humans learn language opportunistically. Toddlers start with a powerful "superchimpanzee" understanding of the real world, and use that to learn words in order to satisfy their needs and desires. Statistical frequency is incidental to what words a toddler learns: what matters is the real-world context. Also note how important it is that infants instinctively understand nonverbal communication.
The most depressing thing about the 2020s AI summer is watching ignorant tech workers use the success of LLMs to launder their own ignorant misanthropy. Your views are many many centuries out of date.
> [St.Augustine quote] I remember this
I’m absolutely positive St. Augustine wasn’t some prodigious person who was able to self-introspect his early language learning and break past infant amnesia. If we’d have any means on verifying, my bet would be on that he just made this all up, post-hoc rationalizing all of that.
He can remember some sensory and pre-language mental states (desire to achieve some other state, like change of body position, or experience a taste, or a different view), but all that “whimpering and grunts and various gestures of my limbs” rationalized as purposeful communication is pure fiction. At least to my understanding, of course.
> Also note how important it is that infants instinctively understand nonverbal communication.
What does “understand” here means, and what infants exactly?
From what we know about human development, before somewhere between 2 and 9 months of age infants are unable to purposefully communicate in principle. No recipient in mind: they don’t track gaze, they don’t point at objects, they don’t share affect. Purely reflexively they can cry when they hear others crying (just like how we yawn with others), but that’s about it - just a resonance, not communication.
Obviously, comparing two different machineries suggests us they’re different (duh!) Humans aren’t LLMs, and language acquisition happens significantly different between two. But stochastic parrot is not an animal, it’s a basic principle, and that principle may possibly hold true to all life with associative learning capabilities.
FWIW nothing in this comment refutes any claims made in the comment it replies to. It's probably not the worst thing in the world for humans to start being a little more humble about themselves and their capabilities. Anthropocentrism has been a fucking disaster.
I really appreciate the effort you put in to this post. Posts like these are what makes HN great. Thank you.
This is a complete misunderstanding of how even idiots function in the real world. There is a lot of thinking that goes into living a human (or even animal) life that models are nowhere near ready to model yet. Even ignoring the physical interaction side, the way any human sets and achieves long term goals (such as getting and maintaining a job), interacting with the huge amount of systems present in day to day life, and learning new tools along the way for decades is far beyond the current abilities of these models - even if they handily beat 90-100% of humanity on some tasks normally considered much harder.
'Stochastic parrots' is a great term, but reading it now, it's quite apparent how bad this paper is.
> It argued that large language models (LLMs) generate text by statistically predicting likely sequences of words rather than understanding what they are saying—a process the authors captured with the metaphor of a “stochastic parrot,” a system that repeats patterns without comprehension.
I don't understand what we're setting the record straight on. This is the core point of dispute, and the author just blazes past it to focus on other things. I'm glad to hear "stochastic parrot" isn't intended as an insult, and I agree that it's not right to think of LLMs as a box with a little homunculus inside replying to you. But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
I think it’s pretty clear that they are repeating without “comprehension” - both mechanistically (as in there is no facility for comprehension in their formulation) and in the ways they fail. The standard rs in strawberry, should I walk or drive to the car wash, etc all play on the fact that they don’t have any real world model or thoughts against which they can judge their output, as do many of the jailbreaks which basically play on the fact that the model has memorized patterns.
There are people who argue semantics, that we can call the pattern matching that LLMs do “understanding”, or the moronic “how do we know that’s isn’t all we do” but for the normal use of comprehension, LLMs at a fundamental level don’t.
> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
So this seems obvious to you, and yet to many others, it is equally obvious that LLMs can/could do the things they routinely do without any meaningful sense of "understanding".
I think it's a mistake to disentangle their abilities from understanding. Just swallow the pill that they have some form of understanding, even if it slightly differs from ours. I really don't see the problem.
I prefer to work the other way around. That is, accept that a lot of human speech (and text) is generated via similar mechanisms to the ones that drive LLMs, but note that there is another kind of behavior - reasoning - which seems to be distinct.
> But to me it seems obvious that LLMs are not repeating patterns without comprehension and do understand what they are saying; otherwise they would not be capable of doing things they routinely do.
Is it possible you're making the following error described in the article?
> The fact that these systems are designed to mimic the way we use language makes it very easy for people to mistake them for other people.
Clearly you don't believe it's actually a person ("it's not right to think of LLMs as a box with a little homunculus inside replying to you"), but you do believe it's doing something a little bit magical. Is it possible because the interface is linguistic, and every other thing in your world that communicates with language is intelligent, that you're projecting something that just isn't there onto the situation?
I'm sorry if this line of questioning is a little invasive. But this is literally the "danger" the original paper talks about, and it seems an awful lot like you've fallen for it.
I'm not offended by the line of questioning! But I don't really follow it. I don't and IIUC Bender doesn't use "understanding" to refer to any kind of magical property. Understanding is the capability of using words as consistent handles to things in the exterior world which the language is describing. And this is something LLMs can clearly do. I just went to ChatGPT and asked this question, which is almost surely not in its training data:
> What would happen if I walked to the top of a skyscraper with a soda can full of Maraschino cherries and let them go?
And its answer (https://chatgpt.com/s/t_6a4bd9ffa5708191901bb6d43c89f43b) clearly demonstrates understanding. It knew that this is a dangerous thing I should not do in real life, and that my question is ambiguous about whether I intend to drop the can, and that this might be intended as a physics problem rather than a real life scenario.
> And this is something LLMs can clearly do. ... > It knew that this is a dangerous thing I should not do in real life
From the ChatGPT response you linked, all I see for sure is some matches on the following patterns:
Then there are some sentences of likely characters following those patterns. You don't need anything more than a basic cartoon-level understanding of how an LLM works to explain this output. I see no evidence of reasoning or understanding here, or any theory of "real life".It also does an incredibly poor job of answering your question. It makes no attempt to explain what might actually happen. If it has been trained on the entire corpus of medical science, and it is indeed intelligent, then surely it can reference ballistics studies and give you a very detailed and thorough theory of what--exactly--injuries you might expect from a 12oz can being dropped from the height of a skyscraper. Calculating the terminal velocity and therefore the momentum of the can is trivial. Characterizing the physics of the impact on various parts of a human body is trivial. If it actually understood your request why didn't it just answer the question?
It's a rhetorical question. LLMs do not "understand". It is completely outside their capability. "Understanding" is something we impose upon their output (to loosely quote TFA). [edit] I think the most powerful evidence for a lack of any understanding whatsoever is all the stuff about the cherries being in or out of the can. Yes, cans contain things. That is not a profound observation, nor is it at all relevant to the question. If you drop an empty can off a skyscraper nothing meaningful will happen. And, no, probably dumping all the cherries out won't hurt anyone or cause a slipping hazard... It's also not particularly relevant to point out that dropping things off skyscrapers is bad behavior. But that's more forgivable from a CYA standpoint.
I believe you are projecting something that is not there onto a completely mindless stochastic process.
But it shouldn't even be contentious like that. It's not a fundamental mystery how these things work. It is for the most part not a valid target for the kind of speculation you seem to want to do about it.
It's not like you can be agnostic, or measured about this. It's like someone explaining a car to you, saying, "look here is where you put the fuel, here is where it ignites, where the axels are turned..." And you, trying to be measured, are like "hm well yes of course that all is clearly important, but there is clearly just a bit of magic here somewhere, between all the different 'parts'."
The "magic here somewhere" in the car is in the design that reference aspects of animal anatomy (facial features, stance) and in the millions of dollars of advertising that prime the public with expectations about how they'll feel driving it, or how to see other people in the car. There's a direct connection there to packaging LLMs as chatbots, it gives them a recognizable shape and behavior that a lot of people interpret as consciousness and personality.
This is a facile point. Lisp expert systems transparently don't understand the meaning of any symbols they process, yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability. The fact that LLMs are less transparent than Lisp expert systems (and easier to program) is extremely bad evidence that they understand language. Especially given that AFAICT Opus does not properly understand concepts like "four."
> yet with enough developer elbow grease they can do all the same things an LLM can do, with much higher reliability
Where can I access such a Lisp expert system?
If I cannot because they don't exist: then they cannot do the same things an LLM can do. And of course one can assert anything and everything about what a non-existing thing could do.
For context, here's the main quote:
> Contrary to how it may seem when we observe its output, an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.
I think this metaphor is so strained as to not be useful. I think key here is that the authors say "without any reference to meaning", which is a heavily loaded term, that does definitely apply to parrots, but does not apply when you apply it to immense bodies of text.
Namely that language embeds meaning in language. A sentence being written by a human (as a starting point) is designed to have consistent meaning. While it is possible to write syntactically correct meaningless text, that is not what most of human language has done; the meaning cannot be removed from the text.
This I think is clarifying, from the same paragraph in the text:
> ... the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.
That's just facially incorrect. The training data is entirely about sharing thoughts with a listener. Else why is the text being written?
I don't accept that it applies to parrots. Certainly not to Congo African Grey parrots.
What I have been doing in many places—the octopus thought experiment, stochastic parrots, the phrase “synthetic text-extruding machines”—it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do
> Meanwhile, O, a hyper-intelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially, but is very good at detecting statistical patterns. Over time, O learns to predict with great accuracy how B will respond to each of A’s utterances. O also observes that certain words tend to occur in similar contexts, and perhaps learns to generalize across lexical patterns by hypothesizing that they can be used somewhat interchangeably. Nonetheless, Ohas never observed these objects, and thus would not be able to pick out the referent of a word when presented with a set of (physical) alternatives.
This seems kind of obviously wrong at least in the context of coding agents. These models get trained on actual output of the previous version of the model doing its job, often "IRL" on a real computer/project. It's like O is in the conversation for years now and learning from his own interactions between A <-> O <-> B, where A is the human and B is the computer.
The idea O ontologically has never "observed" "these objects" or referents is philosophically strained. Have I observed the moon, or a finger pointing at the moon? Have I observed `sed` more than Fable?
I think this is the most measured take I've seen from Bender, and I think it summarizes her only compelling point well (technologies should be referred to specifically rather than generally as AI, and that referring to everything as AI is not useful and helps hype the technology in a way that benefits those selling it).
In her previous interviews, I've found her assertion that LLMs aren't useful and will never be good at anything totally uncompelling. Also laughed at this quote as she's been pretty harsh IMO on "the people who like the systems".
> it’s all about trying to make vivid to people who aren’t in the business of building language technology what these systems actually do, which is not the same thing as insulting the systems or insulting the people who like the systems.
After having used LLMs for some time now, I don't agree with the concept they are just token generators, unless you think that's all humans are too. The way we test in most schools is just picking the right token. We also give them unique problems that they never saw in their training, which is the nature of programming. I realize they are probabilistic token generator models, but I find it harder and harder to accept that somehow there isn't something more going on. I'm not sure whether they are intelligent or not, but for the most part token generation is how you get degrees too. The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.
They are just token generators. It is just that 'just' does a lot of lifting!
Here's the thing: most things people do does not involve tokens of any kind. It is, in fact, stuff that not easily describable. For example, it's trivial for a person to walk, but they cannot verbally describe what muscles they're activating in what order to make that happen.
Cognitive skills such as tool use and complex navigation predate language as well. That means there's a core of reasoning in humans that doesn't depend on "tokens" or "language" of any kind. Language is a tool for communication and forming complex human societies, but it's not cognition.
> The thing is a parrot just says things it has already heard, it doesn't perform complex reasoning on novel situations and then explain it succinctly.
Well a parrot does perform complex reasoning on novel situations all the time. It just doesn't have the wiring to connect that to "tokenized" human language. I suspect LLMs have the opposite problem, where they exist in the domain of their "tokens" and have no way to connect these to truly novel situations that have no existing words to describe them.