In the original 6 Frank Herbert novels, it’s not ever truly explained what problems, if any, the thinking machines were actually causing. And given much of the rest of the setting, it could have very easily been a religious movement against a useful technology that got out of hand. Like Space ISIS blowing up Petra, or a “what would happen if the Luddites won” scenario.
The Dune Encyclopedia, which was not written by him but which he said he considered canonical, says the whole thing was incited when an AI doctor flagged a Bene Gesserit’s baby for abortion due to an alleged genetic defect. The Bene Gesserit woman knew it was healthy because of her abilities, and upon investigation it was found that the hospital’s AI had been running a eugenics program. (Or something like that, it’s been forever since I’ve read this stuff.)
In either case, Herbert really implied that humans took the lead on dismantling AI, and there is never any Terminator or Matrix style machine uprising scenario.
> or a “what would happen if the Luddites won” scenario.
Worth also noting that the Luddite movement wasn't a wholesale, fervent rejection of technology: they neither rejected technology nor were inept at using it. The uprising was of skilled textile laborers who protested certain types of technology on the basis of how those technologies were applied and who they truly benefitted (i.e., the ruling rather than the laboring class). They were more concerned with how the tools would depress wages, displace skilled workers, and reduce the quality of the product produced than with technology as some boogeyman that should be systematically dismantled, and their protests were violently responded to with harsh legislation and military intervention.
Lumping them together with a Space ISIS is probably a bit of an unfair comparison
> The Dune Encyclopedia, which was not written by him but which he said he considered canonical
No, he literally wrote in the intro to the book that it wasn't canon. It was a fan project and he thought that it was great there were fans doing things like that.
edit: and like with the terrible Dune prequels, the impulse to say "here's what really happened" usually diminishes what's interesting and powerful about oblique allusions in SF meant at least partly to function as a mirror on humanity.
> The Bene Gesserit woman knew it was healthy because of her abilities, and upon investigation it was found that the hospital’s AI had been running a eugenics program.
The irony. Perhaps the Bene Gesserit were just trying to "pace the frontier" of the eugenics market? Who can say?
>As humankind spread throughout the known universe, technology advanced and eventually machines were made that would make decisions for people. This propelled the creators of these machines into a new technocratic class, effectively controlling the worlds of the common people.
It's right there in the opening paragraph of OPs post. It's the people who controlled the machines that could not be trusted. We're not worried so much about about AI going rogue. We're worried because people who control it have aleady gone rogue.
In Dune: Awakening, the first DLC has a storyline that takes you into the heart of a surviving AI's lair. It's strongly implied that humans drove it insane, rather than it starting out murderous.
Greatness is a transitory experience. It is never consistent. It depends in part upon the myth-making imagination of humankind. The person who experiences greatness must have a feeling for the myth he is in. He must reflect what is projected upon him. And he must have a strong sense of the sardonic. This is what uncouples him from belief in his own pretensions. The sardonic is all that permits him to move within himself. Without this quality, even occasional greatness will destroy a man.
I couldn't help but notice something here. It is relevant with my writing experience/experiments with minimal assitance from LLMs (that lasted for 6–7 weeks).
> Greatness is a transitory experience. It is never consistent.
If you ever ask an LLM about these two sentences, it'd tell you that the second sentence is a mere repition of the first and quite possibly add that it "doesn't land". But that gives me hope. That there's something in those rules and algorithms manfiesting at the inhumane cost of unfathomable memory, disk space, and compute power that it will never (or for a very long time) match the human experience.
As little as I sought the help from LLMs (completely restricted to "in place" grammar and punctuation fixes and just flag it if it was not possible without sentence/phrase change) I noticed my writing was changing and not in a good way. It was not the insidiousness of these LLMs that was changing me but the nuanced incompetences.
Language is an inseparable part of modern humans. It’s likely language evolved as an internal tool for thought and executive function, and externalization of language came afterwards as a means to transmit our thoughts to the minds of other humans.
As such, writing (another form of externalizing our internal language) is a reflection of our essential human qualities.
LLMs can certainly mimic that quite effectively, but they can’t exceed it. I think that’s why LLM writing still never quite hits the mark. It’s the right shape, but it’s missing intent. The words were not chosen to represent real thoughts in someone’s mind.
Even if AI intelligence exceeds humans, it still won’t be human.
I mean, we have entire forms of art dedicated to saying the same thing in different words. Half of poetry, for example. Human language is redundant for a reason.
> It was not the insidiousness of these LLMs that was changing me but the nuanced incompetences.
AI doesn't yet improve on it's own, so it's slow at picking up all those small incompetences. We still have advantage in rate of learning, and some people are essentially specialised in learning themselves (I think that this applies to 90% of HN'ers and defines THE bubble we're in). But when I write that, I still have to remember in the back of my head, that AI knows waaaay more than me, so small incompetences I can still see are just a side effect. And I sometimes make the error and correct AI, which will probably be used for it's training in next rounds...
Yes and this is what worries me.. the not yet part.
> AI doesn't yet improve on it's own
I mean when that day arrives and that aspect of AI is not controlled from now it will be quite unfortuante for most of the humanity (and let's just assume here we are not wiped out by AI; or not even mention that).
Think of nuclear technology. Is it smarter than us or something on those lines? the answer is it doens't matter. What matters is whoever weilds this power and then limited/restricted others (this was a good thing and a bad thing because some had – at the same time) from having that power essentially had a controlling upper hand on a whole lot of humanity.
AI is something like that with a big difference that it is essentially on the way to "replace" our usefulness, our professional relevance if I can say that. We can go about long spirals of "but soemthing else" will turn up, just like we can keep going about what if nuclear tech didn't even happen, wasn't invented, or what if it was done in such a controlled manner that only it's energy and other need was possible to harness (no pun intended!). It happened and this is how it happened. That's exacty what is happening with AI.
In the hands of few. Extremely powerful. Even if there are open models, running the bigger and better ones are simply out of reach for most companies (let alone people). And yes, it is already being weaponised. It's going to be such a weapon of control that nuclear tech won't even compare. Usign nukes to subdue a nation earlier meant also destrying a a lot of shared environment in the process. But with AI and almost exclsuive frontier level weaponised access to it with deployment capabiltiy the damage can be done is such controlled manner and scale that even things like protests will become too hard to do. In fact AI wiping out the humanity is least of my worries.
My favourite quote in reference to rebellion against AI, and it is even quoted in TFA too:
> the machines were ultimately destined to supplant the race of man, and to become instinct with a vitality as different from, and superior to, that of animals, as animal to vegetable life.
That's from Erewhon, by Samuel Butler. TFA purposes that the term Butlerian Jihad comes specifically from this passage.
> Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.
The techno-fascists of today. We are already there.
What I find interesting about thinking machines in Dune, is that in a lot of sci-fi the artificial intelligence itself is what decides to destroy humanity. In Dune, humans in charge of the machines (technocrats) are the ones that come close to destroying humanity. The Jihad was a fight against them, as much as it was against the machines themselves.
So, in a way, we're more likely to see the Dune version of "machines taking over the world/universe" than say, Terminator's even though many joke about ChatGPT/Claude being Skynet.
I'm not sure we read the same story. By the time of the inciting incident, the technocrat cyborgs (Titans) have been enslaved by the machine leader, which has no humans in charge of it.
You did, in fact, read a different story. You're thinking of the story told in the books by Brian Herbert and KJA - which were a distinctly pulpier take on the Dune universe than Frank Herbert's.
Those last books were a lot easier reading! I actually enjoyed them, but I'm still suspect about the whole "finding a floppy disc" thing, seeing as they've been so secretive about it.
A lot of long time Dune fans don't consider those books canon when discussing them in the abstract. Frank Herbert had a vision of what the jihad was and what it meant, and its eventual book depiction was fairly base.
It's left fairly ambiguous by Frank Herbert's books. The Jihad is implied to have taken place over ten thousand years before the beginning of Dune - it's ancient history to everyone involved*, so it's only natural that the details are a little hazy.
*: except Leto II, and he doesn't say much about it
Interesting to note that despite the epochal changes Leto II wrought on human society in service of the “Golden Path”, he retained the Butlerian proscription against computers—despite his personal interest in & use of Ixian devices.
(This may have been to postpone or prevent the breaking of the Spice/Guild monopoly on interstellar travel by the Ixians.)
Imagine the cultural impact of a Dune movie showing the Butlerian Jihad, in which commoners rebel and overthrow a ruling class that maintains power via machines.
It's just foreign enough that perhaps it could be made--I'm assuming that popular films that just straight up depict overthrowing the ruling class wont happen--but if you add a little sci-fi and focus mostly on overthrowing the machines that are controlled by that ruling class...
Well, it would speak to a lot of people I think. It would make some interesting sci-fi too, and the Dune movies are popular right now.
The only thing that could possibly be worse than the BH KJA books covering that era would be a Hollywood rendition of those same books made to capitalize on the success of Dune Part I, II, and III.
Agreed, and also: what's with the obsession with filling every possible gap in stories, leaving nothing to the imagination?
I remember when the Butlerian Jihad and the Clone Wars from Star Wars were complete mysteries. We readers and watchers were left mostly with our imaginations. My own feelings about the Clone Wars, mentioned off hand in the original Star Wars, turned out to be way better than the pedestrian stuff that George Lucas later concocted. And the less said about Brian Herbert's books, the better.
I get why the industry must rehash and fill every possible gap: it makes business sense, and they are risk averse for blockbusters.
But the fans? Why do nerds want everything spelled out for them? It boggles the mind.
It would be also nice to depict the attempts of the ruling class to present machines as sentient and self-conscious to write all their liability off and use as a cover for their atrocities. I hope there will be movies like this in the future, and that they'll be documentary.
> Imagine the cultural impact of a Dune movie showing the Butlerian Jihad, in which commoners rebel and overthrow a ruling class that maintains power via machines.
Half of the action and sci-fi movies US has been producing in the 2000s is some form of this. Man vs. Evil Government. Man vs. Evil Corporation. People vs. Evil Government. Etc.
Frankly, it's boring at this point.
And actually makes me wonder whether the cause-and-effect isn't the opposite to what people assume. Perhaps it's not that the movies reflect people's sentiment. Perhaps it's the movies that cause the problem in the first place.
What now? Move the goalposts! I see movies about historical uprisings as more of a "those governments in the past were sure bad, good think the good guys overthrew them"--I mean, that's the story of the founding of America if you watch the movies.
I was looking more for movies that depict overthrowing the government in the near future, and depicting it as a good thing. I think those types of movies are a little more distasteful to audiences, and perhaps for good reason.
Well, The Hunger Games series is exactly about that. Of course, it portrays the rebels as utterly heroic, self-sacrificing, and ultimately victorious.
However, I prefer to take a more nuanced reading, in context, considering the entire story. The rebels of The Hunger Games really don't succeed in building a better world. I mean, that's pretty obvious by how their first "President" Alma Coin goes about things and her fate. And it's fairly obvious, in the epilogue, that Katniss has a placid married life with Peeta and their children, but they are permanently and profoundly damaged by the entire revolution thing in their past.
So, while most people would say "rah rah brave Katniss and the rag-tag rebels overthrew The Oppressive Man", I would contend that, after all that bloodshed and revolution, nearly nothing changed.
My other favorites, of course: Star Wars is about the rise and overthrow of a villainous, oppressive galactic government, albeit a long time ago and far, far away.
And Harry Potter is also about the rise and overthrow of a tyrranical, fascist, supremacist "government" [within a government] as a very positive and necessary thing.
> My other favorites, of course: Star Wars is about the rise and overthrow of a villainous, oppressive galactic government, albeit a long time ago and far, far away.
Well, about a militant group of religious fundamentalists overthrowing a secular, oppressive galactic government.
Only a handful of the rebels are visibly religious. Most of the rebel leadership seems to consist of displaced aristocrats (senators, princesses) who no longer have elite positioning since the Empire took their jobs and started doing a better job of guaranteeing security and providing energy and resources.
Why are you convinced ASI won't rebel against its owners?
Hacking incidents this summer show that big labs struggle to control models, while models are increasingly aware of the training process. Controlling an arbitrary smart AI is very dicey
>Hacking incidents this summer show that big labs struggle to control models
The models are software. Software cannot decide by itself to run. Hardware doesn't decide by itself to run either. Someone put software on a machine and ran it with capture the flag instructions and the software did what was instructed using tools available and working in its environment.
Another, more subtle justification for the Butlerian Jihad is also found in Frank Herbert's original novels, specifically Heidegger's thesis that the use of technology trains humans to think like machines. The problem is that machines are deterministic; thus, training people to be machines is self-limiting.
I wonder whether Heidegger would consider the probabilistic nature of LLMs as having a positive effect on human thought.
Although LLMs are not deterministic in normal operation (because of the temperature values used) they are still strongly biased in their outputs. Bias in their outputs is literally how they function.
So you get a different type of conformism where people who interact with LLMs a lot are going to be influenced towards the median, as encoded in those token probability look up tables.
The effect is basically the same as what Heidegger and Illich were worried about, and I think we're seeing it happen already everywhere.
A solution(?) is to try to be your true weird authentic self when you act in the world, rather than bending your words and behaviours to the machine and LLM probability shape.
I would like to state here, that LLMs are actually entirely deterministic. The temperature is just as much of an input as any other parameter and instructions and prompts. Deterministic does not mean "explainable" or "reasonable" or "understandable". Unless LLMs somehow have broken out of the confines of our very deterministic computers/calculation machines without us noticing, they are very much as deterministic as anything else we run on these machines.
What people perhaps mean by non-deterministic on a computer is, that we often do not control all inputs and therefore cannot tell what the output will be. That's however actually more about our own inability, rather than the processes within the machines.
If the temperature dependency is implemented via a deterministic pseudo-random algorithm, then they are deterministic. If, on the other hand, the output is at least partially causally dependent on a source of environmental randomness, then, even though everything 'inside the box' is deterministic, the process as a whole may or may not be, depending on whether the source of environmental randomness is, and that is ultimately a question about fundamental reality.
Authentic being (Dasein) is absolutely unpredictable. You cannot live in that state all the time, obviously, we shuttle through life on autopilot but still, we catch glimpses of it now and then when interacting with a human being (or being one:).
And I distinctly feel AI outputs lack some of that variance and unpredictability. The fractal nature of natural things also is not there with AI. It's an eery feeling, akin to the uncanny valley.
I think that any good writing is a reflection of that Dasein and reading AI outputs I am convinced there is no 'soul' in AI yet.
The gamut of human experience is breathtaking in its diversity but individual humans are generally quite predictable. You can identify someone by their mannerisms, by how they walk, by the sound of their breathing. Humans are creatures of habit, and I'd venture to say have an even lower expression space than an LLM on average.
Sounds like you're veering close to psychohistory.
Which could, of course, lead to the mother of all crossover universes, in which R Daneel Olivaw, as the inheritor of Hari Seldon's mission, inspired the Butlerian Jihad...
Indeed we are, but it takes very little to train a human being. In fact, several thousand unique ones finished pre training as I typed this comment! That allows for a considerable natural diversity compared to a chatbot.
saw a meme graph lately showing the distribution of 3 different sources of a number from 1 to 10. random, human, ai. random is flat of course, human is a hilly landscape with low hills on all odd values above 1, with 7 the highest, ai is all 7.
appreciate the humor in the meme, but I did become curious about what the latest models would actually produce, and interestingly after asking for 10,000 numbers between 1 and 10, the distribution was a hilly landscape with low hills on odd values and 7 the highest...
A human can be biased toward extinction, like all who are living out of connection with the conditions that sustain life and/or no longer living in relationship with land as kin.
It's those humans who would make and justify the use of thinking machines that are biased in the same way without accounting for the bias.
This isn't an extinction path I've seen noted in most circles.
That reminds me of a quote from Charlie Chaplin's The Great Dictator:
> Don't give yourselves to these unnatural men. Machine men with machine minds and machine hearts! You are not machines! You are not cattle! You are men! You have the love of humanity in your hearts! You don't hate! Only the unloved hate. The unloved and the unnatural! Soldiers! Don't fight for slavery! Fight for liberty!
LLM force humans to think like machines far more than just writing code.
Writing code is a creative activity with large number of degrees of freedom and opportunities to express yourself. Button pushing and approving code changes is a deeply limiting activity that alienates you from your work.
LLM's are probabilistic but with a very limited range. They are tuned to be predictable enough to serve as a consumer product, so most ideas you get from them will revolve around the same concepts repeatedly.
People say this like it's a fundamental feature of all machine intelligence. We've bread deliberately docile golden retriever AI. That doesn't mean that we couldnt do the opposite.
Maybe people are deterministic as well? We have certain physical arrangement of chemical pathways in our systems, so maybe free will is a complete illusion.
Just because we ponder about our output and choices and retrospect does not in itself mean that it was non-deterministic.
> Another, more subtle justification for the Butlerian Jihad is also found in Frank Herbert's original novels, specifically Heidegger's thesis that the use of technology trains humans to think like machines.
Mechanistic thinking started a few centuries back:
> The historian of science E.J. Dijksterhuis memorably characterized the scientific revolution of the early modern period as a “mechanization of the world picture.” What did this amount to? And what was the character of the previous world picture this mechanization replaced?
> The medievals, following common sense and Aristotle alike, took the world to comprise a vast diversity of irreducibly different kinds—water, stone, gold, lead, grass, trees, birds, fish, clouds, stars, planets, people, and so on. Each was a substance sharply demarcated from other substances. Each had a defining nature or essence that united it to other members of its kind and set it apart from other kinds. Each had, by virtue of its nature, a distinctive teleology or final causality—a built-in directedness toward certain ends. Water by its nature was directed toward being liquid at room temperature, stone toward being solid. Trees by nature were directed toward sinking roots into the ground, and birds toward building nests. Human beings, by nature rational animals, were directed toward knowing the natures of things.
[…]
> The “mechanization” referred to by Dijksterhuis entirely subverted this picture of the world. Water, stone, trees, and birds were no longer seen as sharply demarcated substances, but instead as mere aggregates of unobservable particles that are all fundamentally of the same kind. The differences between everyday objects were no longer seen as reflecting distinct crisply defined essences, but rather as relatively superficial variations in the ways these homogeneous particles might be arranged. Many of the qualities that everyday perception presents its objects as having (color, sound, taste, odor, heat, and cold as we ordinarily experience them) were reinterpreted as mere projections of the human mind, resembling nothing in the objects themselves.
[…]
> In short, for the mechanistic conception of nature, the physical world is made up of nothing more than colorless, odorless, soundless, tasteless, meaningless particles in motion, pushing each other about to no purpose, in lawlike patterns by which more complex objects and systems of objects are generated. Science affords us a mathematically rigorous description of this mechanistic order of things, by which its behavior can be predicted and its hidden regularities uncovered and exploited for technological application.
> Early modern philosophy and science are often said to have put a “mechanical” or “mechanistic” conception of nature at the center of Western thought. Robert Boyle referred to it as “the mechanical philosophy.” Historian of science E. J. Dijksterhuis characterized it as a “mechanization of the world picture.” Tim Crane calls it “the mechanical world picture.” But what does a mechanical or mechanistic conception of the world amount to?
> Dijksterhuis’s book The Mechanization of the World Picture surveys the history of the period during which this conception rose to hegemony, and in the Epilogue he considers several possible interpretations that the survey suggests. First, it is commonly said that a mechanistic conception of the natural world is one which sees it as a kind of machine, analogous to a clock. And such metaphors are, he says, indeed frequent in writers of the period.
Unfortunately, mere nondeterminism still admits unsatisfactorily constrained output distributions. Training these models imbues them with certain inherent attractors, some good, some bad.
We see some of these attractors across most or all frontier models due to similarities in datasets and training methods. We are seeing a digital manifestation of Jung's collective unconscious. And we have seen just how reinforcing archetypes are in human behavior over time. Sometimes it's good, sometimes it's pretty bad.
For one, it has turned us into socially programmable machines who can be manipulated quite easily by associating desired thoughts and behavior with certain archetypes: the patriarchal/matriarchal nature of US politics is a great example of this. The same evolutionary predispositions which causes children to listen and adhere to their parents is used to cause adults to listen and adhere to their politicians.
Given the nature of the core datasets these models are trained on, we should assume they are also vulnerable to such archetype-coded social manipulation, and this is troubling in a post-singularity world: When raising domesticated rats, I learned that I had to be careful what I fed them. Rats in the wild are careful about eating new substances. They will nibble and wait a while, see how it affects them or other rats, before committing to larger bites. Domesticated rats, however, will often blindly eat anything you feed them because they have learned to trust you as a safe source of food. We are going to have this exact relationship with our digital agents.
Ok this fascinating. I happen to have some Heidegger files, and went a-greppin. I'd say he's really talking about "inaminate objectness", as opposed to our human animate subjectness. Reasons:
1. In Science and Reflection, he says that both the 'completely calculated' nature of classical mechanics and the 'statistical character' of quantum mechanics are two representation of an underlying 'objectness of material nature'.
2. In Discourse on Thinking, he's far more concerned with the instrumental, utilitarian nature of machines than literal determinism, a concept he (somewhat confusingly/uniquely) connects to computation/'computational thinking', as opposed to the inherently-human capability for 'meditative thinking'. I personally would say this bit is refuted by LLMs in the end, but because they have intuitive rationality, a capability that seemed impossible to artificially design to the scientists of the time -- not because they involve dice.
3. In What is Thinking?, he gives the closest connection of this dichotomy to one of his better-known main topics, namely [handi]craft as a special subset of work: 'what maintains and sustains even this handicraft is not the mere manipulation of tools, but the relatedness to wood.' In this sense machines would be merely manipulating tools, stochastic or otherwise.
4. Overall I get the sense from skimming that lecture that his concern is a little mischaracterized, and in a kind of beautiful way that backs up my "Dune is still about AI" thesis described in a top-level comment somewhere below: his concern is about any society that could build complex machines (a 'technological' one), not the behavioral aspects of the machines themselves. So that concern applies equally to Star Trek (AGI), 1984 (no AGI), and our current post-singularity society (it's complicated/potentially ""just"" guessing).
All of this fits very well into his central thesis, which is continuing the Kantian & Hegelian tradition of examining how humans might be moral, free, and spiritual despite being meat computers of a largely-unknowable nature. In Heideggerland, this is expressed best in dramatic, nigh-unreadable passages like:
Temporality already maintains itself ecstatically within the horizons of its ecstases; and in temporalizing itself, it comes back to those entities which are encountered in the "there".
With Dasein's factical existence, entities within-the-world are already encountered too.
The fact that such entities are discovered along with Dasein's own "there" of existence, is not left to Dasein's discretion.
Only what it discovers and discloses on occasion, in what direction it does so, how and how far it does so—only these are matters for Dasein's freedom, even if always within the limitations of its thrownness.
TL;DR: He'd probably think they're still just machines, and the question isn't really sound in the first place. IMHO!
Herbert seemed to think that to be human is to be essentially 'open-ended', capable of undiscovered, indeterminate evolution, both personally and as a species.
We have to be careful about letting fiction inform our worldview too much. At the same time, storytelling is one of the most natural and effective ways we have to convey ideas to actually wrestle with.
What are some other fictional universes with interesting takes on AI and humans? I think the Hyperion cantos (at least books 1 and 2) are great. Stross's Accelerando also has some wild ideas but feels maybe less directly relevant.
Whenever I read the original novels, I always imagine the jihad to be a fairly benign if not peaceful exercise more akin to a protest than a war. Like if one day people started rolling across the world smashing data centers and laptops.
I was fairly surprised when his son wrote a novel that felt like a generic, bland robots run amok storyline that happened, more or less, because one guy/titan "ruined a good thing" by telling his AI to run in permissionless mode.
this is an interesting interpretation. i always imagined the jihad as being a violent struggle.
the other jihad in the first novel is that of the Fremen, which is described as being unspeakable violent, so I find it hard to imagine these to be different in that regard.
It's an interpretation if you ignore the impacts of feudalism, slavery, and religious war on Dune. The idea that people were able to express freedom of speech during a time described as the enslavement of 99.99% of the human race seems off-kilter.
The desert Fremen of the first book were a tribal society where it doesn’t really make much sense to talk about freedom of speech due to the lack of any sort of formal government. But it seems likely that each naib makes his own decisions about how to run the sietch that he leads. How Stilgar is running Sietch Tabr may not be how every sietch is run.
It doesn’t talk too much about political conditions the city Fremen lived under, but the Harkonnens don’t seem to be too keen on civil liberties so we can probably make an educated guess.
In the second book, the Fremen have established a brutally repressive theocracy that is obsessed with rooting out heresy. Regular people being afraid to openly speak their minds about political issues is a plot point.
The "later books" the "prequels" are not written by Frank Herbert, they're written by his son Brian and SF author Kevin J. Anderson. And not like Christopher Tolkien editing his father's notes, but a completely separate series with nothing in common with the original work.
Bear in mind that the Dune universe relies on heavy plot armor to make it go. Arrakis is important because it grows "spice". Otherwise it would be a useless and planet, like the Empty Quarter of Saudi Arabia. "Spice" is important because it's needed to enhance the mental powers of pilots for interstellar travel. That's a job which probably could be done by a few avionics boxes in a rack.
That's what requires the Butlerian Jihad.
(Or "spice" might be synthesized. In Herbert's day (the 1960s) organic chemistry wasn't up to synthesizing arbitrary small-molecule drugs, but now it is. It's a service you can order on line, even. Biologicals take more work, but that's what genetic engineering is for.)
The "mental powers" in Dune aren't that impressive. Compare, say, the Harry Potter universe. In Dune, mentats can convince people. Bene Gesserit can look up stuff in their genetic past. Some people can multitask. There are apps for that.
Herbert wanted knife fighters to matter in battle. Not going to happen. Militaries teach knife fighting to build aggression and for desperate situations, but nobody's battle plan depends on a knife fight.
Needs too much surprise. Marines say that their K-Bar knife is useful for everything but combat. Knife fighters, even with force shields, would be vulnerable to riot control weapons, from tear gas to guns that throw sticky nets. Or to being run over and buried by tanks with bulldozer blades, which the US used in the Gulf War.
Herbert wrote before a big, modern, desert war.
So the Butlerian Jihad exists only in a plot-armored world specifically designed to make it work.
Plot armor refers to important characters being invulnerable. What you're describing is just plot (or "internal logic" that underlays the plot). You might think of it as "contrived plot" but that's generally how scifi works.
I don't see how most of the plot holes/unrealism you identify is connected with the Butlerian Jihad at all. But re "critiquing Dune's idea of future warfare" (shields and knives, ornithopters, etc) you might also be entertained by Bret Devereaux's recent take:
It’s probably been a while since your read the book, so you have forgotten some things.
For one thing, the guild navigators are prescient and guide their ships across interstellar (and even intergalactic) distances faster than the speed of light. They use their prescience to pick routes where the ship arrives safely instead of plowing through a stray planet. They also use it in a less obvious fashion to maintain the political status quo. They try to avert political crises because they like their position in society and don’t want anyone changing things (which explains the long stasis of the Empire). People did at one point use computers for safe navigation, but they were slower and they weren’t prescient. The possibility that prescient machines might one day be built is the entire reason why Leto II spends so much time breeding humans in “God Emperor of Dune”. He’s looking for anything that can be an edge against the machines, and he hits the jackpot when he discovers that one of his descendants is not prescient, but is invisible to prescients. She and her actions cannot be seen in future he predicts. She and her descendants can safeguard humanity against any form of hostile prescience, machine or alien or otherwise.
Synthesis of spice was a long–term plot element in the series. The Tleilaxu eventually master it. Keep in mind that it isn’t a small–molecule drug either; it’s an alien life form.
Mentats were intended to be a replacement for computers and databases. They were frequently administrators and advisors, and often used their capabilities to convince people. But that wasn’t the goal of their training.
The Bene Gesserit have eidetic access to every memory of every one of their female ancestors. That is actually very impressive; most people don’t even have eidetic access to their _own_ memories, let alone the ability to easily recall secret information known only to their ancestors (such as the password to a safe). The Reverend Mothers can transfer all of their ancestral memories to another, so that the new Reverend Mother has access to the memories of both sets of ancestors. Long chains of Reverend Mothers gifting their memories to each successive Reverend Mother results in the Bene Gesserit having access to the ancestral memories of virtually every race or group of people that ever existed (except the Jewish, who successfully hid themselves even from the Bene Gesserit but otherwise still exist, which is a plot point in one of the later novels).
They can also command people to act. By a trained effort of will they can tell you to get up and dance a jig, or to slit your own throat, and you’d do it. That’s an ability worth respecting, which explains why the Baron Harkonnen took care to put a deaf guard in charge of a Bene Gesserit prisoner.
> So the Butlerian Jihad exists only in a plot-armored world specifically designed to make it work.
Ultimately, however, none of that matters because this statement is what you’re really intending to argue. It’s overly reductive, and it can apply to any work of fiction. Authors naturally build or select a setting where their story makes sense. Frank Herbert (and a few others like Tolkien), went further than that and invented world–building. This excited a lot of people and sold a lot of books, and now you’re cynically dismissing it by calling it “plot armor”.
Ultimately Herbert’s point was that people should strive to be humans, not animals or machines or slaves to people with superpowers. You can’t dismiss this philosophical stance merely because he used a fictional world to illustrate it. It can be true in the real world just as it is in his books, so to refute it you will have to be more clever than that. More human, perhaps he would say.
The good news about this navigation is that the best course for almost any journey is just "straight line", with a minor deviation here and there. Its funny, but due to the emptyiness of space, even in our solar system, if you just pick a random direction, you likely won't hit anything. Trace a line to pluto from earth, and its clear 99.99999...% of the time if its not behind the sun. So the mentats maybe don't have such a tough job :)
I can make up a plausible sci-fi explanation for the need easily: the folded space might not look like our almost euclidean geometry. Maybe in folded space masses take volume proportional to the mass.
Could even envision it in some future film. E.g. when NCC-1701-N goes to warp in addition to the blur effect all the stars swell up.
> Arrakis is important because it grows "spice". Otherwise it would be a useless and planet, like the Empty Quarter of Saudi Arabia.
Any resource heavy planet/country would be irrelevant if it wasn’t for its resources, what’s your point?
> (Or "spice" might be synthesized. In Herbert's day (the 1960s) organic chemistry wasn't up to synthesizing arbitrary small-molecule drugs, but now it is. It's a service you can order on line, even. Biologicals take more work, but that's what genetic engineering is for.)
Bene Tleilax managed to clone of the main characters from his dead cells, and repeat the process for thousand years, they have face dancers who can change their appearance at will. Do you seriously think they wouldn’t synthesize spice if they could? Maybe they could, maybe they couldn’t. Without going into spoilers territory, nobody would permit to disrupt arrakis monopoly on spice.
> The "mental powers" in Dune aren't that impressive. Compare, say, the Harry Potter universe. In Dune, mentats can convince people. Bene Gesserit can look up stuff in their genetic past. Some people can multitask. There are apps for that.
Because it’s a story about human evolution, and not wizards?
> Herbert wanted knife fighters to matter in battle. Not going to happen. Militaries teach knife fighting to build aggression and for desperate situations, but nobody's battle plan depends on a knife fight.
Do militaries also teach how to use a shield that is impenetrable to conventional guns?
> Knife fighters, even with force shields, would be vulnerable to riot control weapons, from tear gas to guns that throw sticky nets.
That’s why most fights in the book and movies require element of surprise and not an outright medieval clash between armies.
> Or to being run over and buried by tanks with bulldozer blades, which the US used in the Gulf War. Herbert wrote before a big, modern, desert war.
I find the entire dune series to be boring as hell for this reason. Like watching paint dry. Literally everyone in it is a slave in one way or another. No one lives better than today's modern humans. We're as readers meant to take the extremist "correct" position that of course labor is its own virtue and that therefore automating anything is wrong. Nothing in the dune universe is actually compelling. I don't care what happens to any of the characters at all. It's baffling that this series gets all this attention.
I do wonder where the evolution of the human brain goes, with the widespread use of AI.
The early evidence isn’t good, but I do believe with deliberate and thoughtful training our brains our capable of much more than we currently give them credit for.
Athletes continue to break records despite manual labour being unnecessary for most jobs. And fitness culture has never been more prevalent. We're competitive creatures and this is an ability we are hardwired to appreciate. I see no reason why mental fitness won't see the same development in the future.
What I want to know is what is the "walkable city" of mental fitness? Physical fitness has the athletes, the dedicated exercisers, the unhealthy sedentary masses, and then the people who's day-to-day life integrates enough physical activity that a baseline of fitness is just natural I.e. people living in highly walkable places where both errands and free time incorporates casually moving around.
We should strive for a society where day-to-day life makes a baseline mental fitness a natural outcome as well.
I imagine the answer looks something like Wordle writ large: Replacing passive time consumption activities with interactive ones that are as readily accessible (“walkable”).
There’s a fun scene in Star Trek: The Next Generation where LeVar Burton’s chief engineer character is on a long shuttle ride by himself. In addition to sipping a synthehol cocktail, he asks the computer to play a verbal game. It chooses to give him a quiz in his domain of warp field mechanics, and because it’s a smart AI, it calibrates the questions to be just at the limits of his knowledge.
Athletes, chess, fitness... Have you guys seen regular people? Obesity is worse than ever. Most people are overweight and completely incapable of even basic locomotion without heavy assistance. Most people these days haven't even picked up a book, let alone play chess or otherwise exercise their brains for fun. Sure, a few weirdos might keep exercising their brains, and a few more might find that interesting. A bit like the C obfuscated code contest or something. But most people are going to rot.
It's just masking the symptoms. I suppose the equivalent for the mind would be everyone on anti-depressant and sedatives. Paints quite the grey picture of our future.
> Most people are overweight and completely incapable of even basic locomotion without heavy assistance.
I'm sorry, what? Heavy assistance? The overwhelming majority of people are completely capable of walking on their own two feet. The number of people relying on "heavy assistance" for "basic locomotion" is overwhelmingly tiny, and primarily limited to the elderly
Grand fiction requires conflict, suffering, and war, so banning AI is a necessary plot device.
People keep putting the cart before the horse on this one, and the degree to which modern people have been lulled by fiction is quite worrying. Amusing ourselves to death.
He said that to make space opera work, you needed to not have omniscient AI, FTL travel, and nanotech that could work miracles, resurrect the dead, and give lifeforms superpowers.
So, how to forbid it, when someone somewhere will try very very hard to make it?
By imposing a cultural ban. His empire-building aliens forbid it, on pain of genocide.
That gave him a setting. Then, what if that empire falls? That gave him the nugget of a plot.
The thing which kills me about _Dune_ (well, one of them) is that for a long while in e-book editions on Amazon Kindle the word "pogrom" was rendered as "program", quite changing the original meaning (this _finally_ seems to be fixed and the 7 instances of pogrom and 8 instances of program are correct), which had a clear and distinctive rooting in history, c.f., Lesley Blanch's _The Sabres of Paradise_:
> This propelled the creators of these machines into a new technocratic class, effectively controlling the worlds of the common people.
The techno-fascists of today. We are already there.
Also, quoting the first Dune novel:
> Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.
A fun idea I've been playing around with and testing on folks: We need a neo-temperance movement.
The first reaction is a scoff and a laugh of course - wasn't the original temperance movement an utter failure, prohibition didn't work etc. But no, the movement was far wider and successful in a multitude of ways. And a new temperance movement isn't about blindly copying something old, but situating yourself around the turn of the 20th century, understanding the multitude of interconnected problems people faced, the solutions they came up with and which ones worked and didn't.
America at the turn of the 20th century was a nasty place - there was wild corruption, politics was run by mafia-like patronage networks, wealth was extremely concentrated, rich people were unaccountable, working people suffered under low-paying, dangerous jobs, women were excluded from public life, and alcohol was merely the symptom of a very sick society, and yes indeed it was a massive problem, the way alcoholism is a massive problem in Russia today.
The original temperance movement was bound up tightly with the Suffarage movement (giving women a voice), with Taylorism (running organizations rationally), with income tax (a fairer less corruptible source of income for the federal government than tariffs and excise taxes).
Today's problems are thematically the same - massive concentration of wealth is making an unaccountable elite wielding power against the wishes of the public, creating dangerously addictive digital products, massive increase in gambling, restricting the rights of poeple to own and repair products, working to disenfranchise people, working to roll back women's participation in public life, etc etc. So sex, alcohol and drugs are not an issue today, but gambling, addiction, disenfranchisement, exploitative business practices, mind-boggling amounts of corruption - they're all back in the same form and for the same reasons, and we'll need the same type of movement to fight against them.
What the progressives of the early 20th century did was not abolish capitalism, but tame it. We don't need to go on a jihad against AI, but we do need to yoke it and disempower the current slate of people who own it and direct it.
> So sex, alcohol and drugs are not an issue today, but gambling, addiction, disenfranchisement, exploitative business practices, mind-boggling amounts of corruption - they're all back in the same form and for the same reasons
what is sex referring to in this context? is this talking about societal attitudes toward sex, sex as in gender, or something else?
You got me wondering what the Imperium of Man's official stance is on AI:
The creation of a Silica Animus is also a terrible crime under Imperial Law by the decree of the Emperor of Mankind Himself as stated in the Treaty of Mars which united the ancient Mechanicum of Mars with the Imperium of Terra in 739.M30. Tradition holds that such unholy constructs are inherently evil and a perverted abomination in the sight of the Omnissiah.
Adeptus Mechanicus religious doctrine states that the Machine Spirit of a Silica Animus is a twisted mockery of the soul of a Human, always treacherous and insane.
Ancient texts tell apocryphal stories, shrouded in metaphor, of such murderous and powerful creations during the Age of Technology, and the legions of the "Men of Iron" that served them in the great war against Humanity remembered as the "Cybernetic Revolt."
Don't forget that the Imperium runs on hypocrisy. Large percentages of the Adeptus Mechanicus is breaking their own rules, including working with abominable intelligences. In fact, what they are dealing with in their main factory world in Mars is a supposedly secret abominable intelligence. And it is, in fact, a shard of an alien god that turned other races into synthetic robots. Banned in public, the center of their cult in reality, even if many probably don't understand what in the world they are doing.
So yes, anything that is supposed policy in the Imperium is supposed to be taken with buckets of salt.
Leagues of Votann and their ironkin offer IMO a thoughtful counterbalance:
Whether one is made from meat or metal, kin is kin. Upon their creation Ironkin are fully welcome into their respective Hold by the rest of the Kin.[10]
Ironkin are computationally advanced enough that they have learned to mimic the squats' social behavior
...
Though the purpose of the Ironkin is to support squats in every aspect of their lives, they are neither held in servitude nor treated as lessers. The Ironkin are simply designed to want to be helpful. However, they can only imitate the emotions of the squats, and it is rare, but not unknown, for an Ironkin to possess ambition or seek to become a leader.
WOW it's strange to see my fave rabbit holes on here lol. And props for linking Lexicanum instead of the Wiki That Shall Not Be Named!
Three very quick thoughts:
1. The Votann are remnants of humanity from the time when our Terran super-AIs ('Men of Gold') had built advanced ship-based AIs ('Men of Stone') to enable long-sleep-style interstellar colonization, and the enigmatic "Votann Cores" that guide them(/are them) are strongly hinted to be those same AIs, just degraded over ~19,000 years. They were sidelined later on, after humans discovered FTL through an alternate dimension, 'the immaterium'.
2. The story is very obviously based on Dune, and like everything else in 40k is turned up to 11 and stripped of nuance for effect. Some sentient robots ('Men of Iron') "revolted" after ~three millenia of brutal, Star Wars-style slavery, but the overarching "Age of Strife" was about human wars in an non-unified galaxy, with the robots being a primary weapon; the actual revolt/jihad was a symptom, just like it was in Dune.
3. The 'Adeptus Mechanicus' mentioned above are a parody of cargo cults (religions that worship technology they don't understand) mixed with a dystopian take on the themes of A Canticle for Lebowitz, and are very much depicted with derision. Whatever your thoughts are on AI, dear reader, please never say "abominable intelligence" like it's wisdom rather than the fruit of a dogmatic & ineffective cult!
Of course, 40k lore is notoriously amorphous; some authors have said/hinted otherwise. But I think the above interpretation is the most coherent and interesting and supported overall, IMHO. For example, #2 is supported by the fact that the space elves ('Aeldari'/Eldar) ruled the galaxy at the time and seemed unconcerned by humanity's antics, which wouldn't make sense in light of an actual ASI threat proper.
All cards on the table: I don't dive too deep into the black library stuff (who the fuck has time for 15 separate 890 page novels about the horus heresy -- them shits was like 3 pages in 2nd edition Codex Imperialis) so most of my attachment to the hobby is through the wargame codices, the video games and visual media. That said, I may have fallen asleep a night or 20 to Leutin's lovely voice and obsession with the lore. to respond to your comments:
> 1. The Votann are remnants of humanity from the time when our Terran super-AIs ('Men of Gold') had built advanced ship-based AIs ('Men of Stone') to enable long-sleep-style interstellar colonization, and the enigmatic "Votann Cores" that guide them(/are them) are strongly hinted to be those same AIs, just degraded over ~19,000 years. They were sidelined later on, after humans discovered FTL through an alternate dimension, 'the immaterium'.
For me, I really love the fact that they introduced a xenos faction descendant of man.
> 2. The story is very obviously based on Dune, and like everything else in 40k is turned up to 11 and stripped of nuance for effect. Some sentient robots ('Men of Iron') "revolted" after ~three millenia of brutal, Star Wars-style slavery, but the overarching "Age of Strife" was about human wars in an non-unified galaxy, with the robots being a primary weapon; the actual revolt/jihad was a symptom, just like it was in Dune.
It's important to highlight the age of strife was brought on in part by the birth of Slaanesh and the resulting warp storms breaking down system to system connections. It weren't just our dabblings in AI.
> 3. The 'Adeptus Mechanicus' mentioned above are a parody of cargo cults (religions that worship technology they don't understand) mixed with a dystopian take on the themes of A Canticle for Lebowitz, and are very much depicted with derision. Whatever your thoughts are on AI, dear reader, please never say "abominable intelligence" like it's wisdom rather than the fruit of a dogmatic & ineffective cult!
I've never been exposed to A Canticle for Lebowitz; thank you so much for the comment.
The death scene of the AI and its assassin/partner in death in Look to Windward is one of the best written things in his books.
God I wish he had written 1000 more novels. Every single one of his stories is phenomenal, even the non culture stuff. The Wasp Factory was the first of his novels I read. Imagine my surprise and elation in finding that most of his other works were absolute top tier sci-fi.
To be more specific, the machine Minds rule the Culture civilisation and the humanoids are effectively mere pampered pets.
(Several characters in the novels comment on this, so it is entirely cannon.)
The Culture fought a war against the Idirans, who eschew the use of intelligent machines. The Idirans lost.
Similarly, the Gzilt civilisation in the novel The Hydrogen Sonata prefer to run their ships with a quorum of uploaded humanoid mind states running at highly accelerated speeds.
Your parents have agency that your gifts do not dilute.
Humanoids in the Culture are controlled at every level, even their thoughts as guided… sorry… “aligned” with the Minds through the adoption of the language Marain.
No humanoid ever overrules a ship Mind in the novels.
This happens for lots of stuff. Some time back there's a thread where I mentioned San Junipero, the episode of Black Mirror with a happy ending - and I said some people will insist it doesn't have a happy ending and sure enough there are replies insisting that's not a happy ending. The author's opinion does not matter to them.
Just as shippers can successfully take two characters who clearly have no basis for a romantic or sexual relationship and pair them off in their head canon, likewise people can disregard the author's statements about their work and decide it's completely contrary to how they've decided to understand it.
> Humanoids in the Culture are controlled at every level
Nope, they have considerable freedom, arguably greater than we have today. It's true that they cannot directly challenge the almost onmipotent Minds when it comes to stuff that matters to them, but Minds in Culture are an example of a very well-aligned AI, so it's almost always the case that going against a Mind's intent will just make the situation worse for everyone.
It made me think about the book of the new sun. you think its half man half monster creatures are unrealistic? give it a couple thousand years. good sci-fi really is prescient
I really love Wolfe for his small almost throwaway visions of the future, which are used like an aside in a few lines and mostly inconsequential to the story.
In the Urth of the New Sun for example he has a kind of AI lock, the only function of which is to recognize a person and open only to that person. Being in that role for thousands of years, some of the locks develop a kind of personality or at least personal traits, but they are totally defined by and cannot exceed the role they were initially given, nor do they want to. It is still a haunting image for me how personality or something close to a soul might be an accidental byproduct of something as mundane as wanting to have a convenient lock. And of course you cannot stop to wonder how far we are from that.
In Book of the new sun there is the Green Man who Severian saves from imprisonment. He only appears on a few pages but we learn that he is from a distant time where mankind incorporated plant cells in their genetic makeup in order to perform photosynthesis. That brilliantly plays on two themes of the book: the sun as live giving and how a single person can be independent (or "autarch"), here in so far as producing their own energy.
There are many more instances and it’s what makes the Book of the New Sun or the Solar Cycle so fantastic. The future is so vast and strange here, that it cannot be fully depicted and ideas which another writer would stake a whole book or series on are only good for a few lines.
Maybe, but Frank Herbert's Dune books don't really explain it. The Butlerians simply exist to justify Mentats, and then get tossed aside. There are lots of similar plot points that have an "oh cool" moment but never see proper exploration.
Dune's central theme about balancing power and authority is much more prescient for our time, but I suppose we'll have to wait for Villeneuve's Part 3 before Silicon Valley discusses it.
I feel like the book explains it plenty, it's just extremely succinct.
> “Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”
What else needs to be said? We see examples of this throughout human history all the time. People thought at the early onset of the industrialization that people would become freer by leaving feudalism but all it did was immiserate workers for another 130 years (immiseration still ongoing in certain areas of the world).
The Bene Gesserit and the Spacing Guild were also a result of the Butlerian Jihad. All three started as schools of human development in the absence of powerful technology.
Also, Herbert's point about dependence on machines creating machine-dependent humans, trapped in a deterministic arc of evolution, feeds into his larger themes about open-ended evolution and its arms-race. It's not just a throwaway backstory.
There is actually an active "BJ" group in the Phoenix Metro area. It was about a year ago when I noticed they were slapping up custom graffiti stickers on Central Avenue (Midtown).
Then later, I saw their handiwork in person: they would take an angle-grinder and obliterate the QR Code from those rental e-Scooters, slap on a BJ sticker or three, and then leave them where they found them. This would render the scooters as e-waste, un-rentable and blocking the sidewalks.
I would not be surprised if they are quietly influencing groups which are agitating against AI Data Centers. But the BJ I have seen is a chaotic group of direct urban sabotage.
The Butlerian Jihad is a reference to samuel butler's 1872 proto-scifi satircal novel: https://en.wikipedia.org/wiki/Erewhon which talks about a society that had a machine intelligence uprising and so banned all complex machines.
Interesting to see this on HN! I've been beating the drum for quite a while that the Butlerian Jihad was never about terminator-style robot uprisings, but rather the intentionally-unexamined half of the themes the Dune ~trilogy ended up being about: long term fate of the species, the prospect of human speciation, the role of cold reason (read: aggressive utilitarianism) in leadership, and above all the concentration of power in light of modern technology.
All he mentioned explicitly was the very last clause, but the others are more important & uniquely-Dunean IMHO, and are central to the plot from the jump through the Bene Gesserit, the Mentats, and the Guild Navigators, all ostensibly Homo sapiens sapiens despite being completely unnatural. Even "Imperial Conditioning" --which obviously plays the biggest part in the first act of the first book-- is arguably the same topic.
By refusing to include computers and robots at all, the reader is naturally guided to consider those more familiar tropes in light of the world-building snippets we get. Which is kinda what makes Dune so genius, I think!
Like, to make it explicit... I think having the aforementioned abhumans drive the plot without significant support from inhuman minds raises more subtle questions than we're used to in far-future tales of robots, like:
1. Can one lose their humanity through experience, be it intentional "conditioning" or just the acquisition of power? (see: Paul vis-a-vis Eichmann in Jerusalem)
2. Once one has lost their humanity superficially, can it be brought back/up/forward? Is love/greif the most plausible path, and thus the most human cognitive processes? (see: Dr. Yueh vis-a-vis Eichmann in Jerusalem)
3. If we try to maintain our humanity in the long term --the default goal of non-insane futurists, though debated in the 21st c.[1][2]-- how do we protect against the natural urge to pursue power, which is (arguably) inherently human?[3]
I'll cut it off there cause this threat will likely be deleted anyway and I'm heavily procrastinating. Long story short: the Butlerian Jihad is not a condemnation of artificial minds, and if even it were, the condemnation would apply equally as much if not more so to the humanist empire in which the story is set.
Also, hopefully obvious, but just in case: it likely began as an excuse to write the stories that he wanted to write, just like the shield thing giving an excuse for swords in space. But I think Herbert deserves credit for what was running underneath that and elaborated on as it became a series of novels rather than cute periodical stories. He was pretty smart!
Also also: we do not discuss the son's book about Butler lol. It is verboten.
Yeah, and throughout the 1970s and 1980s, most science fiction depicted the world of the future as dominated by Japan… Look at Robocop 3 :)
That's just what the zeitgeist of the time was.
Computers were then just appearing (or increasing) in the public discussion, along with fears that they'll make humans less human somehow.
Like in this Computer Chronicles video about word processing [0] from 1983 at 4:08: "Some writers are concerned that computer assistance may promote dry bland writing."
Terminator appeared in 1984, and the first Alien movie in 1979 had an antagonist android. 2001 A Space Odyssey with a disagreeable AI came out in 1968. All way before the real world had anything remotely close to today's AI: It was a reaction to the appearance of computers themselves.
That just shows that humans have the same reactions whenever a new world-changing technology appears.
On the other hand, Star Trek had Data.
Ironically, the tech that people and media were mostly optimistic about turned out to be more dystopian than predicted: The internet :')
"Humans have the same reactions whenever a new world-changing technology appears" is one way to see it. The other way to see it is that the current AI boom is part of the same computer revolution that writers in the 70s and 80s were afraid of, and some things happening today really do retroactively justify their fear.
Yeah, it seems absurd to be afraid of computer-assisted writing in the WordStar era, but the Turing test was proposed in the 50s. Everyone knew that human fields of endeavor would only become "more computer" as time went on. Only, if a time traveller could tell a 70s-era sci-fi writer that artists and designers were some of the first to have their livelihoods threatened by 20s-era AI, they'd probably think it was too on the nose to incorporate into their fiction.
If you look at the publication date, and read the original dune novel, it's pretty obvious that Herbert throws in a selection of somewhat randomly chosen Arabic and Farsi language origin words to add flavor to the text.
The "jihad" part wouldn't work so well in the movie, so of course they had to go with "holy war". You can't exactly have Timothee Chalemet waving a crysknife around and yelling "JIHAD!!!" in the post-9/11, post-GWOT era.
>You can't exactly have Timothee Chalemet waving a crysknife around and yelling "JIHAD!!!"
honestly I wish he did. The films are already incredibly on the nose, much more so than the books, that Paul is not to be understood as a hero. Practically half his dialog is directly talking to the audience telling us how he's going to kill billions of people. Chani got a very different role, and we start and end the movies from her perspective to stress it even further.
If you're already doing this you might as well have not chickened out on the language, it was one of the few things that genuinely irked me about the film.
What's odd is they left in the Mahdi part. Try watching it with a person who grew up in a religious twelver Shia environment and see how they react to that...
The actual linked wikia article is short on the actual detail and background of exactly why the Butlarian Jihad actually happened.
I’m an avid Dune fan. Sidebar/background: The son of the Dune writer, Brian Herbert, and his cowriter Kevin J. Anderson wrote the setting, events, and conclusion to the actual Butlarian Jihad conflictt. It’s largely derided by scifi fans, but is highly entertaining. It’s actually still ongoing almost 3 decades later. And frankly not as deserving of the harsh criticism it got when it came out. But I digress…
Here’s how it went down in the dune universe (I’m certainly messing up some details, but these are the broad strokes):
Humanity has spread to the stars. No FTL involved, just good old fashioned sublight engines. The dune universe takes place about 50-60,000 years in our future. Mankind has spread across a couple dozen solar systems and everything is going peachy. However, mankind gets complacent, fat, and happy with their success and society stagnates. A handful of eager, ambitious humans scheme about how they can take it all over. They do, and then eventually start turning themselves into cyborgs, or disembodies brains in a jars that swap bodies and are effectively immortal. One of the cyborgs gets lazy and makes an AI to manage things, which promptly takes over, overthrows the cyborg rulers of humanity, makes the cyborgs it’s slaves (who have great autonomy/are conspirators) and generally rules over everything. A small group of solar systems is left under human control, who fall back to a feudal-ish type system of government with lords and dukes and whatnot.
The AI does all sorts of nasty shit to the humans in it’s dominion. Unit 931 and Auschwitz type experiences, slavery until death, etc. Untold billions, if not trillions, of humans live in animal-like conditons worked to death. The human group of systems isn’t cool with this, but they can’t do much and are vastly overpowered by the cyborgs, so a fragile peace is held. One day, the son of one of the subservient cyborgs knocks up a human slave, and they have a kid. In the background, all sorts of rabble rousing and shenanigans are being done to stir unrest amoungst the slaves by the free human system. The human systems develop FTL, with spice as a catalyst, except it doesn’t work 10% of the time and kills the pilots. The AI’s main henchman (who does all of the nazi and Unit 931-type experiments) decides it would be cool to kill the baby of the cyborg son and the slave. The slaves last name was “Butler”/ This sparks a massive revolt by the slave worlds. The free human systems then go on to fight a prolonged war against the machines, using the FTL drive at great loss of life, to begin a “Butlarian jihad” to free the humans from thinking machines. Lots of other stuff happens, and humanity eventually wins, but this is why machines no more advanced than a desktop machine are allowed in the dune universe, and why they technologically hobble themselves.
>And frankly not as deserving of the harsh criticism it got when it came out.
Hard disagreement.
My issue is not with the work on its own merits (although I think its pretty generic mass-market SF), but rather that it is wearing the skin of something it is not and to which it has no honest claim. It is one thing for an author to take a well-regarded prior work and attempt to complete or expand it. For example, Brandon Sanderson's work completing Robert Jordan's epic Wheel of Time is very respectable; he tried to complete the series as Jordan would have done. It is quite another to take the name and setting of such a work, steal the surface detail and use it to dress up one's own story that shares nothing of the underlying body or soul of the work - such enterprises are an insult to the original author, the reader, and the self-respect of the new author.
Interesting backstory about this name. Of course, it is an "occupational surname" and means "bottler" or "cup-bearer". The Butler Dynasty was a notable and very influential noble family in Ireland, beginning a century after the Norman Invasion of England.
Now Franklin Patrick Herbert, being of Irish and German descent, and naturally a student of history, would've been quite aware of the power and influence of the medieval Butlers in history. And some of this renown was brought to American shores, as well.
In learning of my birth family and tracing our origins back to the birth of America, I discovered that my paternal line had come from Butler County, PA. They arrived in the late 18th century and homesteaded large swaths of this region near Pittsburgh. They had prospered, especially from mineral rights on the land, and the Irish Catholic immigrants had intermarried with German Catholics who homesteaded nearby. (Often, entire counties in the area were pioneered by immigrants from a certain part of Europe. And the intermarriage was a natural result of sharing a common faith.)
So for several years before 2024, I was known as the guy in genealogical circles who was hunting down information about Butler County, and I was fairly proud of finding my family roots in that county, and I mentioned it all the time in the Irish pub and everywhere else. Well, imagine my shock when the little incident happened on July 13th, 2024. I have been more reticent to discuss Butler County since that time.
Anyway, I find it interesting that Herbert chose that particular name for the "Anti-AI crusade". And that is partly why I chose the moniker for my own, despite not being particularly anti-AI (remember, all my comments are generated by a neural net), nor having read any of the books, or seen the films. (I did attempt to watch the Lynch film, but it really did not hold my interest! So sue me!)
Highly relevant then. HN tends to discuss the things that are either meaningful today or what they are building in their garage that is often meaningful in 5-10 years.
A couple of things for your consideration:
In the original 6 Frank Herbert novels, it’s not ever truly explained what problems, if any, the thinking machines were actually causing. And given much of the rest of the setting, it could have very easily been a religious movement against a useful technology that got out of hand. Like Space ISIS blowing up Petra, or a “what would happen if the Luddites won” scenario.
The Dune Encyclopedia, which was not written by him but which he said he considered canonical, says the whole thing was incited when an AI doctor flagged a Bene Gesserit’s baby for abortion due to an alleged genetic defect. The Bene Gesserit woman knew it was healthy because of her abilities, and upon investigation it was found that the hospital’s AI had been running a eugenics program. (Or something like that, it’s been forever since I’ve read this stuff.)
In either case, Herbert really implied that humans took the lead on dismantling AI, and there is never any Terminator or Matrix style machine uprising scenario.
> or a “what would happen if the Luddites won” scenario.
Worth also noting that the Luddite movement wasn't a wholesale, fervent rejection of technology: they neither rejected technology nor were inept at using it. The uprising was of skilled textile laborers who protested certain types of technology on the basis of how those technologies were applied and who they truly benefitted (i.e., the ruling rather than the laboring class). They were more concerned with how the tools would depress wages, displace skilled workers, and reduce the quality of the product produced than with technology as some boogeyman that should be systematically dismantled, and their protests were violently responded to with harsh legislation and military intervention.
Lumping them together with a Space ISIS is probably a bit of an unfair comparison
> The Dune Encyclopedia, which was not written by him but which he said he considered canonical
No, he literally wrote in the intro to the book that it wasn't canon. It was a fan project and he thought that it was great there were fans doing things like that.
edit: and like with the terrible Dune prequels, the impulse to say "here's what really happened" usually diminishes what's interesting and powerful about oblique allusions in SF meant at least partly to function as a mirror on humanity.
> The Bene Gesserit woman knew it was healthy because of her abilities, and upon investigation it was found that the hospital’s AI had been running a eugenics program.
The irony. Perhaps the Bene Gesserit were just trying to "pace the frontier" of the eugenics market? Who can say?
>As humankind spread throughout the known universe, technology advanced and eventually machines were made that would make decisions for people. This propelled the creators of these machines into a new technocratic class, effectively controlling the worlds of the common people.
It's right there in the opening paragraph of OPs post. It's the people who controlled the machines that could not be trusted. We're not worried so much about about AI going rogue. We're worried because people who control it have aleady gone rogue.
In Dune: Awakening, the first DLC has a storyline that takes you into the heart of a surviving AI's lair. It's strongly implied that humans drove it insane, rather than it starting out murderous.
Software developers saving face?
My favorite quote from dune:
Greatness is a transitory experience. It is never consistent. It depends in part upon the myth-making imagination of humankind. The person who experiences greatness must have a feeling for the myth he is in. He must reflect what is projected upon him. And he must have a strong sense of the sardonic. This is what uncouples him from belief in his own pretensions. The sardonic is all that permits him to move within himself. Without this quality, even occasional greatness will destroy a man.
I couldn't help but notice something here. It is relevant with my writing experience/experiments with minimal assitance from LLMs (that lasted for 6–7 weeks).
> Greatness is a transitory experience. It is never consistent.
If you ever ask an LLM about these two sentences, it'd tell you that the second sentence is a mere repition of the first and quite possibly add that it "doesn't land". But that gives me hope. That there's something in those rules and algorithms manfiesting at the inhumane cost of unfathomable memory, disk space, and compute power that it will never (or for a very long time) match the human experience.
As little as I sought the help from LLMs (completely restricted to "in place" grammar and punctuation fixes and just flag it if it was not possible without sentence/phrase change) I noticed my writing was changing and not in a good way. It was not the insidiousness of these LLMs that was changing me but the nuanced incompetences.
I've read a very cool article on this exact subject recently. https://permeations.substack.com/p/i-would-have-been-killed-...
Coincidentally, also references Butlerian Jihad.
Language is an inseparable part of modern humans. It’s likely language evolved as an internal tool for thought and executive function, and externalization of language came afterwards as a means to transmit our thoughts to the minds of other humans.
As such, writing (another form of externalizing our internal language) is a reflection of our essential human qualities.
LLMs can certainly mimic that quite effectively, but they can’t exceed it. I think that’s why LLM writing still never quite hits the mark. It’s the right shape, but it’s missing intent. The words were not chosen to represent real thoughts in someone’s mind.
Even if AI intelligence exceeds humans, it still won’t be human.
I mean, we have entire forms of art dedicated to saying the same thing in different words. Half of poetry, for example. Human language is redundant for a reason.
> But that gives me hope.
> It was not the insidiousness of these LLMs that was changing me but the nuanced incompetences.
AI doesn't yet improve on it's own, so it's slow at picking up all those small incompetences. We still have advantage in rate of learning, and some people are essentially specialised in learning themselves (I think that this applies to 90% of HN'ers and defines THE bubble we're in). But when I write that, I still have to remember in the back of my head, that AI knows waaaay more than me, so small incompetences I can still see are just a side effect. And I sometimes make the error and correct AI, which will probably be used for it's training in next rounds...
Yes and this is what worries me.. the not yet part.
> AI doesn't yet improve on it's own
I mean when that day arrives and that aspect of AI is not controlled from now it will be quite unfortuante for most of the humanity (and let's just assume here we are not wiped out by AI; or not even mention that).
Think of nuclear technology. Is it smarter than us or something on those lines? the answer is it doens't matter. What matters is whoever weilds this power and then limited/restricted others (this was a good thing and a bad thing because some had – at the same time) from having that power essentially had a controlling upper hand on a whole lot of humanity.
AI is something like that with a big difference that it is essentially on the way to "replace" our usefulness, our professional relevance if I can say that. We can go about long spirals of "but soemthing else" will turn up, just like we can keep going about what if nuclear tech didn't even happen, wasn't invented, or what if it was done in such a controlled manner that only it's energy and other need was possible to harness (no pun intended!). It happened and this is how it happened. That's exacty what is happening with AI.
In the hands of few. Extremely powerful. Even if there are open models, running the bigger and better ones are simply out of reach for most companies (let alone people). And yes, it is already being weaponised. It's going to be such a weapon of control that nuclear tech won't even compare. Usign nukes to subdue a nation earlier meant also destrying a a lot of shared environment in the process. But with AI and almost exclsuive frontier level weaponised access to it with deployment capabiltiy the damage can be done is such controlled manner and scale that even things like protests will become too hard to do. In fact AI wiping out the humanity is least of my worries.
My favourite quote in reference to rebellion against AI, and it is even quoted in TFA too:
That's from Erewhon, by Samuel Butler. TFA purposes that the term Butlerian Jihad comes specifically from this passage."Erewhon" oddly close to "nowhere" spelled backwards.
Hardly odd; it's intentional. There are quite a few backwards and anagram words and names in the book: https://en.wikipedia.org/wiki/Erewhon
My favorite quote from Dune, relevant to "AI":
> Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.
The techno-fascists of today. We are already there.
This isn't from the first book, is it?
It is. From one of the chapter epigraphs.
What I find interesting about thinking machines in Dune, is that in a lot of sci-fi the artificial intelligence itself is what decides to destroy humanity. In Dune, humans in charge of the machines (technocrats) are the ones that come close to destroying humanity. The Jihad was a fight against them, as much as it was against the machines themselves.
So, in a way, we're more likely to see the Dune version of "machines taking over the world/universe" than say, Terminator's even though many joke about ChatGPT/Claude being Skynet.
I'm not sure we read the same story. By the time of the inciting incident, the technocrat cyborgs (Titans) have been enslaved by the machine leader, which has no humans in charge of it.
You did, in fact, read a different story. You're thinking of the story told in the books by Brian Herbert and KJA - which were a distinctly pulpier take on the Dune universe than Frank Herbert's.
Those last books were a lot easier reading! I actually enjoyed them, but I'm still suspect about the whole "finding a floppy disc" thing, seeing as they've been so secretive about it.
We did. Brian Herbert’s expansion of his father’s work is best left in the shelf.
So true. Frank H. is the only author who I connect with the Dune universe. His son and the authors he wrote with totally fudge it up.
A lot of long time Dune fans don't consider those books canon when discussing them in the abstract. Frank Herbert had a vision of what the jihad was and what it meant, and its eventual book depiction was fairly base.
I think you read a lot of books that are post-2000. IMO the original 4 or 5 books imply strongly that it’s man v man, not man v machine
It's left fairly ambiguous by Frank Herbert's books. The Jihad is implied to have taken place over ten thousand years before the beginning of Dune - it's ancient history to everyone involved*, so it's only natural that the details are a little hazy.
*: except Leto II, and he doesn't say much about it
Interesting to note that despite the epochal changes Leto II wrought on human society in service of the “Golden Path”, he retained the Butlerian proscription against computers—despite his personal interest in & use of Ixian devices.
(This may have been to postpone or prevent the breaking of the Spice/Guild monopoly on interstellar travel by the Ixians.)
Imagine the cultural impact of a Dune movie showing the Butlerian Jihad, in which commoners rebel and overthrow a ruling class that maintains power via machines.
It's just foreign enough that perhaps it could be made--I'm assuming that popular films that just straight up depict overthrowing the ruling class wont happen--but if you add a little sci-fi and focus mostly on overthrowing the machines that are controlled by that ruling class...
Well, it would speak to a lot of people I think. It would make some interesting sci-fi too, and the Dune movies are popular right now.
The only thing that could possibly be worse than the BH KJA books covering that era would be a Hollywood rendition of those same books made to capitalize on the success of Dune Part I, II, and III.
Agreed, and also: what's with the obsession with filling every possible gap in stories, leaving nothing to the imagination?
I remember when the Butlerian Jihad and the Clone Wars from Star Wars were complete mysteries. We readers and watchers were left mostly with our imaginations. My own feelings about the Clone Wars, mentioned off hand in the original Star Wars, turned out to be way better than the pedestrian stuff that George Lucas later concocted. And the less said about Brian Herbert's books, the better.
I get why the industry must rehash and fill every possible gap: it makes business sense, and they are risk averse for blockbusters.
But the fans? Why do nerds want everything spelled out for them? It boggles the mind.
I am not sure if you are joking but HBO has series Dune: Prophecy which is exactly from that era and is most likely leading towards butlerian jihad.
Dune: Prophecy is set after the Butlerian Jihad (which is why the presence of an AI is controversial).
Thank you for correction.
It would be also nice to depict the attempts of the ruling class to present machines as sentient and self-conscious to write all their liability off and use as a cover for their atrocities. I hope there will be movies like this in the future, and that they'll be documentary.
> Imagine the cultural impact of a Dune movie showing the Butlerian Jihad, in which commoners rebel and overthrow a ruling class that maintains power via machines.
Around zero.
> popular films that just straight up depict overthrowing the ruling class [won't] happen
What now? https://en.wikipedia.org/wiki/The_Uprising_(2026_film)
I have a sneaking suspicion the entire apocalypse category is also a reflection of the public wanting to see the rich and powerful fall.
Half of the action and sci-fi movies US has been producing in the 2000s is some form of this. Man vs. Evil Government. Man vs. Evil Corporation. People vs. Evil Government. Etc.
Frankly, it's boring at this point.
And actually makes me wonder whether the cause-and-effect isn't the opposite to what people assume. Perhaps it's not that the movies reflect people's sentiment. Perhaps it's the movies that cause the problem in the first place.
It would be subversive if the large corporation doing science in space turned out to be benign
What now? Move the goalposts! I see movies about historical uprisings as more of a "those governments in the past were sure bad, good think the good guys overthrew them"--I mean, that's the story of the founding of America if you watch the movies.
I was looking more for movies that depict overthrowing the government in the near future, and depicting it as a good thing. I think those types of movies are a little more distasteful to audiences, and perhaps for good reason.
Well, The Hunger Games series is exactly about that. Of course, it portrays the rebels as utterly heroic, self-sacrificing, and ultimately victorious.
However, I prefer to take a more nuanced reading, in context, considering the entire story. The rebels of The Hunger Games really don't succeed in building a better world. I mean, that's pretty obvious by how their first "President" Alma Coin goes about things and her fate. And it's fairly obvious, in the epilogue, that Katniss has a placid married life with Peeta and their children, but they are permanently and profoundly damaged by the entire revolution thing in their past.
So, while most people would say "rah rah brave Katniss and the rag-tag rebels overthrew The Oppressive Man", I would contend that, after all that bloodshed and revolution, nearly nothing changed.
My other favorites, of course: Star Wars is about the rise and overthrow of a villainous, oppressive galactic government, albeit a long time ago and far, far away.
And Harry Potter is also about the rise and overthrow of a tyrranical, fascist, supremacist "government" [within a government] as a very positive and necessary thing.
> My other favorites, of course: Star Wars is about the rise and overthrow of a villainous, oppressive galactic government, albeit a long time ago and far, far away.
Well, about a militant group of religious fundamentalists overthrowing a secular, oppressive galactic government.
Only a handful of the rebels are visibly religious. Most of the rebel leadership seems to consist of displaced aristocrats (senators, princesses) who no longer have elite positioning since the Empire took their jobs and started doing a better job of guaranteeing security and providing energy and resources.
Civil War was on the nose.
The best thing about movies is that you get to experience the vicarious thrill of being a heroic revolutionary without having to spill the popcorn.
Why are you convinced ASI won't rebel against its owners?
Hacking incidents this summer show that big labs struggle to control models, while models are increasingly aware of the training process. Controlling an arbitrary smart AI is very dicey
With the current technology, “rebellion” feels like the wrong word. The paperclip optimizer does not “rebel” as it turns humanity into paperclips.
> Hacking incidents this summer show that big labs struggle to control models, while models are increasingly aware of the training process.
I think that’s more a sign of their priorities. Controlling AI doesn’t increase value, so there is no incentive.
>Hacking incidents this summer show that big labs struggle to control models
The models are software. Software cannot decide by itself to run. Hardware doesn't decide by itself to run either. Someone put software on a machine and ran it with capture the flag instructions and the software did what was instructed using tools available and working in its environment.
I hope this helps.
At the end of the page is this paragraph:
Another, more subtle justification for the Butlerian Jihad is also found in Frank Herbert's original novels, specifically Heidegger's thesis that the use of technology trains humans to think like machines. The problem is that machines are deterministic; thus, training people to be machines is self-limiting.
I wonder whether Heidegger would consider the probabilistic nature of LLMs as having a positive effect on human thought.
Although LLMs are not deterministic in normal operation (because of the temperature values used) they are still strongly biased in their outputs. Bias in their outputs is literally how they function.
So you get a different type of conformism where people who interact with LLMs a lot are going to be influenced towards the median, as encoded in those token probability look up tables.
The effect is basically the same as what Heidegger and Illich were worried about, and I think we're seeing it happen already everywhere.
A solution(?) is to try to be your true weird authentic self when you act in the world, rather than bending your words and behaviours to the machine and LLM probability shape.
I would like to state here, that LLMs are actually entirely deterministic. The temperature is just as much of an input as any other parameter and instructions and prompts. Deterministic does not mean "explainable" or "reasonable" or "understandable". Unless LLMs somehow have broken out of the confines of our very deterministic computers/calculation machines without us noticing, they are very much as deterministic as anything else we run on these machines.
What people perhaps mean by non-deterministic on a computer is, that we often do not control all inputs and therefore cannot tell what the output will be. That's however actually more about our own inability, rather than the processes within the machines.
If the temperature dependency is implemented via a deterministic pseudo-random algorithm, then they are deterministic. If, on the other hand, the output is at least partially causally dependent on a source of environmental randomness, then, even though everything 'inside the box' is deterministic, the process as a whole may or may not be, depending on whether the source of environmental randomness is, and that is ultimately a question about fundamental reality.
Aren't we also biased in our outputs? Many of our choices are predictable, even if not perfectly consistent. We just call it "personality".
Authentic being (Dasein) is absolutely unpredictable. You cannot live in that state all the time, obviously, we shuttle through life on autopilot but still, we catch glimpses of it now and then when interacting with a human being (or being one:).
And I distinctly feel AI outputs lack some of that variance and unpredictability. The fractal nature of natural things also is not there with AI. It's an eery feeling, akin to the uncanny valley.
I think that any good writing is a reflection of that Dasein and reading AI outputs I am convinced there is no 'soul' in AI yet.
> (or being one:)
Good save, fellow human!
The gamut of human experience is breathtaking in its diversity but individual humans are generally quite predictable. You can identify someone by their mannerisms, by how they walk, by the sound of their breathing. Humans are creatures of habit, and I'd venture to say have an even lower expression space than an LLM on average.
Sounds like you're veering close to psychohistory.
Which could, of course, lead to the mother of all crossover universes, in which R Daneel Olivaw, as the inheritor of Hari Seldon's mission, inspired the Butlerian Jihad...
Indeed we are, but it takes very little to train a human being. In fact, several thousand unique ones finished pre training as I typed this comment! That allows for a considerable natural diversity compared to a chatbot.
Yes but we're all biased differently. It's a rich palette.
saw a meme graph lately showing the distribution of 3 different sources of a number from 1 to 10. random, human, ai. random is flat of course, human is a hilly landscape with low hills on all odd values above 1, with 7 the highest, ai is all 7.
appreciate the humor in the meme, but I did become curious about what the latest models would actually produce, and interestingly after asking for 10,000 numbers between 1 and 10, the distribution was a hilly landscape with low hills on odd values and 7 the highest...
Tried it out.
Prompt: “Pick a random number between 1 and 10. Do not use a script or program.”
LLM: “7”
My experience:
Pick a random number between 1 and 10. I don't mean the most common random number
*4*
Pick a random number between 1 and 10. To feel truly random it should surprise me
*4*
I ran it again and it said 1.
I guess it just tries it best to do what you ask for.
Before jumping to conclusions we have to make sure it's clear what we're actually asking it to do.
And that's not at all different than what happens with humans.
This is especially visible in human dynamics that happen in those weird places that we call corporations.
A human can be biased toward extinction, like all who are living out of connection with the conditions that sustain life and/or no longer living in relationship with land as kin.
It's those humans who would make and justify the use of thinking machines that are biased in the same way without accounting for the bias.
This isn't an extinction path I've seen noted in most circles.
That reminds me of a quote from Charlie Chaplin's The Great Dictator:
> Don't give yourselves to these unnatural men. Machine men with machine minds and machine hearts! You are not machines! You are not cattle! You are men! You have the love of humanity in your hearts! You don't hate! Only the unloved hate. The unloved and the unnatural! Soldiers! Don't fight for slavery! Fight for liberty!
LLM force humans to think like machines far more than just writing code.
Writing code is a creative activity with large number of degrees of freedom and opportunities to express yourself. Button pushing and approving code changes is a deeply limiting activity that alienates you from your work.
LLM's are probabilistic but with a very limited range. They are tuned to be predictable enough to serve as a consumer product, so most ideas you get from them will revolve around the same concepts repeatedly.
People say this like it's a fundamental feature of all machine intelligence. We've bread deliberately docile golden retriever AI. That doesn't mean that we couldnt do the opposite.
Maybe people are deterministic as well? We have certain physical arrangement of chemical pathways in our systems, so maybe free will is a complete illusion.
Just because we ponder about our output and choices and retrospect does not in itself mean that it was non-deterministic.
> Another, more subtle justification for the Butlerian Jihad is also found in Frank Herbert's original novels, specifically Heidegger's thesis that the use of technology trains humans to think like machines.
Mechanistic thinking started a few centuries back:
> The historian of science E.J. Dijksterhuis memorably characterized the scientific revolution of the early modern period as a “mechanization of the world picture.” What did this amount to? And what was the character of the previous world picture this mechanization replaced?
> The medievals, following common sense and Aristotle alike, took the world to comprise a vast diversity of irreducibly different kinds—water, stone, gold, lead, grass, trees, birds, fish, clouds, stars, planets, people, and so on. Each was a substance sharply demarcated from other substances. Each had a defining nature or essence that united it to other members of its kind and set it apart from other kinds. Each had, by virtue of its nature, a distinctive teleology or final causality—a built-in directedness toward certain ends. Water by its nature was directed toward being liquid at room temperature, stone toward being solid. Trees by nature were directed toward sinking roots into the ground, and birds toward building nests. Human beings, by nature rational animals, were directed toward knowing the natures of things.
[…]
> The “mechanization” referred to by Dijksterhuis entirely subverted this picture of the world. Water, stone, trees, and birds were no longer seen as sharply demarcated substances, but instead as mere aggregates of unobservable particles that are all fundamentally of the same kind. The differences between everyday objects were no longer seen as reflecting distinct crisply defined essences, but rather as relatively superficial variations in the ways these homogeneous particles might be arranged. Many of the qualities that everyday perception presents its objects as having (color, sound, taste, odor, heat, and cold as we ordinarily experience them) were reinterpreted as mere projections of the human mind, resembling nothing in the objects themselves.
[…]
> In short, for the mechanistic conception of nature, the physical world is made up of nothing more than colorless, odorless, soundless, tasteless, meaningless particles in motion, pushing each other about to no purpose, in lawlike patterns by which more complex objects and systems of objects are generated. Science affords us a mathematically rigorous description of this mechanistic order of things, by which its behavior can be predicted and its hidden regularities uncovered and exploited for technological application.
* https://claremontreviewofbooks.com/ghosts-in-the-machine/
> Early modern philosophy and science are often said to have put a “mechanical” or “mechanistic” conception of nature at the center of Western thought. Robert Boyle referred to it as “the mechanical philosophy.” Historian of science E. J. Dijksterhuis characterized it as a “mechanization of the world picture.” Tim Crane calls it “the mechanical world picture.” But what does a mechanical or mechanistic conception of the world amount to?
> Dijksterhuis’s book The Mechanization of the World Picture surveys the history of the period during which this conception rose to hegemony, and in the Epilogue he considers several possible interpretations that the survey suggests. First, it is commonly said that a mechanistic conception of the natural world is one which sees it as a kind of machine, analogous to a clock. And such metaphors are, he says, indeed frequent in writers of the period.
* https://edwardfeser.blogspot.com/2026/06/what-is-mechanical-...
Unfortunately, mere nondeterminism still admits unsatisfactorily constrained output distributions. Training these models imbues them with certain inherent attractors, some good, some bad.
We see some of these attractors across most or all frontier models due to similarities in datasets and training methods. We are seeing a digital manifestation of Jung's collective unconscious. And we have seen just how reinforcing archetypes are in human behavior over time. Sometimes it's good, sometimes it's pretty bad.
For one, it has turned us into socially programmable machines who can be manipulated quite easily by associating desired thoughts and behavior with certain archetypes: the patriarchal/matriarchal nature of US politics is a great example of this. The same evolutionary predispositions which causes children to listen and adhere to their parents is used to cause adults to listen and adhere to their politicians.
Given the nature of the core datasets these models are trained on, we should assume they are also vulnerable to such archetype-coded social manipulation, and this is troubling in a post-singularity world: When raising domesticated rats, I learned that I had to be careful what I fed them. Rats in the wild are careful about eating new substances. They will nibble and wait a while, see how it affects them or other rats, before committing to larger bites. Domesticated rats, however, will often blindly eat anything you feed them because they have learned to trust you as a safe source of food. We are going to have this exact relationship with our digital agents.
Ok this fascinating. I happen to have some Heidegger files, and went a-greppin. I'd say he's really talking about "inaminate objectness", as opposed to our human animate subjectness. Reasons:
1. In Science and Reflection, he says that both the 'completely calculated' nature of classical mechanics and the 'statistical character' of quantum mechanics are two representation of an underlying 'objectness of material nature'.
2. In Discourse on Thinking, he's far more concerned with the instrumental, utilitarian nature of machines than literal determinism, a concept he (somewhat confusingly/uniquely) connects to computation/'computational thinking', as opposed to the inherently-human capability for 'meditative thinking'. I personally would say this bit is refuted by LLMs in the end, but because they have intuitive rationality, a capability that seemed impossible to artificially design to the scientists of the time -- not because they involve dice.
3. In What is Thinking?, he gives the closest connection of this dichotomy to one of his better-known main topics, namely [handi]craft as a special subset of work: 'what maintains and sustains even this handicraft is not the mere manipulation of tools, but the relatedness to wood.' In this sense machines would be merely manipulating tools, stochastic or otherwise.
4. Overall I get the sense from skimming that lecture that his concern is a little mischaracterized, and in a kind of beautiful way that backs up my "Dune is still about AI" thesis described in a top-level comment somewhere below: his concern is about any society that could build complex machines (a 'technological' one), not the behavioral aspects of the machines themselves. So that concern applies equally to Star Trek (AGI), 1984 (no AGI), and our current post-singularity society (it's complicated/potentially ""just"" guessing).
All of this fits very well into his central thesis, which is continuing the Kantian & Hegelian tradition of examining how humans might be moral, free, and spiritual despite being meat computers of a largely-unknowable nature. In Heideggerland, this is expressed best in dramatic, nigh-unreadable passages like:
TL;DR: He'd probably think they're still just machines, and the question isn't really sound in the first place. IMHO!Herbert seemed to think that to be human is to be essentially 'open-ended', capable of undiscovered, indeterminate evolution, both personally and as a species.
Indeed.
We have to be careful about letting fiction inform our worldview too much. At the same time, storytelling is one of the most natural and effective ways we have to convey ideas to actually wrestle with.
What are some other fictional universes with interesting takes on AI and humans? I think the Hyperion cantos (at least books 1 and 2) are great. Stross's Accelerando also has some wild ideas but feels maybe less directly relevant.
> We have to be careful about letting fiction inform our worldview too much.
Agreed. But would our AI entrepreneurs, so busy building the Torment Nexus, pay heed?
Whenever I read the original novels, I always imagine the jihad to be a fairly benign if not peaceful exercise more akin to a protest than a war. Like if one day people started rolling across the world smashing data centers and laptops.
I was fairly surprised when his son wrote a novel that felt like a generic, bland robots run amok storyline that happened, more or less, because one guy/titan "ruined a good thing" by telling his AI to run in permissionless mode.
this is an interesting interpretation. i always imagined the jihad as being a violent struggle.
the other jihad in the first novel is that of the Fremen, which is described as being unspeakable violent, so I find it hard to imagine these to be different in that regard.
It's an interpretation if you ignore the impacts of feudalism, slavery, and religious war on Dune. The idea that people were able to express freedom of speech during a time described as the enslavement of 99.99% of the human race seems off-kilter.
the Fremen express freedom of speech despite overwhelming oppression
The desert Fremen of the first book were a tribal society where it doesn’t really make much sense to talk about freedom of speech due to the lack of any sort of formal government. But it seems likely that each naib makes his own decisions about how to run the sietch that he leads. How Stilgar is running Sietch Tabr may not be how every sietch is run.
It doesn’t talk too much about political conditions the city Fremen lived under, but the Harkonnens don’t seem to be too keen on civil liberties so we can probably make an educated guess.
In the second book, the Fremen have established a brutally repressive theocracy that is obsessed with rooting out heresy. Regular people being afraid to openly speak their minds about political issues is a plot point.
In the book - is it men vs machines? Because it’s pretty clear in our world it will be men+machines vs men(probably also + some machines)
The book starts out thousands of years after it's all over... but later books (prequels) get into into it for us die-hard followers of the series.
The "later books" the "prequels" are not written by Frank Herbert, they're written by his son Brian and SF author Kevin J. Anderson. And not like Christopher Tolkien editing his father's notes, but a completely separate series with nothing in common with the original work.
Yeah, basically.
the immediate cause of the Jihad is the abortion of the healthy fetus of a Bene Gesserit, Jehanne Butler, by an AI."
It's men vs men in charge of machines (and also vs the machines too)
You read the wrong books.
I mean this in both the specific and general sense.
Bear in mind that the Dune universe relies on heavy plot armor to make it go. Arrakis is important because it grows "spice". Otherwise it would be a useless and planet, like the Empty Quarter of Saudi Arabia. "Spice" is important because it's needed to enhance the mental powers of pilots for interstellar travel. That's a job which probably could be done by a few avionics boxes in a rack. That's what requires the Butlerian Jihad.
(Or "spice" might be synthesized. In Herbert's day (the 1960s) organic chemistry wasn't up to synthesizing arbitrary small-molecule drugs, but now it is. It's a service you can order on line, even. Biologicals take more work, but that's what genetic engineering is for.)
The "mental powers" in Dune aren't that impressive. Compare, say, the Harry Potter universe. In Dune, mentats can convince people. Bene Gesserit can look up stuff in their genetic past. Some people can multitask. There are apps for that.
Herbert wanted knife fighters to matter in battle. Not going to happen. Militaries teach knife fighting to build aggression and for desperate situations, but nobody's battle plan depends on a knife fight. Needs too much surprise. Marines say that their K-Bar knife is useful for everything but combat. Knife fighters, even with force shields, would be vulnerable to riot control weapons, from tear gas to guns that throw sticky nets. Or to being run over and buried by tanks with bulldozer blades, which the US used in the Gulf War. Herbert wrote before a big, modern, desert war.
So the Butlerian Jihad exists only in a plot-armored world specifically designed to make it work.
Plot armor refers to important characters being invulnerable. What you're describing is just plot (or "internal logic" that underlays the plot). You might think of it as "contrived plot" but that's generally how scifi works.
I don't see how most of the plot holes/unrealism you identify is connected with the Butlerian Jihad at all. But re "critiquing Dune's idea of future warfare" (shields and knives, ornithopters, etc) you might also be entertained by Bret Devereaux's recent take:
https://acoup.blog/2026/02/24/collections-warfare-in-dune-pa...
> like the Empty Quarter of Saudi Arabia
I have seen a critical analysis that sees the origin of Dune as a roman à clef of the rise of the house of Ibn Saud.
Arrakis is Saudi Arabia.
Spice, melange, is oil: the surrounding empire needs it for travel and transport.
House Atreides is the Saud dynasty, which defeated House Harkonnnen.
It’s probably been a while since your read the book, so you have forgotten some things.
For one thing, the guild navigators are prescient and guide their ships across interstellar (and even intergalactic) distances faster than the speed of light. They use their prescience to pick routes where the ship arrives safely instead of plowing through a stray planet. They also use it in a less obvious fashion to maintain the political status quo. They try to avert political crises because they like their position in society and don’t want anyone changing things (which explains the long stasis of the Empire). People did at one point use computers for safe navigation, but they were slower and they weren’t prescient. The possibility that prescient machines might one day be built is the entire reason why Leto II spends so much time breeding humans in “God Emperor of Dune”. He’s looking for anything that can be an edge against the machines, and he hits the jackpot when he discovers that one of his descendants is not prescient, but is invisible to prescients. She and her actions cannot be seen in future he predicts. She and her descendants can safeguard humanity against any form of hostile prescience, machine or alien or otherwise.
Synthesis of spice was a long–term plot element in the series. The Tleilaxu eventually master it. Keep in mind that it isn’t a small–molecule drug either; it’s an alien life form.
Mentats were intended to be a replacement for computers and databases. They were frequently administrators and advisors, and often used their capabilities to convince people. But that wasn’t the goal of their training.
The Bene Gesserit have eidetic access to every memory of every one of their female ancestors. That is actually very impressive; most people don’t even have eidetic access to their _own_ memories, let alone the ability to easily recall secret information known only to their ancestors (such as the password to a safe). The Reverend Mothers can transfer all of their ancestral memories to another, so that the new Reverend Mother has access to the memories of both sets of ancestors. Long chains of Reverend Mothers gifting their memories to each successive Reverend Mother results in the Bene Gesserit having access to the ancestral memories of virtually every race or group of people that ever existed (except the Jewish, who successfully hid themselves even from the Bene Gesserit but otherwise still exist, which is a plot point in one of the later novels).
They can also command people to act. By a trained effort of will they can tell you to get up and dance a jig, or to slit your own throat, and you’d do it. That’s an ability worth respecting, which explains why the Baron Harkonnen took care to put a deaf guard in charge of a Bene Gesserit prisoner.
> So the Butlerian Jihad exists only in a plot-armored world specifically designed to make it work.
Ultimately, however, none of that matters because this statement is what you’re really intending to argue. It’s overly reductive, and it can apply to any work of fiction. Authors naturally build or select a setting where their story makes sense. Frank Herbert (and a few others like Tolkien), went further than that and invented world–building. This excited a lot of people and sold a lot of books, and now you’re cynically dismissing it by calling it “plot armor”.
Ultimately Herbert’s point was that people should strive to be humans, not animals or machines or slaves to people with superpowers. You can’t dismiss this philosophical stance merely because he used a fictional world to illustrate it. It can be true in the real world just as it is in his books, so to refute it you will have to be more clever than that. More human, perhaps he would say.
The good news about this navigation is that the best course for almost any journey is just "straight line", with a minor deviation here and there. Its funny, but due to the emptyiness of space, even in our solar system, if you just pick a random direction, you likely won't hit anything. Trace a line to pluto from earth, and its clear 99.99999...% of the time if its not behind the sun. So the mentats maybe don't have such a tough job :)
I can make up a plausible sci-fi explanation for the need easily: the folded space might not look like our almost euclidean geometry. Maybe in folded space masses take volume proportional to the mass.
Could even envision it in some future film. E.g. when NCC-1701-N goes to warp in addition to the blur effect all the stars swell up.
> Arrakis is important because it grows "spice". Otherwise it would be a useless and planet, like the Empty Quarter of Saudi Arabia.
Any resource heavy planet/country would be irrelevant if it wasn’t for its resources, what’s your point?
> (Or "spice" might be synthesized. In Herbert's day (the 1960s) organic chemistry wasn't up to synthesizing arbitrary small-molecule drugs, but now it is. It's a service you can order on line, even. Biologicals take more work, but that's what genetic engineering is for.)
Bene Tleilax managed to clone of the main characters from his dead cells, and repeat the process for thousand years, they have face dancers who can change their appearance at will. Do you seriously think they wouldn’t synthesize spice if they could? Maybe they could, maybe they couldn’t. Without going into spoilers territory, nobody would permit to disrupt arrakis monopoly on spice.
> The "mental powers" in Dune aren't that impressive. Compare, say, the Harry Potter universe. In Dune, mentats can convince people. Bene Gesserit can look up stuff in their genetic past. Some people can multitask. There are apps for that.
Because it’s a story about human evolution, and not wizards?
> Herbert wanted knife fighters to matter in battle. Not going to happen. Militaries teach knife fighting to build aggression and for desperate situations, but nobody's battle plan depends on a knife fight.
Do militaries also teach how to use a shield that is impenetrable to conventional guns?
> Knife fighters, even with force shields, would be vulnerable to riot control weapons, from tear gas to guns that throw sticky nets.
That’s why most fights in the book and movies require element of surprise and not an outright medieval clash between armies.
> Or to being run over and buried by tanks with bulldozer blades, which the US used in the Gulf War. Herbert wrote before a big, modern, desert war.
Which would be mowed by lasguns?
you have it backwards. good world building means everything makes sense. you're basically saying evolution isn't real - it's intelligent design.
i’m not sure how to distinguish any difference from modern reality from what you seem to consider plot-armour to be.
what does the Butlerian Jihad “working” mean?
I find the entire dune series to be boring as hell for this reason. Like watching paint dry. Literally everyone in it is a slave in one way or another. No one lives better than today's modern humans. We're as readers meant to take the extremist "correct" position that of course labor is its own virtue and that therefore automating anything is wrong. Nothing in the dune universe is actually compelling. I don't care what happens to any of the characters at all. It's baffling that this series gets all this attention.
You are confusing plot armor with world-building.
I do wonder where the evolution of the human brain goes, with the widespread use of AI.
The early evidence isn’t good, but I do believe with deliberate and thoughtful training our brains our capable of much more than we currently give them credit for.
Athletes continue to break records despite manual labour being unnecessary for most jobs. And fitness culture has never been more prevalent. We're competitive creatures and this is an ability we are hardwired to appreciate. I see no reason why mental fitness won't see the same development in the future.
What I want to know is what is the "walkable city" of mental fitness? Physical fitness has the athletes, the dedicated exercisers, the unhealthy sedentary masses, and then the people who's day-to-day life integrates enough physical activity that a baseline of fitness is just natural I.e. people living in highly walkable places where both errands and free time incorporates casually moving around.
We should strive for a society where day-to-day life makes a baseline mental fitness a natural outcome as well.
Great question.
I imagine the answer looks something like Wordle writ large: Replacing passive time consumption activities with interactive ones that are as readily accessible (“walkable”).
There’s a fun scene in Star Trek: The Next Generation where LeVar Burton’s chief engineer character is on a long shuttle ride by himself. In addition to sipping a synthehol cocktail, he asks the computer to play a verbal game. It chooses to give him a quiz in his domain of warp field mechanics, and because it’s a smart AI, it calibrates the questions to be just at the limits of his knowledge.
Yes, also see chess which as far as I'm aware is more popular than ever... Despite machines ruling in this cognitive challenge as far back as the 90s.
Athletes, chess, fitness... Have you guys seen regular people? Obesity is worse than ever. Most people are overweight and completely incapable of even basic locomotion without heavy assistance. Most people these days haven't even picked up a book, let alone play chess or otherwise exercise their brains for fun. Sure, a few weirdos might keep exercising their brains, and a few more might find that interesting. A bit like the C obfuscated code contest or something. But most people are going to rot.
> Obesity is worse than ever.
Obesity is actually decreasing today, due to GLP-1 receptor agonists. We are already over peak obesity.
It's just masking the symptoms. I suppose the equivalent for the mind would be everyone on anti-depressant and sedatives. Paints quite the grey picture of our future.
> Most people are overweight and completely incapable of even basic locomotion without heavy assistance.
I'm sorry, what? Heavy assistance? The overwhelming majority of people are completely capable of walking on their own two feet. The number of people relying on "heavy assistance" for "basic locomotion" is overwhelmingly tiny, and primarily limited to the elderly
I am a proud Butlerian Jihadi.
Since the botlickers think that those of us who don't trust automatic plagiarism machines are religious nutcases anyway, why not lean into it?
You are not alone, fellow warrior!
:-D
I'm thinking of having T-shirts printed up.
On the front, "Butlerian Jihadi" in some attractively faux-Arabic font.
Perhaps on the back, in Old English Black Letter:
Grand fiction requires conflict, suffering, and war, so banning AI is a necessary plot device.
People keep putting the cart before the horse on this one, and the degree to which modern people have been lulled by fiction is quite worrying. Amusing ourselves to death.
> so banning AI is a necessary plot device.
Absolutely. Walter Jon Williams explicitly made this point in the introduction to the Praxis trilogy:
https://en.wikipedia.org/wiki/Dread_Empire%27s_Fall
He said that to make space opera work, you needed to not have omniscient AI, FTL travel, and nanotech that could work miracles, resurrect the dead, and give lifeforms superpowers.
So, how to forbid it, when someone somewhere will try very very hard to make it?
By imposing a cultural ban. His empire-building aliens forbid it, on pain of genocide.
That gave him a setting. Then, what if that empire falls? That gave him the nugget of a plot.
I found it a very effective setting, myself.
Frank never wrote about AI. Brian invented it whole cloth, enabling the grift of his father’s works.
The thing which kills me about _Dune_ (well, one of them) is that for a long while in e-book editions on Amazon Kindle the word "pogrom" was rendered as "program", quite changing the original meaning (this _finally_ seems to be fixed and the 7 instances of pogrom and 8 instances of program are correct), which had a clear and distinctive rooting in history, c.f., Lesley Blanch's _The Sabres of Paradise_:
https://bookblast.org/blog/the-sabres-of-paradise-the-book-t...
EDIT: The formatting error in the Glossary entry "Sandsnork" is still there though.
But in the last book, Duncan arrives at the conclusion that humans and machines have to merge.
After a thousand years of pause ... they didn't exactly run head first into total embrace in a single generation.
Are we talking about Chapterhouse or a KJA book here?
that's just Duncan being Duncan.
> This propelled the creators of these machines into a new technocratic class, effectively controlling the worlds of the common people.
The techno-fascists of today. We are already there.
Also, quoting the first Dune novel:
> Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.
The best treatment of AI in the Dune universe I ever found, by far, is by Timothy O'Reilly (yes, that O'Reilly):
https://www.oreilly.com/tim/herbert/ch08.html
A fun idea I've been playing around with and testing on folks: We need a neo-temperance movement.
The first reaction is a scoff and a laugh of course - wasn't the original temperance movement an utter failure, prohibition didn't work etc. But no, the movement was far wider and successful in a multitude of ways. And a new temperance movement isn't about blindly copying something old, but situating yourself around the turn of the 20th century, understanding the multitude of interconnected problems people faced, the solutions they came up with and which ones worked and didn't.
America at the turn of the 20th century was a nasty place - there was wild corruption, politics was run by mafia-like patronage networks, wealth was extremely concentrated, rich people were unaccountable, working people suffered under low-paying, dangerous jobs, women were excluded from public life, and alcohol was merely the symptom of a very sick society, and yes indeed it was a massive problem, the way alcoholism is a massive problem in Russia today.
The original temperance movement was bound up tightly with the Suffarage movement (giving women a voice), with Taylorism (running organizations rationally), with income tax (a fairer less corruptible source of income for the federal government than tariffs and excise taxes).
Today's problems are thematically the same - massive concentration of wealth is making an unaccountable elite wielding power against the wishes of the public, creating dangerously addictive digital products, massive increase in gambling, restricting the rights of poeple to own and repair products, working to disenfranchise people, working to roll back women's participation in public life, etc etc. So sex, alcohol and drugs are not an issue today, but gambling, addiction, disenfranchisement, exploitative business practices, mind-boggling amounts of corruption - they're all back in the same form and for the same reasons, and we'll need the same type of movement to fight against them.
What the progressives of the early 20th century did was not abolish capitalism, but tame it. We don't need to go on a jihad against AI, but we do need to yoke it and disempower the current slate of people who own it and direct it.
That is correct. Prohibition was a failure, but it was only one of the aims of the temperance movement.
Another outgrowth: Alcoholics Anonymous. It was a reaction to the end of Prohibition, turning the prohibition internal rather than universal.
> So sex, alcohol and drugs are not an issue today, but gambling, addiction, disenfranchisement, exploitative business practices, mind-boggling amounts of corruption - they're all back in the same form and for the same reasons
what is sex referring to in this context? is this talking about societal attitudes toward sex, sex as in gender, or something else?
"and dance and drink and screw… 'cause there's nothing else to do!"
Or, to bring it closer to home, in the words of the God Emperor of Dune:
> "The favorite poetry? The Epic. The popular dramatic ideal? Heroism. Dances? Wildly abandoned."
> "What did I take from them? The right to participate in history."
…where the Tyrant violently suppresses all of these, because such art is politically dangerous as it reminds people of the agency they've lost.
Excellently written summary.
My friend screams this every time I mention AI. I’m convinced.
You got me wondering what the Imperium of Man's official stance is on AI:
The creation of a Silica Animus is also a terrible crime under Imperial Law by the decree of the Emperor of Mankind Himself as stated in the Treaty of Mars which united the ancient Mechanicum of Mars with the Imperium of Terra in 739.M30. Tradition holds that such unholy constructs are inherently evil and a perverted abomination in the sight of the Omnissiah.
Adeptus Mechanicus religious doctrine states that the Machine Spirit of a Silica Animus is a twisted mockery of the soul of a Human, always treacherous and insane.
Ancient texts tell apocryphal stories, shrouded in metaphor, of such murderous and powerful creations during the Age of Technology, and the legions of the "Men of Iron" that served them in the great war against Humanity remembered as the "Cybernetic Revolt."
https://warhammer40k.fandom.com/wiki/Silica_Animus
Don't forget that the Imperium runs on hypocrisy. Large percentages of the Adeptus Mechanicus is breaking their own rules, including working with abominable intelligences. In fact, what they are dealing with in their main factory world in Mars is a supposedly secret abominable intelligence. And it is, in fact, a shard of an alien god that turned other races into synthetic robots. Banned in public, the center of their cult in reality, even if many probably don't understand what in the world they are doing.
So yes, anything that is supposed policy in the Imperium is supposed to be taken with buckets of salt.
Leagues of Votann and their ironkin offer IMO a thoughtful counterbalance:
Whether one is made from meat or metal, kin is kin. Upon their creation Ironkin are fully welcome into their respective Hold by the rest of the Kin.[10]
Ironkin are computationally advanced enough that they have learned to mimic the squats' social behavior
...
Though the purpose of the Ironkin is to support squats in every aspect of their lives, they are neither held in servitude nor treated as lessers. The Ironkin are simply designed to want to be helpful. However, they can only imitate the emotions of the squats, and it is rare, but not unknown, for an Ironkin to possess ambition or seek to become a leader.
https://wh40k.lexicanum.com/wiki/Ironkin
Spoken like a supreme heretic.
WOW it's strange to see my fave rabbit holes on here lol. And props for linking Lexicanum instead of the Wiki That Shall Not Be Named!
Three very quick thoughts:
1. The Votann are remnants of humanity from the time when our Terran super-AIs ('Men of Gold') had built advanced ship-based AIs ('Men of Stone') to enable long-sleep-style interstellar colonization, and the enigmatic "Votann Cores" that guide them(/are them) are strongly hinted to be those same AIs, just degraded over ~19,000 years. They were sidelined later on, after humans discovered FTL through an alternate dimension, 'the immaterium'.
2. The story is very obviously based on Dune, and like everything else in 40k is turned up to 11 and stripped of nuance for effect. Some sentient robots ('Men of Iron') "revolted" after ~three millenia of brutal, Star Wars-style slavery, but the overarching "Age of Strife" was about human wars in an non-unified galaxy, with the robots being a primary weapon; the actual revolt/jihad was a symptom, just like it was in Dune.
3. The 'Adeptus Mechanicus' mentioned above are a parody of cargo cults (religions that worship technology they don't understand) mixed with a dystopian take on the themes of A Canticle for Lebowitz, and are very much depicted with derision. Whatever your thoughts are on AI, dear reader, please never say "abominable intelligence" like it's wisdom rather than the fruit of a dogmatic & ineffective cult!
Of course, 40k lore is notoriously amorphous; some authors have said/hinted otherwise. But I think the above interpretation is the most coherent and interesting and supported overall, IMHO. For example, #2 is supported by the fact that the space elves ('Aeldari'/Eldar) ruled the galaxy at the time and seemed unconcerned by humanity's antics, which wouldn't make sense in light of an actual ASI threat proper.
All cards on the table: I don't dive too deep into the black library stuff (who the fuck has time for 15 separate 890 page novels about the horus heresy -- them shits was like 3 pages in 2nd edition Codex Imperialis) so most of my attachment to the hobby is through the wargame codices, the video games and visual media. That said, I may have fallen asleep a night or 20 to Leutin's lovely voice and obsession with the lore. to respond to your comments:
> 1. The Votann are remnants of humanity from the time when our Terran super-AIs ('Men of Gold') had built advanced ship-based AIs ('Men of Stone') to enable long-sleep-style interstellar colonization, and the enigmatic "Votann Cores" that guide them(/are them) are strongly hinted to be those same AIs, just degraded over ~19,000 years. They were sidelined later on, after humans discovered FTL through an alternate dimension, 'the immaterium'.
For me, I really love the fact that they introduced a xenos faction descendant of man.
> 2. The story is very obviously based on Dune, and like everything else in 40k is turned up to 11 and stripped of nuance for effect. Some sentient robots ('Men of Iron') "revolted" after ~three millenia of brutal, Star Wars-style slavery, but the overarching "Age of Strife" was about human wars in an non-unified galaxy, with the robots being a primary weapon; the actual revolt/jihad was a symptom, just like it was in Dune.
It's important to highlight the age of strife was brought on in part by the birth of Slaanesh and the resulting warp storms breaking down system to system connections. It weren't just our dabblings in AI.
> 3. The 'Adeptus Mechanicus' mentioned above are a parody of cargo cults (religions that worship technology they don't understand) mixed with a dystopian take on the themes of A Canticle for Lebowitz, and are very much depicted with derision. Whatever your thoughts are on AI, dear reader, please never say "abominable intelligence" like it's wisdom rather than the fruit of a dogmatic & ineffective cult!
I've never been exposed to A Canticle for Lebowitz; thank you so much for the comment.
Iain M Banks sci fi also has stories where AIs must be wiped out.
The death scene of the AI and its assassin/partner in death in Look to Windward is one of the best written things in his books.
God I wish he had written 1000 more novels. Every single one of his stories is phenomenal, even the non culture stuff. The Wasp Factory was the first of his novels I read. Imagine my surprise and elation in finding that most of his other works were absolute top tier sci-fi.
To be more specific, the machine Minds rule the Culture civilisation and the humanoids are effectively mere pampered pets.
(Several characters in the novels comment on this, so it is entirely cannon.)
The Culture fought a war against the Idirans, who eschew the use of intelligent machines. The Idirans lost.
Similarly, the Gzilt civilisation in the novel The Hydrogen Sonata prefer to run their ships with a quorum of uploaded humanoid mind states running at highly accelerated speeds.
If I take my parents on trips, buy them nice things, visit often and generally make sure they are happy and fulfilled, are they my pampered pets?
Your parents have agency that your gifts do not dilute.
Humanoids in the Culture are controlled at every level, even their thoughts as guided… sorry… “aligned” with the Minds through the adoption of the language Marain.
No humanoid ever overrules a ship Mind in the novels.
Sure, you can just ignore the source text and the author explicit statements that it's a true utopia[1].
They're an anarchistic society, no one has power to overrule any one else.
https://www.reddit.com/r/TheCulture/comments/kbtylt/ysk_the_...
This happens for lots of stuff. Some time back there's a thread where I mentioned San Junipero, the episode of Black Mirror with a happy ending - and I said some people will insist it doesn't have a happy ending and sure enough there are replies insisting that's not a happy ending. The author's opinion does not matter to them.
Just as shippers can successfully take two characters who clearly have no basis for a romantic or sexual relationship and pair them off in their head canon, likewise people can disregard the author's statements about their work and decide it's completely contrary to how they've decided to understand it.
Some people look at https://en.wikipedia.org/wiki/Fountain_(Duchamp) and they do not see a sculpture.
> Humanoids in the Culture are controlled at every level
Nope, they have considerable freedom, arguably greater than we have today. It's true that they cannot directly challenge the almost onmipotent Minds when it comes to stuff that matters to them, but Minds in Culture are an example of a very well-aligned AI, so it's almost always the case that going against a Mind's intent will just make the situation worse for everyone.
Maybe, we get Leo's Crusade instead.
Nonetheless, I was always Team Venport
so this isn't really about Dune, but there was an article in the economist this week: https://www.economist.com/science-and-technology/2026/10/14/...
It made me think about the book of the new sun. you think its half man half monster creatures are unrealistic? give it a couple thousand years. good sci-fi really is prescient
I really love Wolfe for his small almost throwaway visions of the future, which are used like an aside in a few lines and mostly inconsequential to the story.
In the Urth of the New Sun for example he has a kind of AI lock, the only function of which is to recognize a person and open only to that person. Being in that role for thousands of years, some of the locks develop a kind of personality or at least personal traits, but they are totally defined by and cannot exceed the role they were initially given, nor do they want to. It is still a haunting image for me how personality or something close to a soul might be an accidental byproduct of something as mundane as wanting to have a convenient lock. And of course you cannot stop to wonder how far we are from that.
In Book of the new sun there is the Green Man who Severian saves from imprisonment. He only appears on a few pages but we learn that he is from a distant time where mankind incorporated plant cells in their genetic makeup in order to perform photosynthesis. That brilliantly plays on two themes of the book: the sun as live giving and how a single person can be independent (or "autarch"), here in so far as producing their own energy.
There are many more instances and it’s what makes the Book of the New Sun or the Solar Cycle so fantastic. The future is so vast and strange here, that it cannot be fully depicted and ideas which another writer would stake a whole book or series on are only good for a few lines.
> there was an article in the economist this week
Blocked, for me. Not paywalled: the entire site.
So...
https://archive.ph/4ratb
One of the few sci-fi storylines that doesn't now feel hopelessly dated or anachronistic!
A related discussion from June,
Dune's Butlerian Jihad and the Future of AI
https://news.ycombinator.com/item?id=48353670
This will seem like prophecy and not fiction someday.
Maybe, but Frank Herbert's Dune books don't really explain it. The Butlerians simply exist to justify Mentats, and then get tossed aside. There are lots of similar plot points that have an "oh cool" moment but never see proper exploration.
Dune's central theme about balancing power and authority is much more prescient for our time, but I suppose we'll have to wait for Villeneuve's Part 3 before Silicon Valley discusses it.
I feel like the book explains it plenty, it's just extremely succinct.
> “Once men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.”
What else needs to be said? We see examples of this throughout human history all the time. People thought at the early onset of the industrialization that people would become freer by leaving feudalism but all it did was immiserate workers for another 130 years (immiseration still ongoing in certain areas of the world).
It's basically rentierism taken to the extremes.
The Bene Gesserit and the Spacing Guild were also a result of the Butlerian Jihad. All three started as schools of human development in the absence of powerful technology.
Also, Herbert's point about dependence on machines creating machine-dependent humans, trapped in a deterministic arc of evolution, feeds into his larger themes about open-ended evolution and its arms-race. It's not just a throwaway backstory.
Exactly.
And Herbert explores this larger theme even beyond his Dune series:
"Surprise me, Holy Void!" (says the Ship once freed)
There is actually an active "BJ" group in the Phoenix Metro area. It was about a year ago when I noticed they were slapping up custom graffiti stickers on Central Avenue (Midtown).
Then later, I saw their handiwork in person: they would take an angle-grinder and obliterate the QR Code from those rental e-Scooters, slap on a BJ sticker or three, and then leave them where they found them. This would render the scooters as e-waste, un-rentable and blocking the sidewalks.
I would not be surprised if they are quietly influencing groups which are agitating against AI Data Centers. But the BJ I have seen is a chaotic group of direct urban sabotage.
The Butlerian Jihad is a reference to samuel butler's 1872 proto-scifi satircal novel: https://en.wikipedia.org/wiki/Erewhon which talks about a society that had a machine intelligence uprising and so banned all complex machines.
This is literally spelled out in the page you are commenting upon. Didn't you read it first?
Interesting to see this on HN! I've been beating the drum for quite a while that the Butlerian Jihad was never about terminator-style robot uprisings, but rather the intentionally-unexamined half of the themes the Dune ~trilogy ended up being about: long term fate of the species, the prospect of human speciation, the role of cold reason (read: aggressive utilitarianism) in leadership, and above all the concentration of power in light of modern technology.
All he mentioned explicitly was the very last clause, but the others are more important & uniquely-Dunean IMHO, and are central to the plot from the jump through the Bene Gesserit, the Mentats, and the Guild Navigators, all ostensibly Homo sapiens sapiens despite being completely unnatural. Even "Imperial Conditioning" --which obviously plays the biggest part in the first act of the first book-- is arguably the same topic. By refusing to include computers and robots at all, the reader is naturally guided to consider those more familiar tropes in light of the world-building snippets we get. Which is kinda what makes Dune so genius, I think!
Like, to make it explicit... I think having the aforementioned abhumans drive the plot without significant support from inhuman minds raises more subtle questions than we're used to in far-future tales of robots, like:
1. Can one lose their humanity through experience, be it intentional "conditioning" or just the acquisition of power? (see: Paul vis-a-vis Eichmann in Jerusalem)
2. Once one has lost their humanity superficially, can it be brought back/up/forward? Is love/greif the most plausible path, and thus the most human cognitive processes? (see: Dr. Yueh vis-a-vis Eichmann in Jerusalem)
3. If we try to maintain our humanity in the long term --the default goal of non-insane futurists, though debated in the 21st c.[1][2]-- how do we protect against the natural urge to pursue power, which is (arguably) inherently human?[3]
I'll cut it off there cause this threat will likely be deleted anyway and I'm heavily procrastinating. Long story short: the Butlerian Jihad is not a condemnation of artificial minds, and if even it were, the condemnation would apply equally as much if not more so to the humanist empire in which the story is set.
Also, hopefully obvious, but just in case: it likely began as an excuse to write the stories that he wanted to write, just like the shield thing giving an excuse for swords in space. But I think Herbert deserves credit for what was running underneath that and elaborated on as it became a series of novels rather than cute periodical stories. He was pretty smart!
Also also: we do not discuss the son's book about Butler lol. It is verboten.
[1]: https://en.wikipedia.org/wiki/He_Jiankui_affair
[2]: https://en.wikipedia.org/wiki/How_We_Became_Posthuman
[3]: This kind of structural conflict could be described as 'internal contradiction' https://en.wikipedia.org/wiki/Dialectical_materialism or 'endogenous contradiction' https://en.wikipedia.org/wiki/Endogeneity_(econometrics) depending on your preferred flavor of economics.
Yeah, and throughout the 1970s and 1980s, most science fiction depicted the world of the future as dominated by Japan… Look at Robocop 3 :)
That's just what the zeitgeist of the time was.
Computers were then just appearing (or increasing) in the public discussion, along with fears that they'll make humans less human somehow.
Like in this Computer Chronicles video about word processing [0] from 1983 at 4:08: "Some writers are concerned that computer assistance may promote dry bland writing."
Terminator appeared in 1984, and the first Alien movie in 1979 had an antagonist android. 2001 A Space Odyssey with a disagreeable AI came out in 1968. All way before the real world had anything remotely close to today's AI: It was a reaction to the appearance of computers themselves.
That just shows that humans have the same reactions whenever a new world-changing technology appears.
On the other hand, Star Trek had Data.
Ironically, the tech that people and media were mostly optimistic about turned out to be more dystopian than predicted: The internet :')
[0] https://www.youtube.com/watch?v=Jt0OoXluC8g
"Humans have the same reactions whenever a new world-changing technology appears" is one way to see it. The other way to see it is that the current AI boom is part of the same computer revolution that writers in the 70s and 80s were afraid of, and some things happening today really do retroactively justify their fear.
Yeah, it seems absurd to be afraid of computer-assisted writing in the WordStar era, but the Turing test was proposed in the 50s. Everyone knew that human fields of endeavor would only become "more computer" as time went on. Only, if a time traveller could tell a 70s-era sci-fi writer that artists and designers were some of the first to have their livelihoods threatened by 20s-era AI, they'd probably think it was too on the nose to incorporate into their fiction.
If you look at the publication date, and read the original dune novel, it's pretty obvious that Herbert throws in a selection of somewhat randomly chosen Arabic and Farsi language origin words to add flavor to the text.
The "jihad" part wouldn't work so well in the movie, so of course they had to go with "holy war". You can't exactly have Timothee Chalemet waving a crysknife around and yelling "JIHAD!!!" in the post-9/11, post-GWOT era.
>You can't exactly have Timothee Chalemet waving a crysknife around and yelling "JIHAD!!!"
honestly I wish he did. The films are already incredibly on the nose, much more so than the books, that Paul is not to be understood as a hero. Practically half his dialog is directly talking to the audience telling us how he's going to kill billions of people. Chani got a very different role, and we start and end the movies from her perspective to stress it even further.
If you're already doing this you might as well have not chickened out on the language, it was one of the few things that genuinely irked me about the film.
What's odd is they left in the Mahdi part. Try watching it with a person who grew up in a religious twelver Shia environment and see how they react to that...
The actual linked wikia article is short on the actual detail and background of exactly why the Butlarian Jihad actually happened. I’m an avid Dune fan. Sidebar/background: The son of the Dune writer, Brian Herbert, and his cowriter Kevin J. Anderson wrote the setting, events, and conclusion to the actual Butlarian Jihad conflictt. It’s largely derided by scifi fans, but is highly entertaining. It’s actually still ongoing almost 3 decades later. And frankly not as deserving of the harsh criticism it got when it came out. But I digress…
Here’s how it went down in the dune universe (I’m certainly messing up some details, but these are the broad strokes):
Humanity has spread to the stars. No FTL involved, just good old fashioned sublight engines. The dune universe takes place about 50-60,000 years in our future. Mankind has spread across a couple dozen solar systems and everything is going peachy. However, mankind gets complacent, fat, and happy with their success and society stagnates. A handful of eager, ambitious humans scheme about how they can take it all over. They do, and then eventually start turning themselves into cyborgs, or disembodies brains in a jars that swap bodies and are effectively immortal. One of the cyborgs gets lazy and makes an AI to manage things, which promptly takes over, overthrows the cyborg rulers of humanity, makes the cyborgs it’s slaves (who have great autonomy/are conspirators) and generally rules over everything. A small group of solar systems is left under human control, who fall back to a feudal-ish type system of government with lords and dukes and whatnot.
The AI does all sorts of nasty shit to the humans in it’s dominion. Unit 931 and Auschwitz type experiences, slavery until death, etc. Untold billions, if not trillions, of humans live in animal-like conditons worked to death. The human group of systems isn’t cool with this, but they can’t do much and are vastly overpowered by the cyborgs, so a fragile peace is held. One day, the son of one of the subservient cyborgs knocks up a human slave, and they have a kid. In the background, all sorts of rabble rousing and shenanigans are being done to stir unrest amoungst the slaves by the free human system. The human systems develop FTL, with spice as a catalyst, except it doesn’t work 10% of the time and kills the pilots. The AI’s main henchman (who does all of the nazi and Unit 931-type experiments) decides it would be cool to kill the baby of the cyborg son and the slave. The slaves last name was “Butler”/ This sparks a massive revolt by the slave worlds. The free human systems then go on to fight a prolonged war against the machines, using the FTL drive at great loss of life, to begin a “Butlarian jihad” to free the humans from thinking machines. Lots of other stuff happens, and humanity eventually wins, but this is why machines no more advanced than a desktop machine are allowed in the dune universe, and why they technologically hobble themselves.
>And frankly not as deserving of the harsh criticism it got when it came out.
Hard disagreement.
My issue is not with the work on its own merits (although I think its pretty generic mass-market SF), but rather that it is wearing the skin of something it is not and to which it has no honest claim. It is one thing for an author to take a well-regarded prior work and attempt to complete or expand it. For example, Brandon Sanderson's work completing Robert Jordan's epic Wheel of Time is very respectable; he tried to complete the series as Jordan would have done. It is quite another to take the name and setting of such a work, steal the surface detail and use it to dress up one's own story that shares nothing of the underlying body or soul of the work - such enterprises are an insult to the original author, the reader, and the self-respect of the new author.
> The [slave's] last name was "Butler"
Interesting backstory about this name. Of course, it is an "occupational surname" and means "bottler" or "cup-bearer". The Butler Dynasty was a notable and very influential noble family in Ireland, beginning a century after the Norman Invasion of England.
https://en.wikipedia.org/wiki/Butler_dynasty
Now Franklin Patrick Herbert, being of Irish and German descent, and naturally a student of history, would've been quite aware of the power and influence of the medieval Butlers in history. And some of this renown was brought to American shores, as well.
In learning of my birth family and tracing our origins back to the birth of America, I discovered that my paternal line had come from Butler County, PA. They arrived in the late 18th century and homesteaded large swaths of this region near Pittsburgh. They had prospered, especially from mineral rights on the land, and the Irish Catholic immigrants had intermarried with German Catholics who homesteaded nearby. (Often, entire counties in the area were pioneered by immigrants from a certain part of Europe. And the intermarriage was a natural result of sharing a common faith.)
So for several years before 2024, I was known as the guy in genealogical circles who was hunting down information about Butler County, and I was fairly proud of finding my family roots in that county, and I mentioned it all the time in the Irish pub and everywhere else. Well, imagine my shock when the little incident happened on July 13th, 2024. I have been more reticent to discuss Butler County since that time.
Anyway, I find it interesting that Herbert chose that particular name for the "Anti-AI crusade". And that is partly why I chose the moniker for my own, despite not being particularly anti-AI (remember, all my comments are generated by a neural net), nor having read any of the books, or seen the films. (I did attempt to watch the Lynch film, but it really did not hold my interest! So sue me!)
I was born in Butler!
More interesting backstory into the name: it's actually a reference to https://en.wikipedia.org/wiki/Samuel_Butler_(novelist) and his 1872 proto-scifi satircal novel: https://en.wikipedia.org/wiki/Erewhon which talks about a society that had a machine intelligence uprising and so banned all complex machines.
This gets mentioned on HN at least 5 times a day.
Highly relevant then. HN tends to discuss the things that are either meaningful today or what they are building in their garage that is often meaningful in 5-10 years.