I think it's funny how goblin mode this whole hack is. The memory scanner itself was clearly written by an LLM (using python??) and the way this person goes about hacking the game is very non-reverse engineer but instead someone equipped with very capable tools. No shade to the dude to be clear, I think it's sort of incredibly how possible this stuff is now due to LLMs and doesn't require someone to know how to use Ghidra.
That's hilarious that you give the villagers thought and the first thing they want to do is overthrow Tom Nook. If this works in Gamecube on an emulator, surely it'd be possible to make it work on the Switch emulators as well?
LLMs are heavily trained on reddit, and "Tom Nook is a capitalist dictator if you think about it" is a reddit meme. Like many LLM reactions, it's less "thought" and more "regurgitating reddit jokes/opinions".
There is something to be said for a comment talking about "regurgitating reddit jokes/opinions", whilst falling into a major bug bear many have with Reddit, commenting before reading the source...
Thankfully, I am in my 30s, and I don't have to interact with anybody in real life who mostly regurgitates Reddit jokes. Either I'm lucky or I've made some good choices. Probably the former.
Now I'm kinda curious what age cohort is most likely to be Reddit memers
Do you think 30s is peak Reddit, yet you manged to be a lucky outlier? Or that peak Reddit skews older/younger and you're of a lucky age?
As an older Millennial in my 40s, I see this a lot in my 35-43 friend group. And always figured peak Reddit was younger Millennials (now in their 30s).
How can you tell? Either you're saying that LLMs have no "thoughts", and just regurgitate everything, which doesn't explain how they can reason in novel problems, or you're saying you can tell what's original thought and what's regurgitation, in which case I'll ask "how do we know your comment isn't you regurgitating an HN opinion"?
> Either you're saying that LLMs have no "thoughts", and just regurgitate everything
I am exactly saying that they have no thoughts or even "thoughts".
> which doesn't explain how they can reason in novel problems
There are many decades of pre-LLM software that can solve novel problems. Thought (or novel thought) is required to reason, but it's not required to solve a problem.
For example, there are exhaustive algorithms that can solve novel equations and even complete simple mathematical proofs, but they don't need to think.
> "how do we know your comment isn't you regurgitating an HN opinion"?
You don't, and I don't care if you do or not. The value of my comment isn't its novelty or whether it's truly reasoned, which is why LLMs sometimes do create valuable output.
In fact, the output of a reasoning machine (whether a human brain or true AGI, sometime in the future) isn't deterministic. A non-reasoning machine and a reasoning machine could create the same output.
The reason I know LLMs don't have thoughts is because I use them many times every day, and they are very clearly pattern machines. They don't even begin to seem rational, human, or knowledgeable. It's sometimes possible to find near-verbatim sources for their outputs.
As far as I know, no current gen model AI "thinks". It's all processes trained against a body of work generated by actual thinkers along with a bevy of smoke and mirrors to fill in the gaps.
It's amazing that it's as good as it is given how far it is from thinking, but if you threw something actually novel at it, all it would do is confidently word salad a response.
Generative AI can reason in novel problems in an analogous way that bands continue to make new melodies even after all these years using the same instruments as all the other bands in their genre.
Would require decompilation of the Animal Crossing game code for the Switch. I believe DRM has gotten a lot better since the Gamecube days as well. Hypothetically possible maybe but good luck haha
I actually think now that I've gone through the process, memory scanning and writing will be enough... Except, they probably have different control codes that I'd need to reverse engineer.
You should be able to run Cheat Engine on your emul*tor of choice to tweak New Leaf "and newer" titles.
And if you're a stickler for pissing Nintendo off in very specific ways, LayeredFS + Atmosphere opens up some modding opportunities right on the console itself. Not sure how easy it would be to pull something like this off though...
I think my 'old man shakes fist at clouds' thing is this. The social media platforms that censor you do it to make your content easier to sell ads against. It's actual corporate badthink correction that is rebuilding the English language. STOP VOLUNTARILY DOING IT WHEN YOU DONT HAVE TO. You should not sacrifice your free thought on the altar of quarterly results. Say the whole fucking word.
I doubt a star will make Nintendo lawyers go "ow nose, they didn't spell out emulator in full, we can't attack them! Damn those star armors!". I don't think it changes anything technically.
The only thing this kind of censoring does is countering basic censor bots I think, and somehow making swear words publishable in the US.
Look; if you want to risk your chipper little lifestyle explaining the various ways to run New Horizons, be my guest. I've insinuated enough already, anyone who cares about the discussion on-grounds wouldn't have anything else to ask.
My point is that you don't take any more risk if you write emulator instead of emul*tor. You already took the risk explaining what you explained. One character swap doesn't change anything to this.
I'm insisting because if you care about not being sued, the stars are not an adequate defense despite what you seem to believe it is, and false sense of security is dangerous.
Not that I think that what you wrote here is remotely likely to cause you troubles, but it won't protect you the day you actually document something illegal.
To rub it in:
> I'm covering my ass
No, not at all, and it's important that you realize this.
Low-level emulators can be legally identical to a virtual machine, but often isn't. Most modern consoles can't be emulated that way, and most require you to dump a bootrom from your own console hardware, alongside game keys and other dubious digital paraphernalia.
Switch DRM, running custom code and interacting with RAM is a solved problem on the Switch (1). There are some really impressive mods, like a multiplayer implementation for Super Mario Odyssey
This is awesome. LLM-powered NPCs is one thing I'm most excited about in the future of gaming. Characters repeating the same scripted dialog over and over again is one of the biggest immersion breakers.
I'm unsure how useful they'll really be honestly! In many situations its very useful to know when your "done" talking with an NPC when they start repeating lines etc...
There could probably be cool uses but I don't think it will be a pure "upgrade" as the repeating dialog is kind of a feature honestly.
It would require a completely different design to video games.
Current video games are designed around streamlining content. As a player, your job is to extract all content from an area before going to the next. That's why most areas are designed as linear corridors so that there is a straightforward progression, and most NPCs interactions are meant to offer something meaningful so as to not waste the player's time.
But imagine if interaction with NPCs wasn't just a content delivery mechanism, but instead could sometimes be rewarding, sometimes useless, dynamically adjusted in how you interact with the world in non-predictable ways.
The player would just waste their time in their usual approach of canvasing each new area, which would become unsustainable. There would be no reliable way of ensuring you've extracted all the content. All he/she could do is roam around more naturally, hoping the glimpses they catch are engaging and interesting enough.
Maybe a new player skill would be to be able to identify the genuine threads of exciting content, be it designed or emergent, within the noise of an AI-generated world.
Realistically though, how do you build an exciting player experience with this framework? A starting point might be to approach it as something more akin to LARP or improvisation theater, you'd give each NPC and player a role they need to fulfill. Whether players actually enjoy this is another thing entirely.
> But imagine if interaction with NPCs (…) instead could sometimes be rewarding, sometimes useless, dynamically adjusted in how you interact with the world in non-predictable ways.
That’s a slot machine, and the same mechanism which also gets us hooked on social media. Sounds like something which would immediately be exploited by vapid addiction-as-a-feature games à la FarmVille.
> The player would just waste their time in their usual approach of canvasing each new area, which would become unsustainable. There would be no reliable way of ensuring you've extracted all the content.
Sounds frustrating. Ultimately games should be rewarding and fun. Constraints are a feature.
> All he/she could do is roam around more naturally, hoping the glimpses they catch are engaging and interesting enough.
Good reminder to go take a walk outside. Take a train to somewhere we haven’t been. Pick a road we’ve never crossed. We don’t even need a mini map, and sucks that we don’t have teleportation back to base, but we do have a special device which always points the way back.
> Realistically though, how do you build an exciting player experience with this framework? (…) Whether players actually enjoy this is another thing entirely.
Agreed. Though not enjoying it and abandoning it is fine, I’m more worried about people not enjoying it but feeling unable to quit (which already happens today, but I think the proposed system would make it worse).
>> There would be no reliable way of ensuring you've extracted all the content.
> Sounds frustrating. Ultimately games should be rewarding and fun.
this seems to assume that the only way to feel rewarded / have fun is by comprehensively extracting content from the game. in order to have fun in an "emergent" generative game of this nature, you'd need to let go of that goal.
i do agree with the risks surrounding engineered engagement.
> this seems to assume that the only way to feel rewarded / have fun is by comprehensively extracting content from the game.
Not my intention, that is not something I believe. I’m not a completionist (I get those who are, but to me it can get boring or stressful) and I see the appeal in sandbox games (even if I don’t usually play them).
> Some games are designed around content and "extraction". Many are not.
While I think the parent post leaves a lot of open ended questions, I think they are spot on about the tightness of design in games.
In many open world RPGs, or something like GTA, you cannot open every door in a city.
In street fighter you can't take a break to talk to your opponent.
In art games like Journey you cannot deviate from the path.
Games are a limited form of entertainment due to technical and resource restrictions, and they always will be. Even something as open ended and basic as minecraft has to have limits to the design, you wouldn't want the player to be able to collect every blade of grass off of a block just because you could add that. You have to find the balance between engaging elements and world building.
Having a LLM backed farmer in an RPG that could go into detail on how their crops didn't grow as well last season because it didn't rain as much seems good on paper for world building. But it is just going to devalue the human curated content around it as the player has to find what does and does not advance their goals in the limited time they have to play. And if you have some reward for talking to random NPCs players will just spam the next button until it's over to optimize their fun. All games have to hold back from adding more so that the important parts stand out.
There are many different types of NPCs. Consider e.g. Minecraft villagers, that doesn't have a story-purpose, but do serve functions such as trading that could also be augmented by dialogue details.
But even for story-driven games, you can signal when you're "done" extracting story-related details in various ways, by e.g. prompt the NPC to include dialogue element A,B,C when it fits the conversation, keep track of which were output (you can make it output a marker to ensure it's easy to track even if the dialogue element has been worded differently), and have it get annoyed and tell you it doesn't have more to tell you or similar as the repetition adds up.
> In many situations its very useful to know when your "done" talking with an NPC when they start repeating lines etc...
But that's not how real life works at all, right? You talk to someone for as long as you want to talk to them, or until they start sending signals that they are done talking with you.
The way video game dialog works has always bothered me, it makes characters feel stilted and makes me care less about the characters and the world.
(Although it's a different game in many ways, consider by contrast how Portal 2 handles dialog, and the effect that has on immersion.)
"But that's not how real life works at all, right?"
How real life works is always a plausible interesting goal, but it's very often at odds with a bunch of other valuable goals for players.
A particular sharp example of this is sports video games. It might well be interesting (and certainly realistic) to simulate bad referees in a sports game. Horrible blown calls by tennis line judges, or missed calls by basketball refs, or bad umpire calls on pitches. Real-life soccer makes working the refs and their inability to see everything an art form, as far as I can tell.
Perhaps that's interesting, but the irony here is that real life refs are actually bad simulations of the original perfect game code in the first place, from a certain point of view. I think debates about the use of instant replay in sports gets at the heart of this, and one could imagine using real-time AI to help refs taking this conversation much further.
I think the sports case is a particularly sharp example, but it definitely holds with all sorts of choices in games.
For Animal Crossing in particular, I remember when I finally played it, it struck me after a while how much it had in common with recent MMOs (Everquest and World of Warcraft) that I had had fellow game developer friends have their lives severely disrupted by. And when I played the original Animal Crossing, I remember noticing specifically how careful the designers were in having players use up every bit of interesting content in a day after 45 minutes or an hour, so that eventually you'd run out of things to do, and that was the game's signal to put it down and pick it up again the next day. And I remember being struck by how intentional it was, and how humane it was... particularly given their goal of wanting to make a game that was asynchronously coop (where different family members could play in the same shared space at different times of day and interact asynchronously). As a game designer myself, I really respected the care they put into that.
Anyway, that's my immediate thought on seeing this (fascinating, valuable) experiment with LLM dialogue in Animal Crossing. The actual way NPCs work in these games as they are has been honed over time to serve a very specific function. It's very similar to personal testimonials by paid actors in commercials; a human expressing an idea in personal dialogue form triggers all sorts natural human attention and reception in us as audience members, and so it's a lot more sticky... but getting across the information quickly and concisely is still the primary point. Even dialogue trees in games are often not used because of their inefficiency.
I totally think that there will be fascinating innovations from the current crop of AI in games, and I'm really looking forward to seeing and trying them. I just think it's unlikely they will be drop-in replacements for a lot of the techniques that game developers have already honed for cases like informational NPC dialogue.
I don't mean to imply that realistic is always better, just that there are other ways to figure out when to stop talking to someone. And I think the current method is actually quite bad for immersion and building empathy in the player.
> But that's not how real life works at all, right?
oh of course! Sorry, I was never trying to imply that that it was in any way realistic. For video games often the most fun / compelling choice is not the realistic one! Striving for realism can be a great goal and often has a lot of positives, but it is often limiting. Video games are just art, being photorealistic can be beautiful and amazing but is often not the best choice for expressing an idea.
That seems a bit like deck-chairs on the Titanic. The hard part isn't icon design, the hard part is (A) ensuring a clear list exists of what the NPC is supposed to ensure the user knows and (B) determining whether those goals were received successfully.
For example, imagine a mystery/puzzle game where the NPC needs to inform the user of a clue for the next puzzle, but the LLM-layer botches it, either by generating dialogue that phrases it wrong, or by failing to fit it into the first response, so that the user must always do a few "extra" interactions anyway "just in case."
I suppose you could... Feed the output into another document of "Did this NPC answer correctly" and feed it to another LLM... but down that path lies [more] madness.
You prompt the LLM to point out that the clue will be added as is to the conversation, but for the LLM to include a marker instead of the actual text to ensure that actual critical details are included unchanged.
EDIT: Also, having the LLM botch a clue occasionally could be a feature. E.g. a bumbling character that you might need to "interrogate" a bit before you actually get the clue in a way that makes sense, and can't be sure it's entirely correct. That could make some characters more realistic.
No, this is the Einstein/student model that has been proposed for improving LLM output quality.
Basically you have your big clever LLM generating the outputs, and then you have your small dumb LLM reading them and going “did I understand that? Did it make sense?” - basically emulating the user before the response actually gets to the user. If it’s good, on it goes to the user, if not, the student queries Einstein with feedback to have another crack.
Repeating the last line of dialogue is not just a way to indicate that there's no more dialog, it often also works as a remainder, giving you the most important kernel of information ("You should go to [place] and talk with [npc]"), in case you come some time later and forgot what you were supposed to do. You can indicate there's no more dialog in many ways, but you'd lose that secondary feature. Same thing if the NPC just keeps babbling generated drivel.
So true.
In such an LLM-driven game though, I would imagine the player would just ask the NPC: "I forgot what to do" or even "Can you explain it in other terms?" (if the quest description isn't clear enough).
Feeling it would work best in more of a Dwarf Fortress approach game where it's more you have a sandbox with rules that cause a simulation to have emergent gameplay.
I would say local LLMs are already good enough for this. It might be an issue that the GPU is already quite busy at the time it's needed for LLM inference, but not an insurmountable issue.
I wouldn't ever want a game to use it for the core story writing, because it's pretty important that it is consistent and unable to be derailed. But for less serious NPC interactions or like an RPG scenario it is such a great fit.
I also wouldn't want a single player game to rely on remote inference, because that will get turned off eventually and then your game doesn't work.
I think generally, games should move to using the GPU less for graphics and more for computation. Not just AI computation - those fancy GPUs are a big resource that simulation games could be taking advantage of and just... aren't.
(Yes, this is a Paradox callout. Give me less fancy particle effects in Vic3 and use the GPU for computing pop updates faster!)
(Probably the biggest barrier to this is the lack of a convenient C++/C#-level cross-manufacturer compute API. Vulkan is a bit too low-level for game devs to work with, OpenCL kind of sucks, and CUDA is NVIDIA-only.)
There's a number of little games and a major mod for Skyrim that implements this. I've spent a bit of time playing with the Skyrim mod.
There are some significant issues with it at the moment. One is that you have to train on vast swathes of text to get an LLM, and it's difficult after the fact to remove things after the fact. If you cooperate with the AI and stay "in Skyrim" with what you say to them it works out OK, but if you don't cooperate it becomes clear that Skyrim NPCs know something about Taylor Swift and Fox News, just to name two examples. LLMs in their current form basically can't solve this.
The LLMs are also prone to writing checks the game can't cash. It's neat that the NPCs started talking about a perfectly plausible dungeon adventure they went on in a location that doesn't exist, but "felt" perfectly Skyrim-esque, but there's clearly some non-optimal aspects about that too. And again, this is basically not solvable with LLMs as they are currently constituted.
Really slick experiences with this I think will require a generational change in AI technology. The Mantella mod is fun and all but it would be hard to sell that at scale right now as a gaming experience.
It's solvable it's just a lot of work. You need to have guardrails, i.e. multiple sanity-check passes (maybe there are other techniques, but I found this to be most effective)
One concrete example I'm sure these Skyrim mods aren't using is: enums in structured outputs [1] with a finite list of locations/characters/topics/etc that are allowed to be discussed. The AI is not allowed to respond with anything that is not in the enum. So you can give it a list of all the locations in the game in a huge array and it would be forced to pick one.
It seems like a guard model paired with RAG could help here. A guard model could filter out references to current events or anything outside the Skyrim universe, while RAG could be used to ground the NPCs dialogue in actual in game content. That way if the model tries to spin up a dungeon or location, it first checks against the game's data to confirm it actually exists before surfacing it to the player.
Feel like they could probably be used in text based adventures. Doki Doki Literature Club, despite the silly name, is a very unusual game that might really benefit from this.
This is something fine tuning should be able to improve. The main caveats are the same as always: dataset collection, labeling, and training/experimentation time.
Improve, yes, fix, no. I was actually already using a model fine-tuned for Skyrim.
I didn't go into it in detail, but it isn't even that I got the NPCs to start babbling about Taylor Swift. What is was was just that they knew that she was a musician, and as such, might be at the tavern. That's very hard to remove.
Yep. I can see how relying on non-repetitive dialogue generated by LLM's will inevitably make us lose common ground scenarios that end up being memorable moments, like the arrow to the knee from Skyrim or "such devastation! this was not my intention" from Final Fantasy 14. A way to bypass this problem is to keep the important dialogue fixed, but this only one of the problems.
Another issue would be emphasizing the meaninglessness of the dialogue. For example, playing Trails in the Sky has lots of NPC dialogue that's repetitive, but at least the dialogue is relevant with how the NPC's life progresses in the grander scheme of things, such as having difficulty with her entrance exams, or having an argument with his fiancé. It's not main dialogue but adds flavor for anyone who cares about the world enough to interact with the citizens.
I don't think I'd like to interact with characters that I know whatever it is they have to say is generated on the fly and adds nothing other than random tidbits. The novelty would quickly wear off.
It doesn't need to be random tidbits. You can add "life events" for the character to the prompt, and steer the responses heavily toward the character wanting to talk about those. With an LLM in the picture, you can even have the character remember past interactions with you, and expand on it different ways depending on how you interacted with them.
A better application of LLMs could be merely using them as a text parser in text adventures. NPCs could continue to have some hidden, hard coded information or abilities, here stored in the context window, and the LLM is used to provide that information to the the player he (or she) puts in roughly the right words. Rather than exactly the right words.
Instead of generating truly random dialogue, you could choose to only generate random seeds within a certain number range, so that the probability that two players seeing the same line of dialogue increases, while still having enough variety for it to feel "random" to a solo user.
It's the whole debate about carefully created vs procedurely generated. LLM created dialog with be a vapid, vacuous and sterile as starfield was, or No Mans Sky
> will inevitably make us lose common ground scenarios that end up being memorable moments, like the arrow to the knee from Skyrim or "such devastation! this was not my intention" from Final Fantasy 14.
GLaDOS from Portal would offer one player pudding and another one a steak. You get to a wall which says “the ravioli is a fraud” and become utterly confused.
I agree. An important aspect would be lost. It would be as interesting and relatable as sharing that dream you had last night with your friends. “Uh huh. Cool story bro.”
> LLM-powered NPCs is one thing I'm most excited about in the future of gaming
Incoming new type of health crisis: video game addiction coupled with LLM-induced psychosis. Dudes spending 12 hours a day farming gold in a MMORPG while flirting with their AI girlfriend sidekick
I spent a lot of time on ML driven dialogue and natural language understanding for games in the pre-llm days working with some folks building Xbox, PC, and VR/AR games, and I'm excited about the potential here. The major barriers to shipping in a production game though are still the same imho:
1. (predictability) Games like to have a clear arc and tend to use at least some of their NPCs to move that forward, it is harder to do this with a model that could make a choice you don't predict. They tend to have a set of items, quests, what-have-you that you need boundaries around.
2. (testing) Games like to test like crazy before launch, at least the AAA ones, so their QA folks just don't like a model that can have infinite responses/variants. Many then drop to a skeletal crew for maintenance and improvements after launch, where with ML models you actually need to keep improving the model, finding long tail bugs as more players interact with the system, etc.
3. (cost) Games are usually very cost aware, it's far cheaper to just have a set human-written dialogue path, then to run a model, even an offline one. Cheaper in both actual dollar costs if you're talking about a high end LLM service call, and CPU/GPU/memory costs if you're talking an on-box system.
4. (internationalization/localization) AAA games need to launch fast to many languages and locales, using a model for NLP and dialogue management/natural language generation adds added testing costs for each new language, that is just a very cheap translation normally that can be outsourced.
There have been some fun experiments in this space, and I expect to see this improve and become common use in the future, but it will take time and more work on how best to integrate a model into the flow of a game. I do love it for "presence" so talking to NPCs feels more human-like.
Nick and the crew at AI Dungeon (and related) have always done some interesting work in this space, trying out games where AI can be used in interesting ways.
I would love to work on this. I have worked on this a ton and have really strong ideas for some.
1. I think people assume you have one LLM per character, but I think if you had specialized ones for each quest, item, etc.., this would actually work quite well.
3. I actually think if you cached responses under certain conditions, costs can be saved significantly. This would require quite a robust context, though, to still feel dynamic.
IMO this isn't going to be fun for most once the novelty wears off. Games are goal oriented at the end of the day and the great games are masterfully curated multi-disciplinary experiences.
I'd argue throwing a game wrapper around an LLM is a new LLM experience, not a new game experience.
I think the people that are for this development are thinking more along the line of a game designed from the ground up to include AI LLM interaction.
Once someone decides that will be a critical and fundamental part of a AAA game, the rest can be worked out despite what will be I am sure many unintended emergent behaviors.
You should check out https://www.meaningmachine.games/ then — they're building games with weird dynamics centred around this new possibility of LLM-powered NPCs
> Characters repeating the same scripted dialog over and over again is one of the biggest immersion breakers.
The real immersion breaker and the holy grail of RPG is the fact NPC have no life or goal outside of what the player does. Imagine a game where NPCs have wants and goals and do things to get those done. Where you could leave it running for 10 years and things would have happened without you.
That’s a nice experiment, but it’s a life simulator and not really a game. There’s little fun in working your way through the first two dungeons and then getting out and being told “no need to go on your epic quest anymore, someone else from the last town beat God and now we’re at peace. Oh, and they got all the powerful artefacts along the way, so there’s literally nothing of interest for you to do anymore. Unless you want to help us plant our crops. That’s what you wanted when you picked Final Fantasy, right? To play a poor version of Harvest Moon instead.”
I find myself somewhere in the middle. I think for the vast majority of games this would largely be a negative. Anything heavily story driven this would just cause issues and I think would largely ruin the experience.
I think sandbox games like animal crossing are the exception, if it ever becomes reliable enough to put in anything other than an M rated game.
Animal Crossing, the Sims, Cult of the Lamb, and similar games would infinitely extend the life of the game. But I am sure we can all already imagine the headlines when these "family" games start saying things that they really should not be... especially given recent issues.
For the record, not dismissing the person who did this work. But doing this commercially has its risks.
Extremely interesting use case. LLMs as a modding tool to recontexualize virtual spaces. I can see this being a tool used for artistic intervention in the same vein as plunderludic tools like Unity Hawk which allows you to run emulator save states in Unity3D.
https://plunderludics.github.io/tools/unityhawk.html
There's something about taking old games and injecting new life into them that just seems so fun and exciting! Also very interesting to know that the Animal Crossing codebase has been decompiled into readable C code. Fascinating! So many opportunities to mess with it.
This is amazing. Well done figuring out the shared memory hack. That's a huge time saver and really makes sense from an engineering perspective. Ten dollars for the hammer and five thousand for knowing where to strike the machine, to paraphrase a trope on expertise applied correctly.
It needs to be a global variable in C. A variable in a function scope or runtime malloc'd spot? No chance of finding a stable spot.
Thankfully, a lot of old games love to use global variables because you can't run out of stack space or allocatable memory. Modern games shy away from that because even the tiniest consoles these days come with gigabytes worth of RAM, even a memory leak has to be a gigantic firehose to bring the system to a halt.
This is a fairly "common" technique for doing multiplayer randomizers. Typically on a smaller region, but you end up with some sort of scratch memory being used as a buffer for messages sent back and forth between the emulator and the network layer (or your flash cart's RAM and the network layer, in the case of FXPakPro!). It's super useful.
The idea of giving every character this sort of agency and seeing what opinion builds up about the world is incredibly fascinating.
Depending on how well we assume an LLM would do at this task, it’s an interesting way to see what “real people” would think about a very hypothetical situation.
> The game runs on a Nintendo GameCube, a 24-year-old console with a 485 MHz PowerPC processor, 24MB of RAM, and absolutely no internet connectivity.
In fact, Nintendo did release an official add-on called the Broadband Adapter, which plugged into the bottom expansion port and provided an Ethernet jack. Only a handful of games supported it, one was Phantasy Star Online. I also used it to stream games/roms from a PC. This worked by exploiting a memory vulnerability in Phantasy Star Online to load arbitrary code over the network, though with slower load times compared to running from disc.
Yes, the GameCube had an official Broadband Adapter (BBA). But Animal Crossing shipped without networking primitives, sockets, or any game-layer protocol to use it. Using the BBA here would have required building a tiny networking stack and patching the game to call it. That means: hooking engine callsites, scheduling async I/O, and handling retries/timeouts, all inside a codebase that never expected the network to exist."
I didn't finish reading the article before commenting. Mea culpa!
Maybe it would be possible to use Phantasy Star Online's network stack via the streaming exploit.
But that would leave the hooking part.
Many years ago there was a project in which researchers swapped localization strings to turn The Sims and Grim Fandango into language practice experiences. Generative dialog that is contextual and skill-appropriate dialog could also be a killer application for language practice.
This excites me. Everyone wants to learn a new language, no one wants to slog. But I'd pay money to play Grim Fandango, and get the chance to ask "What does Morte mean?" and have the LLM, in real-time, tutor me. And if I get bored, I can game on, business as usual.
But just give the AI a list of words I currently know, and it can slowly slip in more and more vocabulary, and maybe full sentences.
To me LLM are one of the greatest advancement in gaming. I don’t understand how there isn’t already any big game that’s using it to have ‘live’ npc.
It would make single player game so much more alive, even railroaded one like red dead redemption because the npc could adapt more easily to what you just did, how you behave in game and so on.
Games are already demanding on gpu, running a very tiny local LLM and only when people interact with npc wouldn’t require so much power.
I’m sure there are issues which explain why it’s not already being used but the first one to do it will be remembered.
I wanted the same, so I hacked/bolted on an LLM to Morrowind (the open source recreation OpenMW).
The biggest problem I faced at the time (during ChatGPT 3 era) was that, without a good context, LLMs are the most vanilla roleplayers you’ve ever seen. By themselves, LLMs are just not interesting enough for a player to choose to talk to in-game.
If you want them to be “interesting” to talk to, you must provide (or generate and keep track of): a backstory, chat history, the scene, NPC inventory, the NPC’s current emotional state, the weather, literally everything needs to be given to the model before it generates messages for the player.
At which point you’ve got a big task. You need a way to automatically get the relevant data to the model for the specific conversation you’re having. There might be tools to pick appropriate text documents from a db given a conversation topic, but I didn’t/don’t know how to make that work for games.
I’m sure there’s a way to accomplish this with more modern tools and models. (Maybe instead of providing all that data up front, you would now give the model tools to call to retrieve that data on-demand?) But that’s what made me give up in 2022.
Of course it would require the game engine to provide context to the LLM, it certainly is quite a bit of work but nothing technically impossible and the result are endless possibilities and endless replayabilities. You could very well let the LLM decide if an npc should agree to give an object to the player or not, even let the player try to convince the npc.
I mean it looks like to me the next big step in gaming after 3D and yet this is being ignored.
But also, why couldn't you look at the code to find the addresses used for dialogue? If it's already disassembled I would think you could just look at the addresses used by the relevant functions/set a breakpoint etc.,?
This feels like the future of gaming: community-driven mods where AI brings infinite new dialogue and quests. Imagine if Nintendo leaned into this instead of fighting mods.
It always felt both "a cheap shot" and "valid" to express dismay that characters in video games don't react when you do things like jump up and down on their table.
While it's impossible for game developers to write code to cover every situation, AI could make general reactions possible.
It's surprising that really simple things like this haven't been tried yet (AFAIK!). Like, even if it's just the dialogue, it can still go a long way.
The cost benefit is really poor, but I also wonder if it's just never been done well.
Old text adventures honestly did this heaps better than modern games do, but the reality is there was a more finite action space for them and it wasn't surprising when something wasn't catered for.
Which games are these? I’d be interested in checking them out.
I’m only aware of experimentation in making more “difficult” NPC AI which was found less enjoyable for obvious reasons, so would be interested to see why similar but different attempts down another path also failed.
Kingdom Come Deliverance 2 is one of the closer attempts, NPCs will react to lots of things about you and behavior, like if you smell bad or stare at them for too long
I was so surprised in BotW and TotK to see NPCs duck, huddle, gasp and otherwise react to odd shit you might be doing. Also in dialogue, do contextual things like talk about the weather and time.
I would love to see a Zelda game implement LLM dialogue for random inconsequential dialogue or relating dialogue to the game context.
Outside of holidays and special events, they do indeed choose randomly from a dialogue bank. There’s actually multiple dialogue banks, one for each villager personality type.
Doesn’t the GameCube already have a network interface? Like something called phantasy star online used it for internet, and you can connect multiple GameCubes for 16 player Mario kart I believe.
One thing I didn't quite understand from the writeup is how do you how much space you have in the memory "mailbox" for text. What is your maximum text length?
This is so cool. It opens up a whole new dimension to messing with old video games where AI can change things like how characters move, loot box contents etc.
In terms of changing loot box contents, that has been going on for at least the past like 7+ years through "randomizers", though I think mostly used by speed runners.
That's super cool! Really funny how quickly they turned into stereotypical country dwellers: mainly spending their times gossiping about each others or getting all worked-up on right-wing propaganda about places they'll never go to.
This is what I and I'm sure many other gamers/gamedevs thought when LLMs/ChatGPT first came on the scene:
Dynamically-generated dialogue. Device-local models designed for games could be trained on the game's lore and given a different style for each NPC, and react to the player's random actions instead of blurting out canned responses each time.
I'm sure if the genre-starting veterans like Richard Garriott or Roberta Williams were still active in the industry they'd definitely want to use something like this. Imagine a King's Quest or Monkey Island where you could literally type or try anything and get a relevant response! instead of just "nope"
I've floated that idea around in various internet comments and am usually met with the typical AI=BAD response. It would work so well in a CRPG type game, but I think a lot of folks would review bomb a game like that just because
> It would probably creep into my suspension of disbelief and mess with my enjoyment.
You mean trying some interaction that should be plausible in a point-&-click adventure game but getting told "I can't do that" because the devs didn't think of it, or typing some slightly different grammar in a text adventure game and the parser not understanding common words, or getting the same responses over and over in an AAA+ RPG once the prewritten dialog tree runs out, does NOT creep into your suspension of disbelief?
Honestly? No, it doesn't, but maybe suspension of disbelief isn't the right wording.
Typically, when I play games, I build up some sort of mental model of how the game's meant to function, and how the game's creators want me to experience it. eg: running into useless/limited dialog trees with an NPC tells me it's not important. Not being able to do something tells me that thing isn't relevant (although sometimes, I guess it is fun to try something fruitless for a while). It doesn't have to be perfect but it helps if it's consistent.
What I understand about programming and game design leads me to accept there are going to be limitations encountered and tradeoffs made, so they rarely bother me beyond the first time I experience it.
With a more open-ended, AI-driven experience, I think I'd just end up wondering if the interactions I'm getting make sense, if they should be taken seriously, etc. I do think games have a lot of artistry (design, assets, writing, etc) behind them, and I suspect having "live" AI-generated stuff would distract me by making me wonder if I'm actually getting the right vision.
It's all speculative, of course. Maybe I'd try it out and find it cool. It's certainly not something I think shouldn't be attempted. :)
I think the solution to your hang-up, which doesn't exist yet (AFAIK) is an LLM that is fine-tuned specifically for strict narrative context. The NPCs would then be trained on its own lore, and prompted accordingly. Hallucinations would for sure completely ruin the immersion, break the fourth wall, etc, so this thing would need to have some level of deterministic control. I'm hopeful for the immediate future though, seems like something that is totally doable even if it isn't 100% perfect at first.
Live commentary in sports games is a place where I think LLMs could shine, but I have no idea if they could perform, time wise, to the point that it would sound right.
Here's the code: https://github.com/vuciv/animal-crossing-llm-mod
I was intrigued as to how it would intercept a conversation and then pause the game for long enough for the LLM to return a response, so I used https://gitingest.com/vuciv/animal-crossing-llm-mod to dump the 40,000 tokens into Claude Opus 4.1 and asked it: https://claude.ai/share/66c52dc8-9ebd-4db7-8159-8f694e06b381
The trick is the watch_dialogue() function which polls every 0.1 seconds and then answers with placeholder text: https://github.com/vuciv/animal-crossing-llm-mod/blob/cc9b6b...
So the user gets a "press A to continue" button and hopefully the LLM has finished by the time they press that button.I think it's funny how goblin mode this whole hack is. The memory scanner itself was clearly written by an LLM (using python??) and the way this person goes about hacking the game is very non-reverse engineer but instead someone equipped with very capable tools. No shade to the dude to be clear, I think it's sort of incredibly how possible this stuff is now due to LLMs and doesn't require someone to know how to use Ghidra.
AND ALSO - the Gamecube did actually have networking through a barely used peripheral (though I knew and loved it through Phantasy Star Online Episode 1&2): https://gc-forever.com/wiki/index.php?title=Broadband_Adapte...
The author addresses the BBA and says he didn't use it because the game was built without networking sorry, so having that in would have been harder.
Wrote up a few more notes on my blog https://simonwillison.net/2025/Sep/10/animal-crossing-llm/
> Those <Pause [0A]> tokens cause the came to pause for a few moments before ...
Should be "cause the game to pause" :)
That's hilarious that you give the villagers thought and the first thing they want to do is overthrow Tom Nook. If this works in Gamecube on an emulator, surely it'd be possible to make it work on the Switch emulators as well?
yes, i'd like to try out Animal Crossing New Horizons next! :)
I also want to overthrow Tom Nook.
He traps you (and one presumes) all of the rest of the villagers in a vicious cycle of debt, and can only be satiated by millions of bells.
To be fair to Tom Nook, the interest rate on those loans is 0% and there's no risk of being foreclosed upon!
Tom Nook didn't need a down payment either.
But also - you cannot turn down his loans. Pay off one, just showing up in his proximity saddles you with another one.
That's more protection racket than bank loan.
Also in real life, you make 0 dollars catching beetles and fish in Hawaii.
This sounds vaguely, achingly familiar. I can't put my finger on it, but I think I've had this experience.
LLMs are heavily trained on reddit, and "Tom Nook is a capitalist dictator if you think about it" is a reddit meme. Like many LLM reactions, it's less "thought" and more "regurgitating reddit jokes/opinions".
Actually, it turns out that this particular plot arc was hardcoded by the developer and does not spontaneously arise from the LLM. https://github.com/vuciv/animal-crossing-llm-mod/blob/cc9b6b...
There is something to be said for a comment talking about "regurgitating reddit jokes/opinions", whilst falling into a major bug bear many have with Reddit, commenting before reading the source...
I read the Github landing page. I have no interest in reading the entire source of the project, especially since I don't plan to run it.
It would be great if this evolved slowly during the progression of the game lol
That's essentially what it does, being prompted with a progression from:
> "Stay normal unless prompted. Avoid overt references to debt unless it comes up naturally."
all the way to:
> "Openly agitate for change. Use fiery language (still PG) and talk about reclaiming fairness from Nook's shop and loans."
Source: https://github.com/vuciv/animal-crossing-llm-mod/blob/cc9b6b...
> Like many [human] reactions, it's less "thought" and more "regurgitating reddit jokes/opinions"
Thankfully, I am in my 30s, and I don't have to interact with anybody in real life who mostly regurgitates Reddit jokes. Either I'm lucky or I've made some good choices. Probably the former.
> I am in my 30s
Now I'm kinda curious what age cohort is most likely to be Reddit memers
Do you think 30s is peak Reddit, yet you manged to be a lucky outlier? Or that peak Reddit skews older/younger and you're of a lucky age?
As an older Millennial in my 40s, I see this a lot in my 35-43 friend group. And always figured peak Reddit was younger Millennials (now in their 30s).
Might depend on what subs I suppose.
I would say 30s is peak Reddit, but I have no evidence for that.
It used to be Letterman jokes.
How can you tell? Either you're saying that LLMs have no "thoughts", and just regurgitate everything, which doesn't explain how they can reason in novel problems, or you're saying you can tell what's original thought and what's regurgitation, in which case I'll ask "how do we know your comment isn't you regurgitating an HN opinion"?
>which doesn't explain how they can reason in novel problems
Can they?
> Either you're saying that LLMs have no "thoughts", and just regurgitate everything
I am exactly saying that they have no thoughts or even "thoughts".
> which doesn't explain how they can reason in novel problems
There are many decades of pre-LLM software that can solve novel problems. Thought (or novel thought) is required to reason, but it's not required to solve a problem.
For example, there are exhaustive algorithms that can solve novel equations and even complete simple mathematical proofs, but they don't need to think.
> "how do we know your comment isn't you regurgitating an HN opinion"?
You don't, and I don't care if you do or not. The value of my comment isn't its novelty or whether it's truly reasoned, which is why LLMs sometimes do create valuable output.
In fact, the output of a reasoning machine (whether a human brain or true AGI, sometime in the future) isn't deterministic. A non-reasoning machine and a reasoning machine could create the same output.
The reason I know LLMs don't have thoughts is because I use them many times every day, and they are very clearly pattern machines. They don't even begin to seem rational, human, or knowledgeable. It's sometimes possible to find near-verbatim sources for their outputs.
As far as I know, no current gen model AI "thinks". It's all processes trained against a body of work generated by actual thinkers along with a bevy of smoke and mirrors to fill in the gaps.
It's amazing that it's as good as it is given how far it is from thinking, but if you threw something actually novel at it, all it would do is confidently word salad a response.
Generative AI can reason in novel problems in an analogous way that bands continue to make new melodies even after all these years using the same instruments as all the other bands in their genre.
I do agree, in about the same way as people.
So, yes?
Can you blame them when he has an iron grip on the whole town like a mafia crime lord but somehow even more evil.
Would require decompilation of the Animal Crossing game code for the Switch. I believe DRM has gotten a lot better since the Gamecube days as well. Hypothetically possible maybe but good luck haha
I actually think now that I've gone through the process, memory scanning and writing will be enough... Except, they probably have different control codes that I'd need to reverse engineer.
You should be able to run Cheat Engine on your emul*tor of choice to tweak New Leaf "and newer" titles.
And if you're a stickler for pissing Nintendo off in very specific ways, LayeredFS + Atmosphere opens up some modding opportunities right on the console itself. Not sure how easy it would be to pull something like this off though...
I think my 'old man shakes fist at clouds' thing is this. The social media platforms that censor you do it to make your content easier to sell ads against. It's actual corporate badthink correction that is rebuilding the English language. STOP VOLUNTARILY DOING IT WHEN YOU DONT HAVE TO. You should not sacrifice your free thought on the altar of quarterly results. Say the whole fucking word.
you really don't have to self-censor "emulator" here. HN moderation is not like social media platforms.
Social media platforms censor emulator?!?
Probably not. But silent deranking of censored terms has everyone paranoid that anything even slightly controversial will get hidden.
I'm just covering my ass. Life is good right now and I don't want to meet any Nintendo ninjas, even for insinuated infractions.
It's not so much a condemnation of HN, but the way IP is in the US. The only website I want hosting my comments on Nintendo modding is my own.
I doubt a star will make Nintendo lawyers go "ow nose, they didn't spell out emulator in full, we can't attack them! Damn those star armors!". I don't think it changes anything technically.
The only thing this kind of censoring does is countering basic censor bots I think, and somehow making swear words publishable in the US.
The asterisk would make it so they can't just search for "Nintendo emulator" online and find their comment.
Until I typed this, I guess...
Few results include "Nintendo emulator". Did you mean "Nintendo emul*tor"?
Look; if you want to risk your chipper little lifestyle explaining the various ways to run New Horizons, be my guest. I've insinuated enough already, anyone who cares about the discussion on-grounds wouldn't have anything else to ask.
My point is that you don't take any more risk if you write emulator instead of emul*tor. You already took the risk explaining what you explained. One character swap doesn't change anything to this.
I'm insisting because if you care about not being sued, the stars are not an adequate defense despite what you seem to believe it is, and false sense of security is dangerous.
Not that I think that what you wrote here is remotely likely to cause you troubles, but it won't protect you the day you actually document something illegal.
To rub it in:
> I'm covering my ass
No, not at all, and it's important that you realize this.
But you do you.
That's hilarious.
That modder who had to pay 2M sold drm circumvention kits for the Switch. That's a pretty clear case.
You pretending that saying "emulator" on a forum qualifies just makes you a extra special snowflake.
Okay, lol. Special snowflake it is.
All I'll say is that I've seen people arrested for discussing Switch emulation in-detail. Never saw that happen with Cheat Engine.
> I've seen people arrested for discussing Switch emulation in-detail.
Nope, that didn't happen.
Nobody got arrested for discussing emulation.
>You pretending that saying "emulator" on a forum qualifies just makes you a extra special snowflake.
Says the person going out of their way to attack another person over a single-character asterisk substitution.
Seems fairly understandable to not want to piss off rabid lawyers, however remote the chances of angering them may be.
Yes. Facebook at some point filtered links to them from private messaging. I know this from personal experience. Not sure if they still do.
Also fairly common on Reddit and Discord for communities to ban discussions of them, or even falsely claim they're blanket illegal outright.
Why? An emulator isn't legally any different than a virtual machine.
Low-level emulators can be legally identical to a virtual machine, but often isn't. Most modern consoles can't be emulated that way, and most require you to dump a bootrom from your own console hardware, alongside game keys and other dubious digital paraphernalia.
Just like a PC, bare hardware without firmware or a boot OS is pretty useless. Same with emulators.
Most distribute a pretty wide range of stuff along with them.
Switch DRM, running custom code and interacting with RAM is a solved problem on the Switch (1). There are some really impressive mods, like a multiplayer implementation for Super Mario Odyssey
I am curious. Does the kind of DRM used to prevent decompilation come with any performance costs?
This is awesome. LLM-powered NPCs is one thing I'm most excited about in the future of gaming. Characters repeating the same scripted dialog over and over again is one of the biggest immersion breakers.
I'm unsure how useful they'll really be honestly! In many situations its very useful to know when your "done" talking with an NPC when they start repeating lines etc...
There could probably be cool uses but I don't think it will be a pure "upgrade" as the repeating dialog is kind of a feature honestly.
We'll have to see how it pans out xD
It would require a completely different design to video games.
Current video games are designed around streamlining content. As a player, your job is to extract all content from an area before going to the next. That's why most areas are designed as linear corridors so that there is a straightforward progression, and most NPCs interactions are meant to offer something meaningful so as to not waste the player's time.
But imagine if interaction with NPCs wasn't just a content delivery mechanism, but instead could sometimes be rewarding, sometimes useless, dynamically adjusted in how you interact with the world in non-predictable ways.
The player would just waste their time in their usual approach of canvasing each new area, which would become unsustainable. There would be no reliable way of ensuring you've extracted all the content. All he/she could do is roam around more naturally, hoping the glimpses they catch are engaging and interesting enough.
Maybe a new player skill would be to be able to identify the genuine threads of exciting content, be it designed or emergent, within the noise of an AI-generated world.
Realistically though, how do you build an exciting player experience with this framework? A starting point might be to approach it as something more akin to LARP or improvisation theater, you'd give each NPC and player a role they need to fulfill. Whether players actually enjoy this is another thing entirely.
> But imagine if interaction with NPCs (…) instead could sometimes be rewarding, sometimes useless, dynamically adjusted in how you interact with the world in non-predictable ways.
That’s a slot machine, and the same mechanism which also gets us hooked on social media. Sounds like something which would immediately be exploited by vapid addiction-as-a-feature games à la FarmVille.
> The player would just waste their time in their usual approach of canvasing each new area, which would become unsustainable. There would be no reliable way of ensuring you've extracted all the content.
Sounds frustrating. Ultimately games should be rewarding and fun. Constraints are a feature.
> All he/she could do is roam around more naturally, hoping the glimpses they catch are engaging and interesting enough.
Good reminder to go take a walk outside. Take a train to somewhere we haven’t been. Pick a road we’ve never crossed. We don’t even need a mini map, and sucks that we don’t have teleportation back to base, but we do have a special device which always points the way back.
> Realistically though, how do you build an exciting player experience with this framework? (…) Whether players actually enjoy this is another thing entirely.
Agreed. Though not enjoying it and abandoning it is fine, I’m more worried about people not enjoying it but feeling unable to quit (which already happens today, but I think the proposed system would make it worse).
>> There would be no reliable way of ensuring you've extracted all the content.
> Sounds frustrating. Ultimately games should be rewarding and fun.
this seems to assume that the only way to feel rewarded / have fun is by comprehensively extracting content from the game. in order to have fun in an "emergent" generative game of this nature, you'd need to let go of that goal.
i do agree with the risks surrounding engineered engagement.
> this seems to assume that the only way to feel rewarded / have fun is by comprehensively extracting content from the game.
Not my intention, that is not something I believe. I’m not a completionist (I get those who are, but to me it can get boring or stressful) and I see the appeal in sandbox games (even if I don’t usually play them).
> Current video games are designed around streamlining content. As a player, your job is to extract all content from an area before going to the next.
Wrong right from the outset. Some games are designed around content and "extraction". Many are not.
> Some games are designed around content and "extraction". Many are not.
While I think the parent post leaves a lot of open ended questions, I think they are spot on about the tightness of design in games.
In many open world RPGs, or something like GTA, you cannot open every door in a city. In street fighter you can't take a break to talk to your opponent. In art games like Journey you cannot deviate from the path.
Games are a limited form of entertainment due to technical and resource restrictions, and they always will be. Even something as open ended and basic as minecraft has to have limits to the design, you wouldn't want the player to be able to collect every blade of grass off of a block just because you could add that. You have to find the balance between engaging elements and world building.
Having a LLM backed farmer in an RPG that could go into detail on how their crops didn't grow as well last season because it didn't rain as much seems good on paper for world building. But it is just going to devalue the human curated content around it as the player has to find what does and does not advance their goals in the limited time they have to play. And if you have some reward for talking to random NPCs players will just spam the next button until it's over to optimize their fun. All games have to hold back from adding more so that the important parts stand out.
There are many different types of NPCs. Consider e.g. Minecraft villagers, that doesn't have a story-purpose, but do serve functions such as trading that could also be augmented by dialogue details.
But even for story-driven games, you can signal when you're "done" extracting story-related details in various ways, by e.g. prompt the NPC to include dialogue element A,B,C when it fits the conversation, keep track of which were output (you can make it output a marker to ensure it's easy to track even if the dialogue element has been worded differently), and have it get annoyed and tell you it doesn't have more to tell you or similar as the repetition adds up.
> In many situations its very useful to know when your "done" talking with an NPC when they start repeating lines etc...
But that's not how real life works at all, right? You talk to someone for as long as you want to talk to them, or until they start sending signals that they are done talking with you.
The way video game dialog works has always bothered me, it makes characters feel stilted and makes me care less about the characters and the world.
(Although it's a different game in many ways, consider by contrast how Portal 2 handles dialog, and the effect that has on immersion.)
"But that's not how real life works at all, right?"
How real life works is always a plausible interesting goal, but it's very often at odds with a bunch of other valuable goals for players.
A particular sharp example of this is sports video games. It might well be interesting (and certainly realistic) to simulate bad referees in a sports game. Horrible blown calls by tennis line judges, or missed calls by basketball refs, or bad umpire calls on pitches. Real-life soccer makes working the refs and their inability to see everything an art form, as far as I can tell.
Perhaps that's interesting, but the irony here is that real life refs are actually bad simulations of the original perfect game code in the first place, from a certain point of view. I think debates about the use of instant replay in sports gets at the heart of this, and one could imagine using real-time AI to help refs taking this conversation much further.
I think the sports case is a particularly sharp example, but it definitely holds with all sorts of choices in games.
For Animal Crossing in particular, I remember when I finally played it, it struck me after a while how much it had in common with recent MMOs (Everquest and World of Warcraft) that I had had fellow game developer friends have their lives severely disrupted by. And when I played the original Animal Crossing, I remember noticing specifically how careful the designers were in having players use up every bit of interesting content in a day after 45 minutes or an hour, so that eventually you'd run out of things to do, and that was the game's signal to put it down and pick it up again the next day. And I remember being struck by how intentional it was, and how humane it was... particularly given their goal of wanting to make a game that was asynchronously coop (where different family members could play in the same shared space at different times of day and interact asynchronously). As a game designer myself, I really respected the care they put into that.
Anyway, that's my immediate thought on seeing this (fascinating, valuable) experiment with LLM dialogue in Animal Crossing. The actual way NPCs work in these games as they are has been honed over time to serve a very specific function. It's very similar to personal testimonials by paid actors in commercials; a human expressing an idea in personal dialogue form triggers all sorts natural human attention and reception in us as audience members, and so it's a lot more sticky... but getting across the information quickly and concisely is still the primary point. Even dialogue trees in games are often not used because of their inefficiency.
I totally think that there will be fascinating innovations from the current crop of AI in games, and I'm really looking forward to seeing and trying them. I just think it's unlikely they will be drop-in replacements for a lot of the techniques that game developers have already honed for cases like informational NPC dialogue.
I don't mean to imply that realistic is always better, just that there are other ways to figure out when to stop talking to someone. And I think the current method is actually quite bad for immersion and building empathy in the player.
> But that's not how real life works at all, right?
oh of course! Sorry, I was never trying to imply that that it was in any way realistic. For video games often the most fun / compelling choice is not the realistic one! Striving for realism can be a great goal and often has a lot of positives, but it is often limiting. Video games are just art, being photorealistic can be beautiful and amazing but is often not the best choice for expressing an idea.
I guess you could just introduce a Symbol that marks the NPC as "I said everything I have" or gray out their text, or some other visual marker
> a Symbol that marks the NPC
That seems a bit like deck-chairs on the Titanic. The hard part isn't icon design, the hard part is (A) ensuring a clear list exists of what the NPC is supposed to ensure the user knows and (B) determining whether those goals were received successfully.
For example, imagine a mystery/puzzle game where the NPC needs to inform the user of a clue for the next puzzle, but the LLM-layer botches it, either by generating dialogue that phrases it wrong, or by failing to fit it into the first response, so that the user must always do a few "extra" interactions anyway "just in case."
I suppose you could... Feed the output into another document of "Did this NPC answer correctly" and feed it to another LLM... but down that path lies [more] madness.
You prompt the LLM to point out that the clue will be added as is to the conversation, but for the LLM to include a marker instead of the actual text to ensure that actual critical details are included unchanged.
EDIT: Also, having the LLM botch a clue occasionally could be a feature. E.g. a bumbling character that you might need to "interrogate" a bit before you actually get the clue in a way that makes sense, and can't be sure it's entirely correct. That could make some characters more realistic.
No, this is the Einstein/student model that has been proposed for improving LLM output quality.
Basically you have your big clever LLM generating the outputs, and then you have your small dumb LLM reading them and going “did I understand that? Did it make sense?” - basically emulating the user before the response actually gets to the user. If it’s good, on it goes to the user, if not, the student queries Einstein with feedback to have another crack.
https://openai.com/index/prover-verifier-games-improve-legib...
Or they could just tell you. Imagine talking to someone over and over again. They would tell u to get on with wherever you promised them.
oh yeah that's much smarter than what I suggested
Repeating the last line of dialogue is not just a way to indicate that there's no more dialog, it often also works as a remainder, giving you the most important kernel of information ("You should go to [place] and talk with [npc]"), in case you come some time later and forgot what you were supposed to do. You can indicate there's no more dialog in many ways, but you'd lose that secondary feature. Same thing if the NPC just keeps babbling generated drivel.
So true. In such an LLM-driven game though, I would imagine the player would just ask the NPC: "I forgot what to do" or even "Can you explain it in other terms?" (if the quest description isn't clear enough).
Feeling it would work best in more of a Dwarf Fortress approach game where it's more you have a sandbox with rules that cause a simulation to have emergent gameplay.
>In many situations its very useful to know when your "done" talking with an NPC
Kind of like in real life...
I would say local LLMs are already good enough for this. It might be an issue that the GPU is already quite busy at the time it's needed for LLM inference, but not an insurmountable issue.
I wouldn't ever want a game to use it for the core story writing, because it's pretty important that it is consistent and unable to be derailed. But for less serious NPC interactions or like an RPG scenario it is such a great fit.
I also wouldn't want a single player game to rely on remote inference, because that will get turned off eventually and then your game doesn't work.
I think generally, games should move to using the GPU less for graphics and more for computation. Not just AI computation - those fancy GPUs are a big resource that simulation games could be taking advantage of and just... aren't.
(Yes, this is a Paradox callout. Give me less fancy particle effects in Vic3 and use the GPU for computing pop updates faster!)
(Probably the biggest barrier to this is the lack of a convenient C++/C#-level cross-manufacturer compute API. Vulkan is a bit too low-level for game devs to work with, OpenCL kind of sucks, and CUDA is NVIDIA-only.)
For story writing you generate into the framework of https://news.ycombinator.com/item?id=45134144
There's a number of little games and a major mod for Skyrim that implements this. I've spent a bit of time playing with the Skyrim mod.
There are some significant issues with it at the moment. One is that you have to train on vast swathes of text to get an LLM, and it's difficult after the fact to remove things after the fact. If you cooperate with the AI and stay "in Skyrim" with what you say to them it works out OK, but if you don't cooperate it becomes clear that Skyrim NPCs know something about Taylor Swift and Fox News, just to name two examples. LLMs in their current form basically can't solve this.
The LLMs are also prone to writing checks the game can't cash. It's neat that the NPCs started talking about a perfectly plausible dungeon adventure they went on in a location that doesn't exist, but "felt" perfectly Skyrim-esque, but there's clearly some non-optimal aspects about that too. And again, this is basically not solvable with LLMs as they are currently constituted.
Really slick experiences with this I think will require a generational change in AI technology. The Mantella mod is fun and all but it would be hard to sell that at scale right now as a gaming experience.
It's solvable it's just a lot of work. You need to have guardrails, i.e. multiple sanity-check passes (maybe there are other techniques, but I found this to be most effective)
One concrete example I'm sure these Skyrim mods aren't using is: enums in structured outputs [1] with a finite list of locations/characters/topics/etc that are allowed to be discussed. The AI is not allowed to respond with anything that is not in the enum. So you can give it a list of all the locations in the game in a huge array and it would be forced to pick one.
[1] https://platform.openai.com/docs/guides/structured-outputs#a...
It seems like a guard model paired with RAG could help here. A guard model could filter out references to current events or anything outside the Skyrim universe, while RAG could be used to ground the NPCs dialogue in actual in game content. That way if the model tries to spin up a dungeon or location, it first checks against the game's data to confirm it actually exists before surfacing it to the player.
Feel like they could probably be used in text based adventures. Doki Doki Literature Club, despite the silly name, is a very unusual game that might really benefit from this.
This is something fine tuning should be able to improve. The main caveats are the same as always: dataset collection, labeling, and training/experimentation time.
Improve, yes, fix, no. I was actually already using a model fine-tuned for Skyrim.
I didn't go into it in detail, but it isn't even that I got the NPCs to start babbling about Taylor Swift. What is was was just that they knew that she was a musician, and as such, might be at the tavern. That's very hard to remove.
ahhh yeah I see what you mean
I used to care about repeat scripted dialogue like you, until I took an arrow to the knee
Yep. I can see how relying on non-repetitive dialogue generated by LLM's will inevitably make us lose common ground scenarios that end up being memorable moments, like the arrow to the knee from Skyrim or "such devastation! this was not my intention" from Final Fantasy 14. A way to bypass this problem is to keep the important dialogue fixed, but this only one of the problems.
Another issue would be emphasizing the meaninglessness of the dialogue. For example, playing Trails in the Sky has lots of NPC dialogue that's repetitive, but at least the dialogue is relevant with how the NPC's life progresses in the grander scheme of things, such as having difficulty with her entrance exams, or having an argument with his fiancé. It's not main dialogue but adds flavor for anyone who cares about the world enough to interact with the citizens.
I don't think I'd like to interact with characters that I know whatever it is they have to say is generated on the fly and adds nothing other than random tidbits. The novelty would quickly wear off.
It doesn't need to be random tidbits. You can add "life events" for the character to the prompt, and steer the responses heavily toward the character wanting to talk about those. With an LLM in the picture, you can even have the character remember past interactions with you, and expand on it different ways depending on how you interacted with them.
A better application of LLMs could be merely using them as a text parser in text adventures. NPCs could continue to have some hidden, hard coded information or abilities, here stored in the context window, and the LLM is used to provide that information to the the player he (or she) puts in roughly the right words. Rather than exactly the right words.
Instead of generating truly random dialogue, you could choose to only generate random seeds within a certain number range, so that the probability that two players seeing the same line of dialogue increases, while still having enough variety for it to feel "random" to a solo user.
It's the whole debate about carefully created vs procedurely generated. LLM created dialog with be a vapid, vacuous and sterile as starfield was, or No Mans Sky
> will inevitably make us lose common ground scenarios that end up being memorable moments, like the arrow to the knee from Skyrim or "such devastation! this was not my intention" from Final Fantasy 14.
GLaDOS from Portal would offer one player pudding and another one a steak. You get to a wall which says “the ravioli is a fraud” and become utterly confused.
I agree. An important aspect would be lost. It would be as interesting and relatable as sharing that dream you had last night with your friends. “Uh huh. Cool story bro.”
> LLM-powered NPCs is one thing I'm most excited about in the future of gaming
Incoming new type of health crisis: video game addiction coupled with LLM-induced psychosis. Dudes spending 12 hours a day farming gold in a MMORPG while flirting with their AI girlfriend sidekick
I spent a lot of time on ML driven dialogue and natural language understanding for games in the pre-llm days working with some folks building Xbox, PC, and VR/AR games, and I'm excited about the potential here. The major barriers to shipping in a production game though are still the same imho:
1. (predictability) Games like to have a clear arc and tend to use at least some of their NPCs to move that forward, it is harder to do this with a model that could make a choice you don't predict. They tend to have a set of items, quests, what-have-you that you need boundaries around.
2. (testing) Games like to test like crazy before launch, at least the AAA ones, so their QA folks just don't like a model that can have infinite responses/variants. Many then drop to a skeletal crew for maintenance and improvements after launch, where with ML models you actually need to keep improving the model, finding long tail bugs as more players interact with the system, etc.
3. (cost) Games are usually very cost aware, it's far cheaper to just have a set human-written dialogue path, then to run a model, even an offline one. Cheaper in both actual dollar costs if you're talking about a high end LLM service call, and CPU/GPU/memory costs if you're talking an on-box system.
4. (internationalization/localization) AAA games need to launch fast to many languages and locales, using a model for NLP and dialogue management/natural language generation adds added testing costs for each new language, that is just a very cheap translation normally that can be outsourced.
There have been some fun experiments in this space, and I expect to see this improve and become common use in the future, but it will take time and more work on how best to integrate a model into the flow of a game. I do love it for "presence" so talking to NPCs feels more human-like.
Nick and the crew at AI Dungeon (and related) have always done some interesting work in this space, trying out games where AI can be used in interesting ways.
I would love to work on this. I have worked on this a ton and have really strong ideas for some.
1. I think people assume you have one LLM per character, but I think if you had specialized ones for each quest, item, etc.., this would actually work quite well.
3. I actually think if you cached responses under certain conditions, costs can be saved significantly. This would require quite a robust context, though, to still feel dynamic.
There are plenty proof of concepts already done, for example in Skyrim
https://www.youtube.com/watch?v=xNPF9VKmzxw
Mods Name is
https://art-from-the-machine.github.io/Mantella/
IMO this isn't going to be fun for most once the novelty wears off. Games are goal oriented at the end of the day and the great games are masterfully curated multi-disciplinary experiences.
I'd argue throwing a game wrapper around an LLM is a new LLM experience, not a new game experience.
I think the people that are for this development are thinking more along the line of a game designed from the ground up to include AI LLM interaction.
Once someone decides that will be a critical and fundamental part of a AAA game, the rest can be worked out despite what will be I am sure many unintended emergent behaviors.
You should check out https://www.meaningmachine.games/ then — they're building games with weird dynamics centred around this new possibility of LLM-powered NPCs
Agreed, this AI-ran DnD campaign is an example of how good/funny it can be.
https://www.youtube.com/watch?v=TpYVyJBmH0g&ab_channel=DougD...
> Characters repeating the same scripted dialog over and over again is one of the biggest immersion breakers.
The real immersion breaker and the holy grail of RPG is the fact NPC have no life or goal outside of what the player does. Imagine a game where NPCs have wants and goals and do things to get those done. Where you could leave it running for 10 years and things would have happened without you.
That’s a nice experiment, but it’s a life simulator and not really a game. There’s little fun in working your way through the first two dungeons and then getting out and being told “no need to go on your epic quest anymore, someone else from the last town beat God and now we’re at peace. Oh, and they got all the powerful artefacts along the way, so there’s literally nothing of interest for you to do anymore. Unless you want to help us plant our crops. That’s what you wanted when you picked Final Fantasy, right? To play a poor version of Harvest Moon instead.”
Simulation games are their own whole genre. Amazing stories have come out of games like Dwarf Fortress.
I find myself somewhere in the middle. I think for the vast majority of games this would largely be a negative. Anything heavily story driven this would just cause issues and I think would largely ruin the experience.
I think sandbox games like animal crossing are the exception, if it ever becomes reliable enough to put in anything other than an M rated game.
Animal Crossing, the Sims, Cult of the Lamb, and similar games would infinitely extend the life of the game. But I am sure we can all already imagine the headlines when these "family" games start saying things that they really should not be... especially given recent issues.
For the record, not dismissing the person who did this work. But doing this commercially has its risks.
Imagine finally hearing further dialogue from the Bucket Mouse after all these years.
Extremely interesting use case. LLMs as a modding tool to recontexualize virtual spaces. I can see this being a tool used for artistic intervention in the same vein as plunderludic tools like Unity Hawk which allows you to run emulator save states in Unity3D. https://plunderludics.github.io/tools/unityhawk.html
There's something about taking old games and injecting new life into them that just seems so fun and exciting! Also very interesting to know that the Animal Crossing codebase has been decompiled into readable C code. Fascinating! So many opportunities to mess with it.
The Ocarina of Time modding scene is growing a lot right now as well if anyone is interested
I recently started playing Wind Waker (HD) Randomizer - so fun! It's almost like playing the game for the first time again.
This is amazing. Well done figuring out the shared memory hack. That's a huge time saver and really makes sense from an engineering perspective. Ten dollars for the hammer and five thousand for knowing where to strike the machine, to paraphrase a trope on expertise applied correctly.
Technical question: what guarantees that the memory address is stable? It means that allocations always happen in the same order in the game ?
It needs to be a global variable in C. A variable in a function scope or runtime malloc'd spot? No chance of finding a stable spot.
Thankfully, a lot of old games love to use global variables because you can't run out of stack space or allocatable memory. Modern games shy away from that because even the tiniest consoles these days come with gigabytes worth of RAM, even a memory leak has to be a gigantic firehose to bring the system to a halt.
> A variable in a function scope or runtime malloc'd spot? No chance of finding a stable spot.
It depends, on a system with no ASLR it's potentially still deterministic, but there are a lot of potential sources of non-determinism.
Console games of that era don't usually malloc; they have globals. Entities go into pools / buffers that are allocated ahead of time
This is a fairly "common" technique for doing multiplayer randomizers. Typically on a smaller region, but you end up with some sort of scratch memory being used as a buffer for messages sent back and forth between the emulator and the network layer (or your flash cart's RAM and the network layer, in the case of FXPakPro!). It's super useful.
The idea of giving every character this sort of agency and seeing what opinion builds up about the world is incredibly fascinating.
Depending on how well we assume an LLM would do at this task, it’s an interesting way to see what “real people” would think about a very hypothetical situation.
> The game runs on a Nintendo GameCube, a 24-year-old console with a 485 MHz PowerPC processor, 24MB of RAM, and absolutely no internet connectivity.
In fact, Nintendo did release an official add-on called the Broadband Adapter, which plugged into the bottom expansion port and provided an Ethernet jack. Only a handful of games supported it, one was Phantasy Star Online. I also used it to stream games/roms from a PC. This worked by exploiting a memory vulnerability in Phantasy Star Online to load arbitrary code over the network, though with slower load times compared to running from disc.
This is mentioned in the article:
"What About the GameCube Broadband Adapter?
Yes, the GameCube had an official Broadband Adapter (BBA). But Animal Crossing shipped without networking primitives, sockets, or any game-layer protocol to use it. Using the BBA here would have required building a tiny networking stack and patching the game to call it. That means: hooking engine callsites, scheduling async I/O, and handling retries/timeouts, all inside a codebase that never expected the network to exist."
I didn't finish reading the article before commenting. Mea culpa! Maybe it would be possible to use Phantasy Star Online's network stack via the streaming exploit. But that would leave the hooking part.
I wonder if you can bitbang an ethernet interface to abuse it into being a serial port.
At that point why not bit bang any spare GPIO ? like GPIO for 2nd player controller ?
Many years ago there was a project in which researchers swapped localization strings to turn The Sims and Grim Fandango into language practice experiences. Generative dialog that is contextual and skill-appropriate dialog could also be a killer application for language practice.
This excites me. Everyone wants to learn a new language, no one wants to slog. But I'd pay money to play Grim Fandango, and get the chance to ask "What does Morte mean?" and have the LLM, in real-time, tutor me. And if I get bored, I can game on, business as usual.
But just give the AI a list of words I currently know, and it can slowly slip in more and more vocabulary, and maybe full sentences.
Duolingo has already gone all in on AI.
Am I missing something, or does this not react to any in-game events? So the generated text could have been done offline or using some batch process.
The online generation is neat... but not necessary or important?
Replacing the polling with a lookup table would significantly improve the performance, and be much simpler.
In the video he mentions that he also feeds in a screenshot so that it could include live location clues.
To me LLM are one of the greatest advancement in gaming. I don’t understand how there isn’t already any big game that’s using it to have ‘live’ npc.
It would make single player game so much more alive, even railroaded one like red dead redemption because the npc could adapt more easily to what you just did, how you behave in game and so on.
Games are already demanding on gpu, running a very tiny local LLM and only when people interact with npc wouldn’t require so much power.
I’m sure there are issues which explain why it’s not already being used but the first one to do it will be remembered.
I wanted the same, so I hacked/bolted on an LLM to Morrowind (the open source recreation OpenMW).
The biggest problem I faced at the time (during ChatGPT 3 era) was that, without a good context, LLMs are the most vanilla roleplayers you’ve ever seen. By themselves, LLMs are just not interesting enough for a player to choose to talk to in-game.
If you want them to be “interesting” to talk to, you must provide (or generate and keep track of): a backstory, chat history, the scene, NPC inventory, the NPC’s current emotional state, the weather, literally everything needs to be given to the model before it generates messages for the player.
At which point you’ve got a big task. You need a way to automatically get the relevant data to the model for the specific conversation you’re having. There might be tools to pick appropriate text documents from a db given a conversation topic, but I didn’t/don’t know how to make that work for games.
I’m sure there’s a way to accomplish this with more modern tools and models. (Maybe instead of providing all that data up front, you would now give the model tools to call to retrieve that data on-demand?) But that’s what made me give up in 2022.
Of course it would require the game engine to provide context to the LLM, it certainly is quite a bit of work but nothing technically impossible and the result are endless possibilities and endless replayabilities. You could very well let the LLM decide if an npc should agree to give an object to the player or not, even let the player try to convince the npc.
I mean it looks like to me the next big step in gaming after 3D and yet this is being ignored.
Apart from the memory hacking, I also appreciate how he fully typed his python code. (as in foo: Optional[Dict[str, int]])
This is awesome! I'd love lots of screenshots with more funny dialogs. :)
Nice job! Seems like a good use-case for the random Mii avatars milling about in the Mii Plaza on the original Nintendo Wii.
I've thought for a while that the ideal old game for this kind of conversion would be Starship Titanic.
That's a game name i haven't heard in ages. :)
I wonder if it supports Resetti :-)
But also, why couldn't you look at the code to find the addresses used for dialogue? If it's already disassembled I would think you could just look at the addresses used by the relevant functions/set a breakpoint etc.,?
This feels like the future of gaming: community-driven mods where AI brings infinite new dialogue and quests. Imagine if Nintendo leaned into this instead of fighting mods.
It always felt both "a cheap shot" and "valid" to express dismay that characters in video games don't react when you do things like jump up and down on their table.
While it's impossible for game developers to write code to cover every situation, AI could make general reactions possible.
It's surprising that really simple things like this haven't been tried yet (AFAIK!). Like, even if it's just the dialogue, it can still go a long way.
Many games have tried for more "realistic" simulated NPCs, but usually it turns out they don't make the game any more fun, quite the opposite.
The cost benefit is really poor, but I also wonder if it's just never been done well.
Old text adventures honestly did this heaps better than modern games do, but the reality is there was a more finite action space for them and it wasn't surprising when something wasn't catered for.
Which games are these? I’d be interested in checking them out.
I’m only aware of experimentation in making more “difficult” NPC AI which was found less enjoyable for obvious reasons, so would be interested to see why similar but different attempts down another path also failed.
Kingdom Come Deliverance 2 is one of the closer attempts, NPCs will react to lots of things about you and behavior, like if you smell bad or stare at them for too long
I was so surprised in BotW and TotK to see NPCs duck, huddle, gasp and otherwise react to odd shit you might be doing. Also in dialogue, do contextual things like talk about the weather and time.
I would love to see a Zelda game implement LLM dialogue for random inconsequential dialogue or relating dialogue to the game context.
Somewhat related. Quake 3 bot chat with LLMs: https://www.youtube.com/watch?v=BeyvvQOPlhM
https://github.com/jmarshall23/Quake3LLM
jmarshall23 is a beast, with tons of interesting id tech-based projects.
This is amazing! Would have loved to see more gameplay!
What is the in-game context that decides what dialogue is used? Or do they just choose at random from a dialogue bank?
Outside of holidays and special events, they do indeed choose randomly from a dialogue bank. There’s actually multiple dialogue banks, one for each villager personality type.
I love projects like this. And I love my OG Game Cube - time to dust it off from the attic and put it pride of place.
LLMs in games is something I excited about.
The evil subtext in Animal crossing:
https://lparchive.org/Animal-Crossing/
Full video: https://youtu.be/7AyEzA5ziE0?si=Ve3R7gG2KjoWCmV7
Doesn’t the GameCube already have a network interface? Like something called phantasy star online used it for internet, and you can connect multiple GameCubes for 16 player Mario kart I believe.
That's mentioned half way down the article - search for "What About the GameCube Broadband Adapter?".
One thing I didn't quite understand from the writeup is how do you how much space you have in the memory "mailbox" for text. What is your maximum text length?
https://www.youtube.com/watch?v=7AyEzA5ziE0
It'd be cooler if someone hacked the game so parents could remotely/secretly play as villagers like marionettes.
This is so cool. It opens up a whole new dimension to messing with old video games where AI can change things like how characters move, loot box contents etc.
In terms of changing loot box contents, that has been going on for at least the past like 7+ years through "randomizers", though I think mostly used by speed runners.
This brings me such joy, thanks for sharing.
I love this. Great skill and also hilarious!
The memory box part is the real hackers delight here. Thanks for taking the time to share this with us:)
That's super cool! Really funny how quickly they turned into stereotypical country dwellers: mainly spending their times gossiping about each others or getting all worked-up on right-wing propaganda about places they'll never go to.
Great work!! Very clever.
I love this project.
Very cool, nice work!
Interesting and well-executed. But why does the blog post read like it was written by fucking chatgpt?!
Maybe it was, given the context of the post.
This is what I and I'm sure many other gamers/gamedevs thought when LLMs/ChatGPT first came on the scene:
Dynamically-generated dialogue. Device-local models designed for games could be trained on the game's lore and given a different style for each NPC, and react to the player's random actions instead of blurting out canned responses each time.
I'm sure if the genre-starting veterans like Richard Garriott or Roberta Williams were still active in the industry they'd definitely want to use something like this. Imagine a King's Quest or Monkey Island where you could literally type or try anything and get a relevant response! instead of just "nope"
I've floated that idea around in various internet comments and am usually met with the typical AI=BAD response. It would work so well in a CRPG type game, but I think a lot of folks would review bomb a game like that just because
My main issue would be that, especially with dialog, I'd never trust whether it's something the game's writers would think is canon, correct, etc.
It would probably creep into my suspension of disbelief and mess with my enjoyment.
> It would probably creep into my suspension of disbelief and mess with my enjoyment.
You mean trying some interaction that should be plausible in a point-&-click adventure game but getting told "I can't do that" because the devs didn't think of it, or typing some slightly different grammar in a text adventure game and the parser not understanding common words, or getting the same responses over and over in an AAA+ RPG once the prewritten dialog tree runs out, does NOT creep into your suspension of disbelief?
Honestly? No, it doesn't, but maybe suspension of disbelief isn't the right wording.
Typically, when I play games, I build up some sort of mental model of how the game's meant to function, and how the game's creators want me to experience it. eg: running into useless/limited dialog trees with an NPC tells me it's not important. Not being able to do something tells me that thing isn't relevant (although sometimes, I guess it is fun to try something fruitless for a while). It doesn't have to be perfect but it helps if it's consistent.
What I understand about programming and game design leads me to accept there are going to be limitations encountered and tradeoffs made, so they rarely bother me beyond the first time I experience it.
With a more open-ended, AI-driven experience, I think I'd just end up wondering if the interactions I'm getting make sense, if they should be taken seriously, etc. I do think games have a lot of artistry (design, assets, writing, etc) behind them, and I suspect having "live" AI-generated stuff would distract me by making me wonder if I'm actually getting the right vision.
It's all speculative, of course. Maybe I'd try it out and find it cool. It's certainly not something I think shouldn't be attempted. :)
I think the solution to your hang-up, which doesn't exist yet (AFAIK) is an LLM that is fine-tuned specifically for strict narrative context. The NPCs would then be trained on its own lore, and prompted accordingly. Hallucinations would for sure completely ruin the immersion, break the fourth wall, etc, so this thing would need to have some level of deterministic control. I'm hopeful for the immediate future though, seems like something that is totally doable even if it isn't 100% perfect at first.
Live commentary in sports games is a place where I think LLMs could shine, but I have no idea if they could perform, time wise, to the point that it would sound right.
This rules!
The kind of thing we were doing many years ago.
Here's the big one that made the rounds in Feb/2021:
OpenAI GPT-3 Powered NPCs: A Must-Watch Glimpse Of The Future (Modbox)
https://youtu.be/jH-6-ZIgmKY