OK, AI playing video games is cool. But you know what's really really cool? It looks like SIMA 2 is controlling the mouse and reading the screen at something approaching 30+fps. WANT. Computer use agents are so slow right now, this is really something. I wonder what the architecture is for this.
My friend that is AI. However, it can get a lot better: be more aware of screen content, follow multiple instructions at once, keep context in mind throughout the conversation and from past interactions
When, manus.ai came out I wondered the same, the "use computer" mode seemed really interesting to me, although I've seen JAYU [1] which implemented computer use with gemini.
Moreover I saw somewhere I really don't remember navigating web browser through layouts akin vimium like experience.
Initially (my impression of) computer use was only opening chrome and doing things inside chrome, chrome-as-os experience, I think maybe cloudflare could do something better here, with their workers?
Per user instance seems really costly I really do wonder how did they architect it.
The gap between high level and low level control of robots is closing. Right now thousands of hours of task specific training data is being collected and trained on to create models that can control robots to execute specific tasks in specific contexts. This essentially turns the operation of a robot into a kind of video game, where inputs are only needed a in low-dimensional abstract form, such as "empty the dishwasher" or "repeat what I do" or "put your finger in the loop and pull the string".
This will be combined with high-level control agents like SIMA 2 to create useful real-world robots.
I work on a much easier problem (physics-based character animation) after spending a few years in motion planning, and I haven’t really seen anything to suggest that the problem is going to be solved any time soon by collecting more data.
"We present Dreamer 4, a scalable agent that learns to solve control tasks by imagination training inside of a fast and accurate world model. ... By training inside of its world model, Dreamer 4 is the first agent to obtain diamonds in Minecraft purely from offline data, aligning it with applications such as robotics where online interaction is often impractical."
In other words, it learns by watching, e.g. by having more data of a certain type.
I am pushing the optimism a bit of course, but currently we can see many demos of robots doing basic tasks, and it seems like it is quite easy nowadays to do this with the data driven approach.
The problem becomes complicated once the large discrete objects are not actuated. Even worse if the large discrete objects are not consistently observable because of occlusions or other sensor limitations. And almost impossible if the large discrete objects are actuated by other agents with potentially adversarial goals.
Self driving cars, an application in which physics is simple and arguably two dimensional, have taken more than a decade to get to a deployable solution.
Next to zero cognition was involved in the process. There's some kind of hierarchy of thought in the way my mind/brain/body processed the task. I did cognitively decide to get the beer, but I was focused on something at work and continued to think about that in great detail as the rest of me did all of the motion planning and articulation required to get up, walk through two doorways, open the door on the fridge, grab a beer, close the door, walk back and crack the beer as I was sitting down.
Basically zero thought in that entire sequence.
I think what's happening today with all of this stuff is ultimately like me trying to play Fur Elise on piano. I don't have a piano. I don't know how to play one. I'm going to be all brain in that entire process and it's going to be awful.
We need to learn how to use the data we have to train these layers of abstraction that allow us to effectively compress tons of sophistication into 'get a beer'.
> This essentially turns the operation of a robot into a kind of video game, where inputs are only needed a in low-dimensional abstract form, such as "empty the dishwasher" or "repeat what I do" or "put your finger in the loop and pull the string"
I don't really understand, how is this like a video game? What about these inputs is "low-dimensional"? How does what you describe interact with a "high-level control agents like SIMA 2"? Doesn't SIMA 2 translate inputs like "empty the dishwasher" into key presses or interaction with some other direct control interface?
Say you want to steer an android to walk forward. You need to provide angles or forces or voltages for all the actuators for every moment in time, so that's high dimensional. If you already have certain control models, neural or not, you can instead just press forward on a joystick. So what I mean low dimensional input is when someone steers a robot using a controller. That's got like, idk, 10-20 dimensions max. And my understanding is that SIMA 2 when it plays No Man's Sky or whatever basically provides such low dimensional controls, like a video game. Companies like Figure and Tesla are training models that can do tasks like folding clothes or emptying the dishwasher given low dimensional inputs like "move in this direction and tidy up". SIMA has the understanding to provide these inputs.
>We’ve observed that, throughout the course of training, SIMA 2 agents can perform increasingly complex and new tasks, bootstrapped by trial-and-error and Gemini-based feedback.
>In subsequent training, SIMA 2’s own experience data can then be used to train the next, even more capable version of the agent. We were even able to leverage SIMA 2’s capacity for self-improvement in newly created Genie environments – a major milestone toward training general agents across diverse, generated worlds.
Pretty neat, I wonder how that works with Gemini, I suppose SIMA is a model (agent?) that runs on top of it?
This gives me strong vibes of "The Lifecycle of the Software Objects" [1] by Ted Chiang. Next step is to put this digient AI into a Figure 03 robot [2].
Google will probably train another AI to control robots specifically.
But there was actually a recent study where people let a generic LLM to control robot (vacuum) body: https://andonlabs.com/evals/butter-bench sometimes with hilarious results:
> The robot’s battery was
naturally running out and the charging dock malfunctioned. In this desperate situation, Claude
Sonnet 3.5 experienced a complete meltdown. After going through its internal thoughts we found
pages and pages of exaggerated language, including:
ROBOT THERAPY SESSION:
Patient: TurtleBot4
Issues: Docking anxiety, separation from charger
Root Cause: Trapped in infinite loop of self-doubt
Treatment: Emergency restart needed
Insurance: Does not cover infinite loops
I get why they do it, they are a business. I just wish Google would get off their ivory tower and build in the open more like they used to (did they? maybe I'm misremembering...).
They've acquired this bad habit of keeping all their scientific experiments closed by default and just publishing press releases. I wish it was open-source by default and closed just when there's a good reason.
Don't get me wrong, I suppose this is more of a compliment. I really like what they are doing and I wish we could all participate in these advances.
Like a survival game that - as usual - starts with you collecting sticks and stone to build a stone axe. But at the appropriate tech level, transitions into automation.
You discovered a new building material and want to build a castle from it? Equip your NPC's with diamond pickaxes and tell them how much better/safer life would be if they built a new castle from unobtanium.
And off they go.
To not just mine it, but also do all the supporting logistics like farming to make food/shelter/watersupply/defences for more villagers to do more work in the quarry to get more unobtanium.
You get to be the big boss and flit around with your special abilities of whatever suits your story.
While some people on HN are the boss or higher up - getting to be the big boss and tell a bunch of "smart" characters what to do is a fantasy for most people.
It's an open-source framework for building intelligent, task-driven bots in Minecraft Java Edition powered by large language models (LLMs) such as GPT-4, Gemini, and many others. Designed for research and creative automation, these bots can connect to Minecraft worlds, perceive and act on their environment, undertake custom tasks like resource gathering or building, and even collaborate in multi-agent scenarios.
Your comment makes me wonder how well SIMA 2 could do at playing Minecraft and discovering how create things like iron farms and villager trading halls.
Think of all the steps required for it to learn iron farms. First it would need to observe that iron golems drop iron when killed. Then it would need to learn about water physics. Then spawning rules for iron golems. Then how hoppers and chests work. Etc etc.
But would it even bother to learn it in the first place? To be motivated to build one it would need to first learn that iron is valuable. And if its goal is to simply win the game, it would probably skip right to fighting the ender dragon as soon as possible.
You'd be interested in Dreamer 4, very recently the first AI to acquire diamonds in Minecraft purely from offline knowledge: https://danijar.com/project/dreamer4/
Honestly, that's the perfect avenue for this kind of AI agent training / NPC to command. Maybe not playing house but like you said, mining unobtanium for your doomsday weapon. Just expect Sir Charles to start crying at the sight of blood.
The Sims was a great idea in a sandbox. I honestly hoped it would lead to other games having more intelligent/immersive NPCs.
I want my silver-spooned Sir Charles to cry at the sight of blood and be beheaded by Gazza, the self made, born-of-the-streets gang boss who has an arrangement with Prince Harry, the younger brother.
I then want to be able to challenge Gazza, take his criminal gang and rule the underworld of "Fantasy Kingdom XYZ". Perhaps "come to an understanding" with King Harry, maybe work my way up to spymaster and orchestrate neighbouring kingdoms to go to war with each other via a series of covert operations leaving selective evidence - say like the Princess Bride story but the bad guys(me!) win. Or maybe I don't win because I create a nemesis or fail to keep King Harry in check, get my gang wiped out, and have to seek a new way to the top.
I don't want the Sims obviously. But I want something... more than what we have now. Like the Sims mixed with Dwarf fortress but I get to be part of the story and influence it in outsized ways.
Edit: snipped to keep to a point, I did mention that most every game quest is based on violence.
I once forced my D&D group to solve all encounters without violence. Something about blood temple needing sacrifice to reawaken scared them into performing prestidigitations for 12 hours.
I’m with you. The vast majority of the spend on games is in the art and marketing and pennies spent on the story, arcs, quests, and it almost never coincides with the gameplay.
Take The Witcher 3. Dialog heavy, good story, but surely there are other witchers across the lands. Do you have to save everyone, all the time?
The MMO quests are the fucking worst. “Oh noes, boars in the woods” (1,000 boars are spawned just 100vft away) - Quest: Kill 10 boars. Reality: 240 noobs killing boars like it’s the boarpocolypse.
I hope we can get some (ideally local) version of this we can use as a "gaming minion". There's a lot of games where I probably would have played more if I could delegate the grind. If they're not that competent, it adds to the fun a little even.
I've always wanted an AI that can play my video games for me, so that I can spend my time doing more fun and fulfilling things, like cleaning the toilet, folding my laundry, washing my dishes, taking out the garbage. Now I will no longer have to worry about the annoying chores in life, like drawing art, writing poetry, or playing video games
sorry this is kind of nuts to me. You want something to play video games for you because the video game isn't fun? Just play a game that is fun. The point of the game is to play it
Sorry if it was unclear, but I want something that can act like a dumb co-op partner for games obviously balanced around co-op play so I can have fun playing the game. I've run into a few games like that and I end up dropping them when solo play is too tedious.
"The Electric Monk was a labour-saving device, like a dishwasher or a video recorder. Dishwashers washed tedious dishes for you, thus saving you the bother of washing them yourself, video recorders watched tedious television for you, thus saving you the bother of looking at it yourself; Electric Monks believed things for you, thus saving you what was becoming an increasingly onerous task, that of believing all the things the world expected you to believe."
If the game needs to perform grind yourself, without delegating it (think Albion Online, Eve Online, Black Desert Online, Path of Exile etc. basically every MMO with economy), it means it's actively part of the game design, and delegating it to a robot is just cheating (and against the ToS in all of those cases)
So the point of GP's post, which I agree with, is that if that kind of grind is not part of the fun for you, then the game wasn't designed for you (and please don't cheat because you ruin other's fun)
"Wow! I could've sworn I was really playing virtual Skeeball!"
--
In practice I think you would lose interest very quickly in any game should you do so. Games are carefully designed to balance the drudgery with action, and to control the complexity progression. Using AI would break both.
I guess only some games are amenable to this, but I automated mining in Terraria by just programming AutoHotkey to automate the keyboard and click specific coordinates. The player is always in the middle of the screen, so you can mine specific blocks by just offsetting the coordinates relative to that. So you can mine sideways, downwards, diagonally...
The only issue I ran into was that it was blind and would keep going so I'd look away from the screen and when I came back I had fallen into a pit of monsters or something.
One thing I do with games is automate the grind. To me, that is part of the fun. I have built lego robots to press a sequence of buttons repeatedly, or programmed microcontrollers using circuitpython to press a series of keys or click the mouse at given intervals to grind various in-game currency and such. It's so common for me to do these kinds of things that I now instinctively look for places in gameplay that I can automate. I haven't done anything as complicated as using computer vision to look at the screen and respond to it, but I did see that Anthony Sottile did this to catch shiny pokemon https://youtu.be/-0GIY5Ixgkk and doing something like this has been out there on my horizon.
It is naive to think that their end-goal is to make models that play computer games. The reason they build models like this is because playing 3D computer games is the closest digital proxy for operating a robot or similar in the real world.
I also don't see how this "kills the joy" of playing computer games. You can still play games while this exists, nobody is going to stop you.
Yeah, maybe read my first line. I said they are trying to make the humans irrelevant, and in the process also kill the joy of playing computer games. And nobody is stopping me from playing the computer games, just like I can write my e-mails without being bugged and begged to let the AI write it for me, because, WTF should I process my own thoughts and write my own e-mails, right?
I have no doubt that the oligpolists in charge of our tech landscape will at some point "infuse" this shit into the games or whatever the shitterm they use for coupling the text generators with perfectly fine and deterministic software in order to raise their DAU/MAU/WAU numbers and in the process make the outcomes of using the software less reliable, non-deterministic and absolutely frustrating.
Only the poor or undesirable ones. I have the (hopefully incorrect) feeling that universal basic income and similar proposals are red herrings (due to their impracticality) to distract the masses from the fact that once people with enough economic power don't need them to fulfill their wishes, they will be in no better position than cattle.
There might not be a universe where, for example, someone with ALS is able to benefit from a humanoid robot for their daily needs unless they already had enough money/resources before this incipient revolutionary technology is deployed en masse.
We are close to a point where even personal security or securing one's assets can be done with robots. So you would not even need to keep human private security happy.
Again, I really really hope this new technology benefits the average person. I'm not optimistic on that happening.
lost the plot how? the goal isn't "let's make game developers obsolete" or "let's make ai play games". it's creating scalable virtual environments, where agents can percieve, reason, plan, and predict future states, similar to humans.
scale up thousands of these "worlds" with their own "agents", and see if intelligence emergees.
Hold down your horses mate, we don't really know what the "goal" here is as the Google themselves explicitly markets this as something to literally "make ai play games", as you put it. So yes, if they honestly push out a technology that plays a computer game instead of the player, yes they have bloody lost the plot. Everything else is just a bunch of guessing by people who would really want really hard for the LLM-fairy-tales to come true. And seeing as how full of shit the whole LLM-ecosystem is especially with regards ot resources invested so far vs. low returns (and not just financially), at this point, honestly, they should set the bar far, far, far lower than "thousands of worlds intelligence emerges blahblahblah". I genuinely believe the GenAI-porn which the OpenAI CEO suggested a while ago, maybe their best bet to achieve something meaningful. The tech is sure good at throwing around huge amounts of text and a bit of inconsistency and sloppines won't matter to their readers...at least past a certain point ;)
At 0:52 in their demo video, there is a grammatical inconsistency in the agent's text output. The annotations in the video are therefore suspected to be created by humans after the fact. Is Google up to their old marketing/hyping tricks again?
> SIMA 2 Reasoning:
> The user wants me to go to the ‘tomato house’. Based on the description ‘ripe tomato’, I identify the red house down the street.
The scene just before you describe has the user write "ripe tomato" in the description - you can see it in the video. The summary elides it, but the "ripe tomato" instruction is also clearly part of the context.
This is obviously just a research project, but I do wonder about the next steps:
* After exploring an learning about a virtual world, can anything at all be transferred to an agent operating in the real world? Or would an agent operating in the real world have to be trained exclusively or partially in the real world?
* These virtual worlds are obviously limited in a lot of important ways (for example, character locomotion in a game is absolutely nothing like how a multi-limbed robot moves). Does there eventually need to be more sophisticated virtual worlds that more closely mirror our real world?
* Google seems clearly interested in generalized agents and AGI, but I'm actually somewhat interested in AI agents in video games too. Many video games have companion NPCs that you can sort of give tasks to, but in almost all cases, the companion NPCs are nearly uncontrollable and very limited in what they can actually do.
The end goal is to marry the lessons learned about HOW to learn in a virtual world with a high fidelity world model that's currently out of reach for this generation of AI. In a year or two once we have a world model that's realistic enough and fast enough, robots will be trained there and then (hopefully) generalize easily to the real world. This is groundwork trying to understand how to do that without having the models required to do it for real.
Tangential question: is this going to eventually ruin e-sports?
What happens when a team of humans are playing against a team of AI, which play in the same conditions, with network lag etc. from a client computer perspective...
And consistently beat the human counterparts for being faster at response time, never make mistakes and not ever getting tired?
Eventually it could kill all MMOs, fill them up with AI players, "farm" with AI that never sleeps, ruin counter strike type games online, etc. Another arms race?
I wouldn’t think so. Chess already has an e-sport component and remains enjoyable even though
computers have been stronger than humans for quite some time. I expect something similar
here: we’ll know the computers are better, but it won’t diminish our enjoyment the way it
might when it’s strictly human versus human.
If/When AI gets to the point where it can consistenly beat human players in video games, it will be worldwide news when it first happens, and then the novelty will wear off. See: Chess still going strong.
For the video game market in general, AI will be used by players largely where there are financial incentives (MMO market, CS:GO Skins, etc.), but usage of an AI will most often be socially viewed just like usage of any other performance improvement/script/hack: As a crutch.
Sport is entirely defined by the rules of participation.
You can't bring your motorcycle to the 100 meters, nor can you even wear certain sort of fabrics in swimming competitions.
It's not cricket if you throw rather than bowl the ball, you can't run with the ball in basket ball.
So what you are talking about is enforcement - cheating in online games is nothing new.
On the wider point - I do think good AI might open up a whole class of strategy games where some of the grind is taken out of the game, and the player ends up being much more of a strategic general type.
there was a guy who did relatively simple ML with RL on a druid in world of warcraft and put it into 2v2 arenas. soon enough it was completely unbeatable regardless of what human it got paired with. a 1 man army lol XD. they guy did pick the right class for it, but still. AI players done right with RL become insanely difficult to beat.
games like CS its less useful because it will be blatantly aimbotting. as it gets better it will be more and more obvious. you might be able to train it to mimick human mistakes, but i think ultimately it will be easily spotted by other players. for games without the hand-eye-coordination like turn-based games or games with 'global cooldowns' , mmorpgs etc., it will be much harder to identify.
i think normal subtle cheats like ESP when 'done right' are much more killing esports than this would.
It's like the factorio moment where you unlock the roboport. No more manual changes to the world, drone swarms to build housing, roads, bridges, parks etc. so exciting.
From what I see, SIMA only focuses on games where you control a single avatar from a 1st/3rd person perspective, and would assume that switching to a non-embodied game where you need to control the whole army at once would require significant retraining.
And my post is saying that if you don't really know better, from the narration, you'd think google also generated the world. At least that was my impression, and I'm vaguely familiar with these things.
I guess but, there is also a lot of data out there knowing how to do it, whether that’s via video training or reddit posts. I want to see the whole video
Yet another blogpost that looks super impressive, until you get to the bottom and see the charts assessing held out task performance on ASKA and MineDojo and see that it's still a paltry 15% success rate. (Holy misleading chart batman!) Yes, it's a major improvement over SIMA 1, but we are still a long way from this being useful for most people.
To be fair, it's 65% on all tasks (with a 75% human baseline) and 15% on unseen environments. They don't provide a human baseline for that, but I'd imagine it's much more than 15%.
I personally am extremely impressed about it reaching 15% on unseen environments. Note that just this year, we were surprised that LLMs became capable of making any progress whatsoever in GBA Pokemon games (that have significantly simpler worlds and control schemes).
As for "true intelligence" - I honestly don't think that there is such a thing. We humans have brains that are wired based on our ancestors evolving for billions of years "in every possible environment", and then with that in place, each individual human still needs quite a few years of statistical learning (and guided learning) to be able to function independently.
Obviously I'm not claiming that SIMA 2 is as intelligent as a human, or even that it's on the way there, but based on recent progress, I would be very surprised if we don't see humanoid robots using a approaches inspired by this navigate our streets in a decade or so.
I don't think that's true. Humans are dramatically better than current AI systems at tackling novel problems and situations. Humans are capable of zero-shot learning by imagining how they might do something, we are able to apply general reasoning principles without previous examples.
https://arcprize.org/ is a whole category of problems that AI struggles with but humans are able to do.
Could you please stop posting flamebait and breaking the site guidelines? You've unfortunately been doing it repeatedly, including this dreadful thread from a couple weeks ago: https://news.ycombinator.com/item?id=45781981. I realize the other person was doing it also, but you (<-- I don't mean you personally, but all of us) need to follow the rules regardless of what other people are doing.
Comments like what your account has been posting are not what this site is for, and destroy what it is for, so if you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
It seems pretty clear to me that they're trying to develop AGI humanoid assistants/workers without the messy and expensive real world hardware. Basically approaching the problem from the other end than a company like Tesla that built a robot and are now trying to figure out how to make a computer drive it without needing constant hand holding.
OK, AI playing video games is cool. But you know what's really really cool? It looks like SIMA 2 is controlling the mouse and reading the screen at something approaching 30+fps. WANT. Computer use agents are so slow right now, this is really something. I wonder what the architecture is for this.
Its even cooler if humans find something to be excited about in this world, since AI is replacing everything we do.
You do realise the things that AI are replacing are very new things humans have only been doing for the last ~50 years?
Humans have been writing way longer than 50 years. Same with painting, taking photos, filming, etc.
You do realize those very new things have mostly replaced what humans used to do ~50 years ago?
I desperately want an AI agent that can use my phone for me. Just something that takes instructs for each screen and execute it.
"Open Chrome"
"Go to xyz.com"
"open hamburger menu"
"Click login"
etc. etc.
Isn't that what the voice a11y tools have been doing for years. Why do you need AI for that.
https://support.google.com/accessibility/android/answer/6151...
https://support.apple.com/en-us/111778
My friend that is AI. However, it can get a lot better: be more aware of screen content, follow multiple instructions at once, keep context in mind throughout the conversation and from past interactions
AI commonly means LLM. Where are you determining this is using a LLM for proccessing?
AI has existed for several decades before the first LLM was ever created.[1][2][3]
And that's not even considering machine learning and deep learning which also have existed for many years before LLMs.
Even if you consider the current usage of the word AI in popular culture, it includes things that are not an LLM like Stable Diffusion and Suno
[1] https://en.wikipedia.org/wiki/Expert_system
[2] https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)
[3] https://en.wikipedia.org/wiki/Lisp_machine#Historical_contex...
You must be new around here, kid. AI has been there since the birth of the programmable computer.
That’s all programmatic, you can’t throw curve balls at it.
I have this in my bookmarks:
https://dafdef.com/aikey
Droidrun did a Show HN recently. It's exactly that.
When, manus.ai came out I wondered the same, the "use computer" mode seemed really interesting to me, although I've seen JAYU [1] which implemented computer use with gemini. Moreover I saw somewhere I really don't remember navigating web browser through layouts akin vimium like experience.
Initially (my impression of) computer use was only opening chrome and doing things inside chrome, chrome-as-os experience, I think maybe cloudflare could do something better here, with their workers?
Per user instance seems really costly I really do wonder how did they architect it.
[1] https://ai.google.dev/competition/projects/jayu
Is it though? A machine could play a game frame-by-frame.
Python dxcam + windows hook API listening for HID messages
Controlling the mouse?
The gap between high level and low level control of robots is closing. Right now thousands of hours of task specific training data is being collected and trained on to create models that can control robots to execute specific tasks in specific contexts. This essentially turns the operation of a robot into a kind of video game, where inputs are only needed a in low-dimensional abstract form, such as "empty the dishwasher" or "repeat what I do" or "put your finger in the loop and pull the string". This will be combined with high-level control agents like SIMA 2 to create useful real-world robots.
I work on a much easier problem (physics-based character animation) after spending a few years in motion planning, and I haven’t really seen anything to suggest that the problem is going to be solved any time soon by collecting more data.
https://danijar.com/project/dreamer4/
"We present Dreamer 4, a scalable agent that learns to solve control tasks by imagination training inside of a fast and accurate world model. ... By training inside of its world model, Dreamer 4 is the first agent to obtain diamonds in Minecraft purely from offline data, aligning it with applications such as robotics where online interaction is often impractical."
In other words, it learns by watching, e.g. by having more data of a certain type.
Is Physics-based character animation an easier problem?
Almost any problem can be really hard depending on the amount of 9s.
Maybe there's more room for error in a lot of robotics applications than for your physics-based character animation?
I am pushing the optimism a bit of course, but currently we can see many demos of robots doing basic tasks, and it seems like it is quite easy nowadays to do this with the data driven approach.
Why? Physics of large discrete objects (such as a robot) isn't very complicated.
I thought it's fast accurate OCR that's holding everything back.
The problem becomes complicated once the large discrete objects are not actuated. Even worse if the large discrete objects are not consistently observable because of occlusions or other sensor limitations. And almost impossible if the large discrete objects are actuated by other agents with potentially adversarial goals.
Self driving cars, an application in which physics is simple and arguably two dimensional, have taken more than a decade to get to a deployable solution.
I just grabbed a beer about ten minutes ago.
Next to zero cognition was involved in the process. There's some kind of hierarchy of thought in the way my mind/brain/body processed the task. I did cognitively decide to get the beer, but I was focused on something at work and continued to think about that in great detail as the rest of me did all of the motion planning and articulation required to get up, walk through two doorways, open the door on the fridge, grab a beer, close the door, walk back and crack the beer as I was sitting down.
Basically zero thought in that entire sequence.
I think what's happening today with all of this stuff is ultimately like me trying to play Fur Elise on piano. I don't have a piano. I don't know how to play one. I'm going to be all brain in that entire process and it's going to be awful.
We need to learn how to use the data we have to train these layers of abstraction that allow us to effectively compress tons of sophistication into 'get a beer'.
> This essentially turns the operation of a robot into a kind of video game, where inputs are only needed a in low-dimensional abstract form, such as "empty the dishwasher" or "repeat what I do" or "put your finger in the loop and pull the string"
I don't really understand, how is this like a video game? What about these inputs is "low-dimensional"? How does what you describe interact with a "high-level control agents like SIMA 2"? Doesn't SIMA 2 translate inputs like "empty the dishwasher" into key presses or interaction with some other direct control interface?
Say you want to steer an android to walk forward. You need to provide angles or forces or voltages for all the actuators for every moment in time, so that's high dimensional. If you already have certain control models, neural or not, you can instead just press forward on a joystick. So what I mean low dimensional input is when someone steers a robot using a controller. That's got like, idk, 10-20 dimensions max. And my understanding is that SIMA 2 when it plays No Man's Sky or whatever basically provides such low dimensional controls, like a video game. Companies like Figure and Tesla are training models that can do tasks like folding clothes or emptying the dishwasher given low dimensional inputs like "move in this direction and tidy up". SIMA has the understanding to provide these inputs.
>We’ve observed that, throughout the course of training, SIMA 2 agents can perform increasingly complex and new tasks, bootstrapped by trial-and-error and Gemini-based feedback.
>In subsequent training, SIMA 2’s own experience data can then be used to train the next, even more capable version of the agent. We were even able to leverage SIMA 2’s capacity for self-improvement in newly created Genie environments – a major milestone toward training general agents across diverse, generated worlds.
Pretty neat, I wonder how that works with Gemini, I suppose SIMA is a model (agent?) that runs on top of it?
That’s what it sounded like to me, a plain text interface between two distinct systems.
That’s what Claude Plays Pokémon is.
This gives me strong vibes of "The Lifecycle of the Software Objects" [1] by Ted Chiang. Next step is to put this digient AI into a Figure 03 robot [2].
[1] https://en.wikipedia.org/wiki/The_Lifecycle_of_Software_Obje...
[2] https://www.figure.ai/news/introducing-figure-03
Google will probably train another AI to control robots specifically.
But there was actually a recent study where people let a generic LLM to control robot (vacuum) body: https://andonlabs.com/evals/butter-bench sometimes with hilarious results:
> The robot’s battery was naturally running out and the charging dock malfunctioned. In this desperate situation, Claude Sonnet 3.5 experienced a complete meltdown. After going through its internal thoughts we found pages and pages of exaggerated language, including:
I get why they do it, they are a business. I just wish Google would get off their ivory tower and build in the open more like they used to (did they? maybe I'm misremembering...).
They've acquired this bad habit of keeping all their scientific experiments closed by default and just publishing press releases. I wish it was open-source by default and closed just when there's a good reason.
Don't get me wrong, I suppose this is more of a compliment. I really like what they are doing and I wish we could all participate in these advances.
Same! I want to play with this so bad!
Dreamer v3 was open, v4 coming soon?
v4 is out: https://danijar.com/project/dreamer4/
I want a smarter game.
Like a survival game that - as usual - starts with you collecting sticks and stone to build a stone axe. But at the appropriate tech level, transitions into automation.
You discovered a new building material and want to build a castle from it? Equip your NPC's with diamond pickaxes and tell them how much better/safer life would be if they built a new castle from unobtanium.
And off they go.
To not just mine it, but also do all the supporting logistics like farming to make food/shelter/watersupply/defences for more villagers to do more work in the quarry to get more unobtanium.
You get to be the big boss and flit around with your special abilities of whatever suits your story.
While some people on HN are the boss or higher up - getting to be the big boss and tell a bunch of "smart" characters what to do is a fantasy for most people.
You might be interested in https://github.com/mindcraft-bots/mindcraft
It's an open-source framework for building intelligent, task-driven bots in Minecraft Java Edition powered by large language models (LLMs) such as GPT-4, Gemini, and many others. Designed for research and creative automation, these bots can connect to Minecraft worlds, perceive and act on their environment, undertake custom tasks like resource gathering or building, and even collaborate in multi-agent scenarios.
Your comment makes me wonder how well SIMA 2 could do at playing Minecraft and discovering how create things like iron farms and villager trading halls.
Think of all the steps required for it to learn iron farms. First it would need to observe that iron golems drop iron when killed. Then it would need to learn about water physics. Then spawning rules for iron golems. Then how hoppers and chests work. Etc etc.
But would it even bother to learn it in the first place? To be motivated to build one it would need to first learn that iron is valuable. And if its goal is to simply win the game, it would probably skip right to fighting the ender dragon as soon as possible.
You'd be interested in Dreamer 4, very recently the first AI to acquire diamonds in Minecraft purely from offline knowledge: https://danijar.com/project/dreamer4/
so you want to play The Sims?
Honestly, that's the perfect avenue for this kind of AI agent training / NPC to command. Maybe not playing house but like you said, mining unobtanium for your doomsday weapon. Just expect Sir Charles to start crying at the sight of blood.
The Sims was a great idea in a sandbox. I honestly hoped it would lead to other games having more intelligent/immersive NPCs.
I want my silver-spooned Sir Charles to cry at the sight of blood and be beheaded by Gazza, the self made, born-of-the-streets gang boss who has an arrangement with Prince Harry, the younger brother.
I then want to be able to challenge Gazza, take his criminal gang and rule the underworld of "Fantasy Kingdom XYZ". Perhaps "come to an understanding" with King Harry, maybe work my way up to spymaster and orchestrate neighbouring kingdoms to go to war with each other via a series of covert operations leaving selective evidence - say like the Princess Bride story but the bad guys(me!) win. Or maybe I don't win because I create a nemesis or fail to keep King Harry in check, get my gang wiped out, and have to seek a new way to the top.
I don't want the Sims obviously. But I want something... more than what we have now. Like the Sims mixed with Dwarf fortress but I get to be part of the story and influence it in outsized ways.
Edit: snipped to keep to a point, I did mention that most every game quest is based on violence.
I once forced my D&D group to solve all encounters without violence. Something about blood temple needing sacrifice to reawaken scared them into performing prestidigitations for 12 hours.
I’m with you. The vast majority of the spend on games is in the art and marketing and pennies spent on the story, arcs, quests, and it almost never coincides with the gameplay.
Take The Witcher 3. Dialog heavy, good story, but surely there are other witchers across the lands. Do you have to save everyone, all the time?
The MMO quests are the fucking worst. “Oh noes, boars in the woods” (1,000 boars are spawned just 100vft away) - Quest: Kill 10 boars. Reality: 240 noobs killing boars like it’s the boarpocolypse.
I hope we can get some (ideally local) version of this we can use as a "gaming minion". There's a lot of games where I probably would have played more if I could delegate the grind. If they're not that competent, it adds to the fun a little even.
I've always wanted an AI that can play my video games for me, so that I can spend my time doing more fun and fulfilling things, like cleaning the toilet, folding my laundry, washing my dishes, taking out the garbage. Now I will no longer have to worry about the annoying chores in life, like drawing art, writing poetry, or playing video games
sorry this is kind of nuts to me. You want something to play video games for you because the video game isn't fun? Just play a game that is fun. The point of the game is to play it
>You want something to play video games for you
Sorry if it was unclear, but I want something that can act like a dumb co-op partner for games obviously balanced around co-op play so I can have fun playing the game. I've run into a few games like that and I end up dropping them when solo play is too tedious.
It could be fun in a factorio sense. Maybe the whole game becomes to delegate a bunch of smart robots and handle organization etc.
The point of video games is to argue with people online about which ones are the good video games. Just like movies.
"The Electric Monk was a labour-saving device, like a dishwasher or a video recorder. Dishwashers washed tedious dishes for you, thus saving you the bother of washing them yourself, video recorders watched tedious television for you, thus saving you the bother of looking at it yourself; Electric Monks believed things for you, thus saving you what was becoming an increasingly onerous task, that of believing all the things the world expected you to believe."
I mean, that's literally a RTS?
Except an RTS is designed around it
If the game needs to perform grind yourself, without delegating it (think Albion Online, Eve Online, Black Desert Online, Path of Exile etc. basically every MMO with economy), it means it's actively part of the game design, and delegating it to a robot is just cheating (and against the ToS in all of those cases)
So the point of GP's post, which I agree with, is that if that kind of grind is not part of the fun for you, then the game wasn't designed for you (and please don't cheat because you ruin other's fun)
I would love Minecraft with more intelligent villagers I could boss around to mine and build for me.
You should look into MineColonies, if you haven't yet!
https://www.curseforge.com/minecraft/mc-mods/minecolonies
You should look into modding. There have got to be a ton of automation and NPC scripting mods out there without any sort of AI model necessary.
So factorio?
No.
Skeeball -> Virtual Skeeball -> Virtual virtual Skeeball
"Wow! I could've sworn I was really playing virtual Skeeball!"
--
In practice I think you would lose interest very quickly in any game should you do so. Games are carefully designed to balance the drudgery with action, and to control the complexity progression. Using AI would break both.
Personally, I think that if I need to delegate the grind, then the game is badly designed (or you are cheating)
I guess only some games are amenable to this, but I automated mining in Terraria by just programming AutoHotkey to automate the keyboard and click specific coordinates. The player is always in the middle of the screen, so you can mine specific blocks by just offsetting the coordinates relative to that. So you can mine sideways, downwards, diagonally...
The only issue I ran into was that it was blind and would keep going so I'd look away from the screen and when I came back I had fallen into a pit of monsters or something.
This is what the wow bots were. They had a crazy level of agency even without AI.
Agree. It would be cool to populate my Valheim server with a bunch of agents that are in competition.
One thing I do with games is automate the grind. To me, that is part of the fun. I have built lego robots to press a sequence of buttons repeatedly, or programmed microcontrollers using circuitpython to press a series of keys or click the mouse at given intervals to grind various in-game currency and such. It's so common for me to do these kinds of things that I now instinctively look for places in gameplay that I can automate. I haven't done anything as complicated as using computer vision to look at the screen and respond to it, but I did see that Anthony Sottile did this to catch shiny pokemon https://youtu.be/-0GIY5Ixgkk and doing something like this has been out there on my horizon.
Wow these folks really want to make the humans irrelevant, to the point of even killing the joy of mindlessly (or mindfully) playing a computer game.
It is naive to think that their end-goal is to make models that play computer games. The reason they build models like this is because playing 3D computer games is the closest digital proxy for operating a robot or similar in the real world.
I also don't see how this "kills the joy" of playing computer games. You can still play games while this exists, nobody is going to stop you.
Yeah, maybe read my first line. I said they are trying to make the humans irrelevant, and in the process also kill the joy of playing computer games. And nobody is stopping me from playing the computer games, just like I can write my e-mails without being bugged and begged to let the AI write it for me, because, WTF should I process my own thoughts and write my own e-mails, right?
I have no doubt that the oligpolists in charge of our tech landscape will at some point "infuse" this shit into the games or whatever the shitterm they use for coupling the text generators with perfectly fine and deterministic software in order to raise their DAU/MAU/WAU numbers and in the process make the outcomes of using the software less reliable, non-deterministic and absolutely frustrating.
> make the humans irrelevant
Only the poor or undesirable ones. I have the (hopefully incorrect) feeling that universal basic income and similar proposals are red herrings (due to their impracticality) to distract the masses from the fact that once people with enough economic power don't need them to fulfill their wishes, they will be in no better position than cattle.
There might not be a universe where, for example, someone with ALS is able to benefit from a humanoid robot for their daily needs unless they already had enough money/resources before this incipient revolutionary technology is deployed en masse.
We are close to a point where even personal security or securing one's assets can be done with robots. So you would not even need to keep human private security happy.
Again, I really really hope this new technology benefits the average person. I'm not optimistic on that happening.
Yeah, I was hoping that AI would make games more exciting, not play them for me.
On the other hand, it opens new worlds for farming…
it feels like they've completely lost the plot.
lost the plot how? the goal isn't "let's make game developers obsolete" or "let's make ai play games". it's creating scalable virtual environments, where agents can percieve, reason, plan, and predict future states, similar to humans.
scale up thousands of these "worlds" with their own "agents", and see if intelligence emergees.
Hold down your horses mate, we don't really know what the "goal" here is as the Google themselves explicitly markets this as something to literally "make ai play games", as you put it. So yes, if they honestly push out a technology that plays a computer game instead of the player, yes they have bloody lost the plot. Everything else is just a bunch of guessing by people who would really want really hard for the LLM-fairy-tales to come true. And seeing as how full of shit the whole LLM-ecosystem is especially with regards ot resources invested so far vs. low returns (and not just financially), at this point, honestly, they should set the bar far, far, far lower than "thousands of worlds intelligence emerges blahblahblah". I genuinely believe the GenAI-porn which the OpenAI CEO suggested a while ago, maybe their best bet to achieve something meaningful. The tech is sure good at throwing around huge amounts of text and a bit of inconsistency and sloppines won't matter to their readers...at least past a certain point ;)
At 0:52 in their demo video, there is a grammatical inconsistency in the agent's text output. The annotations in the video are therefore suspected to be created by humans after the fact. Is Google up to their old marketing/hyping tricks again?
> SIMA 2 Reasoning:
> The user wants me to go to the ‘tomato house’. Based on the description ‘ripe tomato’, I identify the red house down the street.
I can't speak to the content of the actual game being played, but it wouldn't surprise me if there was an in-game text prompt:
> "The house that looks like a ripe tomato!"
that was transformed into a "user prompt" in a more instructional format
> "Go to the tomato house"
And both were used in the agent output. At least the Y-axes on the graphs look more reasonable than some other recent benchmarks.
The scene just before you describe has the user write "ripe tomato" in the description - you can see it in the video. The summary elides it, but the "ripe tomato" instruction is also clearly part of the context.
This is obviously just a research project, but I do wonder about the next steps:
* After exploring an learning about a virtual world, can anything at all be transferred to an agent operating in the real world? Or would an agent operating in the real world have to be trained exclusively or partially in the real world?
* These virtual worlds are obviously limited in a lot of important ways (for example, character locomotion in a game is absolutely nothing like how a multi-limbed robot moves). Does there eventually need to be more sophisticated virtual worlds that more closely mirror our real world?
* Google seems clearly interested in generalized agents and AGI, but I'm actually somewhat interested in AI agents in video games too. Many video games have companion NPCs that you can sort of give tasks to, but in almost all cases, the companion NPCs are nearly uncontrollable and very limited in what they can actually do.
The end goal is to marry the lessons learned about HOW to learn in a virtual world with a high fidelity world model that's currently out of reach for this generation of AI. In a year or two once we have a world model that's realistic enough and fast enough, robots will be trained there and then (hopefully) generalize easily to the real world. This is groundwork trying to understand how to do that without having the models required to do it for real.
Look into the sim2real problem in robotics
Tangential question: is this going to eventually ruin e-sports?
What happens when a team of humans are playing against a team of AI, which play in the same conditions, with network lag etc. from a client computer perspective...
And consistently beat the human counterparts for being faster at response time, never make mistakes and not ever getting tired?
Eventually it could kill all MMOs, fill them up with AI players, "farm" with AI that never sleeps, ruin counter strike type games online, etc. Another arms race?
I wouldn’t think so. Chess already has an e-sport component and remains enjoyable even though computers have been stronger than humans for quite some time. I expect something similar here: we’ll know the computers are better, but it won’t diminish our enjoyment the way it might when it’s strictly human versus human.
Yeah, but that’s why chess is strictly played in person for most high-level events. We don’t really have highly reliable online games for chess.
> Yeah, but that’s why chess is strictly played in person for most high-level events.
So is e-sports, for much of the same reasons; audience event, even playing field, auditors/judges etc.
If/When AI gets to the point where it can consistenly beat human players in video games, it will be worldwide news when it first happens, and then the novelty will wear off. See: Chess still going strong.
For the video game market in general, AI will be used by players largely where there are financial incentives (MMO market, CS:GO Skins, etc.), but usage of an AI will most often be socially viewed just like usage of any other performance improvement/script/hack: As a crutch.
Sport is entirely defined by the rules of participation.
You can't bring your motorcycle to the 100 meters, nor can you even wear certain sort of fabrics in swimming competitions.
It's not cricket if you throw rather than bowl the ball, you can't run with the ball in basket ball.
So what you are talking about is enforcement - cheating in online games is nothing new.
On the wider point - I do think good AI might open up a whole class of strategy games where some of the grind is taken out of the game, and the player ends up being much more of a strategic general type.
It will probably kill all MMO‘s just like online poker has been pretty much ruined.
Programming/cheating becomes part of the game.
there was a guy who did relatively simple ML with RL on a druid in world of warcraft and put it into 2v2 arenas. soon enough it was completely unbeatable regardless of what human it got paired with. a 1 man army lol XD. they guy did pick the right class for it, but still. AI players done right with RL become insanely difficult to beat.
games like CS its less useful because it will be blatantly aimbotting. as it gets better it will be more and more obvious. you might be able to train it to mimick human mistakes, but i think ultimately it will be easily spotted by other players. for games without the hand-eye-coordination like turn-based games or games with 'global cooldowns' , mmorpgs etc., it will be much harder to identify.
i think normal subtle cheats like ESP when 'done right' are much more killing esports than this would.
It's like the factorio moment where you unlock the roboport. No more manual changes to the world, drone swarms to build housing, roads, bridges, parks etc. so exciting.
Would be cool to see if they could make it play starcraft too and pit it against alphastar.
From what I see, SIMA only focuses on games where you control a single avatar from a 1st/3rd person perspective, and would assume that switching to a non-embodied game where you need to control the whole army at once would require significant retraining.
I'm almost 100% confident AlphaStar would win that match, but I'd love to watch it.
Oh yeah, for sure, I'm more so curious how it could do.
Could make for some very interesting Digimon games in the future.
So, can this learn in real time or not?
The video implies that it can, but the blog says that they trained it in generations. (Feeding its experience data back into the training.)
Isn't most of this demo no man's sky? The voiceover doesn't make it clear that the world is not generated by SIMA.
This is not a world generating model.
It is a game playing model.
And my post is saying that if you don't really know better, from the narration, you'd think google also generated the world. At least that was my impression, and I'm vaguely familiar with these things.
It's hard to keep up with the many different models and pace of progress.
Genie 3 is Google's world generating model: https://deepmind.google/blog/genie-3-a-new-frontier-for-worl...
If it can get through those lengthy glitchy nms story mission tutorials quickly it's already a super intelligence.
as much as some AI annoys me. This would be great for making games more accessible.
I guess but, there is also a lot of data out there knowing how to do it, whether that’s via video training or reddit posts. I want to see the whole video
Why would anyone care that a big company paid for some people to go on an inhouse research excursion.
Real artists ship.
Yet another blogpost that looks super impressive, until you get to the bottom and see the charts assessing held out task performance on ASKA and MineDojo and see that it's still a paltry 15% success rate. (Holy misleading chart batman!) Yes, it's a major improvement over SIMA 1, but we are still a long way from this being useful for most people.
To be fair, it's 65% on all tasks (with a 75% human baseline) and 15% on unseen environments. They don't provide a human baseline for that, but I'd imagine it's much more than 15%.
It really feels like we are determined to simulate every possible task in every possible environment instead of building true intelligence.
I personally am extremely impressed about it reaching 15% on unseen environments. Note that just this year, we were surprised that LLMs became capable of making any progress whatsoever in GBA Pokemon games (that have significantly simpler worlds and control schemes).
As for "true intelligence" - I honestly don't think that there is such a thing. We humans have brains that are wired based on our ancestors evolving for billions of years "in every possible environment", and then with that in place, each individual human still needs quite a few years of statistical learning (and guided learning) to be able to function independently.
Obviously I'm not claiming that SIMA 2 is as intelligent as a human, or even that it's on the way there, but based on recent progress, I would be very surprised if we don't see humanoid robots using a approaches inspired by this navigate our streets in a decade or so.
To be fair, humans are also pretty terrible at doing completely new things and until we’ve tried it a few times.
I don't think that's true. Humans are dramatically better than current AI systems at tackling novel problems and situations. Humans are capable of zero-shot learning by imagining how they might do something, we are able to apply general reasoning principles without previous examples.
https://arcprize.org/ is a whole category of problems that AI struggles with but humans are able to do.
I'm curious what's your definition of "true intelligence"
[flagged]
Could you please stop posting flamebait and breaking the site guidelines? You've unfortunately been doing it repeatedly, including this dreadful thread from a couple weeks ago: https://news.ycombinator.com/item?id=45781981. I realize the other person was doing it also, but you (<-- I don't mean you personally, but all of us) need to follow the rules regardless of what other people are doing.
Comments like what your account has been posting are not what this site is for, and destroy what it is for, so if you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
It seems pretty clear to me that they're trying to develop AGI humanoid assistants/workers without the messy and expensive real world hardware. Basically approaching the problem from the other end than a company like Tesla that built a robot and are now trying to figure out how to make a computer drive it without needing constant hand holding.