I am very new to robots and recently decided to go crazy and buy a humanoid from Unitree. I've spent a while building some tools in order to integrate LLMs with vision capabilities that can control the robot motion in a simple manner... and while I saw Opus and Astra crashing against tables, beds and surfaces (yeah, my LIDAR data isn't being used yet), Argon 4 takes a lot of precautions to avoid crashing against walls and other objects! I told my robot to go to a table, and the guy started walking slowly to the table, started steering at it and described the objects in it. Pretty scary for being something I've vibecoded in a matter of hours at home...
Do you think the R1 you got can actually be taught to clean dishes and put them in a dishwasher in their specific sections? I see it doing so in https://www.youtube.com/watch?v=k1JIyXM5qGw
The one in that video is the G1, which is a bit taller and comes with more sensors, but there's a trick and it is the dexterous hands, which can easily cost 10k each.
Based on my very limited knowledge... if you would have dexterous hands with sensitivity on them you could theoretically train a VLA model to do that task, planned by a high-level model like Argon 4, that orchestrates everything. That would as well require in first place a VLA that has similar tasks in its training set and fine-tune it by means of recording hundreds of examples performing teleoperation of the robot hand and arms movements.
Not easy, but definitely getting more and more doable these days.
PS: Didn't spend 20k on hands for my 6k robot, but can confirm it has the cable harness to support them :D
What's the latency like? Is the llm used to create an instruction set for the robot? How is it interrupted for reflex actions like controlling stumbling?
The LLM right now tells a much lower level neural network controlling the legs, arms and head what and how to move at a ~ 500Hz freq, which controls thing like the stumbling. All motor joints and IMU inputs are given to an ONNX-based RL-trained NN which then outputs motor movements (scary since its works well for the scenarios where its has been trained, but it goes in unexpected ways for untrained scenarios).
In any case latency is acceptable for navigation I guess, but avoiding moving obstacles is definitely a no-go right now.
It's 1.23 and weights ~30kg. Even though only the R1 Edu can be in theory programmed, it can be easily modded to enable SSH and root access to deploy custom movement policies and interact with the different on-board daemons to control the movements, grab the video data, etc.
I bought a second-hand Intel DeepSense camera and plugged it on one of the spare USB ports available in the main board to have 3D sensing (even though I really need to play a bit more with it). Right now I am playing with MuJoCo for RL training of new policies, integration with VLAs like NVIDIA Isaac GR00T and integration with LLMs.
I have to say its being a wild ride, and despite the small size of the robot, I have already injured myself with it when playing with the walking policies (If you activate a policy while its hanging, it starts literally kicking and punching you...) I cannot image those owning with the bigger brother of it, the G1...
As a tip from an owner... if you are planning to play around, maybe even the robot dog (Go2) can be a good start and comes with a LIDAR preinstalled. When humanoid robots enter into zero-torque / damping mode, it's 30 kg falling to the floor. I know it might sound stupid, but I didn't realize how heavy and scary this is, at least for me, once you have it in front of you.
I'm a robotics noob currently building a prototype of a snow sweeping robot (www.frost-e.com) using my own design and I'm encouraged to see similar work coming from real places. Namely:
- Zero shot control using a visual harness and action primitives. I've seen this from a few other places as well (https://tml.stanford.edu/homebody/) and it's essentially having a generalist LLM command a robot via a set of predefined actions. Many robots and many tasks can probably be done with this approach, since not everything needs super intricate 6DoF arms or dexterity
- simulation bring-ups: I have a pipeline that takes in satellite imagery and reconstructs the lots and houses in a 3D environment and snows on it so that my virtual robot can learn to clean. With these latest models, you can probably create a higher fidelity digital twin more directly.
- self-improvement: These models (Argon, Astra, etc) can't operate the robot itself, but would be able to create data to distill into smaller models you can actually utilize.
Of course (the watermark is in the lower right), it's just a visualization. I'm spending my money buying parts for the prototype instead of professional renders.
It’s mildly impressive, but Google should be wiping the floor with its competition with all the resources it has. This intelligence isn’t even as good as figure 3. Unitree robots are basically doing parkour while this thing has to be hung up by a hook.
Assuming that models continue to get 90% more efficient every 18 months, and assuming that hardware continues to get 2x more powerful per watt every 2 years...
Aren't we still like a decade away from an Argon model being able to run on battery power for any reasonable amount of time that would fit on a robot that isn't the size of a school bus?
Add on another couple of years for the speed to get to what it likely needs to be to be generally useful instead of narrowly useful?
I mean - I guess we're maybe 5-7 years away from a reasonably priced robot you could plug into the wall like an old vacuum cleaner (that runs on this type of tech).
I think for 95%+ of applications.. you just need a link to a data center anywhere.
My assumption is the hardware will have one/several smaller models running on it for locomotion, balance, etc.
Then the hardware "phones home" for decision making and planning.
Essentially the planning doesn't need to be close to the device as long as the local models can handle some part of the plan execution and make sure the robot doesn't fall over. The latency shouldn't be an issue. In fact more powerful centralized compute quickly creates an advantage over slower local compute in terms of end to end planning.
> Aren't we still like a decade away from an Argon model being able to run on battery power for any reasonable amount of time that would fit on a robot that isn't the size of a school bus?
Sooner or later, the models will stop improving. After that happens, the best models will be etched into model-specific ASICs with weights fixed in mask ROM (like Taalas is already doing), which will result in massive, at least order of magnitude inference efficiency increase.
They said the same thing for transistor density in new chip nodes. It took 40 years of doubling every 2 years to finally hit a wall.
And although the industry hit a wall in density/area, they switched to scaling in 3D or glueing chiplets together. Moore's Law is still working if you change the definition to transistors/integrated circuit.
You can already run top of the line models on something that isn't the size of a school bus. The couple of KW you need to run them are the bigger problem, though on very high end hardware you might get double the efficiency.
Robots stand-alone AI do not really benefit from things like batch mode so you want to keep your LLM as high level as possible and your robot's subsystems as autonomous as possible, very light coupling between the two, definitely nothing time-critical. And that's the challenge: in robotics almost everything is time critical.
I'm naive, but wouldn't industrial applications be enabled by this anyway? You'll have in-house inference hardware that a robot can talk to via local network.
Also, your model probably doesn't need like a lot of its knowledge to control a robot.
That's only bringing up the horizon maybe 1 year. At industrial scale - I guess something like an Amazon warehouse could start to be interesting with a local datacenter IN the warehouse within 2-3 years.
From a cost perspective, I just can't imagine that heavily, heavily specialized models wouldn't crush a general model, though - which Amazon already does.
I wonder to what degree that holds up for very limited domains. You have to balance model size, power budget and cost which is a completely different ballgame than looking at a stationary server with a near infinite power budget and where the costs can be amortized over much larger numbers of hits.
Robotics tends to be at the worst possible intersection point of all three of those: limited power budget, relatively small model size and cheap enough to be affordable when embedded into an appliance.
Training specialized models for every task is a "general method leveraging computation" in the sense of the Bitter Lesson. Making the best possible model for each task is guaranteed to be at least as good as making one model and using it for everything, and often it'll be better.
I am very new to robots and recently decided to go crazy and buy a humanoid from Unitree. I've spent a while building some tools in order to integrate LLMs with vision capabilities that can control the robot motion in a simple manner... and while I saw Opus and Astra crashing against tables, beds and surfaces (yeah, my LIDAR data isn't being used yet), Argon 4 takes a lot of precautions to avoid crashing against walls and other objects! I told my robot to go to a table, and the guy started walking slowly to the table, started steering at it and described the objects in it. Pretty scary for being something I've vibecoded in a matter of hours at home...
Being new to a hobby and spending 30k on a robot is crazy, props to your wealth.
Do you think the R1 you got can actually be taught to clean dishes and put them in a dishwasher in their specific sections? I see it doing so in https://www.youtube.com/watch?v=k1JIyXM5qGw
The one in that video is the G1, which is a bit taller and comes with more sensors, but there's a trick and it is the dexterous hands, which can easily cost 10k each.
Based on my very limited knowledge... if you would have dexterous hands with sensitivity on them you could theoretically train a VLA model to do that task, planned by a high-level model like Argon 4, that orchestrates everything. That would as well require in first place a VLA that has similar tasks in its training set and fine-tune it by means of recording hundreds of examples performing teleoperation of the robot hand and arms movements.
Not easy, but definitely getting more and more doable these days.
PS: Didn't spend 20k on hands for my 6k robot, but can confirm it has the cable harness to support them :D
I'd like to do this too.
What's the latency like? Is the llm used to create an instruction set for the robot? How is it interrupted for reflex actions like controlling stumbling?
The LLM right now tells a much lower level neural network controlling the legs, arms and head what and how to move at a ~ 500Hz freq, which controls thing like the stumbling. All motor joints and IMU inputs are given to an ONNX-based RL-trained NN which then outputs motor movements (scary since its works well for the scenarios where its has been trained, but it goes in unexpected ways for untrained scenarios).
In any case latency is acceptable for navigation I guess, but avoiding moving obstacles is definitely a no-go right now.
How much was it?
Around 5.9k euros in Germany for the R1 Pro.
It's 1.23 and weights ~30kg. Even though only the R1 Edu can be in theory programmed, it can be easily modded to enable SSH and root access to deploy custom movement policies and interact with the different on-board daemons to control the movements, grab the video data, etc.
I bought a second-hand Intel DeepSense camera and plugged it on one of the spare USB ports available in the main board to have 3D sensing (even though I really need to play a bit more with it). Right now I am playing with MuJoCo for RL training of new policies, integration with VLAs like NVIDIA Isaac GR00T and integration with LLMs.
I have to say its being a wild ride, and despite the small size of the robot, I have already injured myself with it when playing with the walking policies (If you activate a policy while its hanging, it starts literally kicking and punching you...) I cannot image those owning with the bigger brother of it, the G1...
Thank you, that’s insanely cheap. I might consider getting one.
As a tip from an owner... if you are planning to play around, maybe even the robot dog (Go2) can be a good start and comes with a LIDAR preinstalled. When humanoid robots enter into zero-torque / damping mode, it's 30 kg falling to the floor. I know it might sound stupid, but I didn't realize how heavy and scary this is, at least for me, once you have it in front of you.
I'm a robotics noob currently building a prototype of a snow sweeping robot (www.frost-e.com) using my own design and I'm encouraged to see similar work coming from real places. Namely:
- Zero shot control using a visual harness and action primitives. I've seen this from a few other places as well (https://tml.stanford.edu/homebody/) and it's essentially having a generalist LLM command a robot via a set of predefined actions. Many robots and many tasks can probably be done with this approach, since not everything needs super intricate 6DoF arms or dexterity
- simulation bring-ups: I have a pipeline that takes in satellite imagery and reconstructs the lots and houses in a 3D environment and snows on it so that my virtual robot can learn to clean. With these latest models, you can probably create a higher fidelity digital twin more directly.
- self-improvement: These models (Argon, Astra, etc) can't operate the robot itself, but would be able to create data to distill into smaller models you can actually utilize.
Is that an AI generated product demo video? Because I personally would not sign up for a $1000/mo service if the main demo video isn't even real...
Enshittification right from the start...
Of course (the watermark is in the lower right), it's just a visualization. I'm spending my money buying parts for the prototype instead of professional renders.
It’s mildly impressive, but Google should be wiping the floor with its competition with all the resources it has. This intelligence isn’t even as good as figure 3. Unitree robots are basically doing parkour while this thing has to be hung up by a hook.
Awesome, now only if i could get my hands two of those robotic hands and make myself my dish cleaning slave...
Can we just not throw trash on the ground in the first place?
Assuming that models continue to get 90% more efficient every 18 months, and assuming that hardware continues to get 2x more powerful per watt every 2 years...
Aren't we still like a decade away from an Argon model being able to run on battery power for any reasonable amount of time that would fit on a robot that isn't the size of a school bus?
Add on another couple of years for the speed to get to what it likely needs to be to be generally useful instead of narrowly useful?
I mean - I guess we're maybe 5-7 years away from a reasonably priced robot you could plug into the wall like an old vacuum cleaner (that runs on this type of tech).
I think for 95%+ of applications.. you just need a link to a data center anywhere.
My assumption is the hardware will have one/several smaller models running on it for locomotion, balance, etc.
Then the hardware "phones home" for decision making and planning.
Essentially the planning doesn't need to be close to the device as long as the local models can handle some part of the plan execution and make sure the robot doesn't fall over. The latency shouldn't be an issue. In fact more powerful centralized compute quickly creates an advantage over slower local compute in terms of end to end planning.
I can see some liability issues with that approach.
> Aren't we still like a decade away from an Argon model being able to run on battery power for any reasonable amount of time that would fit on a robot that isn't the size of a school bus?
Sooner or later, the models will stop improving. After that happens, the best models will be etched into model-specific ASICs with weights fixed in mask ROM (like Taalas is already doing), which will result in massive, at least order of magnitude inference efficiency increase.
They said the same thing for transistor density in new chip nodes. It took 40 years of doubling every 2 years to finally hit a wall.
And although the industry hit a wall in density/area, they switched to scaling in 3D or glueing chiplets together. Moore's Law is still working if you change the definition to transistors/integrated circuit.
That doesn’t help the RAM situation
You can already run top of the line models on something that isn't the size of a school bus. The couple of KW you need to run them are the bigger problem, though on very high end hardware you might get double the efficiency.
Robots stand-alone AI do not really benefit from things like batch mode so you want to keep your LLM as high level as possible and your robot's subsystems as autonomous as possible, very light coupling between the two, definitely nothing time-critical. And that's the challenge: in robotics almost everything is time critical.
I'm naive, but wouldn't industrial applications be enabled by this anyway? You'll have in-house inference hardware that a robot can talk to via local network.
Also, your model probably doesn't need like a lot of its knowledge to control a robot.
That's only bringing up the horizon maybe 1 year. At industrial scale - I guess something like an Amazon warehouse could start to be interesting with a local datacenter IN the warehouse within 2-3 years.
From a cost perspective, I just can't imagine that heavily, heavily specialized models wouldn't crush a general model, though - which Amazon already does.
Everyone thinks specialized models are the ticket, except then they aren't: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
I wonder to what degree that holds up for very limited domains. You have to balance model size, power budget and cost which is a completely different ballgame than looking at a stationary server with a near infinite power budget and where the costs can be amortized over much larger numbers of hits.
Robotics tends to be at the worst possible intersection point of all three of those: limited power budget, relatively small model size and cheap enough to be affordable when embedded into an appliance.
Kitchen appliances run Linux now. I think the economies of scale might win regardless
Training specialized models for every task is a "general method leveraging computation" in the sense of the Bitter Lesson. Making the best possible model for each task is guaranteed to be at least as good as making one model and using it for everything, and often it'll be better.
Probably not many applications require the full AI compute to be local on device (Self-driving cars and military applications come to mind).
And? 5-7 years is nothing at the scale of our modern society and the outcome of what you describe could be enormous.
I don't disagree.
It just seems like jumping the shark a little bit.
It's basically completely unviable for a few years still.
It's viable today via network.
I don't think it's viable in the sense that they are still in a demo stage.
Also, I would assume these models are going to get bigger before they get smaller (but they will get smaller over the long run)
> It's viable today via network.
For a set of tasks that almost no one would use a robot for...
We're going to end up with a factory to build factories that build factories... you could call one such concept factorio!
I think someone should make a game based on this concept.
Some of us have been working on precisely that for a while; preparing for when the models are ready. And they're getting close!
Does that make us the Biters?
We've always have been. The engineer is simply a misaligned AI that views all humans as unintelligent creatures on the planet.