LLMs are parameter based representations of linguistic representations of the world. Relative to robot predictive control problems, they are low dimensional and static. They are batch trained using supervised learning and are not designed to manage real time shifts in the external world or the reward space. They work because they operate in abstract, rule governed spaces like language and mathematics. They are ill suited to predictive control tasks. They are the IBM 360s of AI. Even so, they are astonishing achievements.
LeCun is right to say that continuous self supervised (hierarchical) learning is the next frontier, and that means we need world models. I'm not sure that JEPA is the right tool to get us past that frontier, but at the moment there are not a lot of alternatives on the table.
See, I don't get why people say that the world is somehow more complex than the world of mathematics. I think that is because people don't really understand what mathematics is. A computer game for example is pure mathematics, minus the players, but the players can also be modelled just by their observed digital inputs / outputs.
So the world of mathematics is really the only world model we need. If we can build a self-supervised entity for that world, we can also deal with the real world.
Now, you may have an argument by saying that the "real" world is simpler and more constrained than the mathematical world, and therefore if we focus on what we can do in the real world, we might make progress quicker. That argument I might buy.
> So the world of mathematics is really the only world model we need. If we can build a self-supervised entity for that world, we can also deal with the real world.
In theory I think you are kind of right, in that you can model a lot of real world behaviour using maths, but it's an extremely inefficient lense to view much of the world through.
Consider something like playing catch on a windy day. If you wanted to model that mathematically there is a lot going on: you've got the ball interacting with gravity, fluid dynamics of the ball moving through the air, the changing wind conditions etc. yet this is a very basic task that many humans can do without really thinking about it.
Put more succinctly, there are many things we'd think of as very basic which need very complex maths to approach.
The world of mathematics is only a language. The (Platonic) concepts go from simple to very complex, but at the base stands a (dynamic and evolving) language.
The real world however is far more complex and perhaps rooted in a universal language, but in one we don’t know (yet) and ultimately try to describe and order by all scientific endeavors combined.
This philosophy is an attempt to point out that you can create worlds from mathematics, but we are far from describing or simulating ‘Our World’ (Platonic concept) in mathematics.
This view of simulation is just wrong and does not correspond at all to human perception.
Firstly, games aren't mathematics. They are low quality models of physics. Mathematics can not say what will happen in reality, mathematics can only describe a model and say what happens in the model. Just mathematics can not say anything about the real world, so a world model just doing mathematics can not say anything about the world either.
Secondly, and far worse for your premise, is that humans do not need these mathematical models. I do not understand the extremely complex mechanical problem of opening a door, to open a door. A world model which tries to understand the world based on mathematics has to. This makes any world model based on mathematics strictly inferior and totally unsuited to the goals.
Danijar Hafner just left DeepMind. He's behind the Dreamer series of models which are IMO the most promising direction for world models anyone has come up with yet. I'm wondering where he's headed. Maybe he could end up at LeCun's startup?
In Dreamer 4 they are able to train an agent to play Minecraft with enough skill to obtain diamonds, without ever playing the game at all. Only by watching humans play. They first build a world model, then train the agent purely in scenarios imagined by the world model, requiring zero extra data or experience. Hopefully it's obvious how generating data from a world model might be useful for training agents in domains where we don't have datasets like the entire internet just sitting around ready-made for us to use.
And the pendulum swings back toward representation. It is becoming clear that the LLM approach is not adequate to reach what John McCarthy called human-level intelligence:
Between us and human-level intelligence lie many problems. They can be summarized as that of succeeding in the "common-sense informatic situation". [1]
I always felt like one of reasons LLMs are so good is that they piggyback on the many years that have gone into developing language as an information representation/compression format. I don’t know if there’s anything similar a world model can take advantage of.
That being said there have been models which are pretty effective at other things that don’t use language, so maybe it’s a non issue.
I think there is a lot of merit to this approach. Ultimately we live in a world guided by physics and macro-level perception driven by our senses and our own motor control. Of course newtonian physics is not the end all be all -- cell biology or quantum mechanics works on a very different level... but what is important here is that we know that human beings understand these things and make novel discoveries on these things using a thinking apparatus that was pre-trained on large scale newtonian physics. I've always found that even in advanced mathematics my mind always uses low level geometric analogies. So the "embeddings" or priors that can be obtained are probably much better than what can be done through text correlation as with LLMs. It's very different to learn the word bounce through observation of a physical model of a ball bouncing vs. seeing what other words it co-occurs with.
I played with Marble yesterday, Fei-Fei/World Labs' new product.
It is the most impressed I've been with an AI experience since the first time I saw a model one-shot material code.
Sure, its an early product. The visual output reminds me a lot of early SDXL. But just look at what's happened to video in the last year and image in the last three. The same thing is going to happen here, and fast, and I see the vision for generative worlds for everything from gaming/media to education to RL/simulation.
I wasn't actually able to use it because the servers were overloaded. What exactly impressed you (or more generally, what does it actually let you do at the moment?).
What you get is a 3D room based on the prompt/image. It rewrites your prompt to a specific format. Overall the rooms tend to be detailed and imaginative.
Then you can fly around the room like in Minecraft creative mode. Really looking forward to more editing features/infill to augment this.
I’m trying to understand the conversation around “world models.” Why is Tesla’s FSD rarely mentioned in these discussions? Their system perceives, reasons, and acts in the physical world, and they train it using large-scale simulation/digital-twin environments. In what sense does FSD not count as a world model—or does it, and I’m missing something?
I don't know why you're focusing on Tesla to the exclusion of more successful self-driving efforts like Waymo, but yeah, cars moving around in and predicting the real world are pretty interesting in this regard.
In "From Words to Worlds: Spatial Intelligence is AI’s Next Frontier" Li states directly "I’m not a philosopher", proceeds to make a philosophical argument that elevates visual perception as basis for evolution of intelligence.
A trillion dollars are now riding on that white whale. An entire naval fleet is being raised for the purposes of chasing down that whale. LeCun and Fei-Fei merely believe that the whale is in a different ocean.
I think video and agentic and multimodal models have led to this point, but actually making a world model may provide to be long and difficult.
I feel LeCun is correct that LLMs as of now have limitations where it needs an architectural overhaul. LLMs now have a problem with context rot, and this would hamper with an effective world model if the world disintegrates and becomes incoherent and hallucinated over time.
It'd doubtful whether investors would be in for the long haul, which may explain the behavior of Sam Altman in seeking government support. The other approaches described in this article may be more investor friendly as there is a more immediate return with creating a 3D asset or a virtual simulation.
Because they are smart enough to realize current LLM tech is nearing a dead end and cannot serve as a full AGI, even ignoring context and hallucination issues, without actual knowledge of the real world.
Everytime I see LeCun talk about world models, I can’t help but think it is also just a tweak on the fundamentals of what is behind current LLM technology. In the end it’s still neural networks. To me, having to “teach” the model how physics works makes me think it can’t be true AGI either.
With all due respect, AI is ultimately a capital game. World models aren’t where real B2B customer revenue comes from—at least compared to today’s LLMs; they’re mainly a better story for raising huge amounts of private capital. Hopefully they figure out how to build the next-gen AI architecture along the way.
I mean both, and in AI today, they’re deeply intertwined. The “capital game” isn’t just about money—it’s about access to compute, talent, and time. Whoever has the resources can experiment, iterate, and potentially uncover the next big architecture. That financial power naturally translates into influence—control over the market, narrative, and ecosystem. In practice, the investment game and the market ruler’s game often become the same thing.
The most useful models are image, video, and audio models. It makes sense that we'd make the video models more 4D aware.
Text really hogged all the attention. Media is where AI is really going to shine.
Some of the most profitable models right now are in music, image, and video generation. A lot of people are having a blast doing things they could legitimately never do before, and real working professionals are able to use the tools to get 1000x more done - perhaps providing a path to independence from bigger studios, and certainly more autonomy for those not born into nepotism.
As long as companies don't over-raise like OpenAI, there should be a smooth gradient from next gen media tools to revolutionary future stuff like immersive VR worlds that you can bend like the Matrix or Holodeck.
And I'll just be exceedingly chuffed if we get open source and highly capable world models from the Chinese that keep us within spitting distance of the unicorns.
Fundamentally what AGI is trying to do is to encode ability to logic and reason. Tokens, images, video and audio are all just information of different entropy density that is the output of that logic reasoning process or emulation of logic reasoning process.
AI might be the biggest transfer of wealth from the rich to the poor in history. Billions have been poured into closed sourced models which have led directly and indirectly to open weight models being available to everyone.
It's not just the cost, but the freedom to do what you want... With open weight models I can run them on my own hardware on the edge, work with data I am not cool with uploading, experiment with different interfaces, use them for things the original trainers did not intend, even retrain the model a bit.
I am developing a p2p program where the model runs on the end user's computer. So I don't even need to pay money for each user and have a bunch of infrastructure monetize them. It is a game changer and allows for a completely different architecture.
That’s awesome, but I think we’re kinda talking past each other. I was responding to the claim that these models represent the largest wealth transfer from rich to poor in history. In order for that to be true, these models, closed or open, need to have value for average people. I don’t see that at all. Most use it as a glorified google, some are actively harmed by the sycophantic tendencies of the models.
Edit: I’d like to add that I personally get a lot of value out of the models. They’ve helped me learn to do frontend development very quickly at my job. That said, that hasn’t translated into higher pay. The expectations have risen with employee capacity.
Pretty similar to social media in a lot of ways. They've strip mined the commons and provided us a corporate controlled walled garden to compensate us for our loss.
If I was smarter, I would have predicted that not only would everyone else figure out that world models are a critical step, but that as a direct consequence the term "world model" would lose all meaning. Maybe next time. That said, Le Cunn's concept in the blog post is the only one worthy of the title.
The naming collision here is unfortunate since the two kinds of models described couldn't be any more different in purpose. Maybe JEPA-type world models should explicitly be called "predictive world models".
The LLM grift is burned up, so this is the next thing. It has just enough new magic tricks to wow the VCs who don't really get what's going on here. I think this comment from the article says it all:
“Taking images and turning them into 3D environments using gaussian splats, depth and inpainting. Cool, but that’s a 3D GS pipeline, not a robot brain.”
One problem with VR and VFX is how expensive it is in terms of man hours to create immersive worlds. This significantly reduces the cost and has applications in all sorts of ways and could realistically improve the availability of content in VR and reduce movie production costs. And that’s just the obvious applications (ignoring that these world models can be used to train AI itself)
who wants to spend time consuming AI art? If the costs are low, then there is no moat to create movies or gaussian splat VR games, and therefore no reason to spend money on movies or VR splat games.
I'm sure there are other valid reasons, but I think the most obvious one is that LLMs are not improving as fast as money asks for so we're moving to the next buzzword.
LLMs are parameter based representations of linguistic representations of the world. Relative to robot predictive control problems, they are low dimensional and static. They are batch trained using supervised learning and are not designed to manage real time shifts in the external world or the reward space. They work because they operate in abstract, rule governed spaces like language and mathematics. They are ill suited to predictive control tasks. They are the IBM 360s of AI. Even so, they are astonishing achievements.
LeCun is right to say that continuous self supervised (hierarchical) learning is the next frontier, and that means we need world models. I'm not sure that JEPA is the right tool to get us past that frontier, but at the moment there are not a lot of alternatives on the table.
See, I don't get why people say that the world is somehow more complex than the world of mathematics. I think that is because people don't really understand what mathematics is. A computer game for example is pure mathematics, minus the players, but the players can also be modelled just by their observed digital inputs / outputs.
So the world of mathematics is really the only world model we need. If we can build a self-supervised entity for that world, we can also deal with the real world.
Now, you may have an argument by saying that the "real" world is simpler and more constrained than the mathematical world, and therefore if we focus on what we can do in the real world, we might make progress quicker. That argument I might buy.
> So the world of mathematics is really the only world model we need. If we can build a self-supervised entity for that world, we can also deal with the real world.
In theory I think you are kind of right, in that you can model a lot of real world behaviour using maths, but it's an extremely inefficient lense to view much of the world through.
Consider something like playing catch on a windy day. If you wanted to model that mathematically there is a lot going on: you've got the ball interacting with gravity, fluid dynamics of the ball moving through the air, the changing wind conditions etc. yet this is a very basic task that many humans can do without really thinking about it.
Put more succinctly, there are many things we'd think of as very basic which need very complex maths to approach.
A computer game is also textures, audio, maybe 3d models and landscapes, music composition, and data manipulation functions (see Minecraft).
Sure mathematics can be said to be at the core of most of that but you’re grossly oversimplifying.
The world of mathematics is only a language. The (Platonic) concepts go from simple to very complex, but at the base stands a (dynamic and evolving) language.
The real world however is far more complex and perhaps rooted in a universal language, but in one we don’t know (yet) and ultimately try to describe and order by all scientific endeavors combined.
This philosophy is an attempt to point out that you can create worlds from mathematics, but we are far from describing or simulating ‘Our World’ (Platonic concept) in mathematics.
Representing concepts from the real world in terms of mathematics, is exactly what an AI model is internally.
This view of simulation is just wrong and does not correspond at all to human perception.
Firstly, games aren't mathematics. They are low quality models of physics. Mathematics can not say what will happen in reality, mathematics can only describe a model and say what happens in the model. Just mathematics can not say anything about the real world, so a world model just doing mathematics can not say anything about the world either.
Secondly, and far worse for your premise, is that humans do not need these mathematical models. I do not understand the extremely complex mechanical problem of opening a door, to open a door. A world model which tries to understand the world based on mathematics has to. This makes any world model based on mathematics strictly inferior and totally unsuited to the goals.
Danijar Hafner just left DeepMind. He's behind the Dreamer series of models which are IMO the most promising direction for world models anyone has come up with yet. I'm wondering where he's headed. Maybe he could end up at LeCun's startup?
In Dreamer 4 they are able to train an agent to play Minecraft with enough skill to obtain diamonds, without ever playing the game at all. Only by watching humans play. They first build a world model, then train the agent purely in scenarios imagined by the world model, requiring zero extra data or experience. Hopefully it's obvious how generating data from a world model might be useful for training agents in domains where we don't have datasets like the entire internet just sitting around ready-made for us to use.
https://danijar.com/project/dreamer4/
And the pendulum swings back toward representation. It is becoming clear that the LLM approach is not adequate to reach what John McCarthy called human-level intelligence:
Between us and human-level intelligence lie many problems. They can be summarized as that of succeeding in the "common-sense informatic situation". [1]
And the search continues...
[1] https://www-formal.stanford.edu/jmc/human.pdf
I always felt like one of reasons LLMs are so good is that they piggyback on the many years that have gone into developing language as an information representation/compression format. I don’t know if there’s anything similar a world model can take advantage of.
That being said there have been models which are pretty effective at other things that don’t use language, so maybe it’s a non issue.
I will gladly take $10B to find out for you.
I think there is a lot of merit to this approach. Ultimately we live in a world guided by physics and macro-level perception driven by our senses and our own motor control. Of course newtonian physics is not the end all be all -- cell biology or quantum mechanics works on a very different level... but what is important here is that we know that human beings understand these things and make novel discoveries on these things using a thinking apparatus that was pre-trained on large scale newtonian physics. I've always found that even in advanced mathematics my mind always uses low level geometric analogies. So the "embeddings" or priors that can be obtained are probably much better than what can be done through text correlation as with LLMs. It's very different to learn the word bounce through observation of a physical model of a ball bouncing vs. seeing what other words it co-occurs with.
The last line is so true! Extremely excited to see where this research in world models takes us to!
I played with Marble yesterday, Fei-Fei/World Labs' new product.
It is the most impressed I've been with an AI experience since the first time I saw a model one-shot material code.
Sure, its an early product. The visual output reminds me a lot of early SDXL. But just look at what's happened to video in the last year and image in the last three. The same thing is going to happen here, and fast, and I see the vision for generative worlds for everything from gaming/media to education to RL/simulation.
I wasn't actually able to use it because the servers were overloaded. What exactly impressed you (or more generally, what does it actually let you do at the moment?).
You give it a text prompt and optional image.
What you get is a 3D room based on the prompt/image. It rewrites your prompt to a specific format. Overall the rooms tend to be detailed and imaginative.
Then you can fly around the room like in Minecraft creative mode. Really looking forward to more editing features/infill to augment this.
Marble appears to be like HunyuanWorld to me, but this time they marketed it as a first step to a world model, and it has multimodal capabilities.
I’m trying to understand the conversation around “world models.” Why is Tesla’s FSD rarely mentioned in these discussions? Their system perceives, reasons, and acts in the physical world, and they train it using large-scale simulation/digital-twin environments. In what sense does FSD not count as a world model—or does it, and I’m missing something?
I don't know why you're focusing on Tesla to the exclusion of more successful self-driving efforts like Waymo, but yeah, cars moving around in and predicting the real world are pretty interesting in this regard.
How soon is now? In other words, when will the general public have access to an assistant or a coach with selected world models?
In "From Words to Worlds: Spatial Intelligence is AI’s Next Frontier" Li states directly "I’m not a philosopher", proceeds to make a philosophical argument that elevates visual perception as basis for evolution of intelligence.
I don’t know enough about this to be sure, but this feels like a white whale.
Human-level language was a white whale just a few years ago.
A.L.I.C.E. was published in '95.
A trillion dollars are now riding on that white whale. An entire naval fleet is being raised for the purposes of chasing down that whale. LeCun and Fei-Fei merely believe that the whale is in a different ocean.
I think video and agentic and multimodal models have led to this point, but actually making a world model may provide to be long and difficult.
I feel LeCun is correct that LLMs as of now have limitations where it needs an architectural overhaul. LLMs now have a problem with context rot, and this would hamper with an effective world model if the world disintegrates and becomes incoherent and hallucinated over time.
It'd doubtful whether investors would be in for the long haul, which may explain the behavior of Sam Altman in seeking government support. The other approaches described in this article may be more investor friendly as there is a more immediate return with creating a 3D asset or a virtual simulation.
Whether or not this is exactly the same thing, I find this glossary entry from NVIDIA interesting: https://www.nvidia.com/en-us/glossary/world-models/
Because they are smart enough to realize current LLM tech is nearing a dead end and cannot serve as a full AGI, even ignoring context and hallucination issues, without actual knowledge of the real world.
Most world models so far are based on transformers, no?
Le Cunn's talk at Harvard informs how far behind he is.
How so?
Everytime I see LeCun talk about world models, I can’t help but think it is also just a tweak on the fundamentals of what is behind current LLM technology. In the end it’s still neural networks. To me, having to “teach” the model how physics works makes me think it can’t be true AGI either.
Earlier: https://news.ycombinator.com/item?id=45914363
With all due respect, AI is ultimately a capital game. World models aren’t where real B2B customer revenue comes from—at least compared to today’s LLMs; they’re mainly a better story for raising huge amounts of private capital. Hopefully they figure out how to build the next-gen AI architecture along the way.
> World models aren’t where real B2B customer revenue comes from
You could say the same thing about AGI. Ultimately capital will realize intelligence is a drawback.
By capital game, do you mean money investment game or market ruler's game?
I mean both, and in AI today, they’re deeply intertwined. The “capital game” isn’t just about money—it’s about access to compute, talent, and time. Whoever has the resources can experiment, iterate, and potentially uncover the next big architecture. That financial power naturally translates into influence—control over the market, narrative, and ecosystem. In practice, the investment game and the market ruler’s game often become the same thing.
The most useful models are image, video, and audio models. It makes sense that we'd make the video models more 4D aware.
Text really hogged all the attention. Media is where AI is really going to shine.
Some of the most profitable models right now are in music, image, and video generation. A lot of people are having a blast doing things they could legitimately never do before, and real working professionals are able to use the tools to get 1000x more done - perhaps providing a path to independence from bigger studios, and certainly more autonomy for those not born into nepotism.
As long as companies don't over-raise like OpenAI, there should be a smooth gradient from next gen media tools to revolutionary future stuff like immersive VR worlds that you can bend like the Matrix or Holodeck.
And I'll just be exceedingly chuffed if we get open source and highly capable world models from the Chinese that keep us within spitting distance of the unicorns.
>> The most useful models are image, video, and audio models
This is wrong. The vast majority of revenue is being generated by text models because they are so useful.
> they are so useful.
Enterprise doesn't know how to use these models to achieve business outcomes.
These subscriptions will unwind, and when they do, it'll be a bloodbath.
I work in an enterprise using LLMs all over the place, well. Our spending is only going to go one way, up.
> Some of the most profitable models right now are in music, image, and video generation.
Which companies are using these.models to run at a profit?
MidJourney, ElevenLabs, Suno, Kling
>Some of the most profitable models right now are in music, image, and video generation.
I don’t think many of the companies running these make a profit right now
That just sounds like text with extra steps.
Fundamentally what AGI is trying to do is to encode ability to logic and reason. Tokens, images, video and audio are all just information of different entropy density that is the output of that logic reasoning process or emulation of logic reasoning process.
> Fundamentally what AGI is trying to do is to encode ability to logic and reason.
No? The Wason selection task has shown that logic and reason are not really core nor essential to human cognition.
It's really verging on speculation, but see chapter 2 of Jaynes 1976 - in particular the section on spatialization and the features of consciousness.
AI might be the biggest transfer of wealth from the rich to the poor in history. Billions have been poured into closed sourced models which have led directly and indirectly to open weight models being available to everyone.
Open weight models aren’t worth very much money to most people.
They do everything the closed weight models do, slightly less effectively, but for way cheaper. I'd buy that for a dollar!
Just because people aren't spending money on them doesn't mean it won't eat your lunch.
The closed weight models aren’t worth very much money to most people, who find a 20 dollar subscription a bit pricey.
It's not just the cost, but the freedom to do what you want... With open weight models I can run them on my own hardware on the edge, work with data I am not cool with uploading, experiment with different interfaces, use them for things the original trainers did not intend, even retrain the model a bit.
I am developing a p2p program where the model runs on the end user's computer. So I don't even need to pay money for each user and have a bunch of infrastructure monetize them. It is a game changer and allows for a completely different architecture.
That’s awesome, but I think we’re kinda talking past each other. I was responding to the claim that these models represent the largest wealth transfer from rich to poor in history. In order for that to be true, these models, closed or open, need to have value for average people. I don’t see that at all. Most use it as a glorified google, some are actively harmed by the sycophantic tendencies of the models.
Edit: I’d like to add that I personally get a lot of value out of the models. They’ve helped me learn to do frontend development very quickly at my job. That said, that hasn’t translated into higher pay. The expectations have risen with employee capacity.
At the cost of buying the poor's thoughts (training data)
Pretty similar to social media in a lot of ways. They've strip mined the commons and provided us a corporate controlled walled garden to compensate us for our loss.
they were always free. The notion of intellectual property is lofty in the first place
If I was smarter, I would have predicted that not only would everyone else figure out that world models are a critical step, but that as a direct consequence the term "world model" would lose all meaning. Maybe next time. That said, Le Cunn's concept in the blog post is the only one worthy of the title.
The naming collision here is unfortunate since the two kinds of models described couldn't be any more different in purpose. Maybe JEPA-type world models should explicitly be called "predictive world models".
Because it's the new hot thing and bubbles aren't just going to, you know, hype themselves?
The LLM grift is burned up, so this is the next thing. It has just enough new magic tricks to wow the VCs who don't really get what's going on here. I think this comment from the article says it all:
“Taking images and turning them into 3D environments using gaussian splats, depth and inpainting. Cool, but that’s a 3D GS pipeline, not a robot brain.”
One problem with VR and VFX is how expensive it is in terms of man hours to create immersive worlds. This significantly reduces the cost and has applications in all sorts of ways and could realistically improve the availability of content in VR and reduce movie production costs. And that’s just the obvious applications (ignoring that these world models can be used to train AI itself)
who wants to spend time consuming AI art? If the costs are low, then there is no moat to create movies or gaussian splat VR games, and therefore no reason to spend money on movies or VR splat games.
That's great! It's just not a world model as these people are pitching it.
It’s for the VCs who missed out early. Now’s their chance!
I'm sure there are other valid reasons, but I think the most obvious one is that LLMs are not improving as fast as money asks for so we're moving to the next buzzword.