The blog post doesn't seem to mention what strikes me as the most interesting application of a model like this, namely extracting semantic information from its latent space. It mentions robotics applications, but only in the context of generating realistic world models for simulation.
If you have a robot deployed in an environment, generating synthetic views of the environment you're in doesn't have any obvious value. What does have obvious value is the latent knowledge that the model could have used to generate those synthetic views.
For instance, the fact that Atlas is capable of identifying regions of the input images that look like "floors", and smoothly interpolating them and filling in gaps with more floor, suggests that it has a concept of "floor-like walkability" which it's learned from the examples in its training data. And being able to identify the regions of 3D space that correspond to that semantic label would obviously be useful for robot path planning.
There's plenty of literature about e.g. using neural networks to estimate walkable areas from a point cloud. And you could imagine just bolting one of those methods to the front of Atlas, using the synthesized point cloud (instead of traditional photogrammetry or LIDAR) as input. But that seems like it's throwing away a lot of potentially useful semantic information, on top of being needlessly inefficient.
As a World Labs customer, I suspect they are using existing 3D assets when creating their environments. For one thing, a vendor of 3d models for AI mentioned Word Labs as a customer on a webinar.
I don't know if that undercuts what you're saying entirely. But if they're just protecting the splat in real-time based on the camera path through a 3d world, would the model truly understand that it's a floor?
I am not sure why creating synthetic view of the environment should not have any value.
When you plan your path to go somewhere you can imagine how it would look like based on what you know and when new information comes in that contradicts it it gets updated to what it's really like.
The robot could do the same thing. It could create images/views of what path it wants to take and start acting on it without knowing the real situation yet.
Granted, the 3D representation that atlas also provides is probably more useful than just an image but the idea remains the same, we are giving robots a kind of idea how the current world around them most likely is based on previously learned data of other locations
What is the scenario you're thinking about? This is kinda what happens with VLAs, the latent knowledge of an LLM can help in action generation pipelines, but it's also kinda slow.
Comments
The blog post doesn't seem to mention what strikes me as the most interesting application of a model like this, namely extracting semantic information from its latent space. It mentions robotics applications, but only in the context of generating realistic world models for simulation.
If you have a robot deployed in an environment, generating synthetic views of the environment you're in doesn't have any obvious value. What does have obvious value is the latent knowledge that the model could have used to generate those synthetic views.
For instance, the fact that Atlas is capable of identifying regions of the input images that look like "floors", and smoothly interpolating them and filling in gaps with more floor, suggests that it has a concept of "floor-like walkability" which it's learned from the examples in its training data. And being able to identify the regions of 3D space that correspond to that semantic label would obviously be useful for robot path planning.
There's plenty of literature about e.g. using neural networks to estimate walkable areas from a point cloud. And you could imagine just bolting one of those methods to the front of Atlas, using the synthesized point cloud (instead of traditional photogrammetry or LIDAR) as input. But that seems like it's throwing away a lot of potentially useful semantic information, on top of being needlessly inefficient.
As a World Labs customer, I suspect they are using existing 3D assets when creating their environments. For one thing, a vendor of 3d models for AI mentioned Word Labs as a customer on a webinar.
I don't know if that undercuts what you're saying entirely. But if they're just protecting the splat in real-time based on the camera path through a 3d world, would the model truly understand that it's a floor?
I am not sure why creating synthetic view of the environment should not have any value.
When you plan your path to go somewhere you can imagine how it would look like based on what you know and when new information comes in that contradicts it it gets updated to what it's really like.
The robot could do the same thing. It could create images/views of what path it wants to take and start acting on it without knowing the real situation yet.
Granted, the 3D representation that atlas also provides is probably more useful than just an image but the idea remains the same, we are giving robots a kind of idea how the current world around them most likely is based on previously learned data of other locations
What is the scenario you're thinking about? This is kinda what happens with VLAs, the latent knowledge of an LLM can help in action generation pipelines, but it's also kinda slow.
Sounds like something Atlas probably is a great base model for.