Ask HN: Is "3D as software" about to undergo a Cambrian Explosion?
Is code-native 3D about to undergo a Cambrian explosion?
And if that happens, what emergent use cases will we witness? E.g. on-demand 3D, or smart 3D objects that automatically optimize for target environments, etc?
A couple of years ago, I got deeply interested in 3D represented as software, instead of static meshes. More recently, I also built a code-native 3D generation project to explore this line of thinking.
Building it made me realize:
(i) LLMs are increasingly becoming better spatial coders. They're naturally synthesizing sophisticated 3D via code, rather than attempting to output raw vertex data.
(ii) There is an emerging niche of users leveraging Three.js, Python, or C# specifically for dynamic, AI-driven generation at runtime.
So it seems to me we're moving away from pipelines where code just loads a file, toward runtime compilation where code is the file.
If LLMs take the reins of programmable 3D, what happens to the traditional graphics stack?
Comments
This is an interesting question where the answers feel tied to the training data, because with 2D, you primarily have declarative data with HTML/CSS, frontend frameworks, scene graphs. With 3D though, I would think most of the training data is imperative, because of libraries like Three.js and Python scripting for Blender.
For your first point though, I don't think raw vertex data necessarily needs to be the output, it could instead be based on shape primitives with raw points as an escape route, with an architecture similar to the one Quiver/StarVector uses for SVG generation. I think llama-mesh and mesh-gpt prove this semi-viable for 3D?
Even some of those smart 3D object ideas you mentioned could still work with declarative, where you build constraints and resolvers and such into the compiler instead of treating each design as a program.
In my own experimentation with 2D, I haven't actually seen a major gap between the outputs from the agent for imperative vs declarative but I'm curious if you've done any testing with declarative languages for 3D and whether you see a quality gap. It should in theory be much more token efficient?
Tangential thought: In the early 90s there was cutting edge image compression method called Iterated Fractal System (IFS) that was kind of like what you are talking about but for 2D. It achieved impressive image compression and since compression is kind of the inverse of generation by code, I wonder if someone has thought to combine IFS with Gaussian splats, for instance, to draw 3D scenes.
A scene would be represented by trees of splats and scaled copies of trees of splats so the code would "grow" the scene predicted by LLM data from a large database of scenes.