I wasn't trying to suggest that there wasn't end-to-end training. Just that it was based on those models that were subsets of the problem first.
As far as "the drill bit must go near the screw", my idea is maybe there is a representation of 3d geometry, an attached representation of physics interactions, another understanding of types of shapes, another level of drills versus screws.
You don't program it, you train it end to end to reproduce the scenarios, but based on those modules.
I'm not saying it's easy or practical for most projects to do all of that from scratch. But to ke it seems like a sensible program.
As far as voxels versus mesh, I actually think somehow you want it to understand surfaces and high level shapes. Again a pretty big ask.
But the basic concept is that we know these are all important semantic categories and if we can create some modules that encapsulate the understanding then we can get more accurate and more efficient reconstruction.
Comments
I wasn't trying to suggest that there wasn't end-to-end training. Just that it was based on those models that were subsets of the problem first.
As far as "the drill bit must go near the screw", my idea is maybe there is a representation of 3d geometry, an attached representation of physics interactions, another understanding of types of shapes, another level of drills versus screws.
You don't program it, you train it end to end to reproduce the scenarios, but based on those modules.
I'm not saying it's easy or practical for most projects to do all of that from scratch. But to ke it seems like a sensible program.
As far as voxels versus mesh, I actually think somehow you want it to understand surfaces and high level shapes. Again a pretty big ask.
But the basic concept is that we know these are all important semantic categories and if we can create some modules that encapsulate the understanding then we can get more accurate and more efficient reconstruction.