Pretty much yeah. Just to be clear, we only use the color and depth images from the camera. There is actually an offline calibration step to obtain camera intrinsic parameters, which are copied into each scene.
The integrate step runs a SLAM pipeline to compute the trajectory of the camera. Then we run an integration step to obtain the mesh.
Our core philosophy is to not stand in the way once you want to do something custom. So totally, if you want to just read the camera poses and 3D labels and do your own thing, you can totally do that and the data is available in each scene folder.
You only have to label the 3D Bounding Box once. Then you can automatically generate 2D bounding boxes for every frame of the video. So instead of annotating every frame with a 2D box, you only annotate once with a 3D box.
Though, I wonder if the whole hassle of relying on RGB-D sensor of a phone, copying from your phone and using a yet another annotation tool, is worth it, when you can instead use some tracking bbox annotation tool, which interpolates many frames.
With those, you can even annotate moving and distant objects, which I would argue is even better for generalization (since the background changes).
But I bet there are some use cases/users which can profit from it.
Yes if you only care about 2d bbox detection, a smart bounding box annotation tool has some advantages. If you need to solve 3D vision tasks, as is the case in 3D bbox detection, 3D keypoint detection, 6D pose estimation, then you need a tool that can also label the z dimension.
Comments
Just so I understand the idealized pipeline here, a user does the following:
1. Use the Scanner app to take the images and camera pose data
2. Export the scene directory (color and depth images and json files) somehow to your computer
3. Import (integrate, open) the directory via the Stray CLI
4. Annotate voxels via 3D bounding box in Studio GUI
5. Generate labels from the annotated voxels
6. Import data and labels, train and test a detectron model with pytorch
7. Export trained model in torchscript format
8. Profit
I assume you require users to "ETL" the scene directory from your phone to your desktop/laptop via some manual transfer process?
Is there any reason I couldn't stop at step 5 and push my new labeled date to my own training system?
Pretty much yeah. Just to be clear, we only use the color and depth images from the camera. There is actually an offline calibration step to obtain camera intrinsic parameters, which are copied into each scene.
The integrate step runs a SLAM pipeline to compute the trajectory of the camera. Then we run an integration step to obtain the mesh.
Our core philosophy is to not stand in the way once you want to do something custom. So totally, if you want to just read the camera poses and 3D labels and do your own thing, you can totally do that and the data is available in each scene folder.
Do you comprehend how the tool reduces labeling time in (4) and (5) (compared to labeling with eg CVAT) as the post claims?
You only have to label the 3D Bounding Box once. Then you can automatically generate 2D bounding boxes for every frame of the video. So instead of annotating every frame with a 2D box, you only annotate once with a 3D box.
Though, I wonder if the whole hassle of relying on RGB-D sensor of a phone, copying from your phone and using a yet another annotation tool, is worth it, when you can instead use some tracking bbox annotation tool, which interpolates many frames. With those, you can even annotate moving and distant objects, which I would argue is even better for generalization (since the background changes).
But I bet there are some use cases/users which can profit from it.
Yes if you only care about 2d bbox detection, a smart bounding box annotation tool has some advantages. If you need to solve 3D vision tasks, as is the case in 3D bbox detection, 3D keypoint detection, 6D pose estimation, then you need a tool that can also label the z dimension.
Ok, sure. I was just focusing on the 2D detection task, since that seem to be the focus of the linked product page.