Since it's based on 3D Gaussians in space, is there a way to obtain sharp images? Inherently, Gaussian functions extent infinitely, so images always look blurry. Don't they?
Of course, \sigma can be optimized to be small, but then it converges to some point representation, doesn't it?
Maybe some CV/ML people can help me understanding.
Yes. The main way to keep the images sharp is to render the models at near the same size & resolution that they were captured, or slightly smaller in size. It’s the same thing as zooming into an image- if you zoom in it gets blurry because the filtered pixels get too big, the highest frequency in the data is now zoom-factor pixels wide. If you zoom out, the Gaussian splat images become sharper automatically (and eventually you run into aliasing issues). The way to obtain sharp images if you want to zoom in is to let the NN hallucinate some high frequency details based on what it learns about similar objects (or otherwise have external knowledge of the likely geometry and material properties not captured in the original image.)
The theoretical Gaussian function is infinite, but splat rendering doesn’t use infinite extent, and that’s not really the reason images look blurry, nor do they always look blurry. (Lots of anti-aliasing pixel filters have theoretically infinite extent, but that doesn’t matter in practice, i.e., what matters is only sigma, not extent, provided the finite extent doesn’t cut off too early.) There is a near optimal range of Gaussian sizes for image sharpness that will antialias without overblurring. The capture / optimization process of opaque objects will probably produce Gaussians that are near this optimal size at the smallest, so if you render them back at the same size, it will stay near the optimal range. Generally, the optimizers we have so far tend to blur a little bit, which is why rendering the reconstruction slightly smaller than the captured image currently tends to sharpen things.
Thinking in 2D for a second, to get a nice crispy edge, you need a long and opaque splat to mark the boundary. Sometimes the long splat could wisp off leaving fuzzy artifacts.
Peyman Milanfar [1] suggested using bump functions instead. Bump functions would allow you to specify cut off intervals but still make the whole function smooth and continuous (good for my gradient optimization freaks)
Comments
Since it's based on 3D Gaussians in space, is there a way to obtain sharp images? Inherently, Gaussian functions extent infinitely, so images always look blurry. Don't they? Of course, \sigma can be optimized to be small, but then it converges to some point representation, doesn't it?
Maybe some CV/ML people can help me understanding.
Yes. The main way to keep the images sharp is to render the models at near the same size & resolution that they were captured, or slightly smaller in size. It’s the same thing as zooming into an image- if you zoom in it gets blurry because the filtered pixels get too big, the highest frequency in the data is now zoom-factor pixels wide. If you zoom out, the Gaussian splat images become sharper automatically (and eventually you run into aliasing issues). The way to obtain sharp images if you want to zoom in is to let the NN hallucinate some high frequency details based on what it learns about similar objects (or otherwise have external knowledge of the likely geometry and material properties not captured in the original image.)
The theoretical Gaussian function is infinite, but splat rendering doesn’t use infinite extent, and that’s not really the reason images look blurry, nor do they always look blurry. (Lots of anti-aliasing pixel filters have theoretically infinite extent, but that doesn’t matter in practice, i.e., what matters is only sigma, not extent, provided the finite extent doesn’t cut off too early.) There is a near optimal range of Gaussian sizes for image sharpness that will antialias without overblurring. The capture / optimization process of opaque objects will probably produce Gaussians that are near this optimal size at the smallest, so if you render them back at the same size, it will stay near the optimal range. Generally, the optimizers we have so far tend to blur a little bit, which is why rendering the reconstruction slightly smaller than the captured image currently tends to sharpen things.
Hard edges are a challenge right now.
Thinking in 2D for a second, to get a nice crispy edge, you need a long and opaque splat to mark the boundary. Sometimes the long splat could wisp off leaving fuzzy artifacts.
Take this example: https://www.shadertoy.com/view/dtSfDD
Peyman Milanfar [1] suggested using bump functions instead. Bump functions would allow you to specify cut off intervals but still make the whole function smooth and continuous (good for my gradient optimization freaks)
1: https://x.com/docmilanfar/status/1719584410348204233
Not working in the field I don't know relevance, but I thought that the "4D Gaussian Splatting"[0] looked like it makes great efficiency gains.
[0] https://news.ycombinator.com/item?id=37905601