For those going down this rabbit hole, perceptual downscaling is state of the art, and the closest thing we have to a Python implementation is here (with a citation of the original paper): https://github.com/WolframRhodium/muvsfunc/blob/master/muvsf...
Other supposedly better CUDA/ML filters give me strange results.
I really wish there are some better general-purpose imaging libraries that steadily implement/copy these useful filters, so that more people can use them out of the box.
Most of languages I've involved are surprisingly lacking in this regard despite their huge potential use cases.
Like, in case of Python, Pillow is fine but it has nothing fancy. You can't even fine-tune parameters of bicubic, let alone billions of new algorithms from video communities.
OpenCV or ML tools like to re-invent the wheels themselves, but often only the most basic ones (and badly as noted in this article).
I found https://dl.acm.org/doi/10.1145/2766891 but I don't like the comparisons. Any designer will tell you, after down-scaling you do a minimal sharpening pass. The "perceptual downscaling" looks slightly over-sharpened to me.
I'd love to compare something I sharpened in photoshop with these results.
That implementation is pretty easy to run! The whole Python block (along with some imports) is something like:
clip = core.imwri.Read(img)
clip = muf.ssim_downscale(clip, x, y)
clip = core.imwri.Write(clip, imgoutput)
clip.set_output()
Any designer will tell you, after down-scaling you do a minimal sharpening pass
This is probably wisdom from bicubic scaling, but you usually dont need further sharpening if you use a "sharp" filter like Mitchell.
Anyway I havent run butteraugli or ssim metrics vs other scalers, I just subjectively observed that ssim_downscale was preserving some edges in video frames that Spline36, Mitchell, and Bicubic were not preserving.
Comments
For those going down this rabbit hole, perceptual downscaling is state of the art, and the closest thing we have to a Python implementation is here (with a citation of the original paper): https://github.com/WolframRhodium/muvsfunc/blob/master/muvsf...
Other supposedly better CUDA/ML filters give me strange results.
There are so many gems in VapourSynth scene.
I really wish there are some better general-purpose imaging libraries that steadily implement/copy these useful filters, so that more people can use them out of the box.
Most of languages I've involved are surprisingly lacking in this regard despite their huge potential use cases.
Like, in case of Python, Pillow is fine but it has nothing fancy. You can't even fine-tune parameters of bicubic, let alone billions of new algorithms from video communities.
OpenCV or ML tools like to re-invent the wheels themselves, but often only the most basic ones (and badly as noted in this article).
VapourSynth is great for ML stuff actually, as it can ingest/output numpy arrays or PNGs, and work with native FP32.
A big sticking point is variable resolution, which it technically supports but doesn't really like without some workarounds.
But yeah I agree, its kinda tragic that the ML community is stuck with the simpler stuff.
Hm, any examples of that?
I found https://dl.acm.org/doi/10.1145/2766891 but I don't like the comparisons. Any designer will tell you, after down-scaling you do a minimal sharpening pass. The "perceptual downscaling" looks slightly over-sharpened to me.
I'd love to compare something I sharpened in photoshop with these results.
That implementation is pretty easy to run! The whole Python block (along with some imports) is something like:
clip = core.imwri.Read(img)
clip = muf.ssim_downscale(clip, x, y)
clip = core.imwri.Write(clip, imgoutput)
clip.set_output()
This is probably wisdom from bicubic scaling, but you usually dont need further sharpening if you use a "sharp" filter like Mitchell.
Anyway I havent run butteraugli or ssim metrics vs other scalers, I just subjectively observed that ssim_downscale was preserving some edges in video frames that Spline36, Mitchell, and Bicubic were not preserving.