Performing these optimization processes during inference time has never been very practical for generative tasks, as it requires a lot of time, memory (to store the gradient) and the quality is usually mediocre. I still remember VQGAN+CLIP, the optimization process was to find a latent embedding that would maximize the cosine similarity between the CLIP encoded image and the CLIP encoded prompt, It worked but not very practical.
Comments
I've expected NERF + Diffusion models for a while, but it looks like there's still a lot of work needed before it gets practical.
Performing these optimization processes during inference time has never been very practical for generative tasks, as it requires a lot of time, memory (to store the gradient) and the quality is usually mediocre. I still remember VQGAN+CLIP, the optimization process was to find a latent embedding that would maximize the cosine similarity between the CLIP encoded image and the CLIP encoded prompt, It worked but not very practical.