The most interesting thing is that the model is relatively small compared to Imagen, Dalle 2 and Parti. They have specially trained this model so that people can easily use it on their GPUs. I think StabilityAI will train a larger version of Stable Diffusion, perhaps with a larger text encoder, as the one used in this model is quite small and I think that is the biggest bottleneck in this model; Imagen shows how scaling the text encoder is actually more important than scaling the generator.
In the end, the architecture of the model is not very different from LDM-400m that CompVis had already trained, but it is conditioned on CLIP text embeddings instead of text tokens, they trained an autoencoder from 512 instead of 256, and of course Stable Diffusion was trained for much more.
Comments
The most interesting thing is that the model is relatively small compared to Imagen, Dalle 2 and Parti. They have specially trained this model so that people can easily use it on their GPUs. I think StabilityAI will train a larger version of Stable Diffusion, perhaps with a larger text encoder, as the one used in this model is quite small and I think that is the biggest bottleneck in this model; Imagen shows how scaling the text encoder is actually more important than scaling the generator. In the end, the architecture of the model is not very different from LDM-400m that CompVis had already trained, but it is conditioned on CLIP text embeddings instead of text tokens, they trained an autoencoder from 512 instead of 256, and of course Stable Diffusion was trained for much more.