But this specific use case the extracts an embedding from the caption which is where CLIP would skip a lot of overhead by going from the image to the embedding directly.
If you were solely doing reverse image search (submit image, generate embeddings, vector search) yes.
This is LLaVA -> text output -> sentence embedding -> (RAG style-ish) search on sentence embedding output based on query input text (back through the sentence embedding).
You could skip the LLaVA step and use CLIP/BLIP-ish caption output -> sentence embedding but pure caption/classification model text output is pretty terrible by comparison. Not only inaccurate, but very little to no context for semantic and extremely short so the sentence embedding models have poor quality input and not much to go on even when the caption/classification is decently accurate.
CLIP does not generate captions, it's simply an encoder, the image and text encoders are aligned so you don't need to generate a caption, you simply encode the image and you later retrieve it using the vector crated by the text encoder (the query).
I'm using CLIP here generically to refer to families/models generating captions by leveraging CLIP as the encoder - of which there are plenty on "The Hub".
Have you actually done the approach I think you're suggesting for anything more complex than "this is a yellow cat"? Not trying to be snarky, genuinely curious. I've done a few of these projects and this approach never comes close to meeting user expectations in the real world.
Comments
But this specific use case the extracts an embedding from the caption which is where CLIP would skip a lot of overhead by going from the image to the embedding directly.
If you were solely doing reverse image search (submit image, generate embeddings, vector search) yes.
This is LLaVA -> text output -> sentence embedding -> (RAG style-ish) search on sentence embedding output based on query input text (back through the sentence embedding).
You could skip the LLaVA step and use CLIP/BLIP-ish caption output -> sentence embedding but pure caption/classification model text output is pretty terrible by comparison. Not only inaccurate, but very little to no context for semantic and extremely short so the sentence embedding models have poor quality input and not much to go on even when the caption/classification is decently accurate.
CLIP does not generate captions, it's simply an encoder, the image and text encoders are aligned so you don't need to generate a caption, you simply encode the image and you later retrieve it using the vector crated by the text encoder (the query).
I'm using CLIP here generically to refer to families/models generating captions by leveraging CLIP as the encoder - of which there are plenty on "The Hub".
Have you actually done the approach I think you're suggesting for anything more complex than "this is a yellow cat"? Not trying to be snarky, genuinely curious. I've done a few of these projects and this approach never comes close to meeting user expectations in the real world.
Do you an example of a query that should fail by using the CLIP embeddings directly but works with the method describe in the article?