I'm still tryna understand the difference between multimodal models like Llava and projects like JARVIS that connect LLMs to other huggingface models (including object detection models) or clip. Is a multimodal model doing this under the hood?
Object detection models have human-comprehensible outputs. You can feed in a picture and it'll tell you that there's a child and a cat, and it'll draw bounding boxes around them. You can pass that info into an LLM if you want.
The downside to that approach is the LLM can't tell whether the cat is standing in front of the child, or sitting on the child, or the child is holding the cat; the input just tells it there's a child, and a cat, and their bounding boxes overlap.
In contrast, LLaVA feeds feeds the image into a visual encoder called 'CLIP' which doesn't output anything human-comprehensible - it just gives out a bunch of numbers which have something to do with the contents of the image. But the numbers can be fed into the LLM along with text - and they can train the image encoder and the LLM together.
If the training works right, and they have enough training data for the model to figure out the difference between a cat sitting on a lap and one being held, they end up with a model that can figure out that the child is holding the cat.
Comments
I'm still tryna understand the difference between multimodal models like Llava and projects like JARVIS that connect LLMs to other huggingface models (including object detection models) or clip. Is a multimodal model doing this under the hood?
Object detection models have human-comprehensible outputs. You can feed in a picture and it'll tell you that there's a child and a cat, and it'll draw bounding boxes around them. You can pass that info into an LLM if you want.
The downside to that approach is the LLM can't tell whether the cat is standing in front of the child, or sitting on the child, or the child is holding the cat; the input just tells it there's a child, and a cat, and their bounding boxes overlap.
In contrast, LLaVA feeds feeds the image into a visual encoder called 'CLIP' which doesn't output anything human-comprehensible - it just gives out a bunch of numbers which have something to do with the contents of the image. But the numbers can be fed into the LLM along with text - and they can train the image encoder and the LLM together.
If the training works right, and they have enough training data for the model to figure out the difference between a cat sitting on a lap and one being held, they end up with a model that can figure out that the child is holding the cat.