I think it just boils down to what they were trained on, some models do better when the training sets are more specific even if they're smaller sometimes, so the engineers chase better wholesale performance while leaving some of the weirder edge cases to be cleaned up later eg text generation. maybe start with the image and try adding the text after in a separate prompt if you haven't already?