Is this announcement that pinecone is adding their own?
TLDR: they trained their own embeddings model and rely on Cohere for ranking. Pinecone (the database) uses this model automatically to generate and store embeddings.
I assumed that a specific flavour of LLM was needed, an “embedding model” to generate the vectors.
You're mostly right, with one caveat: embeddings models aren't really LLMs in that they're not very large: they just map semantic meaning to numerical space.
This is the golden question. As far as I know, there is no appropriate benchmarking/eval data about this. I think the real value is the first-class integration between their model and their service.
I think the general rule is "the smarter the model the better the embedding" but I can't cite a paper right now. So in theory GPT 4 would give better embeddings (if extracted from the middle layers) but that would be overkill.
Comments
There's more technical detail here: https://www.pinecone.io/blog/integrated-inference/
TLDR: they trained their own embeddings model and rely on Cohere for ranking. Pinecone (the database) uses this model automatically to generate and store embeddings.
You're mostly right, with one caveat: embeddings models aren't really LLMs in that they're not very large: they just map semantic meaning to numerical space.
This is the golden question. As far as I know, there is no appropriate benchmarking/eval data about this. I think the real value is the first-class integration between their model and their service.
I think the general rule is "the smarter the model the better the embedding" but I can't cite a paper right now. So in theory GPT 4 would give better embeddings (if extracted from the middle layers) but that would be overkill.