Looking at this plot by @oca.computer, I feel like I’m peering into the world’s first microscope and spying bacteria, or through a blurry, early telescope, and spotting invisible dots that turn out to be the previously unknown moons of Jupiter… There is something there! New information to be interpreted!
Comments
Any tools to replicate @oca.computer's work?
Once we have the 1000-dim vector embeddings I can make the rest work. Not sure how to go from 20-word span to a 1000-dim vector embedding.
Generating embeddings is relatively simple with a model and Python code. There's plenty of them on HuggingFace, along with code examples.
all-MiniLM-L6-v2 is a really (if not the most) popular one (albeit not SotA), with 384 dimensions: https://huggingface.co/sentence-transformers/all-MiniLM-L6-v...
Edit: A more modern and robust suite of models comes from Nomic, and can generate embeddings with 64 to 768 dimensions (https://huggingface.co/nomic-ai/nomic-embed-text-v1.5).
When the author talks about thousands of dimensions, they're probably talking about the OpenAI embedding models.