Now this is a fun idea. If you think of embeddings as a sort of quantization of latent space, what would happen if you “turned off” that quantization? It would obviously make no sense to us, as we can only understand the output of vectors that map to tokens in languages we speak, but you could imagine a language model writing something in a sort of platonic, infinitely precise language that another model with the same latent space could then interpret.