I've been working on research related to this in the context of diferent LLMs, and I can tell you that similarity != executability.
While you can make embeddings across different LLMs similar (i.e. universally looking) the few percentage of R2 that you are missing in translating the universal representation into a native representation are precisely those that make the hidden states executable in the LLM (which makes them useful).
They do this with simple embedding models because there the purpose of the embeddings is to measure similarity, but if you would like to use this principle to turn latent representations of one LLM into latent representations that are understandable/executably by a different LLM, you will fail.
Comments
I've been working on research related to this in the context of diferent LLMs, and I can tell you that similarity != executability.
While you can make embeddings across different LLMs similar (i.e. universally looking) the few percentage of R2 that you are missing in translating the universal representation into a native representation are precisely those that make the hidden states executable in the LLM (which makes them useful).
They do this with simple embedding models because there the purpose of the embeddings is to measure similarity, but if you would like to use this principle to turn latent representations of one LLM into latent representations that are understandable/executably by a different LLM, you will fail.