I wonder if the NCD metric says something about distillation too. Would you expect that a model that has been distilled/seen traces from other models would have a smaller NCD? It would be really interesting to see if this holds up and provides evidence of distillation or certainly evidence of model outputs being used in the training mix.
Hello! I wrote the above article and did the NCD on model outputs. The same thought crossed my mind when I saw Gemma misclassified as Gemini quite frequently. And GLM almost as Claude and not as Gemini at all. Gave me the feeling as if GLM didn't train on Gemini generated synthetic data at all but mainly on Claude and GPT.
Comments
I wonder if the NCD metric says something about distillation too. Would you expect that a model that has been distilled/seen traces from other models would have a smaller NCD? It would be really interesting to see if this holds up and provides evidence of distillation or certainly evidence of model outputs being used in the training mix.
Hello! I wrote the above article and did the NCD on model outputs. The same thought crossed my mind when I saw Gemma misclassified as Gemini quite frequently. And GLM almost as Claude and not as Gemini at all. Gave me the feeling as if GLM didn't train on Gemini generated synthetic data at all but mainly on Claude and GPT.
https://eqbench.com/results/creative-writing-v3/hybrid_parsi...
This looks spot on, where is it from?