the 15T tokens that got thrown at Llama-3 didn't seem to hurt. Will be interesting to see how well Phi-2 holds up with it's more curated approach, hopefully they don't get disappeared like WizardLM 2 =)
"The quality of the prompts used in SFT and the preference rankings used in PPO and DPO played a crucial role in the performance of the aligned models. Meta's team carefully curated this data and performed multiple rounds of quality assurance on annotations provided by human annotators."
Comments
the 15T tokens that got thrown at Llama-3 didn't seem to hurt. Will be interesting to see how well Phi-2 holds up with it's more curated approach, hopefully they don't get disappeared like WizardLM 2 =)
"The quality of the prompts used in SFT and the preference rankings used in PPO and DPO played a crucial role in the performance of the aligned models. Meta's team carefully curated this data and performed multiple rounds of quality assurance on annotations provided by human annotators."
* https://www.unite.ai/everything-you-need-to-know-about-llama...