It doesn't necessarily mean anything to reach 99% on the public set. All of the public set is known in advance, so it's possible to hardcode rules that make this easy for the models. ARC-AGI-3 is supposed to measure generalization to unseen games, so the only score that matters is the score on the held out private test set that nobody outside the ARC prize foundation has access to. Also, I believe the private set is significantly harder than the public set.
Comments
It doesn't necessarily mean anything to reach 99% on the public set. All of the public set is known in advance, so it's possible to hardcode rules that make this easy for the models. ARC-AGI-3 is supposed to measure generalization to unseen games, so the only score that matters is the score on the held out private test set that nobody outside the ARC prize foundation has access to. Also, I believe the private set is significantly harder than the public set.