I thought ARC-AGI-3 was explicitly a test of raw model performance excluding the harness? Adding the harness back in doesn't tell us anything new. We've known that agents are capable of long horizon reasoning with sufficient harnesses. GPT-4(?) was capable of beating Pokemon 18 months ago but models only became capable of beating it without a harness in the last six months...?
I don't think GPT-4 was ever used for beating pokemon with or without a harness. Successful attempts include Gemini 2.5 and Opus 4.7 both using relatively advanced custom harnesses that give access to game memory, notes systems, and one-off hacks to get around parts of the game the model gets stuck on. More recently, Fable 5 beat FireRed with a _very_ minimal harness (screenshots and button inputs). That's the only example I know of but that is a very sophisticated and very expensive model compared to GPT-4.
Most of this doesn't discredit your overall point, though.
GPT-o3 was the first OpenAI one to beat Red. The harness used by GPT Plays Pokemon is the most featureful one of the main competitors (GPT, Claude, Gemini), IIRC.
The intent in forbidding harnesses was to prevent an ARC-AGI specific harness, which for example presented the game interface in a more agent-friendly way.
What NVIDIA has here is a generic "evolution" harness, which can be used for any problem.
I think it would be fair game to allow OpenClaw, Hermes, Codex, Grok Bot, this NVIDIA thing, to compete, as long as they don't have ARC-AGI specific skills, toolset.
I'm assuming you're referring to a harness that includes memory -- I generally think of the harness as anything beyond executing the generation loop, but I'm not an expert.
True as that may be, it may be better to optimize models for some amount of memory versus forcing some token count based on a reasoning level, right?
GPT-4(?) was capable of beating Pokemon 18 months ago but models only became capable of beating it without a harness in the last six months...?
GPT-4 was decidedly not capable of beating Pokemon 18 months ago. I doubt it would be able to complete a single level. I don't think people realize how large the advances in model capabilities have been. GPT-4 in a modern harness is absolutely horrendous compared to modern models.
Yes. Presumably you're referring to my use of the word "level". I mean here basically every "level" as denoted by the order of locations and places on the town map that you get (which is usually +- some other locations how game runners refer to different sections of the game).
Comments
I thought ARC-AGI-3 was explicitly a test of raw model performance excluding the harness? Adding the harness back in doesn't tell us anything new. We've known that agents are capable of long horizon reasoning with sufficient harnesses. GPT-4(?) was capable of beating Pokemon 18 months ago but models only became capable of beating it without a harness in the last six months...?
I don't think GPT-4 was ever used for beating pokemon with or without a harness. Successful attempts include Gemini 2.5 and Opus 4.7 both using relatively advanced custom harnesses that give access to game memory, notes systems, and one-off hacks to get around parts of the game the model gets stuck on. More recently, Fable 5 beat FireRed with a _very_ minimal harness (screenshots and button inputs). That's the only example I know of but that is a very sophisticated and very expensive model compared to GPT-4.
Most of this doesn't discredit your overall point, though.
GPT-o3 was the first OpenAI one to beat Red. The harness used by GPT Plays Pokemon is the most featureful one of the main competitors (GPT, Claude, Gemini), IIRC.
Community maintained spreadsheet of the runs: https://docs.google.com/spreadsheets/d/e/2PACX-1vQDvsy5Dt_-P...
The intent in forbidding harnesses was to prevent an ARC-AGI specific harness, which for example presented the game interface in a more agent-friendly way.
What NVIDIA has here is a generic "evolution" harness, which can be used for any problem.
I think it would be fair game to allow OpenClaw, Hermes, Codex, Grok Bot, this NVIDIA thing, to compete, as long as they don't have ARC-AGI specific skills, toolset.
These days calling to a server like openai or anthropic effectively results in a server side harness wrapping the weights.
I'm assuming you're referring to a harness that includes memory -- I generally think of the harness as anything beyond executing the generation loop, but I'm not an expert.
True as that may be, it may be better to optimize models for some amount of memory versus forcing some token count based on a reasoning level, right?
GPT-4 was decidedly not capable of beating Pokemon 18 months ago. I doubt it would be able to complete a single level. I don't think people realize how large the advances in model capabilities have been. GPT-4 in a modern harness is absolutely horrendous compared to modern models.
Have you ever played pokemon?
Yes. Presumably you're referring to my use of the word "level". I mean here basically every "level" as denoted by the order of locations and places on the town map that you get (which is usually +- some other locations how game runners refer to different sections of the game).