Yeah, I'm with you on this, I think this is just what fable/opus-5 slop looks like now...
- "A frozen verify.py accepts the claim" (what does it mean to freeze a python script?)
- "which trains the recipe eight times on fixed seeds it can't touch" (what does it mean to not be able to touch a seed)
- "One other detail is that we gave an estimation of the speedrun noise in program.md that was slightly too large. 62 out of ~100 runs measured it themselves instead of trusting our number" (What does it mean for a "run" to "distrust" a noise measurement)
- "One important disclaimer is that our benchmark has a lot of variance" (Actually this one makes sense, but congratulations for burying the lede that your entire article is bogus.)
- "Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind" (This implies the the graph would show every model finding a plateau in whatever metric the experiment is measuring, but I don't see every model scoring the same in the graph)
Those damn invented terms are so annoying. We can't see their reasoning anymore unfortunately but I wonder if it's terminology they invent for themselves early in their reasoning token burn and by the time they're outputting a final message they've used that term talking themselves so many hundreds of times that they start assuming the meaning of the term is common knowledge
They gave 18 frontier models the task of “researching” how to improve a lab-rat nano model’s training. Stopping when it met a quality goal of a target loss rate. During each autonomous research session, the AI repeatedly tried changes, tested them, and used the results to decide what to try next. They repeated the whole research session many times with different seeds to average out variance.
Basically they do 8 runs trying to optimize to under 3.28 loss in the fewest training steps possible under time/token constraint. I dunno why 18 * 8 != 153 (it's 144)
Comments
“We ran 153 autonomous runs across 18 frontier models on the nanoGPT optimizer speedrun.”
Uh.. okay.. but whats a run… read blog
“We want to measure how well frontier models can conduct research….””we ran 153 autonomous runs on the nanoGPT optimizer speedrun across”
Okay but what is a optimiser run and what connection does it have to being good at research?
“For comparison, Anthropic's internal automated AI R&D evaluation optimizes a model on a CPU node,”
So I should go look what Anthropic was doing to understand?
Why not just explain what it means in their blog..
Yeah, I'm with you on this, I think this is just what fable/opus-5 slop looks like now...
- "A frozen verify.py accepts the claim" (what does it mean to freeze a python script?)
- "which trains the recipe eight times on fixed seeds it can't touch" (what does it mean to not be able to touch a seed)
- "One other detail is that we gave an estimation of the speedrun noise in program.md that was slightly too large. 62 out of ~100 runs measured it themselves instead of trusting our number" (What does it mean for a "run" to "distrust" a noise measurement)
- "One important disclaimer is that our benchmark has a lot of variance" (Actually this one makes sense, but congratulations for burying the lede that your entire article is bogus.)
- "Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind" (This implies the the graph would show every model finding a plateau in whatever metric the experiment is measuring, but I don't see every model scoring the same in the graph)
Those damn invented terms are so annoying. We can't see their reasoning anymore unfortunately but I wonder if it's terminology they invent for themselves early in their reasoning token burn and by the time they're outputting a final message they've used that term talking themselves so many hundreds of times that they start assuming the meaning of the term is common knowledge
So maybe this is a simple way to put it..
They gave 18 frontier models the task of “researching” how to improve a lab-rat nano model’s training. Stopping when it met a quality goal of a target loss rate. During each autonomous research session, the AI repeatedly tried changes, tested them, and used the results to decide what to try next. They repeated the whole research session many times with different seeds to average out variance.
Thank you. The primeintellect writing is like an icepick to the forehead.
I think that's explained here:
https://www.primeintellect.ai/blog/measuring-autonomous-rese...
Basically they do 8 runs trying to optimize to under 3.28 loss in the fewest training steps possible under time/token constraint. I dunno why 18 * 8 != 153 (it's 144)