Skip to content

Comment on On the Difficulty of Extrapolation with NN Scalingparent

Comments

I think in theory those techniques should still work in the sense that they give you the best prediction (for some definition of “best”) of the next point to test given all the previous information, but the more hyperparameters you can vary and the further you extrapolate from observations the more likely it is that something surprising happens. You should not expect a fancy tuning algorithm to anticipate surprises—they’re designed to do the opposite by exploiting predictable trends.

AboutSource Built by g1lg1l

Hackerly is an independent reader for Hacker News, built on the public HN API. Not affiliated with Y Combinator.