for those here using pytorch, theres some really nasty data corruption for computations that arent using powers of two as the coordinate axes.
in gradient calcs ive had examples with multiple orders of magnitude of remative or absolute error.
The linked pr has a one line fix and repro test I spent some time distilling and tracking down. The dimensions ive been using in my models are almost adversarially good at catching these issues!
Comments
for those here using pytorch, theres some really nasty data corruption for computations that arent using powers of two as the coordinate axes.
in gradient calcs ive had examples with multiple orders of magnitude of remative or absolute error.
The linked pr has a one line fix and repro test I spent some time distilling and tracking down. The dimensions ive been using in my models are almost adversarially good at catching these issues!