This is PyTorch code, so the Python is setting up a bunch of kernels that are executed on the GPU. The switch from + to += might allow two of those kernels to be fused together or something, and that could lead to the large performance gain.
The Python part only runs a handful of times so JIT vs. non-JIT doesn't really make a difference.
Nah it’s because PyTorch has a different implementation for __iadd__. It’s saving a copy by mutating the LHS in-place, and possibly more divergent as comments report broken code.
I haven't played much with torch, but the game is generally that you have a graph of computations which gets JIT compiled into GPU ops. The compiler may have more or less competence at finding modifications (eg, 'fusions') to reduce the number of GPU ops required to perform the computation.
Depends on the field, python is almost never the bottleneck in CV - but it almost always a bottleneck in NLP or Search research.
Regardless, the code in the github PR should be running strictly in pytorch for the heavy lifting. Clearly something is touching python that shouldn't be, or must be for frustrating reasons.
Comments
This is PyTorch code, so the Python is setting up a bunch of kernels that are executed on the GPU. The switch from + to += might allow two of those kernels to be fused together or something, and that could lead to the large performance gain.
The Python part only runs a handful of times so JIT vs. non-JIT doesn't really make a difference.
Nah it’s because PyTorch has a different implementation for __iadd__. It’s saving a copy by mutating the LHS in-place, and possibly more divergent as comments report broken code.
I haven't played much with torch, but the game is generally that you have a graph of computations which gets JIT compiled into GPU ops. The compiler may have more or less competence at finding modifications (eg, 'fusions') to reduce the number of GPU ops required to perform the computation.
See, for example, XLA: https://www.tensorflow.org/xla
It looks like maybe nvFuser is an equivalent library for pytorch? https://pytorch.org/blog/introducing-nvfuser-a-deep-learning...
The Python code is run every time.
Yes but not nearly as much as the GPU code, which I think is what the parent is saying; it’s typically not the bottleneck.
Depends on the field, python is almost never the bottleneck in CV - but it almost always a bottleneck in NLP or Search research.
Regardless, the code in the github PR should be running strictly in pytorch for the heavy lifting. Clearly something is touching python that shouldn't be, or must be for frustrating reasons.