To clarify, by "compilers" I mean "deep learning compilers".

There's many different paths to optimizing compilers folks use with PyTorch. One with close integration is NVFuser (see https://www.reddit.com/r/MachineLearning/comments/xa75km/p_p...), although there are other compilers like ONNXRuntime.

Yes, handling autograd (during training) is a whole different thing, and not all compilers support that.