Ok, I work on PyTorch, so probably should clear up some misconceptions in this thread.

1. In PyTorch (and other array programming libraries like Numpy), the operations being passed around are tensors/arrays (i.e. large chunks of memory). Thus, += is overloaded to mean "in-place write" to the arrays.

So, `+` vs `+=` is the equivalent of

    a: float[1000]
    b: float[1000]
    for i in [0, 1000]:
        b[i] = a[i] + 2

vs.

    a: float[1000]
    for i in [0, 1000]:
        a[i] = a[i] + 2

The main performance advantage comes in 1. no need to allocate an extra array, 2. you're using less memory overall, so various caching levels can work better. It has nothing to do with python bytecodes.

2. As for whether it generally makes sense to do this optimization manually... Usually, PyTorch users don't use in-place operations as its a bit uglier mathematically and have various foot-guns/restrictions that users find confusing. Generally, it's best to have this optimization be done automatically by an optimizing compiler.

3. PyTorch in general does support using in-place operations during training, albeit with some caveats.

(PS) 4. Putting everything on one line (as some folks suggest) is almost certainly not going to help performance - the primary performance bottlenecks here have almost nothing to do with CPU perf.