staticassertion's point is that the current code's usage of `+=` mutates the x that was passed in by the caller, and their suggestion is to copy x into a function local before mutating it, which is similar to how the original `+` code also worked on a function local x (the result of `attn1() + x`).
That's not the problem though. The problem is that the += operations mutate x in place, but the right hand side reads from x. There is no copy you could insert like that to fix this. You would have to do the following, for example
Instead of
x = op(x) + x -> x += op(x)
Do
x_copy = copy(x)
x += op(x_copy)
If you do
x_copy += op(x_copy)
Then you are still mutating x_copy while op() reads it.
EDIT:
I also don't think copy(x) will copy the actual tensor data, although I'm not super familiar with Pytorch.
Comments
staticassertion's point is that the current code's usage of `+=` mutates the x that was passed in by the caller, and their suggestion is to copy x into a function local before mutating it, which is similar to how the original `+` code also worked on a function local x (the result of `attn1() + x`).
That's not the problem though. The problem is that the += operations mutate x in place, but the right hand side reads from x. There is no copy you could insert like that to fix this. You would have to do the following, for example
Instead of x = op(x) + x -> x += op(x)
Do
x_copy = copy(x) x += op(x_copy)
If you do x_copy += op(x_copy)
Then you are still mutating x_copy while op() reads it.
EDIT: I also don't think copy(x) will copy the actual tensor data, although I'm not super familiar with Pytorch.
Ah, ok, I assumed the issue was happening outside of the function. If the issue is actually those intermediaries being mutated, bummer.
As for copy vs deepcopy, like I said, I have no idea what the type is so I don't know that deepcopy is necessary or not.
In x += op(x)
mutation of LHS x really starts before RHS has been evaluated completely?
I assume so, although if I remember correctly the actual error in the thread is caused by the the gradient computation