FlashAttention is completely orthogonal to this. This work is about speeding up the computation of Q, K and V vectors while FlashAttention is about speeding up the attention algorithm itself.
Then they should have. It isn’t clear to be that this change is measurable when the code is optimized. Even in the presented numbers the difference seems meh at times.
Comments
I feel like FlashAttention is the relevant baseline here.
FlashAttention is completely orthogonal to this. This work is about speeding up the computation of Q, K and V vectors while FlashAttention is about speeding up the attention algorithm itself.
You could combine the two.
Then they should have. It isn’t clear to be that this change is measurable when the code is optimized. Even in the presented numbers the difference seems meh at times.