Because self-attention can be replaced with FFT for a loss in accuracy and a reduction in kWh [1], I suspect that the Quantum Fourier Transform can also be substituted for attention in LLMs.
Our first contribution is Optimised Attention, which performs similarly to standard attention, but has 3/4 as many parameters and one matrix multiplication fewer per head. Next, we introduce Efficient Attention, which performs on par with standard attention with only 1/2 as many parameters as many parameters and two matrix multiplications fewer per head and is up to twice as fast as standard attention. Lastly, we introduce Super Attention, which surpasses standard attention by a significant margin in both vision and natural language processing tasks while having fewer parameters and matrix multiplications.
A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.
The 2D Discrete Fourier Transform (DFT) can replace attention because the 2D DFT looks a lot like attention. The algorithm loops over each pair of values in the input matrix, calculating first a linear combination of each pair that scales based on the distance between the values in the matrix. Closer values have more influence on each other than more distance values. This linear combination is fed into the famous e^(i*pi) equation that transforms it into a sinusoid. The sinusoids are summed up, giving you an output matrix that is a sum of sinusoids representing the frequency domain.
By comparison, the attention algorithm also combines each pair of values in the input matrix, but that linear combination involves learned weights, rather than simple fixed basis functions (sines and cosines) in the DFT.
The FNet paper suggests that the DFT works as a replacement for attention because in making the pairwise calculations to generate the 2D frequency representation of the input, information on these pairwise relationships is surfaced and made available to later feed-forward layers. As a handy add-on feature, the DFT also makes positional embedding unnecessary, because positional information is encoded by the DFT at each layer. That being said, in the paper they still applied positional embeddings so that they could make a direct comparison with the BERT architecture.
Amazing how these old signal processing ideas have made their way into neural networks, still proving their effectiveness and efficiency after 100 years or more (Gauss devised the Fast Fourier Transform algorithm in 1805 but we left it mostly on the shelf until it was reinvented by Cooley and Tukey in 1965).
Can't believe that FNet paper flew under my radar all this time—what a cool idea! It's remarkable to me that it works so well considering I haven't heard anyone mention it before! Do you know if any follow-up work was done?
This is the paper, which appears to be cited in hundreds of others, some of which appear to be about efficiency gains. https://arxiv.org/abs/2105.03824
Comments
Because self-attention can be replaced with FFT for a loss in accuracy and a reduction in kWh [1], I suspect that the Quantum Fourier Transform can also be substituted for attention in LLMs.
[1] https://syncedreview.com/2021/05/14/deepmind-podracer-tpu-ba...
"You Need to Pay Better Attention" (2024) https://arxiv.org/abs/2403.01643 :
"Leave No Context Behind: Efficient Infinite Context Transformers" (2024) https://arxiv.org/abs/2404.07143 :
The 2D Discrete Fourier Transform (DFT) can replace attention because the 2D DFT looks a lot like attention. The algorithm loops over each pair of values in the input matrix, calculating first a linear combination of each pair that scales based on the distance between the values in the matrix. Closer values have more influence on each other than more distance values. This linear combination is fed into the famous e^(i*pi) equation that transforms it into a sinusoid. The sinusoids are summed up, giving you an output matrix that is a sum of sinusoids representing the frequency domain.
By comparison, the attention algorithm also combines each pair of values in the input matrix, but that linear combination involves learned weights, rather than simple fixed basis functions (sines and cosines) in the DFT.
The FNet paper suggests that the DFT works as a replacement for attention because in making the pairwise calculations to generate the 2D frequency representation of the input, information on these pairwise relationships is surfaced and made available to later feed-forward layers. As a handy add-on feature, the DFT also makes positional embedding unnecessary, because positional information is encoded by the DFT at each layer. That being said, in the paper they still applied positional embeddings so that they could make a direct comparison with the BERT architecture.
Amazing how these old signal processing ideas have made their way into neural networks, still proving their effectiveness and efficiency after 100 years or more (Gauss devised the Fast Fourier Transform algorithm in 1805 but we left it mostly on the shelf until it was reinvented by Cooley and Tukey in 1965).
Can't believe that FNet paper flew under my radar all this time—what a cool idea! It's remarkable to me that it works so well considering I haven't heard anyone mention it before! Do you know if any follow-up work was done?
This is the paper, which appears to be cited in hundreds of others, some of which appear to be about efficiency gains. https://arxiv.org/abs/2105.03824
"Fnet: Mixing tokens with fourier transforms" (2021) https://arxiv.org/abs/2105.03824
https://scholar.google.com/scholar?cites=1423699627588508486...
Fourier Transform and convolution.
Shouldn't a deconvolvable NN be more explainable? #XAI
Deconvolution: https://en.wikipedia.org/wiki/Deconvolution
I know AI is moving fast but they were only published within the last month or three.
I was referring to link [1] in GP’s comment, which is to a paper published in 2021, not to either of the more recent papers they published.
Oops, apologies.