Hard cases really do require matching up parts of document A with parts of document B
The hard part here is not processing documents, it's determining which documents need to be "matched". This requires having some sort of a "knowledge map", a semantic search space of "knowledge patterns", or maybe even a traditional search engine, so that given a document A a model can find relevant documents - in its long term memory, or in a dataset, or even on the internet. Once the documents are found, you don't really need to load the whole thing into the attention window. When I read a long paper, I do it section by section - I just need to maintain a high level map of the paper in my head. I process one "knowledge pattern" at a time, and every time I do a lookup or a search for relevant patterns. I shouldn't be limiting that search to only what's in my current attention window, even if the window is a million tokens. But yes, the window should be big enough to hold at least two of such patterns (which map to chunks of text, or images/audio/etc) - the one I'm currently processing, and one that is most similar to it in the knowledge space.
Comments
transformers were not up to the task in Feb 2023, and I am hoping for some progress soon that will help.
I think that's coming, OpenAI is talking about some new "DV" model with up to 32k context window: https://twitter.com/transitive_bs/status/1628118163874516992
Hard cases really do require matching up parts of document A with parts of document B
The hard part here is not processing documents, it's determining which documents need to be "matched". This requires having some sort of a "knowledge map", a semantic search space of "knowledge patterns", or maybe even a traditional search engine, so that given a document A a model can find relevant documents - in its long term memory, or in a dataset, or even on the internet. Once the documents are found, you don't really need to load the whole thing into the attention window. When I read a long paper, I do it section by section - I just need to maintain a high level map of the paper in my head. I process one "knowledge pattern" at a time, and every time I do a lookup or a search for relevant patterns. I shouldn't be limiting that search to only what's in my current attention window, even if the window is a million tokens. But yes, the window should be big enough to hold at least two of such patterns (which map to chunks of text, or images/audio/etc) - the one I'm currently processing, and one that is most similar to it in the knowledge space.