All of this stuff seems like a band-aid solution. These things need to be trained ground-up to maintain and update persistent memory (maybe outside the context window?).
Also seems like a requirement for any sort of continual learning capabilities as well.
Memory is not necessarily a good thing. What we need is a combination of sufficiently large context windows (10-100 million tokens) along with curate, bloat-free data.
Comments
All of this stuff seems like a band-aid solution. These things need to be trained ground-up to maintain and update persistent memory (maybe outside the context window?).
Also seems like a requirement for any sort of continual learning capabilities as well.
Memory is not necessarily a good thing. What we need is a combination of sufficiently large context windows (10-100 million tokens) along with curate, bloat-free data.