Logical direction, but the real killer feature would be for an LLM to read every page you bookmark and not just categorize it once for you, but add it to a queryable knowledge-base that you can reference at any point in future conversations.
I tried to use it, and it just does not work very well, I don't it's because implementation is bad, it's just RAG (retrieval based generation) does not work well outside of some simple use cases.
I think it works better if query is a larger chunk of text. Like, if you have an email from a customer and want to compose a response based on some relevant documentation, it should work well.
But for a use case where you want to retrieve something from browsing history you would mainly use a short search query, just few words. in this case embeddings are too ambiguous and relevance of retrieved content is not great.
That’s not a problem with RAG itself that’s an issue with your retriever. In the original RAG paper they used two vanilla BERT models and cosine similarity but there’s no requirement you do that. Use any retriever that gets you high precision. Use BM25 if you want, it’s simple and cheap.
You’re right in saying there’s not enough semantic meaning in the text of the query. The domain of queries and the domain of documents are very different. That’s why a real retrieval system will train the query encoder and doc encoder to be closer in their embedding space using click data. This is what Google is doing.
"train the query encoder and doc encoder to be closer in their embedding space using click data" <- Any papers/resources you know where I can learn more about this process?
Triplet loss takes an anchor, positive, and negative. In this case the anchor is your query, the positive is a similar doc, and the negative is a dissimilar doc. When you train, backpropagate the loss to both the doc and the query encoder.
Apple and Mozilla aren’t Google. Apple runs their own searches on the App Store and iPhone, but Google pays them to be default. There’s no reason Apple couldn’t also index your search history in a more meaningful way other than perhaps space requirements.
Comments
Logical direction, but the real killer feature would be for an LLM to read every page you bookmark and not just categorize it once for you, but add it to a queryable knowledge-base that you can reference at any point in future conversations.
There is a nice opensource extension that does exactly that: https://github.com/memex-life/memex
I tried to use it, and it just does not work very well, I don't it's because implementation is bad, it's just RAG (retrieval based generation) does not work well outside of some simple use cases.
What are those simple use cases, and where do you see Retrieval Augmented Generation fall over?
I think it works better if query is a larger chunk of text. Like, if you have an email from a customer and want to compose a response based on some relevant documentation, it should work well.
But for a use case where you want to retrieve something from browsing history you would mainly use a short search query, just few words. in this case embeddings are too ambiguous and relevance of retrieved content is not great.
That’s not a problem with RAG itself that’s an issue with your retriever. In the original RAG paper they used two vanilla BERT models and cosine similarity but there’s no requirement you do that. Use any retriever that gets you high precision. Use BM25 if you want, it’s simple and cheap.
You’re right in saying there’s not enough semantic meaning in the text of the query. The domain of queries and the domain of documents are very different. That’s why a real retrieval system will train the query encoder and doc encoder to be closer in their embedding space using click data. This is what Google is doing.
"train the query encoder and doc encoder to be closer in their embedding space using click data" <- Any papers/resources you know where I can learn more about this process?
Triplet loss.
https://pytorch.org/docs/stable/generated/torch.nn.TripletMa...
Triplet loss takes an anchor, positive, and negative. In this case the anchor is your query, the positive is a similar doc, and the negative is a dissimilar doc. When you train, backpropagate the loss to both the doc and the query encoder.
alway wonder why browsers don't keep at least a text version of the pages for search
nowadays I expect browsers to incorporate small LLMs like Mistral out of the box
if history search worked better then you would visit google/bing less, so I expect pushback from browser vendors
also: https://bugs.chromium.org/p/chromium/issues/detail?id=297648
From a comment on that issue page, dating a little over a decade ago:
Yes, those were good times. Nowadays, it's anathema to software vendors, and increasingly even to open source devs.
Not every browser vendor runs a search engine…
Apple and Mozilla aren’t Google. Apple runs their own searches on the App Store and iPhone, but Google pays them to be default. There’s no reason Apple couldn’t also index your search history in a more meaningful way other than perhaps space requirements.
Another advertising revenue win for Query Jerry from marketing.
Opera used to do this! And then when you searched in the top bar, it would also search your visited pages. It was very useful.
Zenfetch (https://www.zenfetch.com/) does an awesome job at this!
the ability to do this is pretty accessible nowadays, its just a little expensive for normal use for the layman
That gets real messy real fast!