From a consumer perspective, this is a super interesting paper because it touches on one of the fundamental issues with most RAG beyond the toy case - that you need to do different stuff depending on what the user is asking for. You also (usually) can't just ask because most users don't know that LLMs are bad at math or semantic search won't be sufficient to answer questions that involve enumeration or totality. And while you can always add more steps to your RAG pipeline, some of those steps may be computationally expensive or not particularly relevant to the question at hand.
That being said, it is a bit frustrating that so much RAG research focuses on multi-hop approaches with LLMs. IME multiple round trips to an LLM is essentially a non-starter for any serious consumer product as it's far too slow. Smaller models can struggle to follow instructions so they often can't be an adequate replacement even for simpler tasks. Curious to hear if other folks working in this space have had any success thinking critically about these types of problems!
That depends on the model, you can run stuff in parallel and sometimes keep everything timely. You shouldn't be waiting till the last second to start running rag, you can be pre-emptively building context based on the current chat (like a human does) so that you've already got stuff summarized and ready to fire off when the final prompt does come.
Think about how a human will draw out a conversation around answering a question and use delaying words and phrases to let them continue answering when they don't have the solution fully formulated. LLMs can use the same tactic.
Sure, if you're running a customer service chatbot, you can ask customers what the problem is, then start running rag async to populate a proper context for a smart LLM, and have the chatbot continue asking some questions to clarify details to give the background RAG process time to fetch data and run a quick summary, then have the chatbot give some indication it's thinking, run the full context query on the smart LLM, generate a summary answer then feed it back to the chat LLM and say "I may have found a solution to your problem" then switch to the response from the smart LLM.
I see what you're saying, but you're assuming that consumer products are always chatbots (and that a small language model can buy time interacting with the user while possibly providing additional context). That being said, I would be interested to see such a system in practice - any examples you can point me to? My more general point was not chat-related; much of the research around RAG seems to use LLMs to parse or route the user's query, improve retrieval, etc. which doesn't often work in practice.
This is where the opportunity for creativity comes in. You could allow a chat based refinement to search queries, or provide popup refinement buttons that narrow the search space, and build the search results iteratively rather than the old paradigm of "search" -> "results"
Comments
From a consumer perspective, this is a super interesting paper because it touches on one of the fundamental issues with most RAG beyond the toy case - that you need to do different stuff depending on what the user is asking for. You also (usually) can't just ask because most users don't know that LLMs are bad at math or semantic search won't be sufficient to answer questions that involve enumeration or totality. And while you can always add more steps to your RAG pipeline, some of those steps may be computationally expensive or not particularly relevant to the question at hand.
That being said, it is a bit frustrating that so much RAG research focuses on multi-hop approaches with LLMs. IME multiple round trips to an LLM is essentially a non-starter for any serious consumer product as it's far too slow. Smaller models can struggle to follow instructions so they often can't be an adequate replacement even for simpler tasks. Curious to hear if other folks working in this space have had any success thinking critically about these types of problems!
That depends on the model, you can run stuff in parallel and sometimes keep everything timely. You shouldn't be waiting till the last second to start running rag, you can be pre-emptively building context based on the current chat (like a human does) so that you've already got stuff summarized and ready to fire off when the final prompt does come.
Think about how a human will draw out a conversation around answering a question and use delaying words and phrases to let them continue answering when they don't have the solution fully formulated. LLMs can use the same tactic.
Could you elaborate a bit more about how that would work in practice?
Sure, if you're running a customer service chatbot, you can ask customers what the problem is, then start running rag async to populate a proper context for a smart LLM, and have the chatbot continue asking some questions to clarify details to give the background RAG process time to fetch data and run a quick summary, then have the chatbot give some indication it's thinking, run the full context query on the smart LLM, generate a summary answer then feed it back to the chat LLM and say "I may have found a solution to your problem" then switch to the response from the smart LLM.
I see what you're saying, but you're assuming that consumer products are always chatbots (and that a small language model can buy time interacting with the user while possibly providing additional context). That being said, I would be interested to see such a system in practice - any examples you can point me to? My more general point was not chat-related; much of the research around RAG seems to use LLMs to parse or route the user's query, improve retrieval, etc. which doesn't often work in practice.
This is where the opportunity for creativity comes in. You could allow a chat based refinement to search queries, or provide popup refinement buttons that narrow the search space, and build the search results iteratively rather than the old paradigm of "search" -> "results"