Because that’s how people use llms. You go to ChatGPT to ask a question and get an answer, rather than searching on Google, revising your search because you didn’t know a term, and then look at 3-5 different links to find the answer to what you were searching for
It seems you might be mixing up different types of "context" in LLM benchmarking. In this case, it refers to the input text directly provided to the model during evaluation by the user (as in in-context learning). This is separate from the text an LLM is trained on or can access via RAG methods.
Comments
Because that’s how people use llms. You go to ChatGPT to ask a question and get an answer, rather than searching on Google, revising your search because you didn’t know a term, and then look at 3-5 different links to find the answer to what you were searching for
It seems you might be mixing up different types of "context" in LLM benchmarking. In this case, it refers to the input text directly provided to the model during evaluation by the user (as in in-context learning). This is separate from the text an LLM is trained on or can access via RAG methods.
That's how people use llms because they (llms) don't seem to be good at the more sophisticated thing
But also because search engines have gotten even worse at answering questions.