From the paper it looks like you would get GPT-4 level quality with the 65B model - but if you just do some random tests you will quickly figure out that is not even remotely the case. There must be something seriously wrong with the benchmarks used.
The evaluation prompts I saw ( https://colab.research.google.com/drive/1kK6xasHiav9nhiRUJjP... ) were all "encyclopedia-style", asking for summaries of information that would have occurred many times in the training data (e.g. "How can I improve my time management skills?") and hence probably reflect ability to retrieve and summarise trained information as opposed to novel problem solving.
Or in the programming questions the ones I saw are also certainly in the training data (e.g. "Implement a Python function to find the longest common subsequence of two input strings using dynamic programming.")
Comments
From the paper it looks like you would get GPT-4 level quality with the 65B model - but if you just do some random tests you will quickly figure out that is not even remotely the case. There must be something seriously wrong with the benchmarks used.
The evaluation prompts I saw ( https://colab.research.google.com/drive/1kK6xasHiav9nhiRUJjP... ) were all "encyclopedia-style", asking for summaries of information that would have occurred many times in the training data (e.g. "How can I improve my time management skills?") and hence probably reflect ability to retrieve and summarise trained information as opposed to novel problem solving.
Or in the programming questions the ones I saw are also certainly in the training data (e.g. "Implement a Python function to find the longest common subsequence of two input strings using dynamic programming.")