The evaluation prompts I saw ( https://colab.research.google.com/drive/1kK6xasHiav9nhiRUJjP... ) were all "encyclopedia-style", asking for summaries of information that would have occurred many times in the training data (e.g. "How can I improve my time management skills?") and hence probably reflect ability to retrieve and summarise trained information as opposed to novel problem solving.
Or in the programming questions the ones I saw are also certainly in the training data (e.g. "Implement a Python function to find the longest common subsequence of two input strings using dynamic programming.")
Comments
The evaluation prompts I saw ( https://colab.research.google.com/drive/1kK6xasHiav9nhiRUJjP... ) were all "encyclopedia-style", asking for summaries of information that would have occurred many times in the training data (e.g. "How can I improve my time management skills?") and hence probably reflect ability to retrieve and summarise trained information as opposed to novel problem solving.
Or in the programming questions the ones I saw are also certainly in the training data (e.g. "Implement a Python function to find the longest common subsequence of two input strings using dynamic programming.")