State-of-the-art models (using BERT) are far from achieving human performance (0.504 vs 0.822).
This is moderately surprising.
In question answering (QA) style tasks (SQUAD, SQUAD 2) we see state of the art models approach human performance. QA is similar to KBC in the sense that the answers are usually extracted from text in a similar way.
I'd imaging there is potential for fairly rapid improvement in this (Knowledge Base Population) task.
As long as we haven't reached AGI I feel like this is true of any new benchmark. BERT wasn't trained or designed for the task, give some smart folks a few months and they can now beat the task. The bigger question is what would we like an AI to be able to do. Is this benchmark a good one? Is there maybe a better choice of questions to get the type of NLP we want?
Comments
State-of-the-art models (using BERT) are far from achieving human performance (0.504 vs 0.822).
This is moderately surprising.
In question answering (QA) style tasks (SQUAD, SQUAD 2) we see state of the art models approach human performance. QA is similar to KBC in the sense that the answers are usually extracted from text in a similar way.
I'd imaging there is potential for fairly rapid improvement in this (Knowledge Base Population) task.
As long as we haven't reached AGI I feel like this is true of any new benchmark. BERT wasn't trained or designed for the task, give some smart folks a few months and they can now beat the task. The bigger question is what would we like an AI to be able to do. Is this benchmark a good one? Is there maybe a better choice of questions to get the type of NLP we want?
Is this benchmark a good one?
Any benchmark which reflects a task that humans do is a good one, unless it has specific weaknesses that a computer exploits.
I'd use models that are written for this is my work, so I find it useful.
I feel like this is true of any new benchmark.... give some smart folks a few months and they can now beat the task.
In NLP work this use not to be the case. 5 years ago we were stuck at a local maximum.
And this undervalues this task - this bridges the gap between unstructured and structured data. In many ways it is the holy grail for many tasks.