It would really be interesting to see a "computers-on-Jeopardy" competition that was open to all comers. (The 'Watson' appearance, while impressive in many ways, was essentially a contrived IBM infomercial.)
I think you are describing TREC[1]. Over on the chatbot arena, there is The Loebner Prize[2], also interesting, but in general the systems are over learned.
TREC, as far as I know, doesn't have a competitive question-answering format, with definitive right or wrong answers.
Jeopardy is a widely-understood game format; a competition-via-API would be understandable to a wide variety of competitors, from leading researchers to precocious cranks.
Sure, Jeopardy is peculiar. Still, it's popular and well-understood, so an open competition, where anyone can upload their playing-agents (quirks and all) could draw big interest. It'd be more like recurring competitive computer chess tournaments than the stage-managed Watson-on-Jeopardy series.
Comments
It would really be interesting to see a "computers-on-Jeopardy" competition that was open to all comers. (The 'Watson' appearance, while impressive in many ways, was essentially a contrived IBM infomercial.)
I think you are describing TREC[1]. Over on the chatbot arena, there is The Loebner Prize[2], also interesting, but in general the systems are over learned.
[1] http://trec.nist.gov/ [2] www.loebner.net/Prizef/loebner-prize.html
TREC, as far as I know, doesn't have a competitive question-answering format, with definitive right or wrong answers.
Jeopardy is a widely-understood game format; a competition-via-API would be understandable to a wide variety of competitors, from leading researchers to precocious cranks.
There's TREC QA, which finished in 2007[1]. This year they are doing TREC Live QA[2] (which looks pretty challenging!)
Outside TREC, there is the CLEF QA track[3]
[1] http://trec.nist.gov/data/qamain.html
[2] https://sites.google.com/site/trecliveqa2015/
[3] http://nlp.uned.es/clef-qa/
Eh, Jeopardy success requires way too much Jeopardy-specific tuning. It has lots of quirks and conventions.
The data they use is much purer, and makes a much better evaluation for general-purpose QA technologies.
Sure, Jeopardy is peculiar. Still, it's popular and well-understood, so an open competition, where anyone can upload their playing-agents (quirks and all) could draw big interest. It'd be more like recurring competitive computer chess tournaments than the stage-managed Watson-on-Jeopardy series.