Ask HN: What are other "uncrackable benchmarks"?
I have my own benchmark for AI just based off my Doctoral Thesis, which isn't public(university constraints). I am wondering if other people have other "hidden" benchmarks? I have a few I have made, and one wonder if talking about them devalues them, and two can a good test set be made that can't be leaked?
Comments
Effective natural language processing of sign languages. Current LLMs are almost completely incapable. And it's a steep hill to climb. No text corpus. Must be learned from video. I do not believe current LLM approaches are capable of this with the amount of training material available, regardless of compute. Though I'm eager to be shown wrong.