Treat it like somebody grading an essay. It is either pass/fail or you can grade from 0 to 100 though you probably want to develop some kind of scoring rules for the latter.
I know a lot about evaluating classification but I am seeing people struggle w( evaluating text genersators; why don’t you look up my profile, send me an email, and we can talk more.
Comments
Are you doing classification, regression, extraction or something different? How did you evaluate the model before it went into production?
Causal text generation. I evaluated it on my own internal dataset but real world data shifts over time.
How do you get a number for text generation?
Exactly my question. I just scanned over the inputs and outputs and compared how "useful" they were.
Treat it like somebody grading an essay. It is either pass/fail or you can grade from 0 to 100 though you probably want to develop some kind of scoring rules for the latter.
I know a lot about evaluating classification but I am seeing people struggle w( evaluating text genersators; why don’t you look up my profile, send me an email, and we can talk more.