Ask HN: How do you track NLP models in prod?
Working on a service which uses GPT-3.5 and was wondering how I would go about tracking the usefulness of it. Wondering what you guys have done, or if you guys even track your NLP models at all.
Working on a service which uses GPT-3.5 and was wondering how I would go about tracking the usefulness of it. Wondering what you guys have done, or if you guys even track your NLP models at all.
Comments
Are you doing classification, regression, extraction or something different? How did you evaluate the model before it went into production?
Causal text generation. I evaluated it on my own internal dataset but real world data shifts over time.
How do you get a number for text generation?
Exactly my question. I just scanned over the inputs and outputs and compared how "useful" they were.
Treat it like somebody grading an essay. It is either pass/fail or you can grade from 0 to 100 though you probably want to develop some kind of scoring rules for the latter.
I know a lot about evaluating classification but I am seeing people struggle w( evaluating text genersators; why don’t you look up my profile, send me an email, and we can talk more.