Model performance is not the same as business performance
This is interesting. Sometimes some people from business side consider that AI is the solution to all problems (as if there was just one catch-them-all AI solution) and some academic people think that the top-performance model for some classification task is the must-go, and all they forget that the goal is to earn money.
That was an interesting result from the original Netflix challenge.
First of all it turned out that the winner wasn't actually all that useful for various reasons such as computational intensity.
But, more interestingly, it also turned out that the goals of the model--"best" recommendations--isn't actually the goal of Netflix at all which is much more interested in customer retention and similar metrics. The two things may be correlated but they're certainly not the same thing.
I don't remember all the details but I thought it was a really good insight at the time.
The big problem for Netflix was that their data was all from DVD rentals, but by the end of the contest, their business model was very streaming oriented. As you might imagine, people have beliefs about what they'll want to watch in a few days that don't exactly match up with what they want to watch now. That difference killed the model.
another problem was that their "user" was actually a household, with a range of (sometimes conflicting) likes and interests - which could result in strange recommendations. They finally added profiles to fix that data problem.
Yep, from my experience with booking.com it seems that instead of using highly trained AIs the decision was made to simply slap every dark pattern known to man onto the site and auto-subscribe every customer to a dozen newsletters.
Comments
This is interesting. Sometimes some people from business side consider that AI is the solution to all problems (as if there was just one catch-them-all AI solution) and some academic people think that the top-performance model for some classification task is the must-go, and all they forget that the goal is to earn money.
That was an interesting result from the original Netflix challenge.
First of all it turned out that the winner wasn't actually all that useful for various reasons such as computational intensity.
But, more interestingly, it also turned out that the goals of the model--"best" recommendations--isn't actually the goal of Netflix at all which is much more interested in customer retention and similar metrics. The two things may be correlated but they're certainly not the same thing.
I don't remember all the details but I thought it was a really good insight at the time.
The big problem for Netflix was that their data was all from DVD rentals, but by the end of the contest, their business model was very streaming oriented. As you might imagine, people have beliefs about what they'll want to watch in a few days that don't exactly match up with what they want to watch now. That difference killed the model.
another problem was that their "user" was actually a household, with a range of (sometimes conflicting) likes and interests - which could result in strange recommendations. They finally added profiles to fix that data problem.
Thanks, this makes more sense than any other explanation I've heard for why they didn't use the Netflix prize model in production.
Relevant news.yc discussion from a month ago: https://news.ycombinator.com/item?id=20876158
Yep, from my experience with booking.com it seems that instead of using highly trained AIs the decision was made to simply slap every dark pattern known to man onto the site and auto-subscribe every customer to a dozen newsletters.