Coming from the math side, I don't really get this. Isn't all of the latency introduced by the learning side of things? Shouldn't the answer side be entirely decoupled from the learning, and simply be plugging data into an equation with a bunch of constant parameters (with values discovered by your learning system, updated at a less than realtime frequency)?
The "equation with a bunch of constant parameters" generated by an ML model can be huge, with thousands of inputs or more. Evaluating that equation for a specific observation can require a huge number of computations, which is why there's a boom in ML inference hardware right now.
Yeah, I don't buy that. It's still just a matrix multiplication (for the linear bits). Incredibly fast. Besides, the old physics rule of thumb is that any real world equation with a bunch of parameters only has 5-7 that actually matter, and only 3 that matter a lot. Everything else can be set to zero without noticeable change in the result.
If you’re making decisions that involve multiple variables you may be doing hundreds to thousands of inferences for a single page load. Keeping latency under 50ms becomes a real challenge.
x1 * a + x2 * b + x3 * c + ... + x1000 * zzz + ...
If a, b, c ... zzz, are all fixed constants already discovered by your learning algorithm. That's a very fast calculation, and doesn't take anything like 50ms.
Also, in the real world, you can establish a significance cutoff for a lot of these constants and get something like this as your final equation:
The inputs to those functions might be coming from external data sources, or aggregated. These have a cost too. But mostly, it just adds up. At a thousand features, you have a 0.05ms budget for each. Without taking into account network latency since you won’t be running those models inside the application server.
Nobody's saying this is an impossible problem. The paper shows how much additional work is required beyond a traditional data science workflow.
The team behind the paper built a model that had good performance on training data. They're a smart lot so they knew they needed to cross-validate. The results held up in cross-validation! Hooray, the model works! ...right?
That's as far as a lot of data scientists go. This paper points out that you need to have a model that does (at least) three things:
1. Generates good scores with training and testing data
2. Outperforms existing models in the real world
3. Runs really really quickly
There are a lot of data scientists who have no idea how to do #2 and #3. This paper says "These parts are really important!!!"
The feature engineering is probably where most of the performance comes from so there is likely a lot of code that turns the raw data into features.
Features likely include more than just the single users history, so they need to be updated often enough for the model to do fast predictions. E.g. you want your model to capture if many people are booking from the same area at once because there were results from a sports game etc, but you dont want to run an expensive query for every user of the page.
Definitely not the first thing to worry about in a startup, but better performance at Booking.com's scale is serious $$$.
Comments
Coming from the math side, I don't really get this. Isn't all of the latency introduced by the learning side of things? Shouldn't the answer side be entirely decoupled from the learning, and simply be plugging data into an equation with a bunch of constant parameters (with values discovered by your learning system, updated at a less than realtime frequency)?
The "equation with a bunch of constant parameters" generated by an ML model can be huge, with thousands of inputs or more. Evaluating that equation for a specific observation can require a huge number of computations, which is why there's a boom in ML inference hardware right now.
Yeah, I don't buy that. It's still just a matrix multiplication (for the linear bits). Incredibly fast. Besides, the old physics rule of thumb is that any real world equation with a bunch of parameters only has 5-7 that actually matter, and only 3 that matter a lot. Everything else can be set to zero without noticeable change in the result.
If you’re making decisions that involve multiple variables you may be doing hundreds to thousands of inferences for a single page load. Keeping latency under 50ms becomes a real challenge.
But it comes down to this doesn't it:
x1 * a + x2 * b + x3 * c + ... + x1000 * zzz + ...
If a, b, c ... zzz, are all fixed constants already discovered by your learning algorithm. That's a very fast calculation, and doesn't take anything like 50ms.
Also, in the real world, you can establish a significance cutoff for a lot of these constants and get something like this as your final equation:
x13 * m + x523 * cdf + x777 * wdc + x893 * ydz
The inputs to those functions might be coming from external data sources, or aggregated. These have a cost too. But mostly, it just adds up. At a thousand features, you have a 0.05ms budget for each. Without taking into account network latency since you won’t be running those models inside the application server.
So why not load the calculated constants to the application server to reduce network latency?
And the learning side of things should have culled that list of thousand features down to a list of 5 - 10 that mattered.
It really sounds like the off-the-shelf stuff isn't built for efficiency.
Nobody's saying this is an impossible problem. The paper shows how much additional work is required beyond a traditional data science workflow.
The team behind the paper built a model that had good performance on training data. They're a smart lot so they knew they needed to cross-validate. The results held up in cross-validation! Hooray, the model works! ...right?
That's as far as a lot of data scientists go. This paper points out that you need to have a model that does (at least) three things: 1. Generates good scores with training and testing data 2. Outperforms existing models in the real world 3. Runs really really quickly There are a lot of data scientists who have no idea how to do #2 and #3. This paper says "These parts are really important!!!"
The feature engineering is probably where most of the performance comes from so there is likely a lot of code that turns the raw data into features.
Features likely include more than just the single users history, so they need to be updated often enough for the model to do fast predictions. E.g. you want your model to capture if many people are booking from the same area at once because there were results from a sports game etc, but you dont want to run an expensive query for every user of the page.
Definitely not the first thing to worry about in a startup, but better performance at Booking.com's scale is serious $$$.