In my company we use this approach, instead of infer online, we just run all our models overnight and save the results on a database that we serve trough an API, that gives you constant time on the latency, its a shotgun approach as much of the recommendations aren't served (specially if are user facing and not item facing ones) but works really well.
All our models are balanced using multi-armed bandits, so for our recommendations engine, we run lots of arms that depends on the incoming channel, were in the app is being shown the recommendation, etc and just combine the outputs of the models.
Comments
In my company we use this approach, instead of infer online, we just run all our models overnight and save the results on a database that we serve trough an API, that gives you constant time on the latency, its a shotgun approach as much of the recommendations aren't served (specially if are user facing and not item facing ones) but works really well.
All our models are balanced using multi-armed bandits, so for our recommendations engine, we run lots of arms that depends on the incoming channel, were in the app is being shown the recommendation, etc and just combine the outputs of the models.