We might have been able to get away with raw SQL, but because of some of the complexities and dependencies on this same code, I felt like it wasn't the prudent choice during a fire. I wouldn't have had time to properly test it.
As for moving the inserts to an async job, that was one of our thoughts, too. Really it's what we did, but we just have the intermediary step of MongoDB. We wanted that step (rather than just immediately returning to the user and having the client not know when the inserts actually occur) because we wanted to keep the user experience great. Our app immediately sends you to personalized recs. We didn't want the initial experience to be: open the app => sent to a blank page of "recommendations."
An async job that didn't wait for any sort of confirmation would have left us in this position (or we would have had to implement a poller to check if the inserts were complete, which seemed wasteful and would need a push to Android Market which seemed less feasible.)
They did move their writes to an async job. They write the apps list into mongodb, then a daemon later copies it into mysql. Mongodb is being used as a cache (the frontend queries mongo then mysql)... this way they can read the data while also throttling writes to their main db.
Comments
Seems like this could have been solved without mongodb by:
1. Moving the inserts to a adync background job
2. Use raw SQL instead of activerecord for the inserts.
I'm used to postgres though, I know it can handle heavy writes at the same time as heavy reads just fine without replication.
We might have been able to get away with raw SQL, but because of some of the complexities and dependencies on this same code, I felt like it wasn't the prudent choice during a fire. I wouldn't have had time to properly test it.
As for moving the inserts to an async job, that was one of our thoughts, too. Really it's what we did, but we just have the intermediary step of MongoDB. We wanted that step (rather than just immediately returning to the user and having the client not know when the inserts actually occur) because we wanted to keep the user experience great. Our app immediately sends you to personalized recs. We didn't want the initial experience to be: open the app => sent to a blank page of "recommendations."
An async job that didn't wait for any sort of confirmation would have left us in this position (or we would have had to implement a poller to check if the inserts were complete, which seemed wasteful and would need a push to Android Market which seemed less feasible.)
Good call!
They did move their writes to an async job. They write the apps list into mongodb, then a daemon later copies it into mysql. Mongodb is being used as a cache (the frontend queries mongo then mysql)... this way they can read the data while also throttling writes to their main db.