If you're growing at that pace, you can't do just one thing; you've got to attack it on multiple fronts. I approach things like this doing something like this:
4) Partition the data in such a way that you serve queries and users quickly
If you do the above, you should easily be able to scale any decent hardware into 10x where you are now. That will buy you some time to then build a data warehouse and backend support/reporting systems. At some point you're going to be in a spot where you either have to invest $5m+ in hardware to keep serving your current data, or you're going to have to start archiving some of this data to a data warehouse.
$5m+ in hardware to keep serving your current data
Can you explain this a little further? Are you speaking specifically about buying hardware, or do you mean buying virtual servers/ database infrastructure?
Comments
If you're growing at that pace, you can't do just one thing; you've got to attack it on multiple fronts. I approach things like this doing something like this:
1) Optimize data type usage
2) Optimize indexing - removing redundant indexes, adding new indexes, changing existing indexes
3) Optimize storage - get your SAN/disks in order
4) Partition the data in such a way that you serve queries and users quickly
If you do the above, you should easily be able to scale any decent hardware into 10x where you are now. That will buy you some time to then build a data warehouse and backend support/reporting systems. At some point you're going to be in a spot where you either have to invest $5m+ in hardware to keep serving your current data, or you're going to have to start archiving some of this data to a data warehouse.
$5m+ in hardware to keep serving your current data
Can you explain this a little further? Are you speaking specifically about buying hardware, or do you mean buying virtual servers/ database infrastructure?