TLDR; Split databases, split tables, prefetch queries with cron jobs or database functions, use separate servers, don't collect redundant data
As a BI developer, I'd also like to second the idea of being able to split the table up and changing your architecture. Ideally, you would leave your production database untouched and have a replicated database to work with. The replicated database will of course still probably be really massive, so you will want to create tables specifically for querying (attempts to simulate a data mart) where you only grab the data that you need for querying. Another idea would be to have cron jobs pre-query at say midnight so when your users hit the database, they are fetching pre-fetched data.
Also, if it's rising exponentially, I would have a look to see if all that data is really what you need. It's great to collect every single piece of data if you can but if you can't cover the costs to do that and there's no added value to all that data, it doesn't make sense to keep everything.
Thanks a lot for your insights, splitting the database up and having a replicate/production database setup is an interesting Idea, I'm going to look into a bit more
Comments
TLDR; Split databases, split tables, prefetch queries with cron jobs or database functions, use separate servers, don't collect redundant data
As a BI developer, I'd also like to second the idea of being able to split the table up and changing your architecture. Ideally, you would leave your production database untouched and have a replicated database to work with. The replicated database will of course still probably be really massive, so you will want to create tables specifically for querying (attempts to simulate a data mart) where you only grab the data that you need for querying. Another idea would be to have cron jobs pre-query at say midnight so when your users hit the database, they are fetching pre-fetched data.
Also, if it's rising exponentially, I would have a look to see if all that data is really what you need. It's great to collect every single piece of data if you can but if you can't cover the costs to do that and there's no added value to all that data, it doesn't make sense to keep everything.
Thanks a lot for your insights, splitting the database up and having a replicate/production database setup is an interesting Idea, I'm going to look into a bit more