You have get already very good tips. I could add this ones for completion:
Review your backups and contingency plans (and measure your restore times).
Usually you will want a mysql.conf for production mode, and a different one for full restore (with different parameters). A full restore of a so big database with the mysql defaults, can take looooooooong.
As an anecdote, in the most big mysql system I've ever touched, there was a "performance problem", when I got the credentials and started to review, 3 different sysadmins, had scheduled 3 different full backups daily, using dumps. Some of them at office hours.
Yes, this is just plainly stupid, but it's an example of real life.
On a big database system, backups (and restores) is something who is convenient to design with care, to monitor, and to measure/adapt periodically.
If you go to master/slave(s) usually you can make backups from some slave, without disrupt the master operations.
Also, you maybe interested in play with things like mysqlproxy to split reads/writes, so you can have writes going to master, but reads spread across multiple slaves.
I assume you're already using 64bits, and that you know already about mysql tuning variables.
Thank you for your experiences, Small things like that can cause big problems and they're easy to forget about! I'm thinking we're going to need to write out a lot of processes for doing backups and adjustments. Organization/structure will help squash those incidents you mention, I think...
Comments
You have get already very good tips. I could add this ones for completion:
Review your backups and contingency plans (and measure your restore times).
Usually you will want a mysql.conf for production mode, and a different one for full restore (with different parameters). A full restore of a so big database with the mysql defaults, can take looooooooong.
As an anecdote, in the most big mysql system I've ever touched, there was a "performance problem", when I got the credentials and started to review, 3 different sysadmins, had scheduled 3 different full backups daily, using dumps. Some of them at office hours.
Yes, this is just plainly stupid, but it's an example of real life.
On a big database system, backups (and restores) is something who is convenient to design with care, to monitor, and to measure/adapt periodically.
If you go to master/slave(s) usually you can make backups from some slave, without disrupt the master operations.
Also, you maybe interested in play with things like mysqlproxy to split reads/writes, so you can have writes going to master, but reads spread across multiple slaves.
I assume you're already using 64bits, and that you know already about mysql tuning variables.
Thank you for your experiences, Small things like that can cause big problems and they're easy to forget about! I'm thinking we're going to need to write out a lot of processes for doing backups and adjustments. Organization/structure will help squash those incidents you mention, I think...