Don't just monitor general things like CPU, I/O... monitor what your application does as well. The point of monitoring is to give you the ability to spot the interconnectedness of symptoms to their cause.
Knowing that CPU spiked shortly after networking became saturated isn't as helpful as knowing that your calls were doing requests of a large number of records and returning more data than needed. So if you fail to monitor your actual application too, you only have half of the picture.
Comments
I failed to give you a starting point on monitoring, try these:
http://graphite.wikidot.com/ (I prefer this)
http://munin-monitoring.org/ (widely used and fairly easy to find resources for)
Don't just monitor general things like CPU, I/O... monitor what your application does as well. The point of monitoring is to give you the ability to spot the interconnectedness of symptoms to their cause.
Knowing that CPU spiked shortly after networking became saturated isn't as helpful as knowing that your calls were doing requests of a large number of records and returning more data than needed. So if you fail to monitor your actual application too, you only have half of the picture.