xtacy on GitHub: Recent Load Balancer Problems (RCA)

Comment on GitHub: Recent Load Balancer Problems (RCA)

The post mentions that heartbeats timeout when the load spikes momentarily. I have a few questions, would love to hear answers if it's okay to share :-)

    1. What load spiked?  Is it the network/CPU load?

    2. By spiked (be it network or CPU), do you mean
       the load went all the way to 100%?   Or was it
       some threshold like say 90% of the available
       capacity?

    3. What's the heartbeat time interval?

Thanks, (EDIT: spacing)

jnewland 14y

1. IO and CPU load spiked so much that the system was basically unresponsive over SSH. We think it was due another Xen VM swapping out of control.

2. Was 10 seconds with a 10 second timeout (way to low to run `xm list` in a loaded situation). It's now 90 seconds with a 90 second timeout.

Comments