Comment on Root cause analysis: significantly elevated error rates on 2019‑07‑10parentComments−NikolaeVarius7yIn many HA setups, you're supposed to not have to care if any single thing goes down because it should auto recoverThe article said that the node stalled in a way that was unforseen which may have caused standard recovery mechanisms to silently fail.−laCour7yRight, but they didn't recover speedily. To have the cluster in such a state for so long sounds like poor monitoring to me because this can knowingly interfere with an election later.−kortilla7yThe health check said it was ok. How would they know it needed to be recovered?The fault was the bad health check. Not the process.−laCour7yThey only just clarified that monitoring was in place and they were reporting as healthy. See the comments above.
Comments
In many HA setups, you're supposed to not have to care if any single thing goes down because it should auto recover
The article said that the node stalled in a way that was unforseen which may have caused standard recovery mechanisms to silently fail.
Right, but they didn't recover speedily. To have the cluster in such a state for so long sounds like poor monitoring to me because this can knowingly interfere with an election later.
The health check said it was ok. How would they know it needed to be recovered?
The fault was the bad health check. Not the process.
They only just clarified that monitoring was in place and they were reporting as healthy. See the comments above.