you really can't just debug a single machine in production when you have a cluster
Somehow I ended up debugging, tracing, monitoring and even hotpatching individual machines in the cluster. Yeah the easy problems will show up in the monitoring and logs. The harder ones won't.
That must have been a pain in the butt :) . And for sure your right there are always exceptions.
I guess I haven't ran into those issues probably because I run JVMs but I suppose if you have native code or an interpreter using native code I can see how it would be helpful to just SSH and figure out what the issue is.
Now that I recall I have actually had to SSH a bunch of times because of Rackspace network interfaces randomly failing so I am a big hypocrite :)
It wasn't a huge pain. It was Erlang, so I could do those very easily. But it still had to be done by logging into a few machines and poking around. I can't imagine if that was somehow a bunch of C code combined with a kernel running in the same memory space.
Comments
Somehow I ended up debugging, tracing, monitoring and even hotpatching individual machines in the cluster. Yeah the easy problems will show up in the monitoring and logs. The harder ones won't.
That must have been a pain in the butt :) . And for sure your right there are always exceptions.
I guess I haven't ran into those issues probably because I run JVMs but I suppose if you have native code or an interpreter using native code I can see how it would be helpful to just SSH and figure out what the issue is.
Now that I recall I have actually had to SSH a bunch of times because of Rackspace network interfaces randomly failing so I am a big hypocrite :)
It wasn't a huge pain. It was Erlang, so I could do those very easily. But it still had to be done by logging into a few machines and poking around. I can't imagine if that was somehow a bunch of C code combined with a kernel running in the same memory space.