I'm actually looking into a segfault issue deep in the bowels of a C++ addon we have in node.js (anyone in #node.js will have seen me over the past few weeks ask about it), but what reading this makes me realize is how woefully underequipped I am to hunt for problems of this nature.
My problem is likely in one of our addons, but this kind of debugging, this whole genre of problem solving is entirely beyond me. How do I get to this level? What do I need to learn? To study?
It's just a little depressing to read something like this and see how far the road ahead goes, despite how far I've already traveled...
Debugging severe memory corruption or memory leaks is annoying, and can occasionally take a lot of time, but it's not necessarily that bad. Here are some pointers that may be helpful.
Tools: valgrind and gdb are obvious. But don't forget your compiler! Crank up the warnings, and look through LLVM-clang's -fsanitize=<foo> and warning options. (Also, if you're already on OpenBSD, check out the "S" flag to malloc; if you're on Solaris, check out, well, the blog post.) Finally, Boehm's conservative garbage collector has a "find memory leaks" mode, which looks useful for those cases where you can't get valgrind working. If all else fails, shovel through the memory dump looking for repeated patterns.
Testing: try to reproduce the problem; the first iteration may look something like "it runs out of memory after 36 hours". Then simplify: for instance, the author of the article could have asked "does this still happen if the server closes the connection immediately, without sending any data" and would have found the bug very quickly. (Of course, you're likely to ask a lot of wrong questions before hitting on the right one; experience and a full knowledge of the system you're working on is useful but not sufficient.) Questions like "does this happen more quickly if we ping 100 times per second instead of once every ten minutes" are often useful as well. (Finally, just printing memory usage every N seconds is helpul.)
Coding: be careful when writing code. The usual ways of improving code quality (e.g. code reviews) work to reduce memory leaks, too. Try to run a multiple-hour soak test every so often during development (preferably on a CI server); it's a lot easier to debug "hey, we suddenly run out of memory after yesterday's commits" than "well, something goes wrong in production". If you're doing new development, consider alternatives to malloc() - arena/pool allocation (e.g. libtalloc) is convenient and very fast if your memory use is tree-like (e.g. a connections owns a request owns some memory to sort the data before returning it). In C, goto a single chunk of cleanup-and-return code rather than duplicating the cleanup at every place where you exit from the function.
- Use valgrind (or gdb)! Your segfault should be simpler to find than a memory leak, because you know what line the segfault happens on.
- If you have a value that's getting mangled (pointer getting overwritten by a write to another address) and you can't figure out why, use watchpoints to see when that address is getting touched. http://sourceware.org/gdb/onlinedocs/gdb/Set-Watchpoints.htm...
- Find a minimal program to reproduce the problem. It's gross, but I used to actually just take a copy of the code and cut things out until the bug stopped, then look at the last thing I cut. You can do this as a binary search - only run the first half, check for the bug, only run the second half, check for the bug, repeat on the buggy half.
As I said, segfaults are a lot easier than this kind of problem (not that they're easy when you start out). Don't be discouraged! I would help out too, but you'd need to send everything to reproduce the bug (client code, server code, server platform, etc.)
I can't claim to be at the level of the joyent guys presented here, but I think taking a Operating System class and Computer Architecture class, or reading the respective textbooks helps, and at the same time you have to be familiar with the particular OS you happen to use, probably up to the point of reading and having basic understanding of the source code of the most important subsystems (virtual memory, process scheduling, filesystem handling, TCP/IP stack) and understanding what the system calls are and what they do. Then you need to know a wide range of tools the given OS offers for examining things, so that you do not get hopelessly stuck in the face of an emergency, since you often have to investigate a crash while it happens to even be able to reproduce it, so you need to know how to examine a running process etc. For Linux this means knowing stuff like:
There is a big bunch of tools in the OS very few developers know, sysadmins know more, but they often don't understand the OS and use the tools without understanding their output too well.
Some confirmation of what I have written here is the fact that Joyent forked OpenSolaris to create an OS precisely to make it easier to do things of this kind:
In 2005, Sun Microsystems open sourced Solaris, its renowned Unix operating system, eventually to be released as a distribution called OpenSolaris. Among the earliest adopters and most effective advocates of OpenSolaris was Ben Rockwood, who wrote The Cuddletech Guide to Building OpenSolaris in June, 2005 – the first of his many important contributions to the nascent OpenSolaris community. Meanwhile, Joyent's CTO Jason Hoffman was frustrated by the inability of most operating systems to answer seemingly-simple questions like: "Why is the server down? When will it be back up? ... Now that it's back up, why is my database still slow?"
Jason knew that these questions would be a lot easier to answer on Solaris-based systems, and recognized Sun's open-sourcing initiative as a huge opportunity.
You need determination and experience, and some knowledge of how code is compiled at a low level.
Tools like those described in the article are handy, but aren't absolutely necessary. They save a lot of time, but the same effects can usually be gotten by more laborious means.
You have a segfault. You should know where in the code it's occurring already; it's either an access to bad memory with the instruction pointer (IP) at the point of access, or it's an attempt to execute code with the IP pointing at the bad memory, in which case the top of the stack (or, depending on calling convention, one of the registers) normally contains the place where it came from (necessarily, since the code expected to be returned to).
There are ways to turn an instruction pointer into line number offset when you have appropriate debug info, if you can't get the program running under a debugger.
Given the line number, segfaults can typically be split into three categories: plain bad logic, use after free, and memory corruption. The last is hardest to find IME, most easily done using a debugger and hardware breakpoints on memory address modifications, but you need a stable repro and a consistent memory allocator that gives predictable addresses for every rerun.
If any of the above is meaningless to you, it should give you some clues as to where you need to research.
Yeah I got this far, but the stack trace doesn't have the symbols. The guy I'm working with said it might be because the addons use dynamically linked libraries instead of statically linked libraries...
By the way, I think a jenandre used to work at my company.
What library is missing the symbols? You may be able to tell by the stack trace. You should get symbols for something at least before it's lost. Examine that frame to see what it's doing. Linking some non-debug library w/ a debug executable just means that gdb won't display the symbols when it enters code for that library. But if your addons are built with -g you'll get the symbols for the addon before it starts calling the other library it is using.
Btw, people often write javascript wrappers that manually refer to the build/Release/.node version instead of the debug version (which will get added to build/Debug/.node). Check that first.
Even if you are using dynamically linked libraries (like the zmq addon does), you can always build debug versions of those to get all of the symbols (try CFLAGS=-g when ./configure).
We never make it to symbol-land in the bt, so I'm assuming it's because we never get into a debug library at all. I will make sure the js is referring to the correct version of the addon though.
My next step is probably going to be to try and figure out how to get every library to build with symbols. Thanks for the help. :D
Comments
I'm actually looking into a segfault issue deep in the bowels of a C++ addon we have in node.js (anyone in #node.js will have seen me over the past few weeks ask about it), but what reading this makes me realize is how woefully underequipped I am to hunt for problems of this nature.
My problem is likely in one of our addons, but this kind of debugging, this whole genre of problem solving is entirely beyond me. How do I get to this level? What do I need to learn? To study?
It's just a little depressing to read something like this and see how far the road ahead goes, despite how far I've already traveled...
Debugging severe memory corruption or memory leaks is annoying, and can occasionally take a lot of time, but it's not necessarily that bad. Here are some pointers that may be helpful.
Tools: valgrind and gdb are obvious. But don't forget your compiler! Crank up the warnings, and look through LLVM-clang's -fsanitize=<foo> and warning options. (Also, if you're already on OpenBSD, check out the "S" flag to malloc; if you're on Solaris, check out, well, the blog post.) Finally, Boehm's conservative garbage collector has a "find memory leaks" mode, which looks useful for those cases where you can't get valgrind working. If all else fails, shovel through the memory dump looking for repeated patterns.
Testing: try to reproduce the problem; the first iteration may look something like "it runs out of memory after 36 hours". Then simplify: for instance, the author of the article could have asked "does this still happen if the server closes the connection immediately, without sending any data" and would have found the bug very quickly. (Of course, you're likely to ask a lot of wrong questions before hitting on the right one; experience and a full knowledge of the system you're working on is useful but not sufficient.) Questions like "does this happen more quickly if we ping 100 times per second instead of once every ten minutes" are often useful as well. (Finally, just printing memory usage every N seconds is helpul.)
Coding: be careful when writing code. The usual ways of improving code quality (e.g. code reviews) work to reduce memory leaks, too. Try to run a multiple-hour soak test every so often during development (preferably on a CI server); it's a lot easier to debug "hey, we suddenly run out of memory after yesterday's commits" than "well, something goes wrong in production". If you're doing new development, consider alternatives to malloc() - arena/pool allocation (e.g. libtalloc) is convenient and very fast if your memory use is tree-like (e.g. a connections owns a request owns some memory to sort the data before returning it). In C, goto a single chunk of cleanup-and-return code rather than duplicating the cleanup at every place where you exit from the function.
The fact that someone is saying they don't know where to start seems to indicate this isn't true.
In my experience from debugging C programs:
- Use valgrind (or gdb)! Your segfault should be simpler to find than a memory leak, because you know what line the segfault happens on.
- If you have a value that's getting mangled (pointer getting overwritten by a write to another address) and you can't figure out why, use watchpoints to see when that address is getting touched. http://sourceware.org/gdb/onlinedocs/gdb/Set-Watchpoints.htm...
- Find a minimal program to reproduce the problem. It's gross, but I used to actually just take a copy of the code and cut things out until the bug stopped, then look at the last thing I cut. You can do this as a binary search - only run the first half, check for the bug, only run the second half, check for the bug, repeat on the buggy half.
As I said, segfaults are a lot easier than this kind of problem (not that they're easy when you start out). Don't be discouraged! I would help out too, but you'd need to send everything to reproduce the bug (client code, server code, server platform, etc.)
That's not gross.
I can't claim to be at the level of the joyent guys presented here, but I think taking a Operating System class and Computer Architecture class, or reading the respective textbooks helps, and at the same time you have to be familiar with the particular OS you happen to use, probably up to the point of reading and having basic understanding of the source code of the most important subsystems (virtual memory, process scheduling, filesystem handling, TCP/IP stack) and understanding what the system calls are and what they do. Then you need to know a wide range of tools the given OS offers for examining things, so that you do not get hopelessly stuck in the face of an emergency, since you often have to investigate a crash while it happens to even be able to reproduce it, so you need to know how to examine a running process etc. For Linux this means knowing stuff like:
http://en.wikipedia.org/wiki/Strace
http://en.wikipedia.org/wiki/Lsof
http://en.wikipedia.org/wiki/Vmstat
http://en.wikipedia.org/wiki/Netstat
http://en.wikipedia.org/wiki/DTrace
http://en.wikipedia.org/wiki/Tcpdump
http://en.wikipedia.org/wiki/Magic_SysRq_key
https://perf.wiki.kernel.org/index.php/Main_Page
...
There is a big bunch of tools in the OS very few developers know, sysadmins know more, but they often don't understand the OS and use the tools without understanding their output too well.
Some confirmation of what I have written here is the fact that Joyent forked OpenSolaris to create an OS precisely to make it easier to do things of this kind:
http://wiki.smartos.org/display/DOC/Why+SmartOS+-+ZFS%2C+KVM...
In 2005, Sun Microsystems open sourced Solaris, its renowned Unix operating system, eventually to be released as a distribution called OpenSolaris. Among the earliest adopters and most effective advocates of OpenSolaris was Ben Rockwood, who wrote The Cuddletech Guide to Building OpenSolaris in June, 2005 – the first of his many important contributions to the nascent OpenSolaris community. Meanwhile, Joyent's CTO Jason Hoffman was frustrated by the inability of most operating systems to answer seemingly-simple questions like: "Why is the server down? When will it be back up? ... Now that it's back up, why is my database still slow?"
Jason knew that these questions would be a lot easier to answer on Solaris-based systems, and recognized Sun's open-sourcing initiative as a huge opportunity.
You need determination and experience, and some knowledge of how code is compiled at a low level.
Tools like those described in the article are handy, but aren't absolutely necessary. They save a lot of time, but the same effects can usually be gotten by more laborious means.
You have a segfault. You should know where in the code it's occurring already; it's either an access to bad memory with the instruction pointer (IP) at the point of access, or it's an attempt to execute code with the IP pointing at the bad memory, in which case the top of the stack (or, depending on calling convention, one of the registers) normally contains the place where it came from (necessarily, since the code expected to be returned to).
There are ways to turn an instruction pointer into line number offset when you have appropriate debug info, if you can't get the program running under a debugger.
Given the line number, segfaults can typically be split into three categories: plain bad logic, use after free, and memory corruption. The last is hardest to find IME, most easily done using a debugger and hardware breakpoints on memory address modifications, but you need a stable repro and a consistent memory allocator that gives predictable addresses for every rerun.
If any of the above is meaningless to you, it should give you some clues as to where you need to research.
Crashes (that can be reliably reproduced) should be way easier to debug than memory leaks.
a) build a `debug` version of node (building node from source creates a node_g version which is a debug version)
b) build a `debug` version of all of the c++ addons in your node_modules folder (node-gyp build -d for each addon)
c) start gdb with the debug version of node: `gdb ./node_g`
d) in gdb, run your node script using `run <script.js>` -- add any other options
e) wait for it to crash, and then type `bt` - you'll see the location of the crash which should give you a good starting place.
Yeah I got this far, but the stack trace doesn't have the symbols. The guy I'm working with said it might be because the addons use dynamically linked libraries instead of statically linked libraries...
By the way, I think a jenandre used to work at my company.
What library is missing the symbols? You may be able to tell by the stack trace. You should get symbols for something at least before it's lost. Examine that frame to see what it's doing. Linking some non-debug library w/ a debug executable just means that gdb won't display the symbols when it enters code for that library. But if your addons are built with -g you'll get the symbols for the addon before it starts calling the other library it is using.
Btw, people often write javascript wrappers that manually refer to the build/Release/.node version instead of the debug version (which will get added to build/Debug/.node). Check that first.
Even if you are using dynamically linked libraries (like the zmq addon does), you can always build debug versions of those to get all of the symbols (try CFLAGS=-g when ./configure).
We never make it to symbol-land in the bt, so I'm assuming it's because we never get into a debug library at all. I will make sure the js is referring to the correct version of the addon though.
My next step is probably going to be to try and figure out how to get every library to build with symbols. Thanks for the help. :D
I find you need two things:
- the "troubleshooter" mentality/thinking pattern - extensive system knowledge
I haven't figured out how to teach #1, except maybe for "don't have anyone to ask for help" and #2 is self explanatory.