At Fosdem Volker presented a very great session on how to debug OpenJDK (and Hotspot) with gdb. Roman and Andrew (Dinn) did something similar while speaking about Shenandoah. In the next few days I’ll try to upload their slides on the FOSDEM website so that anyone can access that (and hopefully we will have the recordings this time as well).
There are a few things though that I keep forgetting myself and so I thought it would be useful to sum up in a blog post, hopefully general enough for most people as well as future reference for myself!
Suppose you are trying to detect some funny behaviour in your code, and that the crash is in a native library (or perhaps in some native OpenJDK code which is not hotspot).
What you would usually do with Java code is to start your debugger in Eclipse or IntelliJ or whatever, and go step by step until you figure out what’s wrong.
But when dealing with native code the thing gets complex, Eclipse and NetBeans can’t follow by default the native code, IntelliJ doesn’t even support native code at all (at least on Linux). There is an option though, first, you can still use those tools in process attach mode, they have very good debugging interfaces that make it easier to analyse quickly anything, but you can also use gdb directly, likewise in process attach mode.
Let’s see a couple of common cases here:
1. The application crashes, you want gdb launched automagically:
$ java -XX:OnError="gdb - %p" MyApplication
Roman (thanks!) show me this trick back in 2008! Honestly, I didn’t test that recently, but I suppose this still works 😉
2. You want to start a debugging session yourself rather than automatically on crash.
The trick here is to either start the application in debug mode via Eclipse/Whatever or attaching the Java debugger (including jdb if you enjoy suffering!) remotely:
$ java -Dsun.awt.disablegrab=true \ -Xdebug \ -Xrunjdwp:transport=dt_socket,server=y,address=1080 \ MyApplication
This will produce an output like the following:
Listening for transport dt_socket at address: 1080
Blocking the application until the debugger is attached.
At this point, you can set the breakpoints in your IDE and attach to the Java process remotely. The idea is to set the breakpoint right before the native call (tip: If you follow from there stepping with the java debugger, you’ll also see how native libraries are loaded).
Now to connect gdb all you need to to is to get the pid of the java process, with jps for example:
$ jps 30481 Jps 27162 MyApplication <------
$ gdb -p 27162
Set your breakpoint in the native function of choice. Remember the name mangling, so you need to look up how the methods are actually called in native code, the naming convention is:
But you need to double check exactly everything since there may be method overloads that dictate a slightly different convention.
If instead of using gdb from the command line you want to use your IDE the rule to follow is the same really. Afaik both Eclipse and NetBeans allow their native debugger plugins to attach to a process.
All that is needed now is to set your gdb breakpoints and issue a continue in the gdb shell in order to resume the Java process so that it can then hit the breakpoint you just set. From there, stepping in Java code until you enter the native function will magically continue the stepping inside the native function! If you use Eclipse to do both debugging this is even extremely cool since it’s just like following the program inside the same editor!
There’s one last thing to remember (other than possibly the need to set the source location in gdb or installing the OpenJDK debuginfo package for your distribution).
Hotspot uses segfaults for a number of interesting things, like deoptimise, NullPointerException etc.. Apparently, this is faster than doing specific checks and jumping around the code. This is a problem for gdb though, since it will stop every now and then to some random routines you don’t really (usually!) care about:
(gdb) cont Continuing. Program received signal SIGSEGV, Segmentation fault.
Irritating, since those are all legitimate segfaults.
To avoid that just do the following in the gdb console (or from the IDE in whatever way this is handled there):
(gdb) handle SIGSEGV nostop noprint pass
Now all the interesting work can be done without interruptions 😉