Day 36 - nginx debug and valgrind from Eclipse

Today I'm breaking one of the rules of the blog and not telling you exactly what I did but rather telling you about something I have been doing over time, a little bit at a time. But I still think this is valuable to anybody who's really interested in learning the internals of nginx: debugging the code and running it through valgrind.

As I'm a lazy kind of guy, I use my favorite IDE (you know it's Eclipse) and that's what I'm going to tell you about. However, the principles apply if you are using more "standard" tools like gdb or the plain valgrind command. As a matter of fact, Eclipse integration of those tools is pretty minimal and quite often it is merely a preferences dialog box and a log parser.

When you want to put nginx under the microscope, there are two ways: the easy way and the hard way.

The easy way. I guess at some point Igor got sick of having to follow generation after generation of nginx processes to pursue a bug and decided that for development it would be easier to have only one process. Or maybe it is not the reason why but the fact and the matter is that if you set the following lines in your nginx.conf:

daemon  off;
master_process  off;

then the process that you start by typing nginx will actually be serving HTTP requests (or mail requests if you configured it so, but let's not even get there).

• The daemon directive tells nginx to act as a daemon. So, the first nginx process spawns another itself, detaches it and commit suicide. This is the usual way of daemonizing a process
• The master_process directive tells nginx to keep a "master process". This master process spawns the worker processes (the one actually handling the requests) and watches them. If one of them dies, it will receive a signal and restart a new one. As is mentioned in the documentation: this should always be "on" in production.

This being said, debugging or valgrindind with this configuration under Eclipse is nothing but pressing the big buttons. nginx acts like the good old "Hello world!" program.

The hard way. Sometimes (thank god, not too often) you are interested in what really happens in the forking/signal-handling process of the real thing. For example, I wanted to figure out if the init_process callback of the ngx_module_t was actually called in the first process, master process or worker process. Of course, you cannot figure this out if both daemon and master_process are set to off. So, you set them back to on and that's when things get dirty...

First of all, don't even think about using valgrind from Eclipse: as the valgrind page on the Linux Tools Project explains, the "Child silent after fork" cannot be unset. So, you are good to go back to running valgrind manually. This is probably no big deal: the leaks in code that is executed only when processes get forked are usually not big enough and don't happen often enough to give anybody any discomfort. Not a big loss, I would say.

Now, debugging is kind of tricky. By default, Eclipse will not follow forked children of a process you are debugging. However, since it uses gdb as its backend and gdb lets you change this behavior you can do it. Unfortunately the process is not that obvious (even after reading how to do it). So, I'll give it my own shot:

1. Start your debug configuration like usual by pressing the little green bug.
2. Eclipse stops at the beginning of the main function.
3. This is usually a good time to tell nginx that you want to follow future children of the current process instead of sticking with the father.
4. Click on the gdb process in the Debug View.
   
5. In the console (which is now the GDB console), type set follow-fork-mode child and hit enter:
   
6. You can confirm the setting with show follow-fork-mode
7. From now on, whenever there is a call to fork or vfork the debugger will follow the child (and not the parent).

Of course, if you are only interested in the processing of an http request you usually don't need to go the hard way. If you still want to, you can attach your debugging session to the right nginx process (this tend to be the one with the highest PID, the other one usually being the master process). And unlike when you are trying to debug an Apache mod you don't have to figure out where your request is going to land as there is only one process handling your requests (of course you could change that in your configuration too, but given nginx resource consumption it's not something you're likely to do any time soon ;) ).

Now, all you have to figure out is when the init_process callback/handler is called.

Have fun.