Monday, February 28, 2011

Day 29 - nginx development tool-chain, reviewed

Originally, today I was planning to tell you about ngx_str_t and char * but life being what it is, you will have to wait. Because today, I got a very interesting mail from agentzh on the mailing list. If you're interested in the details, have a look at agentzh's message.

As we were talking about the difficulty there is to getting a nginx module right (in addition to the usual C traps you have the nginx traps) he pointed at a number of interesting tools. Therefore I decided to review/expand my nginx development tool-chain...

You remember how, on Day 24 - Revisiting HTTP verbs and testing, I decided to use Python for my testing. Don't get me wrong, I'm happy with Python. But I was starting to feel the need for something a bit more evolved (and more self-contained) in terms of testing. So, I was starting to contemplate the idea of putting together some way to setup configurations in my test cases, restart nginx and make testing less verbose than it currently is. Luckily, agentzh pointed at his Test::Nginx on CPAN. It already does the things I was contemplating to do. And all you have to do is to learn a little bit of the library syntax. I am starting to really feel like it's worth it. May be not today (because I'm fine with what I have now), but I really feel like it would be a good investment. But I will have to rewrite my test cases for RRD with this new approach. Should be fun...

Now, what I consider to be THE MOST valuable tip agentzh gave me is actually just a patch provided by one of this students that removes the "memory pool" mechanism of nginx: nginx no-pool. It is very valuable because it allows you to run valgrind on nginx (and more specifically on your module). Memory management is always a beast to tame in C and a tool like valgrind is a must for this kind of quest. Unfortunately nginx "memory pool" mechanisms prevent the use of such tools. This patch makes valgrind work on nginx. So, I installed valgrind and the patch right away and I'm trying to figure out how to configure my Eclipse to run with it. And then it will be a full-time member of my nginx development platform.

Agentzh also pointed at EtcProxy. From what I understand, this is a TCP proxy that lets you fragment your packets and delay them as you wish. It's in Erlang (a language I had never heard of before today) and very small but for now, I consider this as an ad-hoc tool to create very specific situations when you cannot change the "thing" nginx is talking to (I think this was MySQL when agentzh used it). I think I'd rather have a "stub server" that replies only a certain type of response with the right fragmentation rather than add an extra component. Anyway, as I don't feel any need for this right now, I'll wait and see.

Another thing I must tell you: I was looking at chaoslawful's github. He is watching Evan Miller's github. So I went and had a look at Evan's page to see what he is working on. I found about ChicagoBoss and a lot of other Erlang things (that's probably where the Erlang presence in nginx community comes from). And I found this: mod_rrd_graph. Yeah, I'm not the only one to have stupid ideas... ;) I promise, I won't look at it before I consider I'm done with my RRD module. And then, I'll see how different our modules are. That will be fun too...

Oh, and another side note: agentzh works for taobao.com. Look at their stats on google trends. I would say nginx does scale... ;)

Friday, February 25, 2011

Day 28 - POST body buffers and first look at logging

You remember, yesterday I mentioned that when you are coding a request body handler you should loop through all the request->request_body->bufs. What I did not tell you is what is in there and why there is more than one buffer. It all boils down to the fact that nginx tries to have the most minimal impact on the planet by saving resources and doing as little as it can. That plus the fact that the network does pretty much what it wants.

To make my point as clear as I can I decided to walk you through an example of what happens when you force the network to do things unnatural. To do so, I wrote a test case:
def test_POST_show_buffers(self):
    conn = httplib.HTTPConnection("localhost", 8000, None, 20)
    params = urllib.urlencode({'value' : 'N:12345:678'+'678'*300})
    conn.putrequest("POST", "/tutu")
    conn.putheader('Content-Type', "application/x-www-form-urlencoded")
    conn.putheader('Content-Length', str(len(params)))
    conn.putheader('Accept', "text/plain")
    conn.endheaders()
    conn.send(params[0:6])
    time.sleep(1)
    conn.send(params[6:15])
    time.sleep(1)
    conn.send(params[15:])
    response = conn.getresponse()
    self.assertEqual(response.status, 200)
    data = response.read();
    conn.close()
    self.assertNotRegexpMatches(data, "Robin");
You got the idea: I want to send the body value=N:12345:678678678... with 678 being repeated 301 times (that's actually value=N%3A12345%3A678678678... after encoding the parameters) in three parts with a little pause between them to make sure the network layer sends them separately.
  • First part is value= and goes with the headers
  • Second part is N%3A12345
  • Third and last part is N%3A678678678...

My nginx being configured with maximum logging (i.e. error_log logs/error.log debug;
), I'll walk you through the interesting parts of the log...

22:16:20 [debug] 5222#0: *22 accept: 127.0.0.1 fd:3
nginx accepted the connection. Note the *22 which is an identifier assigned to this connection. This is what you'll use to keep track of what's going on for this customer in the rest of the log. This is very convenient. I guess Igor coming from a sysadmin background thought of making sysadmins' lives easier, a thing developers tend not to do (as a counterpart, he did not make developers' lives easier but that's another story). So, with this *22 we look a bit further:

22:16:20 [debug] 5222#0: *22 http process request line
22:16:20 [debug] 5222#0: *22 recv: fd:3 168 of 1024
22:16:20 [debug] 5222#0: *22 http request line: "POST /tutu HTTP/1.1"
22:16:20 [debug] 5222#0: *22 http uri: "/tutu"
22:16:20 [debug] 5222#0: *22 http args: ""
22:16:20 [debug] 5222#0: *22 http exten: ""
22:16:20 [debug] 5222#0: *22 http process request header line
22:16:20 [debug] 5222#0: *22 http header: "Host: localhost:8000"
22:16:20 [debug] 5222#0: *22 http header: "Accept-Encoding: identity"
22:16:20 [debug] 5222#0: *22 http header: "Content-Type:
application/x-www-form-urlencoded"
22:16:20 [debug] 5222#0: *22 http header: "Content-Length: 921"
22:16:20 [debug] 5222#0: *22 http header: "Accept: text/plain"
22:16:20 [debug] 5222#0: *22 http header done
And we see that nginx read 168 bytes (for those of you who don't know, recv is the system call to read data from a file descriptor: file descriptor 3 in our case which is the one for the socket resulting of the accept system call we saw earlier). 168 bytes were enough to get the request line plus all the headers. Now, looking even further, we find:

22:16:20 [debug] 5222#0: *22 http client request body preread 6
22:16:20 [debug] 5222#0: *22 http read client request body
22:16:20 [debug] 5222#0: *22 recv: fd:3 -1 of 915
22:16:20 [debug] 5222#0: *22 recv() not ready (11: Resource
temporarily unavailable)
22:16:20 [debug] 5222#0: *22 http client request body recv -2
22:16:20 [debug] 5222#0: *22 http client request body rest 915
nginx knows it has already read 6 bytes. Which is exactly the size of the first part of the body (value=). And, as you can see it knows that it still needs 915 more bytes to read the rest of the body. This is when it figures out that it won't be able to fit the rest of the 1024 buffer it allocated for processing the request line plus its headers. But it would be a shame to throw away all this good memory. So, it keeps the first 6 bytes as the first buffer for the request body and creates a new one for the remaining 915 bytes. A bit later:

22:16:21 [debug] 5222#0: *22 http run request: "/tutu?"
22:16:21 [debug] 5222#0: *22 http read client request body
22:16:21 [debug] 5222#0: *22 recv: fd:3 9 of 915
22:16:21 [debug] 5222#0: *22 http client request body recv 9
22:16:21 [debug] 5222#0: *22 http client request body rest 906
This little web server is good at math: it received the second part (9 bytes), copied them in the second buffer and knows that it needs only 906 more to be done. And finally:

22:16:22 [debug] 5222#0: *22 http run request: "/tutu?"
22:16:22 [debug] 5222#0: *22 http read client request body
22:16:22 [debug] 5222#0: *22 recv: fd:3 906 of 906
22:16:22 [debug] 5222#0: *22 http client request body recv 906
22:16:22 [debug] 5222#0: *22 http client request body rest 0
We are done. And with your favorite debugger you can see that for once request->request_body->bufs is not NULL.

As you noticed, I had to go through quite some effort to exercise this situation, but it can happen. And it gave me an opportunity to walk you through some logging...

Thursday, February 24, 2011

Day 27 - POST body in files and ngx_unescape_uri

I told you yesterday that I would talk about retrieving values form a POST for processing. As I'm a stand-up guy, here we go.

Back in the days when I started this journey, I told you buffers (ngx_buf_t) were interesting beast that were the foundation of this little web server. It's even more true than I thought and I don't think I'm at the end of my surprises with them. They are basically used for almost anything that involves processing data streams (and you do a lot of that in a web server). POST methods are no exception. Basically, whenever nginx is reading data from a client and making a HTTP request (ngx_http_request_t) from it, it is using buffers. And roughly, it all goes like that:
  1. Accept connection from client (SYN<->ACK/SYN<->ACK, if you see what I mean).
  2. Start reading data from client
  3. Parse client request (method, uri, etc.) and header lines as they arrive
  4. Store all that in the ngx_http_request_t structure
  5. When all headers are processed, call the content handler (there is actually a lot more going on at this point but if we start digging here we'll get lost, plus there is a lot I still don't know).
  6. At this point, the header_in field of the ngx_http_request_t structure points to a buffer that contains the "raw" headers as they were read from the client.

This holds true for all HTTP requests (GET, POST, HEAD, etc.). Now, as you know some HTTP methods accept bodies. It is the case for POST. However, the HTTP protocol being nice and everything has a header to tell the server how much data will be in the body. And nginx makes good use of this Content-length header and continues its processing with:
  1. Read all data in the body.
  2. Then trigger the request body handler (I told you at length about this guy yesterday).
  3. At this point, the body is available in request->request_body->bufs where request is a pointer to the ngx_http_request_t structure.
The attentive reader will have noticed the s at the end of bufs. Yes, there can be many buffers. And there are really two situations:
  • Either the body is really small and you have only one buffer.
  • Or, it is big and you have more than one
The point is: your code should loop through this chain and process all data there. There is one more thing you should be aware of: buffers can be files. And if your body is bug enough you can be sure they will be. I tell you because I read about this then completely forgot about it and looked stupid when I realised my nginx segfaulted when I was trying to access the memory of a buffer which was a "file buffer". So, you remember all this was to retrieve data encoded in "application/x-www-form-urlencoded". Of course, data has to be urldecoded. nginx is nice enough to help with the decoding (and the encoding) by providing a pair of functions: ngx_unescape_uri and ngx_escape_uri. Now, that's all good, except that they operate on u_char *, not on buffers. This means the body request MUST be loaded in memory. So, I went ahead and used ngx_read_file(). Unfortunately from what I can see this is blocking. This should not be a problem in my specific scenario because most of the time data will fit in memory and never go to disk. But this is against the principles of nginx. So, I made a note in my code (with a TODO) and will have a look at it another day. May be I'll find some inspiration with Valery's nginx upload module.

Wednesday, February 23, 2011

Day 26 - content handler vs. request body handler

As we have seen in yesterday's episode a request body handler (called when the body of the POST request made it to the server) returns nothing whereas the content handler (called after the headers have been processed) returns a status (NGX_DONE, NGX_AGAIN, etc.) in the form of a ngx_int_t. This thing ended up driving me almost crazy as you will see in a moment.

So, after yesterday's adventure, I was happy, able to test my module with big POST bodies. So, I figured I could start returning stuff. Like the URL-decoded version of the parameter that was submitted to me in the POST. Nothing really fancy. I had some trouble actually getting the data out of the POST body and in memory but I'll tell you about this tomorrow. So, I managed to get the response in a memory buffer and started sending it to the client with something like:
void ngx_http_rrd_body_received(ngx_http_request_t *r) {
[...]
/* Prepare BIG header and response */
[...]
rc = ngx_http_send_header(r);
if (rc != NGX_OK) {
ngx_log_error(NGX_LOG_ALERT, log, 0,
"pb sending header");
return NGX_HTTP_INTERNAL_SERVER_ERROR;
}

return ngx_http_output_filter(r, out_chain);
}
And that's when things started to get interesting (to say the least):
  • When posting a small body everything worked fine.
  • When posting a big body, the webserver was sending only the beginning fo the response and the client ended up timing out.

I went on and tried to figure out what was wrong. Checked 10 times my code to make sure that there was nothing wrong with the memory allocations. Ran it step-by-step in the debugger. Added debug logging for nginx, even watched the packets fly with Wireshark. I also started to dig deep into nginx code with the http_copy_filter, the http_write_filter and all their friends in the output filter chain. Watched them run in my debugger. But all this only got me:
  • A spinning head at realizing how much there is I don't know
  • The confirmation that nginx was sending only the first 65536 bytes of my response

A break under my fig tree did not help, so I went and explained my problem on the development mailing list. I was pretty sure I was doing something wrong but I could not figure what. Maxim Dounim gave me the answer: in the request body handler you should call ngx_http_finalize_request. When you are writing a content handler, you return the output of ngx_http_output_filter and nginx calls ngx_http_finalize_request to do the job for you. Now, in our situation, the content handler just registers a request body handler so it needs a way to tell nginx not to call ngx_http_finalize_request. It does that by returning NGX_DONE. The counterpart is that, later on, when the request body handler executes, wz must call ngx_http_finalize_request. It is fairly simple once you know it. And it explains why the request body handler does not return anything.

Now, if like me you wonder why two different conventions were used for something so close, I don't have the answer. The only comments I got from agentzh were:
Well, just to mind you, the rewrite handler, access handler, and body
handler all use its own convention of return values and they're
different in one way or another ;)

Besides, the specific meaning may change without notice because Igor
Sysoev tends to change his mind :)

If you'd call such things "traps", then I'd say there's tons of them
in the core.

The good thing is that it means nginx still has a lot more surprises in stock for me to discover and share with you... ;)

Tuesday, February 22, 2011

Day 25 - testing BIG request bodies, request body handler

Remember, on Day 19 - Echo module and where nginx stops with posts (or not), I pointed out a forum thread explaining how you should proceed to retrieve data POSTed. Basically it boils down to:
  1. In your content handler, when the request you are processing is a post, register a request body handler with something like:
    ngx_http_read_client_request_body(r, ngx_http_rrd_body_received);
  2. Once the body has been retrieved, your new request body handler (ngx_http_rrd_body_received in my example above) will be called and you can do whatever you want with it.

nginx being asynchronous and all, that left me with a question: is the request body handler called once when the body has been fully loaded or is it called many times as the "chunks" become available. And I know only one way to figure it out: to test it.

Testing this specific case has proven to be more difficult than I thought and I spent quite some time on this. I'll try to walk you through my thought process and the results I got.
  1. I started simple just by POSTing some meaningful value. I quickly realized that by doing that, the body was so small that header+body were fitting in one IP packet and everything was read by nginx in one gulp. As a consequence, in this situation, the call to ngx_http_read_client_request_body() actually results in a call to the request body handler (ngx_http_rrd_body_received). This is nice but not really what I had in mind.
  2. So, I figured that if I increased the size of the body, things would start to get interesting. So, I created a BIG request body and posted it. And, I got stuck with an HTTP status of 413 (I had to lookup the translation on google ; it means "Request Entity too large"). Didn't even know that was possible. But after soem search, I discovered that there is even a directive in nginx to control it : client_max_body_size. I had no reason to change it, so I did not.
  3. Instead, I went for a POSTed entity that would be BIG (bigger than an IP packet) but still fit in client_max_body_size. And...I saw the exact same behavior that I had seen with small entities: ngx_http_read_client_request_body() directly calls the request body handler (ngx_http_rrd_body_received here). I guess this is because everything is local and there is no network latency to slow down things. So, I went for a coffee under my fig tree... ;)
  4. And I came back with an interesting idea: what if I just create the latency myself by waiting between sendign the header and the entity. That led me to researching a bit more the python libraries and I ended up with some testing code like that:
    conn = httplib.HTTPConnection("localhost", 8000, None, 20)
    params = urllib.urlencode({'value' : 'N:12:34:56:78' * 20000})
    conn.putrequest("POST", "/tutu")
    conn.putheader('Content-Type', "application/x-www-form-urlencoded")
    conn.putheader('Content-Length', str(len(params) * 2))
    conn.putheader('Accept', "text/plain")
    conn.endheaders()
    time.sleep(4)
    conn.send(params)
    time.sleep(4)
    conn.send(params)
    response = conn.getresponse()
    self.assertEqual(response.status, 200)
    data = response.read();
    conn.close()
    self.assertRegexpMatches(data, ".*Robin.*");
    The two sleeps are really to figure out whether the handler is called once or twice. With that, I managed to test what I wanted.


Now, you can run it yourself and figure out how many times the request body handler is called. No, I'm kidding, I'll tell you. Here is the sequence as I saw it in my favorite debugger:
  1. The header arrives, nginx fires the content handler.
  2. The content handler registers the request body handler and returns.
  3. The first chunk of the entity arrives: none of the two handlers is called.
  4. The second (and last) chunk arrives
  5. The request body handler is called.
Conclusion: the request body handler is called only once...

This is fairly different from the behavior with small entities:
  1. The whole request arrives, nginx fires the content handler.
  2. The content handler registers the request body handler.
  3. As a consequence, the content handler calls the request body handler.
  4. The request body handler returns.
  5. The request handler returns.

Now, something weird about this request body handler is that its signature forbids you to return anything. This is really weird as the "regular" content handler expects you to return a ngx_int_t (which is pretty much supposed to be the one returned by ngx_http_output_filter()). The only way to ship back an error is by writing the appropriate header/body. The problem is: what can you do if the ngx_http_output_filter fails? Or returns NGX_AGAIN? There is no way to guarantee a module will not return NGX_AGAIN.

That's where we are going tomorrow...

Monday, February 21, 2011

Day 24 - revisiting HTTP verbs and testing

As I already told you many times, I wanted to support only GET and POST. So, that's how I started and tried to implement "meaningful" return codes (405) and error messages for other HTTP verbs. So, I started curling with -X and inadvertently tried to submit a HEAD request and have my reply send a meaningful message. That's when things started to get weird (connection closed before it should and the like). So, I went back to one of my favorite sites: W3C and more specifically to HTTP/1.1 methods definitions. That's how I realized that the HEAD is pretty much a MUST have and that it MUST NOT contain anything (content-length=0). That's where I was failing.

On top of that, this page talks about idempotence. As this is one of my leitmotivs when coding/designing interfaces, I could pass the joy of telling you that idempotence should be one of the drivers for your decision to use one of the verbs or the other. At least, that's what is in the specs. And all verbs/methods are supposed to be idempotent, except POST and CONNECT. Now, I don't think anybody cares, let alone respect that. Think of the number of websites that track the number of views for anything (pictures, videos, profiles, etc.). For all these sites, the GET is not idempotent (it increases the counter each time it processes a GET).

After 20 times typing curl -X WHATEVER -d name=daniel and other variants I finally got bored to death and decided that I needed an easier way to test my module. Using a C unit test library did not even come to my mind. However I thought of using Perl/LWP like agentzh. But that was too easy. I know Perl and I could use what agentzh did. And you know I like challenges. So, as I don't know Python (read a few examples, know there are no curly braces and that the number of spaces is important) and have been interested in learning it for years, I figured I would go for that. And, it has a unittest and a httplib libraries that should make my life easier (I like things that are not too easy but I am no masochist). And so far, a few copy/paste from the web pages of the two libraries mentioned above have already saved me quite some typing. We'll see how long this goes.

Oh, and as a side note: still no news from Valery on the review of my translation work...

Friday, February 18, 2011

Day 23 - revisiting HTTP POST

So, now that the configuration is pretty much sorted out, comes the question of actually getting the data from the request. An easy way would be to take them as arguments of the GET. nginx automatically creates variables for the arguments: nginx arg_PARAMETER. But this is not very REST-like andd I want to be REST-like: POST to send the data and GET to retrieve the graph. So, I have to get the values from the POST. And, never before in my life I wondered how the POST variables get transferred from the client to the server. This has always been managed for me by the "other layers", would their name be Weblogic, JBoss, php or anything else. And I have always been happy like that. Now, I tried to go hardcore with nginx and it's time to start paying the price, I guess. So, I looked a bit on the web and found a few interesting things.

Point one. There is a nginx module that makes the POST arguments available as variables: the Http Form-input module. I could copy/paste from there. But the presence of a test on the Content-type and some logging messages got me wondering: what if it's not application/x-www-form-urlencoded? Or is it even possible? So, I went to the HTTP RFC to figure out what a web server SHOULD and MUST support in terms of ways to encode POST parameters in a request body. It doesn't help much. Reading a bit more got me a to a blog post that lists a few not so uncommon content-types: HTTP POST Content-Type.

Point two. Yes, a form can pretty much be submitted using any kind of Content-Type. But the common ones seem to be: application/x-www-form-urlencoded (this is pretty much putting the query part of the URL from a GET in the body of a POST request), multipart/form-data (a bit trickier but this is very similar to the multi-part encoding mechanisms used in emails and is described in RFC 2388. But text/plain seems to be possible as well.

So, I have yet another design decision to make: what should I support? First version will just support application/x-www-form-urlencoded like everybody else. I'll get multipart/form-data support later (if at all).

Thursday, February 17, 2011

Day 22 - rrd directive, post argument handler, ngx_str_t to ngx_buf_t, patch day

A lot of different things today. So, I'll try to be concise and to the point with each.

I finally stabilised the specs for the configuration of my module. I already told you the HttpRRDModule will have to be activated with a directive. This directive will be valid only at the location level and it will take one mandatory argument: the name of the RRD database. This is all nice and good and can pretty much translate into the following ngx_command_t:
{ ngx_string("rrd"),
NGX_HTTP_LOC_CONF|NGX_CONF_TAKE1,
ngx_conf_set_str_slot,
NGX_HTTP_LOC_CONF_OFFSET,
offsetof(ngx_http_rrd_module_conf_t, db_name),
&ngx_http_rrd_post},
As you can see, since I'm a lazy guy I let nginx do most of the job. That includes parsing the directive argument and setting it into my structure:
typedef struct {
ngx_str_t db_name; /* Name of rrd database. */
} ngx_http_rrd_module_conf_t;
Now, I still want to be able to set the handler for requests and this is not done by ngx_conf_set_str_slot as you probably guessed. However, nginx offers a mechanism to do it. And this mechanism is the post field of ngx_command_t. But don't mistake the void* to be the pointer to the function that will do the "post-configuration job". Or you will end-up exactly where I ended: crash of nginx at configuration load. This is not very forgiving. The trick here is to use as post the ngx_conf_post_t structure:
static char *ngx_http_rrd_post_command(ngx_conf_t *cf, void *data, void *conf);
[...]
static ngx_conf_post_t ngx_http_rrd_post = {
ngx_http_rrd_post_command
};
I'm sure there is a good reason for post being of type void * instead of ngx_conf_post_t * but I have no clue what this reason might be.


Now that I have my configuration read and processed, I wanted to send a response with only a fixed text saying "everything is fine". This is pretty much the Echo module I'm rewriting...only crappier. As soon as you want to send a response, you have to go through a ngx_chain_t of ngx_buf_t. At this point I am "there must be a way to create a buffer from a ngx_str_t !!!". Except that I spent a lot of time looking for it and couldn't find it. So, I wrote one myself:
/*
* Helper function to allocate a buffer from a string.
*/
ngx_buf_t * ngx_http_rrd_create_buf_from_str(ngx_pool_t *pool, ngx_str_t *s)
{
ngx_buf_t *buf;
buf = ngx_calloc_buf(pool);
if (NULL == buf) {
return NULL;
}
buf->start = s->data;
buf->end = s->data + s->len;
buf->pos = buf->start;
buf->last = buf->end;
buf->memory = 1;
return buf;
}
Feel free to use it. If anyone cares, this is officially public-domain.

I think of the license because of something that happened on the mailing-list. Maxim Dounim posted no less than 31 patches against 0.9.4 (what is considered to be the HEAD). And someone asked what was the license. The author answered "BSD, like the rest of nginx". A lot of the patches are backport of patches already submitted against 0.8.X. A few interesting things:
  • Igor does not seem to be in a hurry to integrate other people patches.
  • One of the patches allow to reuse keep-alive connections when nginx needs more connections to handle the load. Pretty nifty and might be useful.
  • Maxim rocks

Wednesday, February 16, 2011

Day 21 - HTTP handler vs. content phase handler

I hope you have been able to handle the suspense. So, there are basically two ways to register your handler for a HTTP request and the two ways are very different.

Way one is the most common: you want your code to run as soon as you use a certain directive in the configuration of a certain location. And as soon as you use it you want your code to run and no other code... To do so, all you have to do is to copy/paste/adapt from ngx_http_flv_module.c:
static ngx_command_t ngx_http_flv_commands[] = {

{ ngx_string("flv"),
NGX_HTTP_LOC_CONF|NGX_CONF_NOARGS,
ngx_http_flv,
0,
0,
NULL },

ngx_null_command
};
[...]
static char *
ngx_http_flv(ngx_conf_t *cf, ngx_command_t *cmd, void *conf)
{
ngx_http_core_loc_conf_t *clcf;

clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module);
clcf->handler = ngx_http_flv_handler; /* this is your specific handler */

return NGX_CONF_OK;
}
The good thing is that nginx will handle most of the plumbing: parsing the location configuration, figuring out which requests match this location and run the appropriate handler for those who do.

Way two is more complicated but can let you do things you never dreamt of. You add an extra handler for the NGX_HTTP_CONTENT_PHASE. The idea here is to use the postconfiguration field of your module definition structure. Here is an example from ngx_http_autoindex_module.c:
static ngx_http_module_t ngx_http_autoindex_module_ctx = {
NULL, /* preconfiguration */
ngx_http_autoindex_init, /* postconfiguration */

NULL, /* create main configuration */
NULL, /* init main configuration */

NULL, /* create server configuration */
NULL, /* merge server configuration */

ngx_http_autoindex_create_loc_conf, /* create location configration */
ngx_http_autoindex_merge_loc_conf /* merge location configration */
};
[...]
static ngx_int_t
ngx_http_autoindex_init(ngx_conf_t *cf)
{
ngx_http_handler_pt *h;
ngx_http_core_main_conf_t *cmcf;

cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module);

h = ngx_array_push(&cmcf->phases[NGX_HTTP_CONTENT_PHASE].handlers);
if (h == NULL) {
return NGX_ERROR;
}

*h = ngx_http_autoindex_handler;

return NGX_OK;
}
The main advantage is that you get a chance to act on each request, not just the ones corresponding to a configuration you explicitly configured to do so. But, as you know from another crappy blockbuster "with great power comes great responsibility". So, you want to be careful with this. In particular, remember that you should politely NGX_DECLINED if you don't want your handler to run for a given request and that "other" modules might decide to handle the request before you. And if they do, you won't get your chance. So, at the end of the day, it's the order in which you specify modules in the config that will decide who has highest priority. That's what Igor means in auto/modules circa line 86 when he comments "# the module order is important".

Now that we know at least two ways of doing things, all we have to do is to decide which one we should use. I went for "Way one" because I think we want this to be enabled only on certain locations and when enabled the RRD module should take precedence over everything else.

Monday, February 14, 2011

Day 20 - TCP Proxy, HttpRRDModule v0.0 and content handler registration

Let's start wiht an accident. Yes, it also happens on virtual journeys. So, I ran into: the nginx TCP proxy module. Doesn't look like it used or lot or like it has a lot of support, but it could be a good base to run a cheap tcp load balancer/NATter. Not really knowing what I am talking about, I would say it's probably less efficient than IPVS. But one year ago, I would have bet nginx is not as good as Apache...

Enough distraction (the web is good at that). I started to focus on the development of my HttpRRDModule. So, here what I went through trying to get the bare bone infrastructure of the module:

  1. Get the nginx headers in the C file (copy/paste from any module, they are all the same).
  2. Create the config file. It's the file used by nginx to integrate your module into its build system. Pretty much all exemples are the same. Take Evan Miller's one.
  3. Build the thing (configure --add-module ...; make). Now, we have a big nothing that compiles. It's a good start. ;)
  4. Code the request handler. You know, the thing taking a ngx_http_request_t as argument and returning a ngx_int_t. You code something close to the empty function and you try to compile it. And that's where troubles start... It does not compile any more. The reason is fairly simple: nginx compiler settings are strict enough to fail the compilation in case of unused functions. I cannot say I'm complaining about this, I love using the compiler as much as possible to detect problems. But, this is honestly the first time I run into a project which default settings are to fail on unused functions.
  5. So, I went ahead and added all the magic mantras that register the handler with nginx. Declaring the ngx_http_module_t, the ngx_command_t and attaching my function as handler with something like:
    clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module);
    clcf->handler = ngx_http_rrd_handler;
    It's pretty much the example you find everywhere.

And at this point I look at the thing and am thinking "I'm creating a directive for nothing". Let's say I don't even want a directive, I just want to register my module and have it handle all requests. Yes, it's extreme as in real life I would probably limit my module to a location. But it's a good exercise. So, instead of having my code that registers the handler invoked as the processor of a directive, I make it called as the postconfiguration method of my module (see appropriate field in ngx_http_module_t structure). And...nothing works. At this point, I start wondering very seriously how this works. So, I start checking: my registration is called. It's just that nginx never gives my module a chance to handle the request. That's when I started adding breakpoints and watched the thing run in details.

Where you have to start looking is the ngx_http_core_content_phase function (ngx_http_core_module.c) there is a test to see if the request has a content_handler: if (r->content_handler) and in my case (handler attached from the postconfiguration) this is NULL. So, nginx runs all the other modules (including the index, auto-index and static built-in modules) and finally declares '404 - NOT FOUND'. All I have to find is "who is setting this r->content_handler. Of course, it's one of the previous phases. It's actually the "find config" phase. To be more specific look at functions ngx_http_core_find_config_phase and ngx_http_update_location_config. As I keep debugging, I notice that the handler is not retrieved from the location configuration of the http_core_module. And that's where the trick is: in the postconfiguration method, I can call ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module) but it pretty much returns crap. Or at least what it returns is not tied to a location. Therefore, I was registering my handler in a location that does not exist.

That's long enough for today. I'll tell you the rest of this story in my next post with the two ways you can register your module: as a HTTP handler or as content phase handler.

Sunday, February 13, 2011

Day 19 - Echo module and where nginx stops with posts (or not)

I keep seeing the echo module mentioned here and there. Plus, it's written by agentzh who is definitely oen of the most active guys on the mailing list. With Maxim Dounim and Igor Himself. These three seem to be the top guns of nginx. When they say something on the mailing list and you think they are wrong you'd better reconsider: they are right and you are wrong...

So, I went ahed and read the documentation of the HttpEchoModule. It's the first one I read from head to toe. And it is very instructive: the directives are very simple, they are well documented and the examples usually point things that are definitely worth understanding. As a side note to this reading, I was wrong on day 15 - no loops in nginx: nginx can loop. Or at least the echo module can. It is not one of the official modules so I guess it does not count and I was right... :D But, the fact and the matter is that you can loop through something like query arguments by using echo_foreach_split/echo_end. But I guess this works only for directives of the echo module. If anybody cares to test it and let me know. If it works for other stuff, I'll take a look at the code because I have no idea how this is possible.

Speaking of arguments and retrieving them. This comes for "free" in nginx for the arguments in the URI as the URI is fully parsed and the corresponding fields in the ngx_http_request_t structure. But (yes, there is a but) it's much more complicated if you want to retrieve what is in the body of your request (like parameters from a POST form). This has to do with the body being potentially a big chunk of data (I heard people upload videos on crappy sites like youtube.com... ;)) and nginx handling this as efficiently as possible. For some insight on how this could/should be done when developing your module, have a look at the forum thread Trouble getting the request body of a HTTP POST.

Ah, and one more thing: I don't know why but this blog made it to the home page of Pingy. I didn't even know about this site 2 hours ago... Another web mystery, I guess.

Saturday, February 12, 2011

Day 18 - translation online, rrd is next

Heard back from Valery. He said the translation is good but there are places where it could be improved. So, I'm waiting for his comments. As of now, he authorized me to publish it as "beta" (which I did). So, it's officially available as Development of modules for nginx.

So, now you guys are wondering what I'm going to be up to... Well, the next step is pretty obvious: I'm going to give a try at my first nginx module. And to start I decided to go for something really useless: a HttpRRDModule. It's probably slightly better than HttpHelloWorldModule but barely. So, the idea is that when this module is enabled on a location (a nginx concept but roughly a URL), it will provide a REST-like API to a RRDtool database. If you POST data, it will be inserted in the database and if you GET this URL, you will have a nice RRD graph. This is useless as I could probably come up with a solution that works in an hour or two with CGI or one of the existing RRDtool bindings (Perl, Python, PHP). But that would not belong to this blog then. And it would not help me understand the russian web server on steroids we all love. Therefore, I'm doing it the hard way: one nginx module with direct calls to the RRDtool API (which I will have to learn for the occasion).

So, here we go:
yum install rrdtool rrdtool-devel

And:

Friday, February 11, 2011

Day 17 - review done, let's get moving...

I'm done with reviewing this doc. I also made sure the thing was XHTML1.0 Strict compliant. I think this is the first time in my life I actually get a green light from W3C... Something like 12 years or so after writing my first HTML. Who said standards are made by standard committees for standard committees ? If only I could be on the W3C HTML committee... ;)

I confirm (and yes, I already said that yesterday but this is so true I have to say it again): reading stuff you wrote just after writing it is boring. At times you really feel like falling asleep (which I probably did in the course of this exercise). Next time, I'll take more time between doing something and reviewing it. At least, that will give me a chance to forget about it.

The good thing is: I'm done with it. I'm sending this first version to Valery for review and moving to something else...

Thursday, February 10, 2011

Day 16 - russian was tough, chinese no hope

I'm done with the translation and am now reviewing the beast I created. Trust me, it's boring. I feel like I'm back to school when I felt I had to review my homework. Which usually proves useless because you don't get more errors on second round.

I was so bored I skimmed through mailing lists and web site and ran into this: nginx internal slides. The slides are pretty good and give a lot of pointers (actually to parts of nginx I never had a look at). The tough thing is that I don't speak chinese and the video is in chinese (or so I guess) :(.

What is it about this project that it's all about oriental languages ? I guess russian doesn't really qualify as oriental but more as Slavic but to me both are a nightmare: I don't even get the alphabet.

Anyway, my point was that the slides are...slides and although it looks very interesting, there is very little chance you will understand things you did not understand before just with the slides and without the comments... That's what slides are all about: supporting a presentation, not as a self-explanatory document.

So, I wanted to share a bit of my frustration.

Tuesday, February 8, 2011

Day 15 - no loops in nginx. Or it's inconscious...

Life can be ironic sometimes. I'm almost done with the translation and I find out about this great tool : Google translator toolkit. This is very easy to use. Or, at least much easier than what I was doing with 10 tabs open on google translate and yahoo babelfish. I am no professional translator but from what I can see, I'm pretty sure the guys who designed the user interface of this were (or at least got a good
enough knowledge of the job to understand what was need). Doesn't happen too often, so I figured I would point that this is a good example of a user interface design that matches the need. The other examples that come to my mind are IntelliJIDEA when they introduced the refactoring features (yeah, that was a few years before Eclipse made them popular) and zen coding.

So, here I am, almost at the end of the document, working on configurations and, coming out of nowhere, here they are again : variables... Argh!!! Well, of course it makes sense that modules can define their own variables but this is not something I was expecting at all. Well, the example in Valery's document is not making a lot of sense to me (it's for setting variable "var"). So, I spent a little time to fish an example from the code. But that's not my point (so see below for the example). Besides the irony of it, my point is that I finally got the answer to one of my 'unanswered questions': what is a variable (see Day 7 - bye bye buffers, hello variables) through this sentence "To generate a variable, you must implement a function that fills the ngx_http_variable_value_t structure using data from a query, from the context, from the configuration of the module or from other sources.". So, a variable is pretty much a name and a value and the value can come from anywhere...

I mentioned an example of variable defined by a module: $modern_variable defined by the Http Browser module. The whole purpose of this module is to let the person writing the configuration decide what a modern browser is, provide this in the configuration and nginx makes $modern_variable available with the right value on each request.

Now, as far as the title of this post goes: nginx has no loop construct in its configuration language. But the irony of today's discoveries made me wonder if at some level nginx is not having me go circles.

Monday, February 7, 2011

Day 14 - directives configuration and number of parameters

Now, our good old friend Igor tried to make modules developers life easier. Probably after noticing that he was always copying/pasting the same piece of code to parse and handle the configuration he came up with his mechanism that allows the definition of directives for a module.

Just to give you an example of what I'm talking about: the gzip directive documented here is defined by the HttpGzipModule. All this module had to do was to tell nginx: "gzip" is one of my directives, it takes one argument that can be a flag and you can find it at any level you wish in the configuration (main, server or location). If, like me in the good old days, you've done some parsing yourself and/or tried to do something nifty with command-line arguments you will recognize this is good and nice.

Now, one thing that I find kind of problematic: the flags that let you define what to expect are not very well designed and some of them overlap. As an example, the flags NGX_CONF_TAKE1 and NGX_CONF_FLAG both tell "there is only one argument". Admittedly, NGX_CONF_FLAG says a bit more than that (it's one argument and it's a flag) but nothing in the interface (a bitwise combination of these flags) forces the module developer to avoid using both. Or worth: to use conflicting ones like NGX_CONF_TAKE6|NGX_CONF_FLAG. And there is no "right" way to implement this. All you can do is document it and hope that the developer will read the documentation (personally, I don't know a lot who do). If they don't, they'll eventually fall in the trap and complain that they got hurt.

Bottom-line: that is exactly why I tend to avoid bitwise flags in interfaces. I'm not smart enough to make sure I get all the possibilities right. So, instead I go for less "efficient" interfaces but interfaces that cannot be misused. Better safe than sorry...

Sunday, February 6, 2011

Day 13 - modules and versions

nginx is modular to hell. I think I already mentioned that and modules are the basic on which you can build pretty much any kind of server using nginx. You heard about nginx being both a mail and an http server. But you could probably make an ftp or dns server with it. And all this thanks to modules.

Basically, nginx defines the notion of module to be a group of handlers (remember my previous post) that it will trigger at different points in its life-cycle (basically starting the master process, starting the children/worker processes and death of those). That's it... Now, the smart thing (but also the confusing one) is that a given module can refine what a module is. And that's exactly what you get with HTTP modules: they are specialized modules. There is one "core" module called "http" (http/ngx_http.c) and this module basically defines what "http" modules are and how they are managed. Other types of modules include "core" modules, "event" modules (that's how nginx copes with various event-based interfaces like epoll, kqueue, select, etc.), "http" modules (the most common ones) and "mail" modules. But you could have your own type of modules. This is a good design and I take for a proof of this that Igor was able to build both a mail and an http server from this.

Now, after saying something nice about this design, there is something in the code that does not make sense to me: the implementation of versions of the module architecture. It's everywhere in the code with reference to the macros NGX_MODULE_V1/NGX_MODULE_V1_PADDING. There does not seem to be any example of modules with anything but version 1. I understand the need to have a version (NGX_MODULE_V1): it gives you a criterion to figure out what is the right version of the struct that is used and allows you to cast safely. But the padding is just killing me: the structure already has spare "slots" and the existing mechanism with module types gives already plenty of room for changing what a module is. And if the structure (without padding) would prove to be not enough, you could always extend it with a "larger" structure and cast it. I guess that's another question I will have to ask the development mailing list. At least, it will get discussions going there.

Thursday, February 3, 2011

Day 12 - welcome to handlers land

For those of you who are really planning to have a close look at nginx and the way it works, there is one thing you should know: this thing is modular to hell (some would say too much for its own good, but that's another story). And modularity is achieved using something a lot of people don't like: pointers to functions. This is a standard way to have a bit of "object-oriented-like-interface-implementation" in C but this is not the most common C and a lot of people get scared by these constructs.

Quite honestly, I cannot say this:
typedef char *(*ngx_conf_handler_pt)(ngx_conf_t *cf,
    ngx_command_t *dummy, void *conf);
is the kind of C I find very easy to read. On the other hand it's much more concise than the Java way where you basically have to define an interface and a class implementing this interface (hello java.lang.Comparable) just to provide a basic function.

So, just a warning: the C in nginx is not for the faint of heart. And given there is practically no comment to be found in the code, you should really consider getting back to your C courses if, like me, you haven't practiced in a few years.

So, I was mentioning handlers as they are pretty much everywhere. In particular, they are the foundation on which all modules are built. For a good idea of what kind of behavior you can change by implementing a module, you should read Valery's post on HTTP request processing phases.

Now, you might notice agentzh comment on NGX_AGAIN/NGX_DONE interpretation. Even Valery was not completely up to date with this. Now, what is really funny, is agentzh post: why nginx does not have an API reference. Even the core developers consider their baby good enough to commit to an API. I guess that's why it's still version 0.X.

Now, the scary part: more than 20M domains are running this...

Wednesday, February 2, 2011

Day 11 - request context : a necessary evil ?

I'm right in the modules and more precisely the module context. First, let me try to explain the idea.

Let's say you write a module that gets a domain as input and submits a request to various search engines (google, yahoo, bing) to figure out how many pages each of them indexes for this domain. You know nginx is asynchronous so you definitely should not be holding back while you are waiting for an answer from each of them. You should return to nginx saying "I have submitted my requests, come back to me when you have news from those connections". Now, nginx being a nice back will do exactly that : get back to you as soon as he has news from one of the search engines. Now, the problem is two-fold : first you have to figure which one and second you have to figure out when you are done. To do so, nginx offers you (the module developper) a mechanism called the module module context. It's fairly simple: it's a pointer to a memory zone you allocated and that nginx will keep for you as associated to this request and your module. Not only will it keep it for you, but it will also give it back to you if you ask nicely and provide a request. This is what I call an explicit context.

Now, when you are developing in a more "traditional way" (let's say PHP or Java) you don't bother about all this because the context is basically managed for you by the engine (it is both in the stack itself and/or in some thread-attached structure). The context is always associated with the thread and you don't have to bother. This is what I call an implicit context.

Both approaches are fine. The explicit context is geared towards performance at the cost of developers nightmares. The implicit context makes developers life easier but is suboptimal (you end up with a bunch of things you don't need in your context.

If I had a magic wand (or if I was smarter) I would create a programming model (or would that need to be a language of itself) where it would be easy to decide what to put in your context. You could decide to put "all sockets I'm going to open from now on" or the good-old stack, or some kind of closure (in the style of javascript where you end up with pretty much everything and the kitchen sink in your context). Food for thought...

As I'm already having trouble figuring out how a web server works, I'll forget about this and get back to my original journey.

Tuesday, February 1, 2011

Day 10 - nginx and regular expressions - a design failure to me

Today, I worked on the "regex" (regular expressions, of course) chapter and I am definitely disappointed. I was expecting this to have more "meat". As a matter of fact, nginx regexes are built on top of the PCRE library (if you read me since the beginning you could have guessed because my post Day 2 - building nginx... mentions this is a requirement). That they use it is fine with me. What I find disappointing is that nginx is wrapping an existing library into its own but is not adding anything to it. I can think of a number of reasons on how this happened but none of them is satisfying to me:
  • Igor thought he could do an awesome job on this subject too but, after spending a few months working on a better implementation, he realised this is even more hardcore than developing the fastest web server around. Regexes have been around for a long time. A lot of people worked on it and a lot of computer science/mathematical theory hides behind the problem and the implementation. If you don't believe me, just have a look at the article on Wikipedia regular expressions and explain to me what a Kleene operator is and how it is related to regexes. So, at some point in time he realizes is not going to do a better job than PCRE and decides to stick with this. Then he should have gone one step further and completely remove ngx_regex_t (which is now just adding complexity to the code and not bringing any value).
  • Igor wanted to add some flexibility and be able to switch from one library to the other. If that was his original intention, then I consider there is a problem with the API as it is designed. It fails to abstract some of the concepts that are tied to PCRE. More specifically, the ngx_regex_exec function has a captures argument which is clearly a copycat of the ovector argument of the pcre_exec family of functions.
  • Igor wanted to "hide" the complexity of the PCRE library (the man page for pcreapi is almost 2000 lines long). That, I can buy. But, if that was his original intent he failed to provide a simple API on the ngx_regex_exec: the captures argument should have been "translated" into something like an array of ngx_str_t. Just understanding how this parameter works took me at least half-an-hour.
Or...I completely missed something and in a few days/weeks/months/years, I will publicly apologize. But, at this point, I feel like the regex API provided by nginx is not bringing any value and it would make the world a better place just to get rid of it...