Friday, March 25, 2011

Day 41 - Setting up ngx_openresty (WAS: testing Test::Nginx)

A bit of background first. As I told you in yesterday's post, I started working on changing Test::Nginx (as a result of my frustration trying to move the tests I wrote to test my RRD module from my original "let's use python" approach to a more community-friendly "let's work with agentzh's Test::Nginx approach). But I was scared to break everything (especially with one of the features I want to introduce that changes the way requests and expected responses work). So, I did the smart thing (for once): tried to figure out a way to test what I was about to do. And more specifically to perform regression testing. But to do so, I needed more tests written with Test::Nginx than the ones I wrote myself for my RRD module.

I had read at some point in the mailing list that agentzh packages a bunch of "3rd party nginx modules" into ngx_openresty : Turning Nginx into a Full-fledged Web App Server. So, I figured agentzh would have all his modules there with all the tests. That sounded like a good place to look for tests for Test::Nginx. Actually, he confirmed it would be «the easiest approach though some of the dependencies may be missing"». He also said (in a follow-up email): «But we ourselves usually use the util/ script to build separate nginx instance for the current module».

As dependencies scare neither me nor my good old pal yum I went for the first approach that would give me a simple way to perform a lot of tests (the ones from all the modules) in one go.

It all started with:

wget '' 
tar xvzf ngx_openresty-
cd ngx_openresty-

Of course, I had to install a few things:

yum install readline-devel libdrizzlesee comments for why this is deleted

At this point everything pretty much looked fine (which honestly surprised me). So, I went for the ultimate in testing:

LD_LIBRARY_PATH="build/libdrizzle-root/usr/local/openresty/libdrizzle/lib:build/lua-root/usr/local/openresty/lua/lib" \
    PATH="build/nginx-0.8.54/objs/:$PATH" TEST_NGINX_NO_SHUFFLE=1 \
    prove -I ~/nginx/test-nginx/lib/ -r bundle/*/t

Let me explain a little bit what is going on here:

  • PATH="build/nginx-0.8.54/objs/:$PATH" makes sure you are using the nginx we just built (i.e. the openresty one).
  • TEST_NGINX_NO_SHUFFLE=1 is my personal favorite (I hate when I can't predict the order in which tests are run: it makes my head spin.
  • -I ~/nginx/test-nginx/lib/ is to use the Test::Nginx I'm working on (not a CPN installed one). You might not need this if using the CPAN version installed normaly.
  • -r bundle/*/t because it's where all the tests I'm interested in are (and some of them have subdirectories, so I want to recurse through that).
  • LD_LIBRARY_PATH="build/..." tells where to look for the drizzle and lua libraries (which are part of openresty). You might want to look at the comments to understand why I need this. This line being way too long, I won't mention it anymore.

At this point, the thing blew up in my face and I had to "fix" the obvious with: mkdir -p t/servroot and set TEST_NGINX_IGNORE_MISSING_DIRECTIVES to 1 to avoid the problems with the uninstalled modules. So, I started using something like:

PATH="build/nginx-0.8.54/objs/:$PATH" TEST_NGINX_NO_SHUFFLE=1 \
    prove -I ~/nginx/test-nginx/lib/ -r bundle/*/t

At this point, I started having tests that worked (not all of them, but most of them). I still needed a few more things:

yum install memcached redis
service mysqld start
service memcached start
service redis start

Also make sure you log into your MySQL DB and create the ngx_test user and database (for the tests of the drizzle module to work):

create user 'ngx_test'@localhost identified by 'ngx_test';
create database ngx_test;
grant all on ngx_test.* to 'ngx_test'@localhost;

That's when things got hairy: tests for ngx_lua refused to work. Lua is packaged with openresty but not everything in lua. Namely, you don't have yajl-lua (Yet Another JSON Library for LUA). So, as yajl-lua is not packaged as a rpm on my platform, I went (wait before you do the same):

yum install yajl yajl-devel cmake
cd bundle
git clone
cd lua-yajl
LUA_DIR=../lua-5.1.4/src cmake .

Unfortunately this will not get you anywhere as it's version 1.0.7 of yajl that is packaged for Fedora Core 14 and you need at least 1.0.9... :( So, I also had to get yajl compiled:

cd .. #now you should be in bundle
git clone git://
cd yajl
cmake .
cd ../lua-yajl
LUA_DIR=../lua-5.1*/src CMAKE_PREFIX_PATH=../yajl/yajl-1.0.* cmake .

Still, test bundle/ngx_lua-0.1.6rc2/t/005-exit.t (and quite a few others) was failing because LUA could not find the yajl library. The only way I managed to fix this was by adding a new TEST_NGINX_LUA_PACKAGE_CPATH variable and change the test file itself:

sed -i.bkp s/lua_package_cpath.\*\$/lua_package_cpath\ \'\$TEST_NGINX_LUA_PACKAGE_CPATH\'\;/g bundle/ngx_lua-0.1.6rc2/t/005-exit.t
sed -i.bkp s/lua_package_cpath.\*\$/lua_package_cpath\ \'\$TEST_NGINX_LUA_PACKAGE_CPATH\'\;/g bundle/ngx_lua-0.1.6rc2/t/*/exit.t

Yes, the line gets out of the blog, so get a bigger screen. ;) And yes, this is a lot of backslahes (and the line gets out of the blog), but I never manage to remember what double-quote escaping does in bash. So, for me it's either single-quote escaping (and no expansion happens but you cannot have single-quotes) or single character escaping (with backslash) and as soon as a character looks weird, I escape it... YMMV, of course.

From this point on, I was running my tests with:

PATH="$PATH:build/nginx-0.8.54/objs/" TEST_NGINX_NO_SHUFFLE=1 \
     TEST_NGINX_LUA_PACKAGE_CPATH="$(pwd)/bundle/lua-yajl/?.so" \
     prove -I ~/nginx/test-nginx/lib/ -r bundle/*/t/

And to be completely honest, at this point, I suspect I'm running the LUA that comes installed with Fedora Core 14 (not the one from openresty). I'll try to check that at some point.

The good news is that at this point, there are not many tests that fail. The most obvious ones are the tests from auth-request and upstream-keepalive. So, I looked at the perl code and it doesn't look at all like tests for Test::Nginx, although the tests have a use Test::Nginx;. They are not using the data-driven paradigm and invoking Test::Nginx->new() (and there has never been such a thing as new on Test::Nginx). As both modules were developed by Maxim Dounim, I guess he is using a "different" Test::Nginx. Could be fun to rewrite those tests with agentzh's Test::Nginx just to see if its expressive power is enough for what Maxim wants to test...

So, that will be it for today (that was one hell of a long post) as we are at a point where the only tests that fail use a different Test::Nginx or are for unknown directives. Next step, I'll try to get rid of the TEST_NGINX_IGNORE_MISSING_DIRECTIVES=1 in my prove.


  1. I think you do not have to install libdrizzle into your system by yum, because ngx_openresty has already bundled that C library.

    The readline-devel package is required (by lua) though :)

  2. First and foremost: thanks and congratulations (in no special order) for being the first one to comment here. ;)

    When I remove libdrizzle from my system it configures and compiles fine but nginx fails to start with "nginx: error while loading shared libraries: cannot open shared object file: No such file or directory". I guess this has to do with the fact that I did not issue the "make install" but I really don't feel that much like installing stuff behind my package manager's back.

  3. Was probably the lack of "make install". Gave me an opportunity to look closer at another thing that was not clear to me (which lua was being used). So, the trick if you don't want to do anything behind you yum or apt-get is to start the prove with LD_LIBRARY_PATH="build/libdrizzle-root/usr/local/openresty/libdrizzle/lib:build/lua-root/usr/local/openresty/lua/lib". Changing the post with reference to this right away...