Engineering: Testing performance and reliability of the module

[kiwix]

Engineering: Testing performance and reliability of the module

[kelson42]

Hmmm, looks like file handles are not closed correctly...

One thing I see is that the reader seems to be instantiated at each request and it should be instantiated at the Apache process creation. Doing so should save N file handle creation, depending how the Apache is configured. https://github.com/kiwix/kiwix-apache/blob/master/mod_kiwix.cpp#L110

Then it looks like the pointer is just set to NULL, instead of destroying properly the object. For me the root cause of the problem is probably there https://github.com/kiwix/kiwix-apache/blob/master/mod_kiwix.cpp#L163

@mgautierfr remarks?

[julianharty]

I agree that seems to be the problem and - so far - I've not found how to close the file handle. I've not found an obvious method or way of closing / freeing the file handle in zim/file.h or the smartPtr or fileImpl classes. I did try setting f = NULL at the end of the try {...} code but that didn't compile. I did notice that I was initialising zim::File unnecessarily in the try {...} code as it's only needed to dump out the list of articles if the user's request isn't found (a legacy capability I've left in for debugging while the code is still immature). I've moved the initialisation code to where the file handle is used (in the else code) and will test that. BTW: The HTTP call worked about 16K times before failing. Good for a low-volume alpha test but not for real use.

…

On 15 April 2017 at 19:51, Kelson ***@***.***> wrote: Hmmm, looks like file handles are not closed correctly... @mgautierfr <https://github.com/mgautierfr> any idea? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#8 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAv945CMWGhtYTM705ZJMpkT8BxrYcH7ks5rwRGwgaJpZM4M-a2M> .

[kelson42]

@julianharty just make this to delete the object propertly:
delete reader;

[julianharty]

My change didn't stop the module from running out of file handles. The error occurs around 16K requests (see below) Paradoxically what httping considers OK is not, it should get a 302 not a 200

--- http://localhost/kiwix/ ping statistics ---
26379 connects, 10059 ok, 61.87% failed, time 308877ms
round-trip min/avg/max = 0.3/0.8/25.0 ms
Transfer speed: min/avg/max = 0.000000/0.000000/0.000000 KB

[julianharty]

@kelson42 I tried your suggestion (delete reader;), it still fails after approximately 16K http requests.

connected to 127.0.0.1:80 (131 bytes), seq=16697 time=  0.35 ms 200 OK 0KB/s
^CGot signal 2
--- http://localhost/kiwix/ ping statistics ---
16698 connects, 336 ok, 97.99% failed, time 200804ms
round-trip min/avg/max = 0.3/1.2/28.5 ms
Transfer speed: min/avg/max = 0.000000/0.000000/0.000000 KB

What next?

[julianharty]

I submitted the latest code (that still fails around 16K requests) fdd3299

[mgautierfr]

From what I can see on fdd3299, there are two codepath where the reader is not deleted :

• When there is a exception,
• When fullUrl.back() == '/', the early return make exit the function without deleting the reader.

The best way to avoid this is to not create the reader on the heap (new) but to create it on the stack as you do for zim::File. This way you are ensure that the reader instance will be destroy when existing the stack and that the destructor is correctly called.

I'm also agree with @kelson42, you should create the reader only once for a content. It will not impact reliability but performance yes.

Of course, the bug can be elsewhere also :) I don't know if it is possible but it would be nice to run the module using valgrind.
From a five minutes research you could try to run something like :
LD_PRELOAD=kiwix_module.so valgrind --leak-check=full apache
Hopefully, it will show you the memory leaks at apache exit.

[julianharty]

Thanks both for your advice. I've managed to significantly improve the reliability of the module by adding a delete of the object just before returning the 302 response (see f648cba for the fix). I'll investigate how to improve the initialisation of the Reader object in the context of Apache modules.

For now, it's good to discover that the server coped with over 800K of requests without error.

connected to 127.0.0.1:80 (228 bytes), seq=835865 time=  1.21 ms 302 Found 0KB/s
connected to 127.0.0.1:80 (228 bytes), seq=835866 time=  1.58 ms 302 Found 0KB/s
^CGot signal 2
--- http://localhost/kiwix/ ping statistics ---
835867 connects, 0 ok, 100.00% failed, time 10135396ms

[julianharty]

Well it seems that removing the new and revising the code accordingly works at least as well in terms of longevity. See 3bd1563

Performance seems to be similar (perhaps the limit is in httping, we may want to find a more effective load generator to test performance). The old code (that used new and delete) averaged 1 request/response pair per 12.12 mS the new code 12.07 mS which seems within the margin of error.

connected to 127.0.0.1:80 (228 bytes), seq=324679 time=  0.73 ms 302 Found 0KB/s
connected to 127.0.0.1:80 (228 bytes), seq=324680 time=  2.01 ms 302 Found 0KB/s
^CGot signal 2
--- http://localhost/kiwix/ ping statistics ---
324682 connects, 0 ok, 100.00% failed, time 3920280ms

Note: there may still be various issues in the code e.g. if/when exceptions occur. Also, I'll investigate ways to improve the performance and - of course - to test the performance at greater scale and throughput than we're currently achieving in a local VM with httping.

[julianharty]

After removing an unnecessary variable in 3c0e500 the performance might be slightly better, measured at 11.93 mS per request/response on approximately 32K requests.

connected to 127.0.0.1:80 (228 bytes), seq=32179 time=  2.57 ms 302 Found 0KB/s
connected to 127.0.0.1:80 (228 bytes), seq=32180 time=  1.20 ms 302 Found 0KB/s
^CGot signal 2
--- http://localhost/kiwix/ ping statistics ---
32181 connects, 0 ok, 100.00% failed, time 383830ms

[julianharty]

Out of interest I ran three parallel sessions of httping using the same command line parameters. The servers coped well and for the main session (started first and stopped last) saw similar performance as previously

connected to 127.0.0.1:80 (228 bytes), seq=14425 time=  0.81 ms 302 Found 0KB/s
connected to 127.0.0.1:80 (228 bytes), seq=14426 time=  1.61 ms 302 Found 0KB/s
^CGot signal 2
--- http://localhost/kiwix/ ping statistics ---
14427 connects, 0 ok, 100.00% failed, time 177311ms

Meanwhile, session 2 results are:

connected to 127.0.0.1:80 (228 bytes), seq=11178 time=  0.93 ms 302 Found 0KB/s
connected to 127.0.0.1:80 (228 bytes), seq=11179 time=  1.91 ms 302 Found 0KB/s
^CGot signal 2
--- http://localhost/kiwix/ ping statistics ---
11180 connects, 0 ok, 100.00% failed, time 138641ms

and session 3:

connected to 127.0.0.1:80 (228 bytes), seq=10006 time=  0.71 ms 302 Found 0KB/s
connected to 127.0.0.1:80 (228 bytes), seq=10007 time=  0.68 ms 302 Found 0KB/s
^CGot signal 2
--- http://localhost/kiwix/ ping statistics ---
10008 connects, 0 ok, 100.00% failed, time 124017ms

This indicates that the server performs better than an individual httping session can load it with at least 35K request/responses in 177311ms

No doubt there's greater potential to improve the performance of the module, however my explorations in finding ways to initialise the Kiwix::Reader once and only once have failed with compilation errors so far. I've lots to learn about C++ performance! :)

[julianharty]

I discovered the apache bench utility today, and it's sufficient to find flaws in the module's performance and reliability when requests arrive in parallel. The module can handle 1,000 requests without a reported error provided there are no more than 4 concurrent threads. Once the number of concurrent threads are >=5 errors emerge and the utility quits early. Here's an example

ab -n1000 -c5 http://localhost:80/kiwix/A/musée.html
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
apr_socket_recv: Connection reset by peer (104)
Total of 786 requests completed

This is a useful development as I expected there'd be some flaws in the current somewhat basic implementation, and in comparison kiwix-serve.cpp includes thread locks. Now we can exercise the module I'm ready to experiment with improving the design and implementation.

[julianharty]

Here are some relevant articles to help with testing the module:

https://nextdime.wordpress.com/2014/07/02/apache-bench-installation-and-tests-ubuntu-14-04/

http://joeandmotorboat.com/2009/01/03/nginx-vs-yaws-vs-mochiweb-web-server-performance-deathmatch-part-2

http://joeandmotorboat.com/2008/02/28/apache-vs-nginx-web-server-performance-deathmatch

http://www.xenoclast.org/autobench/ (not currently being used, but mentioned in the deathmatch testing articles)

[kelson42]

The project is stalled since many years and has not reached the v1.0. Archiving the project. Please contact @julianharty if you are interested to work on the project.