I've been doing some load testing of my app on gunicorn with the gevent arbiter, but under concurrent load the server will completely lock up if I exhaust the available file descriptors (in my case it's opening too many sockets to talk to the database). I can do some tuning on the system and my code to help avoid this, but the app server shouldn't completely lock up.
I've included a minimal test case to reproduce this by quickly creating a bunch of tempfiles to use up the file descriptors.
import tempfile
files = []
def application(environ, start_response):
files.append(tempfile.mkstemp())
start_response('200 OK', [('Content-type', 'text/plain'), ('Content-length', '2')])
return ['ok']
$ ab -n 3000 -c 100 http://127.0.0.1:8000/
$ gunicorn -k egg:gunicorn#gevent breakage
2010-05-12 01:03:34 [32292] [INFO] Arbiter booted
2010-05-12 01:03:34 [32292] [INFO] Listening at: http://0.0.0.0:80
2010-05-12 01:03:34 [32295] [INFO] Worker spawned (pid: 32295)
Traceback (most recent call last):
File "/mnt/application/lib/python2.6/site-packages/gevent/greenlet.py", line 388, in run
File "/mnt/application/lib/python2.6/site-packages/gunicorn/workers/ggevent.py", line 55, in acceptor
File "/mnt/application/lib/python2.6/site-packages/gevent/socket.py", line 255, in accept
error: [Errno 24] Too many open files
<Greenlet at 0x9a286ec: <bound method GEventWorker.acceptor of <gunicorn.workers.ggevent.GEventWorker object at 0x99af26c>>(<Pool at 0x9a30c8c set([<Greenlet at 0x9a28a6c: <b)> failed with error
2010-05-12 01:03:53 [32295] [ERROR] Error processing request.
Traceback (most recent call last):
File "/mnt/application/lib/python2.6/site-packages/gunicorn/workers/async.py", line 29, in handle
File "/mnt/application/lib/python2.6/site-packages/gunicorn/workers/async.py", line 58, in handle_request
File "/mnt/application/lib/python2.6/site-packages/gunicorn/http/request.py", line 181, in read
File "/mnt/application/lib/python2.6/site-packages/gunicorn/http/request.py", line 68, in read
File "/mnt/application/lib/python2.6/site-packages/gevent/socket.py", line 333, in recv
File "/mnt/application/lib/python2.6/site-packages/gevent/socket.py", line 137, in wait_read
File "core.pyx", line 293, in gevent.core.read_event.__init__ (gevent/core.c:3255)
File "core.pyx", line 243, in gevent.core.event.add (gevent/core.c:2414)
IOError: [Errno 2] No such file or directory
2010-05-12 01:03:53 [32295] [ERROR] Error processing request.
Traceback (most recent call last):
File "/mnt/application/lib/python2.6/site-packages/gunicorn/workers/async.py", line 29, in handle
File "/mnt/application/lib/python2.6/site-packages/gunicorn/workers/async.py", line 61, in handle_request
File "/mnt/application/breakage.py", line 5, in application
File "/usr/local/encap/python-2.6.5/lib/python2.6/tempfile.py", line 293, in mkstemp
File "/usr/local/encap/python-2.6.5/lib/python2.6/tempfile.py", line 228, in _mkstemp_inner
OSError: [Errno 24] Too many open files: '/tmp/tmpuJ_8V1'