calpaterson/python-web-perf

Code for testing performance of popular python webservers

Python webserver performance comparison

This repository holds some benchmarking configuration for a number of popular python webserver configurations:

• nginx and aiohttp
• nginx, gunicorn and flask
• nginx, uvicorn and starlette
• nginx, uwsgi and flask

These are benchmarked by apache-bench.

The nginx configuration is included, which listens on port 8001.

The servers are run from shell scripts.

$PWPWORKERS

This variable controls how many workers are started. Most of the async servers don't benefit from many more workers than cores available. Most of the sync workers do benefit from this though (often 2*nproc) is a good starting point.

Except for Daphne - see below.

Daphne

Daphne doesn't have a front facing proxy and so requires nginx to do the load balancing. Not a bad idea but means that to test it you need to edit nginx's conf and start multiple instances.

django/daphne#79 (comment)

Slow ttys

Some terminals are quite slow, for example gnome-terminal. Many of the wsgi servers output an access log to either stdout and stderr (uwsgi, daphne and uvicorn) - that needs to be redirected to a /dev/null to ensure a fair comparison.

Test data

Test data is generated by gen_test_data.py to produce a CSV which you then copy into a postgres database that has schema.sql loaded.

Outcomes

In /runs

python-web-perf's Issues

Clarify CPUs versus Cores in blog post

The readme here talks about cores a bit more accurately than the blog post.

"Async frameworks, due to their IO concurrency, are able to saturate a single CPU with a single worker process." lead me to investigate if you understood how multiprocessing actually works.

Changing CPU to Cores in the blog post would help reduce potential confusion.

Changing "IO" to "cooperative" would be good to. I don't think "IO concurrency" is a thing, or at least I don't think it specifically means cooperative concurrency.

Nginx configuration

Would you mind sharing your top-level nginx configuration used for this test? I consistently ran into problems with too many open file descriptors on the os level and couldn't get throughputs this large.

Why use pgbouncer when asyncio drivers support connection pooling?

Consering the fact asyncio frameworks supports connection pooling, theres no reason to use pgbouncer alongside asyncio

Good experiment, I have reproduced most of your results.

First, thank you for doing such a lot of works, they are really valuable.

Then, I have reproduced most of your results. Except that I change those DB connections with a simple sleep method to simulate infinite DB resources.
Yes, the sync frameworks perform better. It surprises me actually.

Next, I did a further adjustment, I try to simulate a heavy backend communication workload. Which is just simply raising sleep time to 50ms. In this case, async and sync performs pretty close. And when I increase the sleep time more, async ones finally perform a little bit better than the sync ones.

To summarize, I am not sure about the critical reason for this counterintuitive result, but I believe it is because that Python itself is slow. So when we have more native codes, we are better. Whatever, it is still intuitive that async ones perform better when it comes to waiting for a long time.

We need to realize this fact, and hope we will have a better implementation of Python in the future.

Could you also test performance of Tornado?

Many thanks.

meinheld is A blocking solution?

from https://github.com/mopemope/meinheld
The description is:
Meinheld is a high performance asynchronous WSGI Web Server (based on picoev) http://meinheld.org

Found a couple of bugs

Hi, thanks for doing this test.

Wanted to mention a few important bugs that I've found in your tests.

• The first one is pretty simple. For the aiohttp test you used the asyncio worker for Gunicorn. Switching to the uvloop worker gives it a nice bump in the chart.
• The next one is more complex. For all the sync tests you are using psycopg2 to access the database. This is fine when you use a standard server such as gunicorn, but this package is blocking when you use with a server that uses async internally, such as meinheld or gevent. All your meinheld/gevent tests have a max concurrency of 1 db query per worker because while a query is running you are blocking the whole process. For gevent, you can use psycogreen to patch psycopg2 so that it performs non-blocking waits. For meinheld I don't know of any package that does this, but there is a gist with a similar fix.
• I fixed the psycopg2 blocking problem with psycogreen to see what effect that has on the tests, but that led to another serious problem. The psycopg2 pool implementation is really bad, because once the maximum of pool connections is reached (4 in the way you configured the pool) it starts raising errors (see here) instead of waiting for a connection to be returned to the pool. Contrast that with aiopg, which properly waits for a connection here. You never saw this because without the psycogreen patch you never had more than one session active at a time.

Would you be willing to re-run your tests after you fix these problems? I believe they are severe enough to have the potential to change your results in a big way.

Number of connections in pool

Have you tried to increase number of connections in db connection pool for async apps?
One async worker can yield much more queries to DB than one sync worker, so they might be waiting for connection from pool under high load.

Pull requests for other frameworks

Are you open to incorporate pull requests that add other frameworks? I'm happy to add one for Pyramid with a couple of WSGI servers.

If yes, would you run the tests with the same setup and incorporate the results to the original blog post?

Please add a test of quart/hypercorn, which aims to be 'asyncio Flask'

Quart/hypercorn was one of the first ASGI (WSGI successor) frameworks. Please add it to your benchmarks. Note: I have no idea of its performance :) as I believe it's pure python.

Nginx not setup to use HTTP/1.1 and pipelining

The proxy_pass config uses HTTP/1.0 without keep-alive, which wastes a lot of time spawning new TCP connections to the backend server. Consult e.g. https://sanic.readthedocs.io/en/latest/sanic/nginx.html for how to make it fast.

P.S. Sanic's built-in web server (the one used in the above docs) is much faster than uvicorn-ASGI-sanic, which is what you are currently benchmarking.

EDIT: use the technical term keep-alive instead of pipelining to avoid confusion.