etianen/aiohttp-wsgi

WSGI adapter for aiohttp.

aiohttp-wsgi

WSGI adapter for aiohttp.

aiohttp-wsgi is actively maintained and used in prodution, with any bug reports or feature requests answered promptly. It's a small, stable library, so can go a long time without new releases or updates!

Features

• Run WSGI applications (e.g. Django, Flask) on aiohttp.
• Handle thousands of client connections, using asyncio.
• Add websockets to your existing Python web app!

Resources

• Documentation is on Read the Docs.
• Issue tracking and source code is on GitHub.

aiohttp-wsgi's Issues

Need a version compatible with aiohttp>=2

It would be great to have a version compatible with the most recent builds of aiohttp.

Version of Python

I do not see any dependency information. Is python3.4 a requisite for this because aiohttp requires it? And if so, can I use this with an app that was written in python2.7?

aiohttp-wsgi on heroku

Hi Dave,

if I remember correctly you wrote the article on why to use Waitress on Heroku. Now since you came up with this server, how does it do on heroku? Can you recommend using it?
I'd love to hear your thoughts on this.

Cheers
-Joe

[Django] bug with cyrillic chars in URL

 File "/Users/antonogorodnikov/repo/partner/venv3/lib/python3.5/site-packages/django/utils/functional.py", line 33, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/Users/antonogorodnikov/repo/partner/venv3/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 121, in GET
    raw_query_string = get_bytes_from_wsgi(self.environ, 'QUERY_STRING', '')
  File "/Users/antonogorodnikov/repo/partner/venv3/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 241, in get_bytes_from_wsgi
    return value.encode(ISO_8859_1) if six.PY3 else value
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 62-64: ordinal not in range(256)

The problem is here:
"QUERY_STRING": request.rel_url.query_string,
https://github.com/etianen/aiohttp-wsgi/blob/master/aiohttp_wsgi/wsgi.py#L230

The solution is:
"QUERY_STRING": request.rel_url.raw_query_string,

TypeError: handle_request() takes 2 positional arguments but 3 were given

what is wrong with my code

import asyncio
from aiohttp_wsgi import wsgi
from aiohttp import web
async def http_server_handler(request):
	headers = {}
	headers['Content-Type'] = 'text/html'

	response = web.StreamResponse(
        status=200,
        reason='OK',
        headers=headers,
    )
	await response.prepare(request)#没有这行 RuntimeError: Cannot call write() before prepare()
	data=b'2333333'
	await response.write(data) #AssertionError: data argument must be byte-ish
	await response.write_eof()
	return response
app = web.Application()
app.router.add_route('*', '/', http_server_handler)
# web.run_app(app, host='0.0.0.0', port=8080)
# loop=asyncio.get_event_loop()
from concurrent.futures import ThreadPoolExecutor
executor = ThreadPoolExecutor(4) # > 1
application=wsgi.WSGIHandler(
            app,
            loop=None,
            executor=executor,
        )


if __name__ == '__main__':
	from gevent.pywsgi import WSGIServer
	http_server = WSGIServer(('0.0.0.0', 8080), application)
	http_server.serve_forever()

got error

>python3 wsgi.py
Traceback (most recent call last):
  File "C:\QGB\Anaconda3\lib\site-packages\gevent\pywsgi.py", line 999, in handle_one_response
    self.run_application()
  File "C:\QGB\Anaconda3\lib\site-packages\gevent\pywsgi.py", line 945, in run_application
    self.result = self.application(self.environ, self.start_response)
TypeError: handle_request() takes 2 positional arguments but 3 were given
2021-05-11T10:52:03Z {'REMOTE_ADDR': '192.168.43.162', 'REMOTE_PORT': '54957', 'HTTP_HOST': '192.168.43.162:8080', (hidden keys: 25)} failed with TypeError

Deployment instructions...?

Just wondering if there are any caveats/instructions to deployment? (eg. in my case, I'm hoping to set up a quick development server with Heroku and wondering if I can use this wrapper with waitress). I'm assuming no, since aiohttp has it's own http server...but I know there is support for aiohttp with Gunicorn...

Let me know if this makes sense. I'm sure I may not be making the most sense as I'm just getting started with asyncio...and trying to make it work with Django...

What's the benefit for aiohttp-wsgi + django?

It seems that aiohttp-wsgi just runs an application in an executor, which by default is just a thread pool, with a default pool size of 5 threads.

So if the application is designed with blocking APIs, such as flask and django, it will actually be executed via multi-threading and could not take advantage of co-routines. Then what's the benefit for using aiohttp-wsgi?

Memory exhaustion for large request.

The WSGI adapter reads the complete request body into memory. This results in extreme memory consumption for large file uploads and makes it very easy to run denial-of-service attacks.

curl --data-binary "@/dev/zero" http://example.com/wsgi

Running under multiple processes?

Is it possible to run this under multiple processes using a ProcessPoolExecutor to allow scaling past a single process? I have tried to replace the default executor with the ProcessPoolExecutor but get some pickling errors.

If not, is there a way to expose the whole app to apache/nginx/gunicorn?

This looks interesting but I have no idea how to implement it into a project.

Option to instruct aiohttp-wsgi to not touch logging

aiohttp-wsgi shouldn't touch the logging configuration or at least provide an option to instruct it not to.

Here is a real life scenario where the setup done by aiohttp-wsgi is problematic: a Django user has a custom configuration for the aiohttp logger:

# in Django settings.py
LOGGING = {
    "loggers": {
        "aiohttp": {
             "level": "ERROR"
         }
    }
}

What happens is:

• the interpreter logging is configured first by Django (when the WSGI app is imported)
• aiohttp-wsgi logging.basicConfig does nothing because logging is already configured
• the aiohttp logger level gets overridden by the level ofaiohttp-wsgi

It may not sound too bad, but it's tiresome when a user has to track down all third-party libraries doing such things, thus I recommend libraries authors to avoid touching logging and let the user configure this the way he wants in a central place.

Runtime error using Django

Hi,
I have Django project using python 3.7.0
Due to minimum example in the documentation, I'm trying to follow this docs and edit my wsgi like this.

import os

from aiohttp import web
from aiohttp_wsgi import WSGIHandler
from django.core.wsgi import get_wsgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')

application = get_wsgi_application()

wsgi_handler = WSGIHandler(application)
app = web.Application()
app.router.add_route('*', '/{path_info:.*}', wsgi_handler)

But when I run ./manage.py runserver I got these error

Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Unhandled exception in thread started by <function check_errors.<locals>.wrapper at 0x106324158>
Traceback (most recent call last):
  File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/utils/autoreload.py", line 225, in wrapper
    fn(*args, **kwargs)
  File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 137, in inner_run
    handler = self.get_handler(*args, **options)
  File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/contrib/staticfiles/management/commands/runserver.py", line 27, in get_handler
    handler = super().get_handler(*args, **options)
  File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/core/management/commands/runserver.py", line 64, in get_handler
    return get_internal_wsgi_application()
  File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/core/servers/basehttp.py", line 45, in get_internal_wsgi_application
    return import_string(app_path)
  File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/django/utils/module_loading.py", line 17, in import_string
    module = import_module(module_path)
  File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/adiyatmubarak/Documents/Koding/Dewe/rumah-dongeng-pelangi-api/src/rumah_dongeng_pelangi/wsgi.py", line 20, in <module>
    wsgi_handler = WSGIHandler(application)
  File "/Users/adiyatmubarak/.local/share/virtualenvs/rumah-dongeng-pelangi-api-avzb_er2/lib/python3.7/site-packages/aiohttp_wsgi/wsgi.py", line 178, in __init__
    self._loop = loop or asyncio.get_event_loop()
  File "/Users/adiyatmubarak/.pyenv/versions/3.7.0/lib/python3.7/asyncio/events.py", line 644, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'Dummy-1'.

Flask + ProcessPoolExecutor -> TypeError: can't pickle _thread.lock objects

• Python version: 3.7.7 (default, Apr 15 2020, 05:09:04) [MSC v.1916 64 bit (AMD64)]
• aiohttp: 3.6.2
• aiohttp_wsgi: 0.8.2
• flask: 1.1.2

To reproduce:

from concurrent.futures.process import ProcessPoolExecutor
from concurrent.futures.thread import ThreadPoolExecutor

from aiohttp import web
from aiohttp_wsgi import WSGIHandler
from flask import Flask


flaskapp = Flask(__name__)


@flaskapp.route('/')
def index():
    return 'hello from flask'


executor = ProcessPoolExecutor()  # ThreadPoolExecutor() works


aioapp = web.Application()
aioapp.router.add_route('*', '/{path_info:.*}', WSGIHandler(flaskapp, executor=executor))


if __name__ == '__main__':
    web.run_app(aioapp)

Traceback:

======== Running on http://0.0.0.0:8080 ========
(Press CTRL+C to quit)
Error handling request
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "F:\envs\_april\lib\multiprocessing\queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "F:\envs\_april\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "F:\envs\_april\lib\site-packages\aiohttp\web_protocol.py", line 418, in start
    resp = await task
  File "F:\envs\_april\lib\site-packages\aiohttp\web_app.py", line 458, in _handle
    resp = await handler(request)
  File "F:\envs\_april\lib\site-packages\aiohttp\web_urldispatcher.py", line 158, in handler_wrapper
    return await result
  File "F:\envs\_april\lib\site-packages\aiohttp_wsgi\wsgi.py", line 266, in handle_request
    environ,
  File "F:\envs\_april\lib\multiprocessing\queues.py", line 236, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "F:\envs\_april\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle _thread.lock objects

Update to aiohttp 3+

Is there any specific reason aiohttp-wsgi is pinned to aiohttp<3 (in setup.py) @etianen ?

Serving WSGI app from Python code

aiohttp-wsgi has a way to serve WSGI apps from the CLI, but there is no easy way to do it from Python code. By easy I mean with a single function call.

The serve function in __main__ is almost good for this use case, except that it expects args from the CLI.

For inspiration, I really like how waitress does it with waitress.serve.

"socket.send() raised exception."

I return an iterable for WSGI to stream to the client. The iterable can provide data for hours (radio streaming).

If the client disconnects before the iterable is done, I see this message printing on stderr: "socket.send() raised exception.". Worse, the iterable keeps iterating until is done, sucking CPU and RAM. Doing this several times I can bring down my server, just leaving broken connections behind me sucking resources.

Expected result: If a client disconects, the iterable I provided WSGI to send data to the client must be disposed, not called anymore and, eventually, garbage collected. If my iterables had a "close" method, call it before disposing the reference.

Code example showing this issue:

#!/usr/bin/env python3

import time
import aiohttp_wsgi

def application(environ, start_response) :
    def iterable() :
        while True :
            time.sleep(1)
            yield b'1234\r\n'

    start_response('200 OK', [
        ('Content-Length', '9999999999'),
        ('Content-Type', 'text/plain'),
    ])
    return iterable()

aiohttp_wsgi.serve(application, host="127.0.0.1", port=18333)

After making a request to port 18333 and force a disconnection, I will get a "socket.send() raised exception." message in the main window every second, showing that the iterator is still running instead of being disposed.

EnvironTest.testEnviron fails with aiohttp 3.8.1

We've upgraded to aiohttp>=3.8.0 recently in nixpkgs and I noticed this package now fails the following test:

___________________________ EnvironTest.testEnviron ____________________________

self = <tests.test_environ.EnvironTest testMethod=testEnviron>

    def testEnviron(self) -> None:
        with self.run_server(assert_environ) as client:
>           client.assert_response(headers={
                "Foo": "bar",
            })

tests/test_environ.py:99: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/base.py:43: in assert_response
    self._test_case.assertEqual(response.status, 200)
E   AssertionError: 500 != 200
------------------------------ Captured log call -------------------------------
ERROR    aiohttp.server:web_protocol.py:405 Error handling request
Traceback (most recent call last):
  File "/nix/store/cpi0v7m6l2y47banahzihgpca47diryv-python3.9-aiohttp-3.8.1/lib/python3.9/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
    resp = await request_handler(request)
  File "/nix/store/cpi0v7m6l2y47banahzihgpca47diryv-python3.9-aiohttp-3.8.1/lib/python3.9/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/build/source/aiohttp_wsgi/wsgi.py", line 294, in handle_request
    return await self._loop.run_in_executor(
  File "/nix/store/nffa4bgyi43wcrw3g3z0cmwy9zzz528y-python3-3.9.6/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/build/source/aiohttp_wsgi/wsgi.py", line 147, in _run_application
    body_iterable = application(environ, start_response)
  File "/build/source/tests/test_environ.py", line 13, in do_environ_application
    func(environ)
  File "/build/source/tests/test_environ.py", line 23, in assert_environ
    assert environ["CONTENT_TYPE"] == ""
AssertionError: assert 'application/octet-stream' == ''
  + application/octet-stream

Please provide support for aiohttp 0.22

I would like to use the new testing functionality in aiohttp 0.22. But the setup.py for aiohttp-wsgi disallows it.

I ran the aiohttp-wsgi tests against aiohttp 0.22.2 and all tests passed. My application also runs correctly with aiohttp-wsgi and aiohttp 0.22.2.

What is the reason for not allowing aiohttp 0.22 in your setup.py. If you let me know, I will happily work on these issues.

Improper "unquoting"

I am requesting this URL to my WSGI application: "http://127.0.0.1:8080/buffy/Test%20%C3%A1%20%C3%B3".

What I am getting in PATH_INFO is: "/buffy/Test á ó". This have two problems:

1. The string is "unquoted". I would expect to get it "as" I send it. That is what other WSGI servers do.
2. The unquoted is literal, not doing the right utf-8 translation. The right unquoting would be "/buffy/Test á ó".