pnuckowski/aioresponses

Aioresponses is a helper for mock/fake web requests in python aiohttp package.

aioresponses's Issues

Recording mode

First i wanted to say thanks for making this package. Its really helpful.

I would like to have a recording mode and a way to load previous responses from a file. Something like Betamax or vrc. ATM I'm manually adding responses from a file in a decorator. This is helpfull for more complex data structures where adding is inside the test would be to verbose.

response with Exception not cleared out of `_responses`

I have an API Client I've written that uses aiohttp under the hood access various APIs. Amongst its tasks are making sure it handles errors/bad responses from the API well without bringing the program to a screaming halt, and one such problem is handling aiohttp's ClientPayloadError exception -- that it continues to retry the endpoint while logging the response. I wrote a test to make sure this works properly:

    @aioresponses()
    def test_ClientPayloadError_is_handled_properly(self, mocked):
        """
        a ClientPayloadError should log that it occurred, and then retry the same link
        """
        mocked.get(
            'https://test.url/objects/10/', exception=ClientPayloadError('oops'),
        )
        mocked.get(
            'https://test.url/objects/10/', payload={'test': 'data'}, status=200, exception=None,
        )

        with self.assertRaises(APIException) as e:
            with self.assertLogs() as logs:
                data = self.loop.run_until_complete(self.api_client.get_async('objects/10/'))

        self.assertEqual(data, {'test': 'data'})
        self.assertEqual(logs.output, ([
              'expected log output blah blah'
        ]))

Unfortunately, when the test above is run, the ClientPayloadError response is repeatedly raised until the retry limit is raised, rather than running the second (non-Exception) response.

My suspicion is that the _responses array is not being updated in match, since the exception stops the code that removes the URlResponse from _responses from running, but I haven't had a chance to really dig into it. I would, however, be happy to open a PR with a failing test, if it would be helpful for you.

Add methods for check request data

Hello, can be added methods like "assert_called_once_with" from mock.Mock for check POST request?

Relative URLs in redirects don't work

aioresponses handles following redirects created mocked requests with 3xx status codes, but only if they redirect to absolute URLs. Relative URLs are not handled correctly.

Here's a test (requires pytest-asyncio) that works with the absolute URL but not the relative URL:

import aiohttp
import aioresponses
import pytest

@pytest.mark.asyncio
@pytest.mark.parametrize('location', ['../baz', 'http://test.invalid/baz'])
async def test_redirect(location):
    with aioresponses.aioresponses() as rmock:
        rmock.get('http://test.invalid/foo/bar', headers={'Location': location}, status=307)
        rmock.get('http://test.invalid/baz', body='hello world')
        async with aiohttp.ClientSession() as session:
            async with session.get('http://test.invalid/foo/bar') as resp:
                text = await resp.text()
                assert text == 'hello world'

Output:

=================================== FAILURES ===================================
____________________________ test_redirect[../baz] _____________________________

location = '../baz'

    @pytest.mark.asyncio
    @pytest.mark.parametrize('location', ['../baz', 'http://test.invalid/baz'])
    async def test_redirect(location):
        with aioresponses.aioresponses() as rmock:
            rmock.get('http://test.invalid/foo/bar', headers={'Location': location}, status=307)
            rmock.get('http://test.invalid/baz', body='hello world')
            async with aiohttp.ClientSession() as session:
>               async with session.get('http://test.invalid/foo/bar') as resp:

rel_redirect.py:15: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../env3/lib/python3.6/site-packages/aiohttp/client.py:1012: in __aenter__
    self._resp = await self._coro
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <aioresponses.core.aioresponses object at 0x7f3b60819470>
orig_self = <aiohttp.client.ClientSession object at 0x7f3b607ca630>
method = 'GET', url = URL('http://test.invalid/foo/bar'), args = ()
kwargs = {'allow_redirects': True}, url_origin = 'http://test.invalid/foo/bar'
url_str = 'http://test.invalid/foo/bar'
key = ('GET', URL('http://test.invalid/foo/bar'))
kwargs_copy = {'allow_redirects': True}, response = None

    async def _request_mock(self, orig_self: ClientSession,
                            method: str, url: 'Union[URL, str]',
                            *args: Tuple,
                            **kwargs: Dict) -> 'ClientResponse':
        """Return mocked response object or raise connection error."""
        if orig_self.closed:
            raise RuntimeError('Session is closed')
    
        url_origin = url
        url = normalize_url(merge_params(url, kwargs.get('params')))
        url_str = str(url)
        for prefix in self._passthrough:
            if url_str.startswith(prefix):
                return (await self.patcher.temp_original(
                    orig_self, method, url_origin, *args, **kwargs
                ))
    
        key = (method, url)
        self.requests.setdefault(key, [])
        try:
            kwargs_copy = copy.deepcopy(kwargs)
        except TypeError:
            # Handle the fact that some values cannot be deep copied
            kwargs_copy = kwargs
        self.requests[key].append(RequestCall(args, kwargs_copy))
    
        response = await self.match(method, url, **kwargs)
    
        if response is None:
            raise ClientConnectionError(
>               'Connection refused: {} {}'.format(method, url)
            )
E           aiohttp.client_exceptions.ClientConnectionError: Connection refused: GET http://test.invalid/foo/bar

../../env3/lib/python3.6/site-packages/aioresponses/core.py:392: ClientConnectionError

Release 0.1.5

Can you please release the 0.1.5 version with commit 867782f - fix startswith error
It's important fix, package can't be used without it.

Thank you.

passthrough for all non-mocked URLs

At the moment passthrough works as a whitelist, where every not mentioned URL goes through the mock.

Does it make any sense to you, if there would be a switch to enable a "passthough" for every URL not mocked?

I could also imagine that aioresponses works like a long caching proxy where it passes through not-yet-mocked 404's and stores them after 200 - so the first call (test run) would fill up the fixtures and the second gets served from there....

At least in my actual project I need reasonable real mocks, which I fetch and store as fixtures and provide them to my tests, e.g. https://gist.github.com/diefans/aa4eb4a0247331ba6d087edd5a88de9b

Better support for using as a fixture with unittest

If I wish to use pytest fixtures with unittest then aioresponses() needs to be stored at the TestCase class level https://docs.pytest.org/en/3.0.0/unittest.html#mixing-pytest-fixtures-into-unittest-testcase-style-tests

To do this effectively ideally you need to create an aioresponses() object with everything added without having already created the with context. At the moment you can only add routes after you have started the context to monkeypatch in, it would be a lot more powerful if you could just specify all the routes then pull up and down the monkey patching. So instead of having to do:

with aioresponses() as m:
    m.get(blah)
    ...

you could instead do something like

mocked_server = aioresponses()
mocked_server.get(blah)

with mocked_server:
    client.get(blah)

With respect to unittest and pytest fixtures this means you can place the mocked server context generation in the class and just do something like:

@pytest.fixture(scope="class")
def mock_aiohttp_service(request):
    mocked = aioresponses(passthrough=['http://127.0.0.1:'])
    mocked.get('/', payload={'status': 'ok'}, status=200)
    request.cls.mock_aiohttp_service = mocked
    return mocked

@pytest.mark.usefixtures("mock_aiohttp_service")
class TestingClientStuff(TestCase):
    def test_something(self):
        with self.mock_aiohttp_service:
            response = wrap_coroutine(ClientSession().get(blah))
            self.assertDictEqual(response, {"status": "ok"})

no main loop assigned when testing

Thanks @pnuckowski for this great library. We use it in our project and we are experiencing some errors when testing with the latest aiohttp library (0.6.0).

RuntimeError: There is no current event loop in thread 'MainThread'.

You can see how the aiohttp documentation covers this topic exemplifying with AioESService
http://aiohttp.readthedocs.io/en/stable/testing.html#aiohttp-testing-writing-testable-services

Some libraries like motor, aioes and others depend on the asyncio loop for executing the code. When running your normal program, these libraries pick the main event loop by doing asyncio.get_event_loop. The problem during testing is that there is no main loop assigned because an independent loop for each test is created without assigning it as the main one.

could you consider to pass the loop in the client signature?
def __init__(self, endpoints, *, loop=None, **kwargs)

Thank you!!

Allow unregistered url to hit actual server 2

Hi guys!

passthrough doesn't work for me

class SerpTopTestCase(AioHTTPTestCase):

    async def get_application(self):
        app = create_app(loop=self.loop)
        return app

    @unittest_run_loop
    async def test_serp_top(self):
        with aioresponses(passthrough=['https://api.vertifire.com']) as mocked:
            mocked.get(
                VERTIFIRE_SERP_TOP_API_URL, status=200, payload=dict(data=[]))
            request = await self.client.request("GET", "/serp_top/")
            assert request.status == 200
            response_data = await request.json()
            assert "data" in response_data

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/test_utils.py", line 415, in new_func
    return self.loop.run_until_complete(func(self))
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/usr/share/marketmuse/marketmuse/tests/functional/serp_top.py", line 165, in test_serp_top
    request = await self.client.request("GET", "/serp_top/")
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/test_utils.py", line 253, in request
    method, self.make_url(path), *args, **kwargs
  File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 616, in __iter__
    resp = yield from self._coro
  File "/usr/lib/python3.5/asyncio/coroutines.py", line 206, in coro
    res = func(*args, **kw)
  File "/usr/local/lib/python3.5/dist-packages/aioresponses/core.py", line 170, in _request_mock
    if url.startswith(prefix):
AttributeError: 'URL' object has no attribute 'startswith'

what am I doing wrong?

ubuntu 14.04
aiohttp==2.0.7
aioresponses==0.1.4

Is params supported for get()?

Hello,

On aiohttp website, it suggests passing params can be done with

async with session.get('http://httpbin.org/get',
                       params=params) as r:

But I get E TypeError: add() got an unexpected keyword argument 'params' when trying to use this lib?

Am I doing something wrong?

Feature request: allow hdr.METH_ANY to match any HTTP request method

Was porting some code that uses requests + requests_mock to aiohttp + aioresponses, which occasionally uses requests_mock.ANY when adding requests to mocker object.

aiohttp already defines the aiohttp.hdrs.METH_ANY constant (defined simply as "*"). It seems that adding support for this in aioresponses would require only a minor change in the first line of the definition of the RequestMatch.match method should be changed do:

def match(self, method: str, url: URL) -> bool:
-  if self.method != method.lower():
+  if self.method != method.lower() and self.method != hdrs.METH_ANY:
      return False
  return self.match_func(url)

This behavior would exactly match the behavior of requests_mock. I'm not sure how other mock frameworks that support an "ANY" method parameter interpret it, nor if aiohttp has any special semantics around it (I could not find any, but didn't dig very deep--however, it does also define an hdrs.METH_ALL, which is a set value). Also, I have no idea if you have any other plans for extending match semantics (e.g., perhaps allow method to be a set of values?).

However, if this feature sounds reasonable and doesn't conflict with anything else, I could also spend a bit of time to submit a PR (either for just hdrs.METH_ANY support, or any preferred alternative such as set arg value, with appropriate unit tests), if that would help.

Finally, thanks for writing aioresponses, very helpful!

Add mocked requests count or mocked requests left

Give user ability to check if all mocked request was called

Send a mocked request more than once then got ClientConnectionError

when i send a mocked request more than once, got ../../venv/lib/python3.7/site-packages/aioresponses/core.py:392: ClientConnectionError

# -*- coding: utf-8 -*-
import aiohttp
import pytest
from aioresponses import CallbackResult, aioresponses


def callback(url, **kwargs):
    return CallbackResult(status=418)


@pytest.mark.asyncio
async def test_callback(aiohttp_client):
    with aioresponses() as m:
        session = aiohttp.ClientSession()
        m.get('http://example.com', callback=callback)
        for i in range(2): # 1 is ok,2 cannot pass
            resp = await session.get('http://example.com')
            assert resp.status == 418

the error is

self = <aioresponses.core.aioresponses object at 0x10b4360b8>
orig_self = <aiohttp.client.ClientSession object at 0x10b4435f8>, method = 'GET'
url = URL('http://example.com'), args = (), kwargs = {'allow_redirects': True}
url_origin = 'http://example.com', url_str = 'http://example.com'
key = ('GET', URL('http://example.com'))
kwargs_copy = {'allow_redirects': True}, response = None

    async def _request_mock(self, orig_self: ClientSession,
                            method: str, url: 'Union[URL, str]',
                            *args: Tuple,
                            **kwargs: Dict) -> 'ClientResponse':
        """Return mocked response object or raise connection error."""
        if orig_self.closed:
            raise RuntimeError('Session is closed')
    
        url_origin = url
        url = normalize_url(merge_params(url, kwargs.get('params')))
        url_str = str(url)
        for prefix in self._passthrough:
            if url_str.startswith(prefix):
                return (await self.patcher.temp_original(
                    orig_self, method, url_origin, *args, **kwargs
                ))
    
        key = (method, url)
        self.requests.setdefault(key, [])
        try:
            kwargs_copy = copy.deepcopy(kwargs)
        except TypeError:
            # Handle the fact that some values cannot be deep copied
            kwargs_copy = kwargs
        self.requests[key].append(RequestCall(args, kwargs_copy))
    
        response = await self.match(method, url, **kwargs)
    
        if response is None:
            raise ClientConnectionError(
>               'Connection refused: {} {}'.format(method, url)
            )
E           aiohttp.client_exceptions.ClientConnectionError: Connection refused: GET http://example.com

../../venv/lib/python3.7/site-packages/aioresponses/core.py:392: ClientConnectionError

Assertion failed

wonder if anyone has see the problem before, or perhaps my usage is somewhere bad?

Support multipart responses?

Mocking mutlipart responses seems to be fairly difficult, if not impossible.

In this conditional here we always cast the body to bytes. However if you want to do a multipart response you should be able to do something like this?:

with MultipartWriter('mixed') as mp_writer:
    for payload in payloads:
        mp_writer.append(payload, headers={hdrs.CONTENT_TYPE: 'image/png'})
mock_aioresponse.get('URL', body=mp_writer)

not working with aiohttp 3.1.0

self = <aioresponses.core.UrlResponse object at 0x10f8c8160>

    def build_response(self) -> Union[ClientResponse, Exception]:
        if isinstance(self.exception, Exception):
            return self.exception
>       self.resp = self.response_class(self.method, URL(self.url))
E       TypeError: __init__() missing 8 required keyword-only arguments: 'writer', 'continue100', 'timer', 'request_info', 'auto_decompress', 'traces', 'loop', and 'session'

../../.virtualenvs/yui/lib/python3.6/site-packages/aioresponses/core.py:57: TypeError

See this commit history for more info

Can not raise ClientConnectorCertificateError

from aiohttp import ClientSession, client_exceptions

import aioresponses

@pytest.yield_fixture()
def response_mock():
    with aioresponses.aioresponses() as m:
        yield m

@pytest.mark.asyncio
async def test_aws(response_mock):
    response_mock.get(
        API_URL,
        exception=client_exceptions.ClientConnectorCertificateError,
    )

    async with ClientSession() as session:
        async with session.get(API_URL) as resp:
            data = resp.json()  # expect rising error, nothing happened

Only one request can be sent in an aioresponses() context？

I sent two same requests in an aioresponses() context, but the second failed:

aiohttp.client_exceptions.ClientConnectionError: Connection refused: GET http://test.example.com

Here's my code:

import asyncio

import aiohttp
from aioresponses import aioresponses


def test_ctx():
    loop = asyncio.get_event_loop()
    session = aiohttp.ClientSession()
    with aioresponses() as m:
        m.get('http://test.example.com', payload=dict(foo='bar'))

        resp = loop.run_until_complete(session.get('http://test.example.com'))
        data = loop.run_until_complete(resp.json())

        assert dict(foo='bar') == data

        resp = loop.run_until_complete(session.get('http://test.example.com'))
        data = loop.run_until_complete(resp.json())

        assert dict(foo='bar') == data

The question is:
Was it designed for some reasons? And what are these reasons?

Incompatibility with Latest Version?

I tried to use aioresponses version 0.1.2 with aiohttp==1.1.6 and was unable to get it to work. I suspect that maybe I am using an older version of your library or too new a version of aiohttp. Are you able to provide some assistance?

You can see the relevant section of my code in the stack trace below. I'm more or less following the same format as in your examples but it seems like the mock response that is being returned by aiohttp is not being accepted by aiohttp/ClientResponse? Is there a newer version I should be using or something else the problem?

Thanks.

mocked = <aioresponses.core.aioresponses object at 0x10269fcf8>

    @aioresponses()
    def test_econtext_url_query(self, mocked):
        # Set up mocks
        session = aiohttp.ClientSession()
        mocked.post('https://api.econtext.com/v2/classify/url', status=200, body='{"test": 1}')

        # Call the method
        loop = asyncio.get_event_loop()
>       json_response = loop.run_until_complete(session.post('https://api.econtext.com/v2/classify/url'))

src/tests/unit/test_classification.py:335:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/base_events.py:337: in run_until_complete
    return future.result()
/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/futures.py:274: in result
    raise self._exception
/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/tasks.py:239: in _step
    result = coro.send(None)
/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/asyncio/coroutines.py:206: in coro
    res = func(*args, **kw)
../../.virtualenvs/cheesesteak/lib/python3.5/site-packages/aioresponses/core.py:154: in _request_mock
    response = self.match(method, url)
../../.virtualenvs/cheesesteak/lib/python3.5/site-packages/aioresponses/core.py:140: in match
    for i, r in enumerate(self._responses)
../../.virtualenvs/cheesesteak/lib/python3.5/site-packages/aioresponses/core.py:141: in <listcomp>
    if r.match(method, url)]
../../.virtualenvs/cheesesteak/lib/python3.5/site-packages/aioresponses/core.py:50: in build_response
    self.resp = ClientResponse(self.method, self.url)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <[AttributeError("'ClientResponse' object has no attribute '_url_obj'") raised in repr()] ClientResponse object at 0x10287aa90>
method = 'post', url = 'https://api.econtext.com/v2/classify/url'

    def __init__(self, method, url, *, writer=None, continue100=None,
                 timeout=5*60):
>       assert isinstance(url, URL)
E       AssertionError

../../.virtualenvs/cheesesteak/lib/python3.5/site-packages/aiohttp/client_reqrep.py:506: AssertionError```

Few words in documentation :)

Should not print aiohttp version on import

compat.py currently prints aiohttp_version on import. I assume this is some debug code that was left in.

body should be annotated as `Union[bytes, str]`

The body argument is currently annotated as str, but it works perfectly well with bytes for providing binary responses.

Provide an API for accessing request mock history

What's the problem this feature will solve?
This feature will allow developers to test that the right request payload are being sent

Describe the solution you'd like

mock.get('url', body="some json", status=201)
call_function_under_test()
# assert that the function actually made the call with the right payload
assert mock.request_history[0].payload == {"user_id":"1223"}

Add the HEAD verb

Is there any reason why we can't mock HEAD requests?

No name 'HttpProcessingError' in module 'aiohttp'

import asyncio
from aiohttp import ClientSession, HttpProcessingError
from aioresponses import aioresponses

@aioresponses()
def test_how_to_throw_an_exception(m, test_client):
    loop = asyncio.get_event_loop()
    session = ClientSession()
    m.get('http://example.com/api', exception=HttpProcessingError('test'))

Provide a way to mock out response cookies

Hi there!

First of all thanks for the great library and your efforts!

Problem

It seems that now there is no way to mock out response cookies. I initially tried to set cookies by adding "Set-Cookie" to response headers, however this also doesn't work (neither for headers argument of the aioresponses.add method, nor using callback).

In aiohttp.ClientResponse cookies are loaded from headers in its start method source

However aioresponses don't call this method and just set headers without loading cookies.

Workaround

My current workaround to set cookies looks like that:

from aioresponses import CallbackResult
from aiohttp import ClientResponse, hdrs


async def callback(url, **kwargs):
    class Response(ClientResponse):
        @property
        def _headers(self):
            return self.__headers

        @_headers.setter
        def _headers(self, value):
            self.__headers = value
            for hdr in value.getall(hdrs.SET_COOKIE, ()):
                self.cookies.load(hdr)

    return CallbackResult(
        response_class=Response,
        status=200,
        headers=headers,
        ...
    )

Proposal/Solution

So my proposal is to extract cookies from headers somewhere here. Alternatively, we can make additional cookies parameter for aioresponses method.

I personally think the first variant will be better as it covers most of the use cases, while adding a new argument to a method requires more work and had more ambiguity.

What do you think about it? I'll be happy to contribute with that

Missing git tag for 0.6.2 release

It would be nice to keep PyPI releases and git tags in sync :)

Support for aiohttp > 2.0

Make headers sent to mocked endpoint available as part of requests

@aioresponses
fn(mock):
    mock.post(url, status=200)
    method_that_calls_url_with_headers()

    print(mock.requests[('POST', URL(url))][0].kwargs['headers'])

Currently, only args, kwargs are available for a request to a particular (method, url) tuple.

It would be nice to be able to access what headers were sent to a mocked endpoint. Hopefully it's as simple as adding it as a field in kwargs

ResponseHandler - missing 1 required positional argument: 'loop'

Hi there. After an upgrade of aiohttp to version 3.5.0, I'm getting the following exception when running tests using aioresponses 0.5.0:

/usr/local/lib/python3.6/asyncio/coroutines.py:110: in __next__
    return self.gen.send(None)
/usr/local/lib/python3.6/site-packages/aioresponses/core.py:256: in _request_mock
    response = await self.match(method, url)
/usr/local/lib/python3.6/asyncio/coroutines.py:110: in __next__
    return self.gen.send(None)
/usr/local/lib/python3.6/site-packages/aioresponses/core.py:232: in match
    response = await matcher.build_response(url)
/usr/local/lib/python3.6/asyncio/coroutines.py:110: in __next__
    return self.gen.send(None)
/usr/local/lib/python3.6/site-packages/aioresponses/core.py:105: in build_response
    resp.content = stream_reader_factory()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def stream_reader_factory():
>       protocol = ResponseHandler()
E       TypeError: __init__() missing 1 required positional argument: 'loop'

/usr/local/lib/python3.6/site-packages/aioresponses/compat.py:23: TypeError

Looks like in aiottp 3.5.0, with this change aio-libs/aiohttp#3372 , the loop parameter is now required.

Thanks

Allow unregistered url to hit actual server

Ability to pass regexp instead of url

regexp might be usefull.

To think about:
what to do if url match, remove from mocked urls list or leave it???
Leaving it might break functionality with requests counting

aioresponses._request_mock() not compatible with aiohttp's ClientSession._request() signature

aioresponses monkey-patches ClientSession._request() in aiohttp in order to achieve it's mocking behavior. But in doing so, there's a minor incompatibility with that method's signature. Per the aiohttp client documentation, the url param can be either a str or a yarl.URL object. If a call being tested actually passes a yarl.URL object with aioresponses' patch in place, an error results:

    def parse_url(self, url: str) -> str:
        """Normalize url to make comparisons."""
>       _url = url.split('?')[0]
E       AttributeError: 'URL' object has no attribute 'split'
...lib/python3.5/site-packages/aioresponses/core.py:41: AttributeError

This should be quite trivial to fix in aioresponses._request_mock() (though reworking the unit tests in order to parametrize the URL fixture that's currently a hardcoded str would be a bit more of a rethink).

A closely related request: Since yarl.URL is the official native URL type in aiohttp, it would be nice if the same type could be supported in the UrlResponse constructor, as well, so it could be used when building the mock responses.

Considering new release

Hi @pnuckowski

Do you think it would be possible to release a new version of aioresponses with the content of master?

If not, can you tell me what would be the blocking point? Maybe I can be of some assistance?

Thanks again

raise_for_status argument is ignored

Hi,

The raise_for_status argument can be passed to a ClientSession since aiohttp v2.0.0 and to individual requests since v3.4.0. aioresponses seems to ignore it.

The following code should raise a ClientResponseError, but it doesn't:

loop = asyncio.get_event_loop()

# Start a session with the raise_for_status argument
session = aiohttp.ClientSession(raise_for_status=True)

with aioresponses() as m:
    m.get('http://example.com', status=500)
    loop.run_until_complete(session.get('http://example.com'))

If this is indeed a bug, I'm happy to submit a PR. I made changes to the code here.

aiohttp client doesn't follow redirects of mocked requests

Hello,

I'm using aioresponses along with aiohttp client and I'm seeing a problem regarding aiohttp.client.ClientSession not following redirects of a mocked request. Here are two tests that reproduces the problem:

from aiohttp.client import ClientSession
from aioresponses import aioresponses
from asynctest import TestCase


class AIOResponseRedirectTest(TestCase):
    async def test_aiohttp_follows_mocked_redirect(self):
        with aioresponses() as rsps:
            rsps.get(
                "http://10.1.1.1:8080/redirect",
                status=307,
                headers={"Location": "https://httpbin.org"},
            )
            client = ClientSession()
            response = await client.get(
                "http://10.1.1.1:8080/redirect", allow_redirects=True
            )
            self.assertEqual(200, response.status)

    async def test_aiohttp_follows_passthrough_redirect(self):
        with aioresponses(passthrough=["https://httpbin.org"]) as rsps:
            rsps.get(
                "http://10.1.1.1:8080/redirect",
                status=307,
                headers={"Location": "https://httpbin.org"},
            )
            client = ClientSession()
            response = await client.get(
                "https://httpbin.org/redirect-to?status_code=307&url=https://httpbin.org",
                allow_redirects=True,
            )
            self.assertEqual(200, response.status)

What I expected to see is both tests passing, but when I run this testclass, that's what I see:

pipenv run py.test -v tests/test_aiohttp_redirect_bug.py
Loading .env environment variables…
===================================================================================== test session starts =====================================================================================
platform linux -- Python 3.6.8, pytest-3.0.1, py-1.8.0, pluggy-0.3.1 -- /home/daltonmatos/.local/share/virtualenvs/asgard-api-YwspFh3W/bin/python3
cachedir: .cache
rootdir: /home/daltonmatos/src/asgard-api, inifile: 
plugins: cov-2.4.0
collected 2 items 

tests/test_aiohttp_redirect_bug.py::AIOResponseRedirectTest::test_aiohttp_follows_mocked_redirect FAILED
tests/test_aiohttp_redirect_bug.py::AIOResponseRedirectTest::test_aiohttp_follows_passthrough_redirect PASSED

========================================================================================== FAILURES ===========================================================================================
________________________________________________________________ AIOResponseRedirectTest.test_aiohttp_follows_mocked_redirect _________________________________________________________________

self = <tests.test_aiohttp_redirect_bug.AIOResponseRedirectTest testMethod=test_aiohttp_follows_mocked_redirect>

    async def test_aiohttp_follows_mocked_redirect(self):
        with aioresponses() as rsps:
            rsps.get(
                "http://10.1.1.1:8080/redirect",
                status=307,
                headers={"Location": "https://httpbin.org"},
            )
            client = ClientSession()
            response = await client.get(
                "http://10.1.1.1:8080/redirect", allow_redirects=True
            )
>           self.assertEqual(200, response.status)
E           AssertionError: 200 != 307

tests/test_aiohttp_redirect_bug.py:18: AssertionError
============================================================================= 1 failed, 1 passed in 1.18 seconds ==============================================================================

Shouldn't the failed test return a status=200 with the history attribute containing the redirected request? Is there anything that I'm doing wrong?

The versions are:

$ pipenv run pip freeze | grep -E "aiohttp|aioresp"
Loading .env environment variables…
aiohttp==3.4.4
aiohttp-cors==0.7.0
aioresponses==0.6.0

Python is 3.6.8:

$ pipenv run python -V
Loading .env environment variables…
Python 3.6.8

Thanks,

Python 3.9 vs. asynctest Dependency

Most aoiresponses unittests depend on the asynctest package.

The thing is that asynctest currently doesn't play with Python 3.9.

What are your plans regarding the asyntest dependency?

I'm asking because the aoiresponses package currently fails to build in Fedora Rawhide (which already switched to a Python 3.9 preview release).

Websocket support

Does this package also support websocket connection mocking ?

Support for throwing an exception

Does aioresponses support throwing an exception, similar to what https://github.com/getsentry/responses provides?

For example something like:

class MyError(Exception):
    pass

with aioresponses() as m:
    m.get('http://test.example.com', body=MyError())

Together with status_code and body mock out also headers

TypeError: name must be a str, not a MagicMock

Hello,

I have a Strange issue when use aioresponses mocking. I can't reproduce with example script and i can't show the concerned code. So i just hope error can suggest you something.

>       with aioresponses() as mock:

tests/xxxxxxxxxxxxx.py:570: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv3.7/lib/python3.7/site-packages/aioresponses/core.py:203: in __enter__
    self.start()
venv3.7/lib/python3.7/site-packages/aioresponses/core.py:238: in start
    self.patcher.start()
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/unittest/mock.py:1386: in start
    result = self.__enter__()
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/unittest/mock.py:1336: in __enter__
    _name=self.attribute, **kwargs)
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/unittest/mock.py:2238: in create_autospec
    **kwargs)
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/unittest/mock.py:1842: in __init__
    _safe_super(MagicMixin, self).__init__(*args, **kw)
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/unittest/mock.py:936: in __init__
    _spec_state, _new_name, _new_parent, **kwargs
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/unittest/mock.py:394: in __init__
    self._mock_add_spec(spec, spec_set, _spec_as_instance, _eat_self)
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/unittest/mock.py:451: in _mock_add_spec
    _spec_as_instance, _eat_self)
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/unittest/mock.py:90: in _get_signature_object
    return func, inspect.signature(sig_func)
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/inspect.py:3075: in signature
    return Signature.from_callable(obj, follow_wrapped=follow_wrapped)
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/inspect.py:2825: in from_callable
    follow_wrapper_chains=follow_wrapped)
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/inspect.py:2282: in _signature_from_callable
    return _signature_from_function(sigcls, obj)
../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/inspect.py:2166: in _signature_from_function
    kind=_VAR_POSITIONAL))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <[AttributeError("_name") raised in repr()] Parameter object at 0x7f81827e0120>
name = <MagicMock name='_request.__code__.co_varnames.__getitem__()' id='140194217530704'>
kind = <_ParameterKind.VAR_POSITIONAL: 2>

    def __init__(self, name, kind, *, default=_empty, annotation=_empty):
        try:
            self._kind = _ParameterKind(kind)
        except ValueError:
            raise ValueError(f'value {kind!r} is not a valid Parameter.kind')
        if default is not _empty:
            if self._kind in (_VAR_POSITIONAL, _VAR_KEYWORD):
                msg = '{} parameters cannot have default values'
                msg = msg.format(_get_paramkind_descr(self._kind))
                raise ValueError(msg)
        self._default = default
        self._annotation = annotation
    
        if name is _empty:
            raise ValueError('name is a required attribute for Parameter')
    
        if not isinstance(name, str):
            msg = 'name must be a str, not a {}'.format(type(name).__name__)
>           raise TypeError(msg)
E           TypeError: name must be a str, not a MagicMock

../../.pythonz/pythons/CPython-3.7.1/lib/python3.7/inspect.py:2483: TypeError

You can close issue if error say Nothing to you. I go back here if solution found ...

Mock timeout?

Thanks for the awesome library. Installed, used it, loved it. Combined with asynctest it's very pleasant.

Just in case you're interested with feature ideas, I think mocking request timeouts would fall in the scope. In order to test stuff like that:

# Code example
async with session.get(url, headers=headers, timeout=timeout) as response:
    return await response.json()

or

with async_timeout.timeout(timeout):
    async with session.get(url, headers=headers, timeout=None) as response:
        return await response.json()

It could be by specifying a delay parameter to aioresponses.add(...), that would then rely on await asyncio.sleep() for example.

What do you think?

Update for aiohttp v3.0 release

It seems to have been broken since aio-libs/aiohttp#2556, as I get the following error in my test codes using aioresponses:

>       self.resp.content = StreamReader()
E       TypeError: __init__() missing 1 required positional argument: 'protocol'

Document how to make it work with aiohttp's v3 test client

I'm not sure how to go about this issue, maybe a section in the docs is enough, or if this is a real bug, or I am the one doing something wrong... So:

In my test suite, aioresponses is also blocking requests to the test server created when the aiohttp_client fixture is used (I'm testing my web server which in turn the request makes a request to an upstream server):

async def test_upstream_request(aiohttp_client):
    with aioresponses() as m:
        m.get('http://example.com', status=200, body='test')
        client = await aiohttp_client(my_web_app_instance)
        response = await client.get('/endpoint')  # Raises connection refused for '127.0.0.1:<random port number>'

My current workaround is to call aioresponses like:

with aioresponses(passthrough=['http://127.0.0.1:']) as m:
   # ...

Which is currently fine for me because the mock is abstracted away in one fixture, I don't have to repeat the passthrough for every test.

So maybe this should be documented for the newcomer, or perhaps aioresponses should automatically find out the test server's URL and implicitly add it to its internal passthrough list.

Versions:
aiohttp (3.0.5)
aioresponses (0.4.0)
pytest-aiohttp (0.3.0)

Can i used aioresponse for server side in test code?

In my case, use aiohttp in server side..

requests_mock using like this in test case, It's working on server side mock.

with requests_mock.Mocker() as mock:
   mock.get('/', text='text')

but aioresponses is not working on server side. It's working only test case..
can i used aioresponses in test case for mock server side request??

my server app is falcon and I write test case like this.

def test_main(client):
    loop = asyncio.get_event_loop()
    session = aiohttp.ClientSession()
    with aioresponses() as m:
        m.get('http://test.example.com', payload=dict(foo='bar'))

        resp = loop.run_until_complete(session.get('http://test.example.com'))
        data = loop.run_until_complete(resp.json())

        assert dict(foo='bar') == data

        resp = client.simulate_get('/')

        assert resp.status == falcon.HTTP_OK

    session.close()

In my server code

import asyncio

import aiohttp
import falcon


class MainResource:
    def on_get(self, req, resp):
        data = {
            'abcd': 'abcd',
        }

        resp.media = data

        session = aiohttp.ClientSession()
        loop = asyncio.get_event_loop()
        resp = loop.run_until_complete(session.get('http://test.example.com'))


api = falcon.API()
api.add_route('/', MainResource())

• macOS Catalnia 10.15.3
• aiohttp == 3.6.2
• aioreponses == 0.6.3

Version string incorrect

The version string in all tags after 0.4.1, for example 0.6.0 (the latest) is 0.4.1. https://github.com/pnuckowski/aioresponses/blob/0.6.0/aioresponses/__init__.py#L4

This caused a bit of head-scratching for me when trying to troubleshoot – perhaps it could be fixed for future tags? I can open a PR if there is interest in cutting a 0.6.1 that fixes this issue (alternately, I'm missing something).

race condition in match method

I have observed a bug, due to a race condition in aioresponses.core.aioresponses.match: In https://github.com/pnuckowski/aioresponses/blob/master/aioresponses/core.py#L342 the match object gets removed from the respective list. In my case, the index calculated in the loop above (https://github.com/pnuckowski/aioresponses/blob/master/aioresponses/core.py#L332-L337) is no longer valid, because other tasks seem to have modified the list concurrently. Therefore I observe an IndexError: list assignment index out of range.

I discovered this behaviour by setting a callback, which uses asyncio.sleep:

async def callback(url, **kwargs):
        await asyncio.sleep(0.1)
        return CallbackResult(body='test')

def test():
     .....
     with aioresponses() as mocked:
          for i range(20):
              mocked.get(f'{http://example.org/id-{i}',  callback=callback)
     .....

The client code calls the URLs concurrently.

Local environment: aioresponses 0.6.4, Ubuntu 18.04, Python 3.6.9

Introduce callbacks?

Pypi has version 0.4.0 but repo has 0.1.4?

As near as I can tell pypi has 0.4.0 but this repo only has 0.1.4, I'm considering using this for a @mozilla-releng project and would benefit from knowing the repo itself is up to date.

No reason phrase with 404 status incompatible with aiohttp raise_for_status() in v3.5

aioresponses() doesn't seem to provide a reason phrase with a 404 status

I don't think this used to cause problems (my test passes with aiohttp==3.4.4) but as of this aiohttp PR the aiohttp.response.raise_for_status() method includes an assertion that fails if reason is None.

Expected behavior

I expect the following test (pseudo)code to pass:

@pytest.mark.asyncio
async def test_error_when_response_has_error_status_code():
    with aioresponses() as mock_server:
        domain_specific_variable = "nonexistent"
        mock_server.get("localhost://url", status=404, body="No recipe found")
        with pytest.raises(CustomException):
            await utils.get_specific_response(domain_specific_variable)

when I have the method defined like so utils.py:

async def get_specific_response(rdomain_specific_variable: str):
    url = _create_endpoint(recipe_name, recipe_params)
    async with aiohttp.request("GET", url) as resp:
        try:
            routes = await resp.json()
        except aiohttp.client_exceptions.ClientResponseError as e:
            raise CustomException(f"Error retrieving routes from '{url}'") from e

Actual behavior

The test fails with the following error:

    def raise_for_status(self) -> None:
        if 400 <= self.status:
>           assert self.reason  # always not None for started response
E           AssertionError

Environment

aiohttp==3.5.0
aioresponses==0.5.1

TypeError: can't pickle async_generator objects

My tests started fall down after I pull the new version

Traceback (most recent call last):
  File "/Users/project/app/views.py", line 659, in _post
    responses = await asyncio.gather(*tasks)
  File "/Users/project/utils/ncapi/client.py", line 314, in upload_verification_document
    return await self._action("uploadVerificationDocument", "post", data, files=True, single_file=True)
  File "/Users/project/utils/ncapi/client.py", line 181, in _action
    res = await request(self.url(action_name), data=result)
  File "/Users/env-2fy-KEPT/lib/python3.7/site-packages/aioresponses/core.py", line 338, in _request_mock
    self.requests[key].append(RequestCall(args, copy.deepcopy(kwargs)))
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 180, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 280, in _reconstruct
    state = deepcopy(state, memo)
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 240, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 215, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 150, in deepcopy
    y = copier(x, memo)
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 220, in _deepcopy_tuple
    y = [deepcopy(a, memo) for a in x]
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 220, in <listcomp>
    y = [deepcopy(a, memo) for a in x]
  File "/Users/env-2fy-KEPT/lib/python3.7/copy.py", line 169, in deepcopy
    rv = reductor(4)
TypeError: can't pickle async_generator objects
Traceback (most recent call last):
  File "/Users/project/app/views.py", line 659, in _post
    responses = await asyncio.gather(*tasks)
TypeError: can't pickle async_generator objects