kevin1024/vcrpy

Automatically mock your HTTP interactions to simplify and speed up testing

VCR.py 📼

---

This is a Python version of Ruby's VCR library.

Source code

https://github.com/kevin1024/vcrpy

Documentation

https://vcrpy.readthedocs.io/

Rationale

VCR.py simplifies and speeds up tests that make HTTP requests. The first time you run code that is inside a VCR.py context manager or decorated function, VCR.py records all HTTP interactions that take place through the libraries it supports and serializes and writes them to a flat file (in yaml format by default). This flat file is called a cassette. When the relevant piece of code is executed again, VCR.py will read the serialized requests and responses from the aforementioned cassette file, and intercept any HTTP requests that it recognizes from the original test run and return the responses that corresponded to those requests. This means that the requests will not actually result in HTTP traffic, which confers several benefits including:

• The ability to work offline
• Completely deterministic tests
• Increased test execution speed

If the server you are testing against ever changes its API, all you need to do is delete your existing cassette files, and run your tests again. VCR.py will detect the absence of a cassette file and once again record all HTTP interactions, which will update them to correspond to the new API.

Usage with Pytest

There is a library to provide some pytest fixtures called pytest-recording https://github.com/kiwicom/pytest-recording

License

This library uses the MIT license. See LICENSE.txt for more details

vcrpy's Issues

Header checking is too brittle

Should use something like webob.headers.ResponseHeaders to normalize headers

global name `matcher_name` is not defined

A minimal case that would reproduce this problem:

def awesome_matcher(r1, r2):
    return True

my_vcr = vcr.VCR()
my_vcr.register_matcher('awesome', awesome_matcher)

def test_this_works():
    with vcr.use_cassette('recording.yaml', match_on=['awesome']) as cass:
        pass

Produces:

     Failure/Error: global name 'matcher_name' is not defined

     Traceback:
     ...
     File "/usr/local/lib/python2.7/site-packages/vcr/__init__.py", line 8, in use_cassette
         return default_vcr.use_cassette(path, **kwargs)
     File "/usr/local/lib/python2.7/site-packages/vcr/config.py", line 62, in use_cassette
         "match_on": self._get_matchers(matcher_names),
     File "/usr/local/lib/python2.7/site-packages/vcr/config.py", line 44, in _get_matchers
         matcher_name

urllib compatibility

urllib (though deprecated) uses all kinds of weird httplib compat classes that aren't stubbed properly by vcr. I would like to support this someday.

Allow filtering the data saved

Currently, vcrpy saves all of the data from a request. However, sometimes there may be part of a request that you wouldn't want recorded, such as authentication data. It would be nice if vcrpy allowed a way to scrub data from a request before it is saved, similar to the way httreplay does it.

doesn't support for latest requests package

response = urllib2.urlopen('http://www.iana.org/domains/reserved').read()
#response = requests.get('http://www.iana.org/domains/reserved')

In the readme, it states it support requests, while I can't make it work using sample above.

I use requests 1.2.3

BTW: the url http://www.iana.org/domains/example/ is not valid any more

Header case-sensitivity?

cookie = resp.getheader('Set-Cookie') versus cookie = resp.getheader('set-cookie')

My HTTP client library under python 2 used getheader('set-cookie') this returns None when using VCRpy. Practical testing seems to indicate that in production the getheader method is case insensitive.

Is this a bug?

Update stored Response to match stored Request format

Since we are updating the format of cassettes in version 1.0.0, I would like to update the format of the recorded responses to match the format of the recorded requests.

publish to pypi

I'd love to use this but it is much easier to do if published to pypi, could you do this?

Provide basic logging and DEBUG, INFO messages

Not sure if other Pythonistas do this, but I work like this these days:

• write a test (possibly using VCRpy!) with tactical sprinkle of logger.debug() bits of info
• run it
• it fails; I see the useful debug trace leading up to failure (which may be an exception, etc.)
• tweak the test / the source code
• re-run the test... etc.
• rinse and repeat

I expect libraries and tools I use to provide me useful logging debug information. When I turn the log level to DEBUG I should see important decisions that VCRpy has made, such as

1. when a request is detected
2. how exactly vcrpy decided to supply the response:
   What fields of the request did it try to match?
   Could it find any matches in the known cassettes?
   If so, which one? Which exact request?
   If not, the request hits the network. How long did the request take to complete?

This is really important especially now that vcrpy has become much more powerful with customisable request matching. There are many more ways in which it could behave not quite as the user anticipated. e.g. I know that two distinct requests were sent and both were supposed to hit the network. I found that the request matching was too broad. I want debugging to tell me this so that I can make informed decisions about how to improve my test.

I imagine a few lines like this at strategic places in vcrpy code:

logger = logging.getLogger('vcrpy')

# in vcrpy code
logger.debug("Found matching request {1} in cassette {0}. Playing back...".format(cassette, request))

is allow_http_connections_when_no_cassette supported?

The ruby version has this useful config allow_http_connections_when_no_cassette , cant find this in vcrpy. Is there plan to support this or am I missing something

Incorrect example in the README

@vcr.use_cassette('fixtures/vcr_cassettes/synopsis.yaml'):
def test_iana():
    response = urllib2.urlopen('http://www.iana.org/domains/reserved').read()
    assert 'Example domains' in response

There's a colon at the end of the annotation. :)

Save all requests, even if they are for the same URL

Refs #15

WindowsError: [Error 183] Cannot create a file when that file already exists.

Problems in "all" mode: Cannot create a file when that file already exists.

============================= ERRORS =============================
inner_func
func(_args, *_kw)
File "D:\Python27\lib\site-packages\vcr\cassette.py", line 141, in exit
self._save()
File "D:\Python27\lib\site-packages\vcr\cassette.py", line 103, in _save
serializer=self._serializer
File "D:\Python27\lib\site-packages\vcr\persist.py", line 11, in save_cassette
FilesystemPersister.write(cassette_path, data)
File "D:\Python27\lib\site-packages\vcr\persisters\filesystem.py", line 23, in write
cls._secure_write(cassette_path, data)
File "D:\Python27\lib\site-packages\vcr\persisters\filesystem.py", line 16, in _secure_write
os.rename(name, path)
WindowsError: [Error 183]

VCR.py needs to be multipart-boundary agnostic

Let's say I created a nice request bin at http://requestb.in/179gbs61. I run the following tests two times:

• first time with an internet connection available (Wifi connected)
• second time with no internet connection (Wifi disconnected)

I notice for both cases that the relevant VCR files (fixtures/test_vcr/foobar.yaml', fixtures/test_vcr/foobar-file-upload.yaml) are created correctly after the first run.

        # This test SUCCEEDS with no connection
        def test_vcr_works_with_requests():
            with vcr.use_cassette('fixtures/test_vcr/foobar.yaml'):
                r = requests.post("http://requestb.in/179gbs61")

        # This test FAILS with no connection
        def test_vcr_works_with_requests_with_file_upload():
            with vcr.use_cassette('fixtures/test_vcr/foobar-file-upload.yaml'):
                with open (os.path.abspath('lolcat.jpg')) as f:
                    r = requests.post("http://requestb.in/179gbs61", files={'photo':f})

The test failure of test_vcr_works_with_requests_with_file_upload on the second run looks like this:

Failure/Error: HTTPConnectionPool(host='requestb.in', port=80): Max retries exceeded with url: /179gbs61 (Caused by <class 'socket.gaierror'>: [Errno 8] nodename nor servname provided, or not known)

     Traceback:
     ...
     File "spec/upload_spec.py", line 56, in test_vcr_works_with_requests_with_file_upload
         r = requests.post("http://requestb.in/179gbs61", files={'photo':f})
     File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 88, in post
         return request('post', url, data=data, **kwargs)
     File "/usr/local/lib/python2.7/site-packages/requests/api.py", line 44, in request
         return session.request(method=method, url=url, **kwargs)
     File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 335, in request
         resp = self.send(prep, **send_kwargs)
     File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 438, in send
         r = adapter.send(request, **kwargs)
     File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 327, in send
         raise ConnectionError(e)

This happens because of a randomly generated multipart boundary.

VCR's logic to retrieve a response depends on a hash lookup on the request:

# stubs/__init__.py
    def getresponse(self, _=False):
        '''Retrieve a the response'''
        if self._vcr_request in self.cassette:
            # (1) RESPONSE FOUND IN CASSETTE: CACHE HIT
            response = self.cassette.response_of(self._vcr_request)
            # Alert the cassette to the fact that we've served another
            # response for the provided requests
            self.cassette.mark_played(self._vcr_request)
            return VCRHTTPResponse(response)
        else:
            # (2) RESPONSE NOT FOUND IN CASSETTE: CACHE MISS; perform actual request
            ... # ConnectionError naturally occurs here, due to lack of connection

The ConnectionError occurs because we've got a cache miss (2) even though we were expecting a cache hit (1).

The cache miss is because every time a multipart request such as requests.post("http://requestb.in/179gbs61", files={'photo':f}) is performed, a random multipart boundary string is generated by requests and included into the body request. This makes every request look different from the previous. For example, this request was recorded the first time:

- request: !!python/object:vcr.request.Request
    body: "--bfc103a89d184e4e8d72aa787102432e\r\nContent-Disposition: form-data; name=\"file\";
      filename=\"assertions.py\"\r\nContent-Type: text/x-python\r\n\r\ndef assert_cassette_empty(cass):\n
      \   assert len(cass) == 0\n    assert cass.play_count == 0\n\n\ndef assert_cassette_has_one_response(cass):\n
      \   assert len(cass) == 1\n    assert cass.play_count == 1\n\r\n--bfc103a89d184e4e8d72aa787102432e--\r\n"
    headers: !!python/object/apply:__builtin__.frozenset
    - - !!python/tuple [Accept-Encoding, 'gzip, deflate, compress']
      - !!python/tuple [Accept, '*/*']
      - !!python/tuple [User-Agent, python-requests/1.2.3 CPython/2.7.5 Darwin/12.4.0]
      - !!python/tuple [Content-Type, multipart/form-data; boundary=bfc103a89d184e4e8d72aa787102432e]
      - !!python/tuple [Content-Length, !!python/unicode 373]
    host: requestb.in
    method: POST
    path: /179gbs61
    port: 80
    protocol: http

Note boundary=bfc103a89d184e4e8d72aa787102432e.

Run the test again and we get a completely different boundary string, like boundary=d2330e7bd84f4384b70e722347030406.

WORKAROUND: Use a constant as boundary.

SOLUTION: Cleverly remove boundary from body (in the various places it appears...) before calculating hash for the request.

Would it be possible to update the latest version to pypi?

Thanks!

Ignore some requests?

Imagine this code:

# app code
def handle_request():
    requests.get('http://example.com')  # I want to record this

def test_app():
    # Imagine we're doing a live server test.
    # The app is running at localhost, and that handler is mounted at the root.
    with vcr.use_cassette('cassette.yaml'):
        server_response = requests.get('http://localhost/')  # but, this gets recorded too

I'm working around this right now by using pycurl - which VCR.py can't mock - when exercising the app endpoint.

Is there a better way of ignoring some requests? The patch for requests looks pretty deep, and I can't put the vcr context manager inside app code.

VCR.py should be OAuth-timestamp / nonce agnostic

This issue is similar to #28 in spirit: that one was about a changing boundary string, this one is about data within the request generally that changes each time we invoke it: especially request parameters.

Nonces in particular are designed exactly to prevent replay attacks from occurring so unsurprisingly this is potentially a problem for VCR.py. They're usually just passed to the server as a parameter.

Using the Flickr API as an example. Three request parameters necessarily change with every request:

• oauth_signature
• oauth_timestamp
• oauth_nonce

These fields may be found in the request body (vcr.Request.body) (in the case of multipart/formdata or query string (vcr.Request.path). Because both of these are included in the hash of the Request, these changing values cause cache misses and VCR fails retrieve pre-canned responses from cassettes.

Just to give you an idea, requests look a bit like this:

Accept-Encoding: gzip, deflate, compress
User-Agent: python-requests/1.2.3 CPython/2.7.5 Darwin/12.4.0
Content-Length: 40093
Accept: */*
Connection: close
Content-Type: multipart/form-data; boundary=5094bb8eab0e4eb3a0188ee90f9a2f4c
Host: SOME_URL

--5094bb8eab0e4eb3a0188ee90f9a2f4c
Content-Disposition: form-data; name="oauth_nonce"

36636389
--5094bb8eab0e4eb3a0188ee90f9a2f4c
Content-Disposition: form-data; name="oauth_timestamp"

1379361625
...
--5094bb8eab0e4eb3a0188ee90f9a2f4c
Content-Disposition: form-data; name="oauth_signature"

P/0/vYDCD/ztPNUWF07wDcRqODI=

Solution:

The only way for VCR to still work in the light of these pesky authenticated requests is to omit these fields when calculating the hash or identity of the request. The hash or identity should be agnostic to a user-defined set of changing fields such as { oauth_signature, oauth_timestamp, oauth_nonce }.

This should be configurable directly from the VCR.py API in the with cassette() block or something.

It looks like VCR will have to parse the request then purposefully omit the request parameters (whether in query string or in body) before forming a new signature-less request to hash with.

VCRpy getheader is case sensitive

cookie = resp.getheader('Set-Cookie') versus cookie = resp.getheader('set-cookie')

My HTTP client library under python 2 used getheader('set-cookie') this returns None when using VCRpy. Practical testing indicates that in production the getheader method is case insensitive.

Note rfc2109 documentation....

Attributes (names) (attr) are case-insensitive. White space is
permitted between tokens. Note that while the above syntax
description shows value as optional, most attrs require them.

This means values such as set-cookie and Set-Cookie should be treated the same.
http/client.py#L755
headers.py#L90

It looks like at least one version of getheader may be case insensitive.

See: basic_connection.yaml#L55 and commit: fix cookie handler in test

I've had to alter the way my library handles HTTP cookies since the getheader call may be used with a response that has either set-cookie (as recorded from the vSphere server) or it may see Set-Cookie as is normally posted by most other servers.

As far as I can tell the following test should pass but doesn't:

def test_case_insensitivity(tmpdir):
    testfile = str(tmpdir.join('case_insensitivity.yml'))
    with vcr.use_cassette(testfile):
        conn = httplib.HTTPConnection('httpbin.org')
        conn.request('GET', "/cookies/set?k1=v1&k2=v2")
        r1 = conn.getresponse()
        cookie_data1 = r1.getheader('set-cookie')
        conn.request('GET', "/cookies/set?k1=v1&k2=v2")
        r2 = conn.getresponse()
        cookie_data2 = r2.getheader('Set-Cookie')
        assert cookie_data1 == cookie_data2

Tests failing with latest version of requests?

Running tox on master has a few failing targets, the ones that pull down the current version of requests, which is 2.2.1 (released 2014-01-23).

  py26: commands succeeded
  py27: commands succeeded
  py33: commands succeeded
  pypy: commands succeeded
ERROR:   py26requests: commands failed
ERROR:   py27requests: commands failed
ERROR:   pypyrequests: commands failed
  py26oldrequests: commands succeeded
  py27oldrequests: commands succeeded
  pypyoldrequests: commands succeeded

Each of the failing targets fails with the following:

    def __init__(self, host, port=None, key_file=None, cert_file=None,
                 strict=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
                 source_address=None):
        HTTPConnection.__init__(self, host, port, strict, timeout,
>                               source_address)
E       TypeError: unbound method __init__() must be called with HTTPConnection instance as first argument (got HTTPSConnection instance instead)

/usr/local/Cellar/pypy/2.1.0/lib-python/2.7/httplib.py:1153: TypeError

I think the latest Travis CI builds on master will fail too once there is a commit, as I am seeing the same error in the Travis CI build for PR #58 from @asmundg.

Setting to prevent any network requests in the cassette context

Currently it's possible to change a test/functionality and cause the tests to go slow/succeed when they should not.

There should be an option to prevent any network requests and protect the cassette from writes.

http://rubydoc.info:8080/github/vcr/vcr/master/VCR/Cassette#VALID_RECORD_MODES-constant

http://en.wikipedia.org/wiki/Compact_Cassette#Write-protection

Python 3 support

I am running a virtualenv using Python 3.3.2
I get this error when trying to import vcr:

path_to_my_project/lib/python3.3/site-packages/vcrpy-0.3.3-py3.3.egg/vcr/init.py", line 1, in
ImportError: No module named 'config'

:(

Monkey patch socket.

Love the API you have (very similar to Ruby's VRC) and very nice code!

Another project I was toying around with (recently on HN, I believe I saw you comment) does something similar: https://github.com/gabrielfalcao/HTTPretty/

I've opened an issue over there with the idea of adding a record and playback context manager, almost exactly like your implementation. But, of course, patching at the lib level (httplib or urllib or whatever) is almost guaranteed to leave some higher level libraries out. Instead, maybe we could monkey patch sockit and ssl?

The HTTPretty implementation is fairly complex but it looks doable.

This is a pretty big pain point for us, so its definitely something we want to do. Not sure whether to invest the time in your lib or theirs!

Loading urls that do a 301/302 redirect don't work

Add custom request matching

Failure/Error: [Errno 32] Broken Pipe if using httplib

Thanks for writing this tool! Python VCR really is quite awesome.

I'm requesting for httplib support because I'm using python-flickr-api at the moment to, say, upload photos. It uses both urllib2 and httplib libraries.

Without httplib support, I can successfully record the requests, but the playback fails when the canned answers are replayed to my test. It currently looks really simple:

        def test_photo_upload():
            with vcr.use_cassette('upload-a-photo-png.yaml'):
                flickr_auth.prepare_flickr_api()
                file_path = os.path.abspath('photo.png')
                photo = flickr.upload(photo_file=file_path, title="TEST") #  this line blows up

It looks like flickr.upload() just performs a POST request by calling self.sock.sendall(data), a very standard method of httplib. The exact error looks like this:

Failure/Error: [Errno 32] Broken pipe

Traceback:
     File "/usr/local/lib/python2.7/site-packages/flickr_api/upload.py", line 92, in upload
         r = post(UPLOAD_URL,auth.AUTH_HANDLER,args,photo_file)
     File "/usr/local/lib/python2.7/site-packages/flickr_api/upload.py", line 52, in post
         r = multipart.posturl(url,fields,files)
     File "/usr/local/lib/python2.7/site-packages/flickr_api/multipart.py", line 19, in posturl
         return post_multipart(urlparts[1], urlparts[2], fields,files)
     File "/usr/local/lib/python2.7/site-packages/flickr_api/multipart.py", line 33, in post_multipart
         h.send(body)
     File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 805, in send
         self.sock.sendall(data)
     File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 224, in meth
         return getattr(self._sock,name)(*args)

How could I work around this problem meanwhile?

See also my SO question. I'm stumped!

When `record_mode` is set to 'once' and `ignore_localhost` set to True, requests to localhost raise `CannotOverwriteExistingCassetteException`

I was playing with the new ignore_localhost option when configuring vcr (btw thanks for implementing this) when I ran into a problem where CannotOverwriteExistingCassetteException was being thrown when localhost requests were being processed.

It seems that in stubs/__init__.py raises this error whenever a request is not in the cassette and record_mode is set to "once". This is expected behavior in most situations, except when requests should be ignored. If a request is in the ignore_list, then it shouldn't matter what record_mode is set to.

To get around this error, I've set record_mode equal to "new_episodes".

Using VCR.py causes 404 for Flickr upload specifically :S

More problems with posting. This is very strange behaviour:

        def should_talk_to_Flickr():
            r = requests.post("http://api.flickr.com/services/upload/")
            print(r.status_code) # returns with HTTP 200, body contains XML complaining about lack of api_key parameter. Fine. This is normal behaviour

Instead of transparently recording the requests that occur, VCR has completely hi-jacked the call somehow:

        def should_still_talk_to_Flickr():
            with vcr.use_cassette('fixtures/flickr/reach.yaml'): # this file doesn't exist yet
                r = requests.post("http://api.flickr.com/services/upload/")
                print(r.status_code) # returns with HTTP 404 with empty body

AttributeError when recording HTTPS request

Not sure if this is a problem with the HTTPS implementation in vcrpy or the boto implementation specifically, but when recording a s3 request made via boto, VCR throws the following error:

Traceback (most recent call last):
  File "repro.py", line 11, in <module>
    k.set_contents_from_string('hello world i am a string')
  File "/python2.7/site-packages/boto/s3/key.py", line 1379, in set_contents_from_string
    encrypt_key=encrypt_key)
  File "/python2.7/site-packages/boto/s3/key.py", line 1246, in set_contents_from_file
    chunked_transfer=chunked_transfer, size=size)
  File "/python2.7/site-packages/boto/s3/key.py", line 725, in send_file
    chunked_transfer=chunked_transfer, size=size)
  File "/python2.7/site-packages/boto/s3/key.py", line 914, in _send_file_internal
    query_args=query_args
  File "/python2.7/site-packages/boto/s3/connection.py", line 633, in make_request
    retry_handler=retry_handler
  File "/python2.7/site-packages/boto/connection.py", line 1040, in make_request
    retry_handler=retry_handler)
  File "/python2.7/site-packages/boto/connection.py", line 913, in _mexe
    request.body, request.headers)
  File "/python2.7/site-packages/boto/s3/key.py", line 815, in sender
    http_conn.send(chunk)
  File "/python2.7/site-packages/vcr/stubs/__init__.py", line 101, in send
    self._vcr_request.body = (self._vcr_request.body or '') + data
AttributeError: VCRHTTPSConnection instance has no attribute '_vcr_request'

Assuming you have a AWS_CREDENTIAL_FILE, you can reproduce it with this code:

from boto.s3.connection import S3Connection
from boto.s3.key import Key
import vcr

s3_conn = S3Connection()
s3_bucket = s3_conn.get_bucket('some-bucket') # a bucket you can access

with vcr.use_cassette('test.yml'):
    k = Key(s3_bucket)
    k.key = 'test.txt'
    k.set_contents_from_string('hello world i am a string')

If I have time later today I will try and narrow this down to something that doesn't require boto, and if I can, then I should be able to make a fix too. Haven't really dug into vcrpy's source yet so any pointers are welcome.

Add configuration

Possible configuration options (mostly stolen from Ruby's VCR)

• Raise exception if HTTP request is attempted to outside world
• Custom request matching
• Save different requests for same URL (refs #18)

Feature Request: Register matchers for decorator

Using the decorator @vcr.use_cassette means I can't use custom matchers. I'd like to be able to register custom matchers for the decorators to use ... or to talk to the default_vcr object in a more transparent manner.

So far, I've included the following at the top of a file I want to have a custom matcher for use with the decorator... things look like this...

def full_matcher(r1, r2):
    if (r1.uri == r2.uri and
            r1.body == r2.body and
            r1.headers == r2.headers):
        return True
    raise SystemError('no match')


vcr.default_vcr.register_matcher('full_matcher', full_matcher)


class ConnectionTests(unittest.TestCase):

    @vcr.use_cassette('basic_connection.yaml',
                      cassette_library_dir=fixtures_path, record_mode='none',
                      match_on=['full_matcher'])
    def test_basic_connection(self):
         pass

compat.collections getting loaded in Python 3 in some cases

Attempting to set up python 3 testing for our library I ran into this issue...

Installing collected packages: vcrpy, contextdecorator
  Running setup.py install for vcrpy
      File "/Users/hartsocks/.venv/pyvmomi-tools-py3/lib/python3.4/site-packages/vcr/compat/counter.py", line 192
        print doctest.testmod()
                    ^
    SyntaxError: invalid syntax


  Running setup.py install for contextdecorator

HTTPS support

The library appears to fall down when HTTPS connections are used. A simple test like this...

        with vcr.use_cassette('fixtures/connect.yaml') as cass:
            sock = urllib.urlopen('http://www.google.com')
            print sock.read()

... works beautifully ... but when 'https' is used ...

        with vcr.use_cassette('fixtures/connect.yaml') as cass:
            sock = urllib.urlopen('https://www.google.com')
            print sock.read()

... the test produces the following trace ...

<snip/>
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 87, in urlopen
    return opener.open(url)
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 208, in open
    return getattr(self, name)(url)
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib.py", line 425, in open_https
    cert_file=self.cert_file)
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1198, in __init__
    cert_file, strict))
  File "/usr/local/Cellar/python/2.7.6/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1164, in __init__
    source_address)
TypeError: unbound method __init__() must be called with VCRHTTPConnection instance as first argument (got HTTPSConnection instance instead)

Can we get HTTPS support?

Update for requests 2.x

Looks like there are some major API changes in the new version of urllib3 shipped with requests. Requests 2.x support is a priority for me right now.

Add tests for urllib3

I'm testing requests, which tests urllib3 indirectly, but I need to add some tests which will test urllib3 directly, since the modules are monkeypatched at a different path.

how to handle different response for same request ? like status will be changed

// it is more of question

The case or the function has "wait until it is ready" method, the status will be changed with the same request.

For example, I poll the system status using REST API, so the poll request is the same, it will return "not ready", While after several seconds, it will return "ready".

How can I handle this using vcr ? not possible or possible and there is pattern for it.

Request matching based on parameters (let user define exclusion list as well)

Most of the time it is sufficient to discriminate between requests with only the URL.

However, APIs like Flickr have largely the same API endpoint. The actual method to be invoked is defined by a parameter method=.... The parameter list lives in the body of the request.

Arguably this is bad API design, still, to discriminate between method calls in this case matching on [url, method, body] (i.e. 'body' is added to the default request matcher).

However due to #30 this is not sufficient. VCRpy will become confused because every request looks different and old requests will never be retrieved.

The only true solution to this is to parse out parameter lists and provide a way to filter out the ones that are supposed to change all the time; and not to match on those.

JSON serializer can't serialize binary data: UnicodeDecodeError: 'utf8' error

When I try to upload or download a .zip file, i see the above error, If i make a plain requests call it works , but with vcr I see the above error
Guess this is happening while serialising the content , with this error the cassette never gets created

Here is the stack -

File "/Library/Python/2.7/site-packages/vcr/cassette.py", line 127, in exit
self._save()
File "/Library/Python/2.7/site-packages/vcr/cassette.py", line 88, in _save
serializer=self._serializer
File "/Library/Python/2.7/site-packages/vcr/persist.py", line 10, in save_cassette
data = serializer.serialize(cassette_dict)
File "/Library/Python/2.7/site-packages/vcr/serializers/jsonserializer.py", line 34, in serialize
return json.dumps(data, indent=4, default=_json_default)
File "/Library/Python/2.7/site-packages/simplejson/init.py", line 369, in dumps
**kw).encode(obj)
File "/Library/Python/2.7/site-packages/simplejson/encoder.py", line 264, in encode
chunks = list(chunks)
File "/Library/Python/2.7/site-packages/simplejson/encoder.py", line 600, in _iterencode
for chunk in _iterencode_list(o, _current_indent_level):
File "/Library/Python/2.7/site-packages/simplejson/encoder.py", line 455, in _iterencode_list
for chunk in chunks:
File "/Library/Python/2.7/site-packages/simplejson/encoder.py", line 568, in _iterencode_dict
for chunk in chunks:
File "/Library/Python/2.7/site-packages/simplejson/encoder.py", line 568, in _iterencode_dict
for chunk in chunks:
File "/Library/Python/2.7/site-packages/simplejson/encoder.py", line 535, in _iterencode_dict
yield _encoder(value)

Installing vcrpy doesn't include subdirectories

I have tried to install vcrpy using pip and easy install, and the installation does not contain the persisters, compat and serializers subdirectories. I tried cloning vcrpy using git, and the subdirectories were also missing when I did that. They are present when I download vcrpy as a zip file from github, though. Am I doing something wrong, or is there something missing in your config mgmt processes?

Is there a way to exclude a url param from comparisons?

We would like to use vcrpy to test some of our code that calls an external web service, but each time you send a GET or POST request to that api, you need to include a param of the form timestamp=<todays_datetime_as_an_int>. If you omit it, or use an old timestamp value, it doesn't return any results. So any url string that we send to that api is different in that one param than the one that created an entry in the relevant vcrpy yaml file, and every subsequent http request goes out to the actual web service instead of using the data stored in the yaml file. Is there a way to tell vcrpy to ignore that param and its value when comparing a new url with ones that it already has stored data for?

cookies lost

1. Open a requests.session
2. Issue a request
3. Issue another request, the cookies will be losed

with vcr.use_cassette('fixtures/vcr_cassettes/synopsis.yaml')

s = requests.Session()
r1 = s.get(“http://ishare.iask.sina.com.cn/")
r2= s.get(“http://ishare.iask.sina.com.cn/search.php“, params={'key': key})

when issue the first request, the cookies will be set, like 'PHPSESSID'. Then issue the second request, the cookies set from the first request will be losed. with vcr.use_cassette, r2 return the wrong content, without use_cassette, it will be fine.

https:// URLs do not work

It seems that vcrpy will change the https:// to http://

In one window I start up justniffer:

sudo justniffer -i en1 -p 'tcp port 80'

In another window:

~/dev/git-repos/python-carepass$ cat vcrtest/vcrtest-ssl.py 
import vcr
import urllib2

with vcr.use_cassette('fixtures/vcr_cassettes/synopsis.yaml'):
    response = urllib2.urlopen('https://marc-abramowitz.com/wp-content/foo.txt').read()
    assert 'Example Domains' in response

~/dev/git-repos/python-carepass$ .tox/py27/bin/python vcrtest/vcrtest-ssl.py 
Traceback (most recent call last):
  File "vcrtest/vcrtest-ssl.py", line 5, in <module>
    response = urllib2.urlopen('https://marc-abramowitz.com/wp-content/foo.txt').read()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found

Note that marc-abramowitz.com is my site and it only runs on port 80; I don't have an HTTPS server on port 443. So the fact that this raises an HTTP 404 error raises an eyebrow.

Furthermore, if I go back to the justniffer window, I see:

192.168.1.65 - - [01/Jul/2012:21:27:07 -0700] "GET /wp-content/foo.txt HTTP/1.1" 404 0 "" "Python-urllib/2.7"

Thus, there is a bug that when I make a urllib2 request to an https:// URL, vcrpy intercepts it and makes an http:// request instead.

README says Python 2.5 doesn't have context managers

it does!

from __future__ import with_statement

Misleading error message

Cassette.play_response() raises UnhandledHTTPRequestError("The cassette (%r) doesn't contain the request (%r) asked for") when matching requests do in fact exist in the cassette, but their play_counts is not zero. That's a bit misleading, I think a more specific error message would be better.

`AttributeError: 'NoneType' object has no attribute 'settimeout'` when using requests.session and connection gets closed

The code that VCRpy has for removing and setting sock to None (in order to work around some behavior of requests apparently) causes problems if you:

1. Open a requests.session
2. Issue a request that results in the connection being closed
3. Issue another request

The result of these steps is an exception being raised: AttributeError: 'NoneType' object has no attribute 'settimeout'.

Using the code in this gist: https://gist.github.com/msabramo/7664462

(py27.venv)marca@marca-mac2:~/dev/git-repos/vcrpy$ rm test_vcr.yaml; python vcr_py_issue_48.py
Doing request #1...
Got response #1; headers = CaseInsensitiveDict({'content-length': '381', 'server': 'gunicorn/0.17.4', 'connection': 'Close', 'date': 'Tue, 26 Nov 2013 19:22:12 GMT', 'access-control-allow-origin': '*', 'content-type': 'application/json'})
Doing request #2...
Traceback (most recent call last):
  File "vcr_py_issue_48.py", line 12, in <module>
    resp = session.post('http://httpbin.org/post', data={}, headers={'Connection': 'close'})
  File "/Users/marca/dev/git-repos/vcrpy/py27.venv/lib/python2.7/site-packages/requests/sessions.py", line 403, in post
    return self.request('POST', url, data=data, **kwargs)
  File "/Users/marca/dev/git-repos/vcrpy/py27.venv/lib/python2.7/site-packages/requests/sessions.py", line 361, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/marca/dev/git-repos/vcrpy/py27.venv/lib/python2.7/site-packages/requests/sessions.py", line 464, in send
    r = adapter.send(request, **kwargs)
  File "/Users/marca/dev/git-repos/vcrpy/py27.venv/lib/python2.7/site-packages/requests/adapters.py", line 321, in send
    timeout=timeout
  File "/Users/marca/dev/git-repos/vcrpy/py27.venv/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 471, in urlopen
    body=body, headers=headers)
  File "/Users/marca/dev/git-repos/vcrpy/py27.venv/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 308, in _make_request
    conn.sock.settimeout(read_timeout)
AttributeError: 'NoneType' object has no attribute 'settimeout'

cassettes written for https are used for requests for the same http url

smarter vcr.use_cassette paths

In this commit I make use of the fully qualified path 'tests/fixtures/basic_connection.yaml' to load a cassette. This works fine from $ python setup.py test but if I run this from inside pyCharm I will see:

DEBUG:vcr.stubs:Got <Request (POST) https://vcsa:443/sdk>

Error
Traceback (most recent call last):
  File "/Users/hartsocks/.venv/pyvmomi-tools-py2/lib/python2.7/site-packages/contextdecorator.py", line 102, in inner
    _reraise(*exc)
  File "/Users/hartsocks/.venv/pyvmomi-tools-py2/lib/python2.7/site-packages/contextdecorator.py", line 95, in inner
    result = f(*args, **kw)
  File "/Users/hartsocks/PycharmProjects/pyvmomi/tests/test_connect.py", line 34, in test_basic_connection
    pwd='vmware')
  File "/Users/hartsocks/PycharmProjects/pyvmomi/pyVim/connect.py", line 239, in Connect
    keyFile, certFile)
  File "/Users/hartsocks/PycharmProjects/pyvmomi/pyVim/connect.py", line 312, in __Login
    content = si.RetrieveContent()
  File "/Users/hartsocks/PycharmProjects/pyvmomi/pyVmomi/VmomiSupport.py", line 553, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/Users/hartsocks/PycharmProjects/pyvmomi/pyVmomi/VmomiSupport.py", line 362, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/Users/hartsocks/PycharmProjects/pyvmomi/pyVmomi/SoapAdapter.py", line 1212, in InvokeMethod
    resp = conn.getresponse()
  File "/Users/hartsocks/.venv/pyvmomi-tools-py2/lib/python2.7/site-packages/vcr/stubs/__init__.py", line 220, in getresponse
    % (self.cassette._path, self.cassette.record_mode)
vim.fault.HostConnectFault: (vim.fault.HostConnectFault) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = "Can't overwrite existing cassette ('tests/fixtures/basic_connection.yaml') in your current record mode ('none').",
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) []
}

Note: there are some odd interactions with my library's own HTTP wrappers but I think this should be enough for you to see what I'm getting at.

If I write:

    @vcr.use_cassette('fixtures/basic_connection.yaml', record_mode='none')
    def test_basic_connection(self):
        # see: http://python3porting.com/noconv.html
        si = connect.Connect(host='vcsa',
                             user='root',
                             pwd='vmware')
        session_id = si.content.sessionManager.currentSession.key
        # NOTE (hartsock): assertIsNotNone does not work in Python 2.6
        self.assertTrue(session_id is not None)
        self.assertEqual('52773cd3-35c6-b40a-17f1-fe664a9f08f3', session_id)

The test runs fine from the pyCharm IDE but breaks in the CLI environment.

It would be nice if we could specify the relative path 'fixtures/basic_connection.yaml' and allow the lib to find the files relative to the executing class.

"cassette is write protected" exception message is cryptic

The error raised when trying to reuse an existing file for a different request is really confusing. It reads like the file is write-protected by the OS (for instance by another process), rather then informing the user that by design cassete files are immutable.

Regardless, this lib is pretty awesome and I expect I'll be using it more in the future -
Thanks.

how to call response_of method ?

I tried passing the request URI , it fails with below error

AttributeError: 'str' object has no attribute 'url'

And if I pass Request Object I get below error

cass.response_of(cass.requests[0])

File "/Library/Python/2.7/site-packages/vcr/cassette.py", line 76, in response_of
raise KeyError
KeyError

Can you please help

Question: validate form of requests

I spotted a new issue in integration testing vmware/pyvmomi#90

It seems I failed to take into account testing for malformed requests. I have a new bug where the 'b' character is injected into my strings. How can I test just the requests as well as how my code handles responses?

Issues about the fixtures path

Use vrcpy a long time, it was very good. However, there is always a problem plaguing me.
Every time you use the 'use_cassette' function always need to pass a parameter 'path' (This path is often the same, except the file name), I feel a bit repetitive operations.

If you have a function, such as 'auto_cassette', can automatically detect the path, and save the resulting file into it, it will be very convenient. Because many times, especially when you are doing the test, you always want to save the file to the same path (current path + fixtures/vcr_cassettes/), the name of yaml file and the name of the test method is always the same (eg: test_xxx() => fixtures/vcr_cassettes/ test_xxx.yaml).

I hope you can help me, thank you!