lukasa/httpcache

Simple HTTP cache for Python Requests

httpcache: HTTP Caching for Python

Warning

This library is no longer maintained: you should use the excellent CacheControl library, which directly ports httplib2's caching algorithms.

HTTP, like all well designed standards, has multiple confusing mechanisms for caching. httpcache is a HTTP cache that knows how to use HTTP headers and status codes to correctly cache your HTTP traffic. It's built for use with the excellent Requests library, because if you're not using Requests you're probably prepared to roll your own caching library too.

It's gloriously easy to use. If all you want is caching in Requests, all you need to do is plug a transport adapter into your Requests session:

>>> import requests
>>> from httpcache import CachingHTTPAdapter
>>> s = requests.Session()
>>> s.mount('http://', CachingHTTPAdapter())

Away you go!

If you want more control, you can use the cache data-store itself. Store your cache entries like this:

from httpcache import HTTPCache
cache = HTTPCache(capacity=50)
cache.store(response)

And retrieve them like this:

cached_response = cache.retrieve(request)

Simple.

Features

• Tight integration with Requests data structures.
• Understands Expires and Cache-Control headers.
• Knows how to interpret 304 Not Modified responses.
• Can send If-Modified-Since headers.
• Aware of HTTP verbs, e.g. POST.
• RFC 2616-compliant.

Installation

To install httpcache, you want to run:

$ pip install httpcache

If you can't do that, and you really must have httpcache, and you can't install pip, then you can try:

$ easy_install httpcache

I strongly recommend you don't do that though.

Versions

httpcache supports all the versions of Python that Requests does. This means 2.6, 2.7 and 3.3. It is possible that httpcache will work on other versions of Python but we do not test on those versions and will not support them.

Contribute

Contributions are always welcome! Please abide by the following rules when contributing:

1. Check that no-one has opened an issue already covering your bug. If you open a duplicate issue, the maintainer will give you a stern look.
2. Fork the Github repository and start writing your tests. If you're fixing a bug, I recommend writing a failing test first and working until it passes. If you're adding a feature, you're free to add tests after you write the functionality, but please test the functionality thoroughly.
3. Send a Pull Request. If I don't respond within a couple of days, please shout at me on Twitter or via email until I do something about it.

httpcache's Issues

expires_from_cache_control: TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Hi, I've found the following error in function expires_from_cache_control:

we setting the duration to None and then trying to convert None to int, and it doesn't work properly.
We need to fix this code:

def expires_from_cache_control(header, current_time):
    """
    Given a Cache-Control header, builds a Python datetime object corresponding
    to the expiry time (in UTC). This function should respect all relevant
    Cache-Control directives.

    Takes current_time as an argument to ensure that 'max-age=0' generates the
    correct behaviour without being special-cased.

    Returns None to indicate that a request must not be cached.
    """
    # Cache control header values are made of multiple comma separated fields.
    # Splitting them like this is probably a bad idea, but I'm going to roll with
    # it for now. We'll come back to it.
    fields = header.split(', ')
    duration = None

    for field in fields:
        # Right now we don't handle no-cache applied to specific fields. To be
        # as 'nice' as possible, treat any no-cache as applying to the whole
        # request. Bail early, because there's no reason to stick around.
        if field.startswith('no-cache') or field == 'no-store':
            return None

        if field.startswith('max-age'):
            _, duration = field.split('=')
            duration = int(duration)

    interval = timedelta(seconds=int(duration))

    return current_time + interval

and add something like this:

    if no duration:
        return None

    interval = timedelta(seconds=int(duration))

    return current_time + interval

deprecation warning

Hi I have following warning when using your library.

venv/lib/python3.8/site-packages/httpcache/compat.py:11
/Users/cobrakai/Projects/woodmac-sdk-python/venv/lib/python3.8/site-packages/httpcache/compat.py:11: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
from collections import OrderedDict, MutableMapping

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Support for Vary

In this current state, there is no parsing or handling of the Vary header. Without this, the cache is fundamentally broken.

HTTP caching actually looks like a tree. At the top level, you have the url. Leafing off of the url, there are variances of the same url. Think a gzipped response vs not gzipped. Those are both objects accessible at the same url, but are a variant because of the Vary: Accept-Encoding.

This becomes more important if you were to vary on Cookie, for example. Typically this is done to support caching for individual users. Without support for Vary, this is impossible.

I'd be happy to explain this more and provide some advice. I think this library is really interesting and would love to try it out, but this is pretty important when it comes to http caching. :)

Improve the RecentOrderedDict

This structure was hacked together pretty quickly and so has quite a serious overhead cost. __getitem__ and __setitem__ both potentially incur a lookup, a deletion and an insertion each time they're called, which isn't great.

There should be some low-hanging fruit for optimisation here, possibly by replacing the structure entirely.