requests-cache

Requests-cache is a transparent persistent cache for the requests library (version 2+).

Installation

Install with pip:

pip install requests-cache

Usage example

import requests_cache

requests_cache.install_cache('demo_cache')

And all responses with headers and cookies will be transparently cached to demo_cache.sqlite. For example, following the code will take only 1-2 seconds instead of 10, and will run instantly on next launch:

import requests

for i in range(10):
    requests.get('http://httpbin.org/delay/1')

It can be useful when you are creating some simple data scraper with constantly changing parsing logic or data format, and don't want to redownload pages or write complex error handling and persistence.

For more complex workflows, it is possible to cache different requests with different expiration times, or disable caching for a specific request completely:

For more complex workflows, it is possible to cache different requests with different expiry times, or disable caching for a specific request completely:

import time
import requests
import requests_cache

requests_cache.install_cache('demo_cache')

# Hits the URL only 2 times
for i in range(10):
    requests.get('http://httpbin.org/delay/1', expire_after=1)
    time.sleep(0.2)

Related Projects

If requests-cache isn't quite what you need, you can help make it better! See the Contributing Guide for details.

You can also check out these other python cache projects:

CacheControl: An HTTP cache for requests that caches according to uses HTTP headers and status codes
aiohttp-client-cache: An async HTTP cache for aiohttp, based on requests-cache
aiohttp-cache: A server-side async HTTP cache for the aiohttp web server
diskcache: A general-purpose (not HTTP-specific) file-based cache built on SQLite
aiocache: General-purpose (not HTTP-specific) async cache backends