Requests-cache is a transparent persistent cache for the requests library (version 2+).
Install with pip:
pip install requests-cache
import requests_cache
requests_cache.install_cache('demo_cache')
And all responses with headers and cookies will be transparently cached to demo_cache.sqlite
.
For example, following the code will take only 1-2 seconds instead of 10, and will run instantly on next launch:
import requests
for i in range(10):
requests.get('http://httpbin.org/delay/1')
It can be useful when you are creating some simple data scraper with constantly changing parsing logic or data format, and don't want to redownload pages or write complex error handling and persistence.
For more complex workflows, it is possible to cache different requests with different expiration times, or disable caching for a specific request completely:
For more complex workflows, it is possible to cache different requests with different expiry times, or disable caching for a specific request completely:
import time
import requests
import requests_cache
requests_cache.install_cache('demo_cache')
# Hits the URL only 2 times
for i in range(10):
requests.get('http://httpbin.org/delay/1', expire_after=1)
time.sleep(0.2)
If requests-cache
isn't quite what you need, you can help make it better! See the
Contributing Guide
for details.
You can also check out these other python cache projects:
- CacheControl: An HTTP cache for
requests
that caches according to uses HTTP headers and status codes - aiohttp-client-cache: An async HTTP cache for
aiohttp
, based onrequests-cache
- aiohttp-cache: A server-side async HTTP cache for the
aiohttp
web server - diskcache: A general-purpose (not HTTP-specific) file-based cache built on SQLite
- aiocache: General-purpose (not HTTP-specific) async cache backends
- Documentation at readthedocs
- Source code and issue tracking at GitHub
- Working example at Real Python