APIs are essential for modern applications, but excessive API requests can really slow things down. In this comprehensive guide, I'll share how to cache API responses in Python to supercharge performance.
Back when I was first learning web scraping, I ran into major slowdowns because my scripts would repeatedly call the same APIs. After banging my head for a while, I discovered caching - a total game changer! Now I'll show you how to cache like a pro.
Why is Caching Helpful?
Caching allows you to save a copy of an API response and return the cached data instead of making duplicate requests.
This improves speed and reduces traffic to APIs. For instance, say your script fetches a product list from an ecommerce API. Instead of hitting the API on each search, you could cache the response for a certain period. Now your script can read the cached data and only refetch when the cache expires.
Other caching benefits:
Caching is a must-have tool for large web scraping and data collection projects.
Installing Caching Libraries
Python doesn't include caching natively, but we can easily add it. The requests-cache library is purpose-built for caching API responses from Python's super popular requests module.
Install both via pip:
pip install requests requests-cache
Requests-cache supports various storage backends like SQLite, Redis, and MongoDB for saving cached responses. I prefer SQLite since it's a simple single file database.
Some other handy caching tools are cachecontrol, which implements client-side caching, and django-cache-machine for Django apps. We'll focus on requests-cache here since it's easy to use standalone.
Basic Caching Usage
Requests-cache provides a CachedSession class that acts like a requests Session, but with automatic caching behavior.
Here's an example:
import requests
import requests_cache
session = requests_cache.CachedSession('api_cache')
response = session.get('<https://example.com/api/products>')
Now the first call fetches and caches the response. Subsequent calls just read from cache without hitting the API again until the cache expires. Easy enough!
CachedSession is great for simple use cases. But what if you want to enable caching on all requests by default without managing a session?
Requests-cache offers a clever install_cache method that patches requests globally:
import requests
import requests_cache
requests_cache.install_cache('api_cache')
response = requests.get('<https://example.com/api/products>')
This transparently caches all requests through requests. Much easier than wrapping every call in a session!
A common doubt is whether enabling caching affects getting fresh data. By default, requests-cache caches indefinitely. But we can set an expiry period so caches automatically invalidate after some time:
requests_cache.install_cache('api_cache', expire_after=3600)
Now caches last 1 hour (3600 seconds). The next request after that fetches a new response from the API and caches it again.
Advanced Caching Techniques
Now that you've seen basic caching, let's dig into some more advanced tactics:
Conditional Requests
When a cache expires, instead of fetching the full response again, the client can make a conditional request to just check if the resource changed:
GET /data
If-Modified-Since: <date>
This validates the cache against the server's current version. If unchanged, it returns a 304 status without any body, saving bandwidth. The client keeps using the existing cached data.
Cache Invalidation
Sometimes you need to purge the cache and fetch fresh data on demand. Requests-cache provides a cache.delete_url() method for this:
import requests_cache
requests_cache.uninstall_cache() # Clear all caches
requests_cache.delete_url('<https://example.com/api/products>') # Clear one URL
This allows manual cache invalidation when the source data changes.
Cache Hierarchies
Real-world caching uses a hierarchy of layered caches. Requests may check the local browser cache, then a shared proxy cache, then finally the origin server.
We can mimic this in Python by combining requests-cache with client-side caching tools like CacheControl. Add both caches and prioritize CacheControl for low-level caching.
Avoiding Stale Caches
Stale caches can cause hard-to-debug issues if you expect live data. Some tips:
When to Cache Responses
Now let's discuss when caching makes sense and when to avoid it:
Also consider caching these types of requests:
Common Caching Patterns
Here are some useful caching architectures:
Cache-Aside
Load data on demand, check if cached, otherwise fetch from API and cache result. Great for dynamic data.
Cache-Then-Network
Fetch first from cache, return result. Then asynchronously fetch latest from API in background to keep cache fresh.
Read-Through
On cache miss, load from API and add result to cache before returning response. Keeps cache populated.
Write-Through
When updating API, propagate write to cache too. Ensures cache consistency with source.
These patterns help build robust, high-performance caching that stays in sync with APIs.
Handling Caching Gracefully
Caching can fail unexpectedly if the cache is cleared or unavailable. Make sure your code handles these scenarios gracefully:
import requests
import requests_cache
try:
response = requests.get('<https://api.example.com/data>')
except requests_cache.CacheMissingError:
# Cache missing - fall back to live API request
response = requests.get('<https://api.example.com/data>')
Use
from datetime import timedelta
response = cache.get_cached_response(url, timedelta(minutes=15))
if not response:
# Cache expired, fetch from live API
response = requests.get(url)
This allows handling scenarios like expired caches or disabled caching more smoothly in your code.
Caching HTTP Errors
By default, requests-cache caches only 200 responses. To cache 404s, 500s and other HTTP errors:
session = requests_cache.CachedSession(cache_errors=True)
Now all responses including errors will be cached. This avoids hitting APIs repeatedly just to collect error codes.
Set
session = requests_cache.CachedSession(
cache_errors=True,
allowable_codes=[404]
)
Caching errors can improve efficiency in scrapers that process many links. But don't cache sensitive errors containing information leaks!
Conclusion
Caching is a crucial technique for improving API-driven applications. With tools like requests-cache, it's easy to add powerful caching to Python programs.
We covered the basics of enabling caching, as well as advanced topics like cache invalidation, conditional requests, cache hierarchies, and common caching architectures.
Key takeaways:
Now you have all the knowledge to implement caching like a pro!
FAQ
What is the difference between caching and data replication?
Caching stores temporary copies of frequently accessed data to improve speed. Replication creates multiple redundant copies of data on different servers to improve availability if one server goes offline.
How is Redis different than caching?
Redis is a fast in-memory data store often used to build caches. Caching is a general concept for improving performance by saving copies of data. Redis provides an infrastructure to implement caching.
Where is the HTTP cache stored on browsers?
Browser caches are typically stored on disk in a cache folder specific to that browser. For example, Chrome caches are under ~/Library/Caches/Google/Chrome on macOS. Caches may also be kept in memory for even faster access.
Should POST requests be cached?
No, POST requests should not be cached as they are often used for data creation with side-effects. Caching POSTs can lead to duplicate data submissions if replayed from cache.
How can I benchmark my caching performance?
Use a tool like Locust to load test APIs with and without caching enabled. Compare metrics like response times and requests per second to quantify caching gains.
Hope this guide helps you become a caching master! Let me know if you have any other questions.