Persisting Cookies from Initial Request in Python Requests

Feb 3, 2024 ยท 2 min read

When making requests in Python using the requests module, you may want to save and re-use cookies from an initial request in subsequent requests. This allows you to maintain session state and authentication across requests.

Here are some ways to accomplish this:

Save Cookies to Variable

The simplest approach is to save the cookies from the response to a variable and add them to future requests:

import requests

# Initial request to get cookies
response = requests.get('https://example.com') 

# Save cookies to variable
cookies = response.cookies

# Subsequent requests use cookies
response = requests.get('https://example.com/user', cookies=cookies)

This saves all cookies to a Python RequestsCookieJar object which can be passed directly to future requests.

Use a Session

A better way is to use a Session object, which will automatically handle persisting cookies across all requests:

import requests

session = requests.Session()

# Initial request saves cookies to session
response = session.get('https://example.com')  

# Subsequent requests will use those cookies automatically
response = session.get('https://example.com/user') 

This approach is recommended as it avoids manually handling the cookie persistence. The session handles that for you.

Potential Gotchas

A couple things to watch out for when working with cookies:

  • Expiry - Cookies may expire after some time, requiring another initial login request
  • Domain - Cookies may be for a specific domain and path, make sure to match requests
  • Security - Don't save sensitive cookies unencrypted for long periods
  • Summary

    Key points:

  • Save cookies from initial response to variable
  • Use a Session for automatic cookie persistence
  • Mind the expiry, domain specificity, and security of cookies
  • Using sessions is the most robust way to ensure cookies are available to all your script's requests. This allows you to maintain state as a client accesses a cookie-based web application.

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!