Mastering Python Requests Sessions for Power Users

Oct 22, 2023 ยท 5 min read

The Python requests library makes sending HTTP requests simple and intuitive. But did you know requests also provides a powerful Session object that takes your skills to the next level?

In this comprehensive guide, you'll learn how to use request Sessions to boost performance, persist settings across requests, and handle advanced scenarios.

Let's dive in!

An Introduction to Request Sessions

A requests Session allows you to persist certain parameters across multiple requests.

For example, you can use a Session to:

  • Reuse HTTP connection pooling for performance
  • Save cookies and headers between requests
  • Add custom defaults like proxies, auth, and more
  • In short, Sessions allow you to consolidate logic that would otherwise be repetitive.

    You can think of a Session as a configurable context in which requests happen.

    To start using them, instantiate the requests.Session class:

    import requests
    
    session = requests.Session()
    

    Now let's explore some powerful use cases.

    Configuring Session Defaults

    One main purpose of Sessions is persisting settings. This avoids repetitive logic for things like headers.

    For example, to set a default header on a Session:

    session.headers.update({'User-Agent': 'CustomAgent'})
    

    Now all requests made through this Session will send that header automatically.

    You can also set default proxies, authentication, timeouts, and more:

    # Set default proxy
    session.proxies.update({'https': '192.168.0.1'})
    
    # Set default auth
    session.auth = ('user', 'pass')
    
    # Set default timeout
    session.timeout = 60
    

    This applies your custom defaults to all requests with minimal code.

    Automatic Cookie Persistence

    An extremely useful perk of Sessions is automatic cookie handling.

    By default, cookies set in requests made from a Session will persist:

    # First request sets cookie
    session.get('<https://httpbin.org/cookies/set/session_id/1234>')
    
    # Next request will send the cookie
    resp = session.get('<http://httpbin.org/cookies>')
    print(resp.json())
    # {'cookies': {'session_id': '1234'}}
    

    This makes sessions perfect for authenticating to sites and APIs.

    Connection Pooling for Improved Performance

    Another major benefit of Sessions is HTTP keep-alive / connection pooling.

    By reusing connections, Session requests avoid expensive TLS handshakes and DNS lookups. This boosts performance significantly when making multiple calls to the same domain:

    # Reuses connection for all requests
    with requests.Session() as s:
      for _ in range(100):
        s.get('<https://httpbin.org/delay/1>') # Reuses connection
    

    Connection pooling happens automatically when using a session!

    Saving and Reusing Sessions

    Once you've configured a Session, it can be serialized to disk and reused later:

    # Save session to file
    with open('session.pkl', 'wb') as f:
      pickle.dump(session, f)
    
    # Load and reuse session
    with open('session.pkl', 'rb') as f:
      session = pickle.load(f)
    

    This lets you persist Sessions between executions rather than reconfiguring each time.

    Advanced Techniques with Sessions

    Beyond the basics, Sessions unlock more advanced capabilities:

    Streaming large responses:

    # Stream a large response body
    with session.get(url, stream=True) as resp:
      for chunk in resp.iter_content(1024):
        # Process chunk
    

    Using client certificates and SSL:

    # Use a client certificate for SSL
    session.cert = ('/path/client.cert', 'path/client.key')
    

    Event hooks to monitor requests:

    # Trigger events during the request
    def print_url(resp, *args, **kwargs):
      print(resp.url)
    
    session.hooks['response'] = [print_url]
    

    And many other advanced use cases.

    Sessions are a power user's best friend!

    A Session vs Cookie Comparison

    Since Sessions handle cookies automatically, you may wonder how they differ from cookies directly.

    The key differences are:

  • Sessions are server-side, stored on the backend
  • Cookies are client-side, stored in the user's browser
  • Sessions allow any data type, cookies just store strings
  • Cookies are sent automatically, sessions require code to use
  • Sessions identify users, cookies store small amounts of data
  • In practice, sessions identify clients while cookies store client data. You can use both together!

    Common Questions and Answers

    Here are some common questions about using Python requests Sessions:

    Are request sessions thread safe?

    No, Sessions objects themselves are not thread-safe. If accessing a Session from multiple threads, you need to add locking.

    What's the limit of requests in a session?

    There is no limit imposed by requests. You can make as many requests from a session as needed.

    Is there a performance difference between session vs normal requests?

    Yes, sessions allow connection pooling so performance is much better for multiple requests.

    How do I access response cookies from a session?

    The cookies attribute contains a CookieJar with cookies set on responses:

    session.cookies['cookie_name']
    

    What's the best way to store a session for later use?

    Serializing the Session object with pickle works well for persisting it locally.

    Wrap Up

    Requests sessions provide a flexible and efficient way to handle advanced scenarios and consolidate logic across HTTP requests in Python.

    Some key takeaways:

  • Sessions persist settings like headers, auth, and cookies
  • Automatic cookie handling makes auth and scraping easier
  • Connection pooling improves performance for multiple requests
  • Sessions can be saved and reused between executions
  • Hopefully you now feel empowered to use sessions for enhanced speed, convenience, and functionality in your Python requests code!

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!