Mastering Python Requests Sessions for Power Users

The Python requests library makes sending HTTP requests simple and intuitive. But did you know requests also provides a powerful Session object that takes your skills to the next level?

In this comprehensive guide, you'll learn how to use request Sessions to boost performance, persist settings across requests, and handle advanced scenarios.

Let's dive in!

An Introduction to Request Sessions

A requests Session allows you to persist certain parameters across multiple requests.

For example, you can use a Session to:

In short, Sessions allow you to consolidate logic that would otherwise be repetitive.

You can think of a Session as a configurable context in which requests happen.

To start using them, instantiate the requests.Session class:

import requests

session = requests.Session()

Now let's explore some powerful use cases.

Configuring Session Defaults

One main purpose of Sessions is persisting settings. This avoids repetitive logic for things like headers.

For example, to set a default header on a Session:

session.headers.update({'User-Agent': 'CustomAgent'})

Now all requests made through this Session will send that header automatically.

You can also set default proxies, authentication, timeouts, and more:

# Set default proxy
session.proxies.update({'https': '192.168.0.1'})

# Set default auth
session.auth = ('user', 'pass')

# Set default timeout
session.timeout = 60

This applies your custom defaults to all requests with minimal code.

Automatic Cookie Persistence

An extremely useful perk of Sessions is automatic cookie handling.

By default, cookies set in requests made from a Session will persist:

# First request sets cookie
session.get('<https://httpbin.org/cookies/set/session_id/1234>')

# Next request will send the cookie
resp = session.get('<http://httpbin.org/cookies>')
print(resp.json())
# {'cookies': {'session_id': '1234'}}

This makes sessions perfect for authenticating to sites and APIs.

Connection Pooling for Improved Performance

Another major benefit of Sessions is HTTP keep-alive / connection pooling.

By reusing connections, Session requests avoid expensive TLS handshakes and DNS lookups. This boosts performance significantly when making multiple calls to the same domain:

# Reuses connection for all requests
with requests.Session() as s:
  for _ in range(100):
    s.get('<https://httpbin.org/delay/1>') # Reuses connection

Connection pooling happens automatically when using a session!

Saving and Reusing Sessions

Once you've configured a Session, it can be serialized to disk and reused later:

# Save session to file
with open('session.pkl', 'wb') as f:
  pickle.dump(session, f)

# Load and reuse session
with open('session.pkl', 'rb') as f:
  session = pickle.load(f)

This lets you persist Sessions between executions rather than reconfiguring each time.

Advanced Techniques with Sessions

Beyond the basics, Sessions unlock more advanced capabilities:

Streaming large responses:

# Stream a large response body
with session.get(url, stream=True) as resp:
  for chunk in resp.iter_content(1024):
    # Process chunk

Using client certificates and SSL:

# Use a client certificate for SSL
session.cert = ('/path/client.cert', 'path/client.key')

Event hooks to monitor requests:

# Trigger events during the request
def print_url(resp, *args, **kwargs):
  print(resp.url)

session.hooks['response'] = [print_url]

And many other advanced use cases.

Sessions are a power user's best friend!

A Session vs Cookie Comparison

Since Sessions handle cookies automatically, you may wonder how they differ from cookies directly.

The key differences are:

In practice, sessions identify clients while cookies store client data. You can use both together!

Common Questions and Answers

Here are some common questions about using Python requests Sessions:

Are request sessions thread safe?

No, Sessions objects themselves are not thread-safe. If accessing a Session from multiple threads, you need to add locking.

What's the limit of requests in a session?

There is no limit imposed by requests. You can make as many requests from a session as needed.

Is there a performance difference between session vs normal requests?

Yes, sessions allow connection pooling so performance is much better for multiple requests.

How do I access response cookies from a session?

The cookies attribute contains a CookieJar with cookies set on responses:

session.cookies['cookie_name']

What's the best way to store a session for later use?

Serializing the Session object with pickle works well for persisting it locally.

Wrap Up

Requests sessions provide a flexible and efficient way to handle advanced scenarios and consolidate logic across HTTP requests in Python.

Some key takeaways:

Hopefully you now feel empowered to use sessions for enhanced speed, convenience, and functionality in your Python requests code!