Authenticating Python Requests: A Practical Guide to Using Tokens for API Access

Dec 6, 2023 ยท 12 min read

APIs are everywhere these days. Whether you're accessing a SaaS platform or building an integration, you'll likely need to hook into an API at some point.

But there's a catch...

Many of these web APIs require authentication. You can't just directly fetch data or call API endpoints from your Python code. Nope, first you need some type of access token to authorize your requests.

Ugh, tokens? Seriously?? ๐Ÿ˜‘

That was my reaction when I first learned about API tokens. Seemed annoying and unnecessary, why can't they just let me access the data!

But after seeing tokens in action and fighting through some painful 401 Unauthorized errors, I now know better. Tokens are critical for securing web APIs.

And working with them in Python doesn't have to be a pain!

In this post, I'll share what I've learned from lots of web scraping and API projects. We'll dig into:

  • Why tokens matter for API access control
  • Common ways to get tokens for your apps
  • Making authenticated API calls from Python
  • Testing token-protected endpoints
  • Troubleshooting token issues
  • I'll draw directly from real-world examples of clients needing data from restricted sources. No dry theory or spec documentation here!

    Ready to master API authentication? Let's get tokenizing!

    Why Tokens Matter for Access Control

    First question is...

    What even are API tokens?

    Great question. ๐Ÿ‘

    You can think of API tokens sort of like temporary passwords or keys that allow access to web APIs and services.

    For example, when I was building a monitoring tool for a client, I needed to pull data from their internal analytics platform.

    But I couldn't just access the API directly. Nope! First I had to register as an "authorized" application, and they gave me a long hex string called a token.

    Only by passing that token string in my requests was I allowed to fetch analytics data from their systems.

    That hex code played the role of securely identifying me and granting access rights. It acted as my temporary API "key" if you will.

    Makes sense so far?

    Cool ๐Ÿ˜Ž

    Now you might be wondering...

    Why require tokens at all? Why not just let anyone use the API?

    Another excellently fair question!

    Many public APIs are wide open in fact. Like the OpenWeatherMap or CoinDesk APIs. You can fetch weather forecasts or crypto prices without any auth.

    However lots of companies have private data or paid resources access to which needs to be restricted.

    Think platforms like Stripe, Twilio, or MongoDB Atlas. Creating accounts and provisioning API keys controls access to usage-based paid resources.

    For internal enterprise platforms, access control protects sensitive systems and data. Can't have random outsiders messing with payroll systems for example!

    From an engineering perspective, requiring tokens:

  • Lets you handle complex permissions - User X might get read-only access while User Y has broader authorizations. Tokens encode those rights.
  • Enables better analytics - You can track token usage to monitor overall API traffic.
  • Allows revoking access - Compromised tokens can be invalidated without affecting others.
  • So in summary, API tokens enable managing authorization across users and systems in a more granular, secure way.

    Does that all make sense?

    Now that you know why tokens matter, let's look at...

    How To Obtain Authentication Tokens

    Alright, let's say you're convinced of the importance of tokens now for API access control. ๐Ÿ‘

    You rush excitedly to integrate the SuperMega API into your Python app.

    "I can't wait to get my hands on that lovely data," you think.

    So you...

    1. Import requests
    2. Enter the endpoint URL
    3. Write your .get() call
    4. Run your script!

    And boom ๐Ÿ’ฅ browser error 401 Unauthorized.

    Doh!

    You smack your forehead realizing you need an access token first! ๐Ÿคฆโ€โ™‚๏ธ

    No worries, it happens to the best of us.

    The next logical question is...

    How do you actually get API tokens for your app?

    Another fantastic question!

    It depends a bit on your use case, but let's cover some common scenarios:

    Client-Side Single Page Apps

    For JS-heavy front-end only apps, you'll typically use OAuth 2.0 flows involving redirection and access codes that exchange for tokens.

    Here your SPA requests user authorization against the API via the user's browser. Authorization codes are supplied which your app then exchanges server-side for access tokens using a temporary backend proxy.

    There's more detail but that's the basic dance. I walk through a full example in this article on the Auth0 SPA SDK.

    Mobile and Native Apps

    Very similar idea here too for mobile apps - you open an auth session in the native OS web view, exchange codes for tokens behind the scenes, then store for API requests.

    I don't have space to dive into mobile auth here but this post covers the high-level concepts well.

    Command Line Interfaces and Scripts

    For CLIs and scripts that run outside a user context, you instead leverage machine-to-machine app auth flows.

    The way this works is your script programmatically obtains OAuth client credentials from the target API (usually a client ID + secret).

    It then exchanges those credentials for access tokens using the client_credentials OAuth grant type:

    POST <https://api.example.com/oauth/token>
    
    {
      "grant_type": "client_credentials",
      "client_id": "YOUR_ID",
      "client_secret": "YOUR_SECRET",
      "audience": "<https://api.example.com/data>"
    }
    

    This returns your shiny new access token for API requests:

    {
      "access_token": "kst094kg093skg09...",
      "expires_in": 3600
    }
    

    Just make sure to refresh before expiry!

    Strategies for Optimizing Tokens

    A quick tip - in my experience APIs typically throttle usage on a per-token basis.

    So your script churning through requests on one overused token might hit rate limits, while an unused token in parallel continues without issue.

    Consider token optimization strategies:

  • Rotate tokens in a pool to prevent overuse of any single one
  • Set timeouts between requests to stay under rate limits
  • Retry failed requests across multiple tokens
  • Now, on to the best part...

    Making Authenticated API Requests in Python

    Alright at this point hopefully you've managed to get your hands on the precious API access tokens you were yearning for. ๐Ÿ˜Š

    So now what? How do we actually use them to authorize requests in Python?

    Let me show you...

    The process works like this:

    1. Make your standard API request like normal
    2. But add an Authorization header specifying your token

    For example:

    import requests
    
    TOKEN = "jOas3213j21k3JAS"
    
    headers = {
        "Authorization": f"Bearer {TOKEN}"
    }
    
    response = requests.get(
        "<https://api.example.com/data>",
        headers=headers
    )
    
    print(response.json())
    

    Here we:

  • Import requests to make the API call
  • Define our access token and assign to TOKEN variable
  • Construct headers dict with auth header using token
  • Make GET request to API endpoint, passing headers along
  • Print response data
  • And that's it! The API will see our valid access token and allow the request.

    So in practice adding token auth takes just 1 extra line of code to construct the headers. Nice and clean ๐Ÿ˜Š

    Now there's a couple variations you may see:

    Some APIs expect token values prefixed with Bearer like:

    Authorization: Bearer j2n3i29j2299j
    

    While others allow the raw token value without Bearer. Check the docs!

    The API response will also clue you in - 401 means try the other format ๐Ÿ˜›

    Second tip - you'll usually want token values in environment variables or secure stores, not hardcoded.

    But you get the basic idea!

    Let's look at some more examples with different HTTP methods:

    POST Request

    import json
    import requests
    
    url = "<https://api.example.com/notes>"
    
    headers = {
       "Authorization": "Bearer abc123"
    }
    
    data = {
       "title": "My cool note",
       "content": "Some fascinating content"
    }
    
    response = requests.post(url, json=data, headers=headers)
    

    Here we construct a JSON body for creating a note, set the auth header, and POST to the API endpoint to make our request.

    PUT Request

    import requests
    
    url = "<https://api.example.com/notes/123>"
    
    headers = {
       "Authorization": "Bearer abc123"
    }
    
    data = {
       "title": "My even cooler note"
    }
    
    response = requests.put(url, json=data, headers=headers)
    

    Similar idea except we PUT to a specific resource URL to update the note vs POSTing a new one.

    We could also make delete, patch and other requests using the same token auth pattern ...

    But I think you've got the hang of this by now ๐Ÿ˜‰

    The key points again being:

  • Import requests to call API โœ…
  • Construct auth header with token โœ…
  • Pass headers along with request โœ…
  • And that's really all there is to it from the Python code perspective. Nice and straightforward!

    Now what should we test next...

    Testing and Troubleshooting Token-Secured API Access

    Alright, so you've got your API tokens. You're making authenticated requests in Python without issue.

    Life is good! ๐Ÿ˜Ž๐Ÿ–

    Well...

    Maybe you spoke too soon? Because I guarantee things won't be smooth sailing forever!

    Yup, issues will come up with your token integration at some point. Could be hours, days or weeks into your work.

    But mark my words - pretty soon you'll see...

    โœจ 401 Unauthorized โœจ

    Or maybe...

    ๐ŸŒŸ 403 Forbidden ๐ŸŒŸ

    Ugh, API errors. What fun ๐Ÿ™ƒ

    The good news is that once you learn some basic troubleshooting workflows for tokens, resolving these problems gets much easier.

    Let me share some tips from the payload validation trenches...

    Testing Token-Protected Endpoints

    First up, a lesson that cost me a few hours once.

    When integrating an API, be sure to test both open public endpoints as well as the protected/private ones!

    It's easy to see simple endpoints working and assume everything is OK authentication-wise.

    But bypassing token validation when some resources don't require it isn't the same as truly testing your auth flow against secured paths specifically.

    Save yourself dead ends by always checking a special /private or /me style endpoint that requires that access token!

    Along these lines, pay attention to endpoints with scope limitations based on token contents.

    For example some APIs use scoped tokens that only allow certain permissions. Attempting to access unauthorized resources will fail, again though public ones may work normally.

    Double check the token data itself as well as API behaviors when testing.

    Common Token-Related Errors

    If you do start seeing mysterious API failures, first place to check is the status code.

    Codes like 401 Unauthorized or 403 Forbidden tell the tale that tokens are likely involved!

    401 Unauthorized

  • Indicates authentication failed completely
  • Check if token is missing from headers
  • Try passing token in Bearer format
  • 403 Forbidden

  • Authentication succeeded but token lacks proper scope
  • Make sure token has permissions for the attempted resource
  • 429 Too Many Requests

  • The bane of token-based rate limiting ๐Ÿ˜ค
  • See if the API docs mention throttling policies
  • Implement exponential backoff retry in your requests
  • I always wrap my API calls in exception handling that catches these cases specifically and handles accordingly.

    Make sure to have good visibility into the response codes!

    Inspecting Token Contents

    To dig deeper into any token-related issues, it's useful to directly examine their contents.

    For example decode that hex string back to JSON:

    import jwt
    
    decoded = jwt.decode(
        "<token>", options={"verify_signature": False}
    )
    
    print(decoded)
    

    This lets you inspect fields like:

  • Expiration time
  • Issuer
  • Key ID
  • Standard and custom claims
  • I've resolved weird middleware routing issues comparing prod vs. sandbox token contents for example.

    Storing Tokens Securely

    After getting burned by leaked API keys a few times, I take token storage seriously nowadays!

    What's the problem you ask? Well my friend, you can't just toss those unencrypted hex codes around willy-nilly.

    Nope! Tokens grant precious access so gotta keep 'em safe and sound.

    My top tip? Use environment variables!

    import os
    
    TOKEN = os.getenv("MY_SECRET_TOKEN")
    
    

    Most CI/CD platforms like GitHub Actions let you securely configure these too.

    Whatever you do, do not commit that sucker directly in code! People are watching ๐Ÿ‘€

    Which brings me to those fancy JWTs...

    Understanding JSON Web Tokens (JWTs)

    As I worked with more APIs, I started noticing more tokens looking like garbled JSON:

    eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWI...
    

    These fellas are JWTs - essentially JSON bundles of user credentials and permissions for APIs to inspect.

    They have some nice advantages - apps can read claims directly, blacklist bad ones, verify signatures...the works!

    Downside is JWTs do expire eventually. Which is no fun let me tell ya!

    So if you start seeing cryptic JSON tokens, now you'll know - those are JWTs, baby ๐Ÿ˜Ž

    Handling Token Expiration and Rotation

    Speaking of expiry, nothing more annoying than random request failures because your access expired! ๐Ÿ˜ค

    My code would work perfectly one minute, then suddenly...

    โœจ 401 Unauthorized โœจ

    No bueno!

    Two tips for ya:

    1. Check for expiry and force refresh
    2. Rotate new tokens continuously

    Ibotta has a nifty open source project called PyRefresh that helps with auto token rotation.

    Give it a whirl if you start seeing unexpected failures!

    Renewing Expired Tokens

    Which brings me to the last tip - handling expired credentials!

    Depending on the expiration policy, your access tokens could stop working anywhere from hours to months after creation.

    The fix is "rotating" new tokens into your system and retiring old ones.

    Architect this rotation intelligently according to app needs:

  • Schedule periodic batch updates
  • Check expiration and force refresh
  • Streamline renewal on failures
  • However you tackle it, make sure to design and build for access token lifecycles from the start!

    Go Forth and Tokenize!

    And there you have it my friend! You're now armed to the teeth with tokens ;)

    We covered:

  • Why API access tokens matter
  • How to obtain tokens for different apps
  • Using tokens to authorize Python API requests
  • Testing and troubleshooting common issues
  • You learned from my own hard-knocks experiences - so keep that in mind working with real-world APIs!

    I hope mapping out the authentication flow removes some "token terror" holding you back from integrating APIs you need.

    Soon you'll laugh about the days you feared tokens instead of wielding them like a boss! ๐Ÿ˜Ž๐Ÿค˜

    So go forth and start tokenizing those internal REST services. Leverage SaaS platform auth for some sweet integrated apps. ๐Ÿš€

    The world is your API-authorized oyster.

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!