The Complete Guide to Retrying Failed Requests with Axios

Jan 9, 2024 ยท 8 min read

Before jumping into retries, let's first understand why Axios is useful. Axios is a promise-based HTTP client for making requests to APIs and servers in JavaScript. Whether it's fetching data from a REST endpoint or posting data to your backend - Axios makes the process seamless.

For example, here's a simple GET request with Axios:

import axios from 'axios';

axios.get('/api/users')
  .then(response => {
    console.log(response.data);
  })
  .catch(error => {
   console.log(error);
  });

Compared to native methods like XMLHttpRequest or the Fetch API, Axios stands out for its ease of use, browser and Node.js support, and handy response handling via promises. Over the years, it has become a staple in web projects.

Of course, even with Axios, stuff can go wrong. Requests can fail due to network errors, server issues or any number of reasons. Let's look at some common cases...

When Requests Fail

From my experience building web scrapers, some typical situations where requests fail include:

Network Issues: Connectivity blips or DNS failures can cause requests to hang and eventually time out. Last month, while scraping a site behind Cloudflare bot protection, two of my IPs lost DNS resolution mid-script!

Overloaded Servers: An influx of traffic can overwhelm servers, causing them to throw 500 or 503 errors under load. This happened recently when one of our monitored ecommerce APIs got flooded during a sale.

Rate Limiting: APIs often enforce thresholds to prevent abuse, throttling requests once limits are crossed. Not ramping up traffic gradually has tripped rate limits for me.

Bad Deploys: There's nothing like broken releases to ruin your day! I've seen CI pipelines directly deploy half-baked features. API responses suddenly error out after months of stability. Fun times!

The costs of failing requests can be huge, specially if critical transactions are involved. This is why manually retrying failed requests is such a common, if tedious, workaround.

But what if retries could happen automatically in code?

Enter Automated Retries

Here are some benefits of automating retries instead of relying on crude manual interventions:

  • Reliability: Retrying failures increases the likelihood of request success, improving overall reliability.
  • Speed: Developers don't waste time manually diagnosing and restarting scripts.
  • Scalability: Automated mechanisms scale seamlessly instead of costly human effort.
  • Resilience: Systems become more resilient to temporary downstream issues.
  • Let's look at how we can bake automated retries into Axios using interceptors.

    Building a Retry Mechanism

    Axios interceptors allow handling requests before they are sent or after a response is received. They are ideal for cross-cutting tasks like logging, authentication or, you guessed it, retries!

    Here is a simple interceptor that retries failed requests up to 3 times:

    import axios from 'axios';
    
    // Add a request interceptor
    axios.interceptors.request.use(function (config) {
    
        // Do something before request is sent
        return config;
    
      }, function (error) {
    
        // Request error handling
    
        return Promise.reject(error);
    
    });
    
    // Add a response interceptor
    axios.interceptors.response.use(function (response) {
    
        // Any status code that lie within the range of 2xx cause this function to trigger
        // Do something with response data
        return response;
    
      }, async function (error) {
    
        // Any status codes that falls outside the range of 2xx cause this function to trigger
        // Do something with response error
    
        if (error.response.status === 500) {
    
          const config = error.config;
    
          if (!config.retryCounter) {
            config.retryCounter = 1;
          } else {
            config.retryCounter++;
          }
    
          if (config.retryCounter > 3) {
            // Reject with error
            return Promise.reject(error);
          }
    
          // Try again
          return axios(config);
    
        }
    
        // Reject with error
        return Promise.reject(error);
    
    });
    

    There are a few interesting bits here:

  • We first handle the request then the response via separate interceptors.
  • On error responses like 500, we check if retryCounter exceeds 3.
  • If no, we retry by calling Axios again with the same config.
  • Else we reject with the error by exiting the Promise chain.
  • This shows the basic mechanism. But there's a lot more we can configure for effective retries.

    Configuring Retries

    Some key controls we have for configuring retries include:

    Number of Retries

    Too few, and transient issues will still cause failures. Too many, and recovering requests may overload systems. There's no ideal value - start small with 2-3 attempts.

    Delay Between Retries

    Adding a delay between retries allows systems to recover before hammering them again. A one-size-fits-all delay doesn't work well. More on this soon.

    Conditional Retries

    Blindly retrying certain errors like 400 Bad Request won't help. Having a conditional check to retry only specific cases is important.

    Let's enhance our mechanism to incorporate some of these capabilities:

    // Import exponential backoff function
    import { exponentialDelay } from './backoffUtils';
    
    // Response interceptor
    axios.interceptors.response.use(
      function (response) {
        // ...
      },
    
      async function (error) {
    
        // Conditional retry
        if (shouldNotRetry(error)) {
          return Promise.reject(error);
        }
    
        if (error.config.retryCount <= 3) {
    
          // Increase counter
          error.config.retryCount++;
    
          // Apply exponential backoff delay
          const delay = exponentialDelay(retryCount);
    
          // Wait before retrying
          await timeout(delay);
    
          // Return error config again
          return axios(error.config);
    
        }
    
        // Max retries reached
        return Promise.reject(error);
    
      }
    );
    
    // Check if error should not be retried
    function shouldNotRetry(error) {
      // Add logic here
      return false;
    }
    
    

    Observe how we:

  • Limit number of retries via counter check
  • Calculate backoff delay dynamically per retry
  • Added waiting based on delay before next retry
  • Enable conditional retry logic through utility
  • Besides fixed/exponential delays, jitter delays help stagger retry clusters hitting servers in unison. This post has a good Primer.

    Strategies for Effective Retries

    There are additionally some best practices worth highlighting around retrying:

    Idempotent vs Non-idempotent Requests

    Methods like GET and PUT are safe to retry since results will be identical. However, POST/PATCH calls modifying data may duplicate records if blindly retried on certain errors. Handling them correctly is vital.

    Avoiding Retry Overloads

    Too many clients furiously retrying failed requests can directly overwhelm recovering systems. Exponential backoffs and conditional retries based on error cause can help.

    Caching/Queuing

    Sometimes it helps to temporarily cache failed requests and replay from persistent storage once issues are mitigated. This takes pressure off overloaded systems.

    There are advanced strategies like the Circuit Breaker pattern which "trip" after too many errors to prevent clobbering fragile backends. Netflix's Hystrix is a good reference implementation.

    While the examples above directly use Axios interceptors for brevity, let's look at another option...

    Using the Axios-Retry Plugin

    For convenience, there are handy plugins like axios-retry that wrap retrying capabilities into simple packages:

    import axios from 'axios';
    import axiosRetry from 'axios-retry';
    
    // Initialize
    axiosRetry(axios, {
       retries: 3,
       retryCondition: isServerError // Condition method
    });
    
    // Use Axios as normal
    axios.get('/items')
      .then(/* ... */)
      .catch(/* ... */)
    

    The plugin automatically retries failed requests which match criteria like server errors or network issues.

    Some benefits over a manual interceptor are:

  • Handling complexity is abstracted
  • Configuration via clean object options
  • Can be selectively applied on specific clients
  • Advanced retry timings (exponential backoff, jitter)
  • For full documentation on usage and configuration parameters, check the axios-retry README.

    Let's also briefly contrast some alternatives.

    Comparison of Libraries

    Besides axios-retry, there are couple of other shared libraries like retry-axios. They have a lot of overlap in capabilities but few differences exist around implementation:

    Featureaxios-retryraw-axiosretry-axios
    ImplementationInterceptorsInterceptorsInterceptors
    Config OptionsManyManual codeFewer
    UsageEasierComplexEasy
    Request ScopeGlobal onlyPer requestBoth
    Promise ControlLimitedFullFull

    For most use cases, axios-retry strikes a good balance without forcing much config coding.

    However, for advanced needs like custom retry flows or specialized HTTP clients, using raw Axios interceptors directly allows finer-grained control.

    There are also niche tools like axios-circuitbreaker focused solely on specific capabilities like circuit breakers which can complement these libraries.

    Testing and Debugging Retries

    Here are some quick tips around testing and troubleshooting retry logic:

    Unit Testing

    Mock error responses using libraries like Jest to simulate failures and assert retries occur expected times.

    Logging

    Trace retry attempts in interceptor handlers during development by logging metadata like retry counts.

    Monitoring

    Monitor metrics like request counts, retry frequency, errors behind retries etc to optimize configurations.

    Handling Circular Retries

    Beware infinite loops if reused errors re-trigger interceptors! Have fallback termination logic as shown earlier via max counters.

    FAQs

    Why use Axios vs Fetch API?

    Axios provides automatic JSON parsing, support for older browsers, and richer error handling out-of-the-box compared to the Fetch API.

    Does Axios require backend/server code?

    No, Axios is a frontend HTTP client library that does not need corresponding server code. You make requests to external APIs and services.

    Why is Axios widely used in React apps?

    Axios integrates seamlessly with React since it is promise-based. It also avoids opt-in async configurations needed for Fetch.

    Is Axios only meant for REST APIs?

    No, Axios can call any HTTP backend - REST, RPC, GraphQL etc. It simply helps make HTTP calls and handle responses.

    Should all requests be retried by default

    No, blindly retrying non-idepotent requests for example can cause data corruption. Having smart conditional retry logic is important.

    So in summary, automated retries form an integral safety net when building robust apps dealing with remote services. Leverage tools like Axios interceptors and plugins to transparently bake resilience against failures, without expensive manual intervention.

    I hope walking through real-world examples and tips from my web scraping experience helps you be better equipped to handle crashes...because code never lies, but APIs always fail!

    Browse by tags:

    Browse by language:

    Tired of getting blocked while scraping the web?

    ProxiesAPI handles headless browsers and rotates proxies for you.
    Get access to 1,000 free API credits, no credit card required!