Curl 1020 error when trying to scrape page using bash script

Apr 2, 2024 · 3 min read

Web scraping is a useful technique for extracting data from websites to use programmatically. However, you may occasionally run into errors that prevent your scraper from accessing site content.

One common error is cURL error 1020, which indicates that your scraper cannot connect to the target server or page. In this guide, we’ll explore the causes of error 1020 when web scraping and provide fixes to resolve connection issues.

What Causes cURL Error 1020?

cURL error 1020 occurs when cURL, the data transfer library used by many web scrapers, fails to connect to the target web page or server. Some potential reasons you may get this error include:

  • Site blocking scrapers: Many sites actively block scraping bots via blocking rules or CAPTCHAs. If the site detects your scraper, it may block access.
  • Connection issues: Network problems, slow internet, or high site traffic could also prevent connections.
  • Incorrect URLs: An invalid or mistyped URL would cause cURL to fail locating the target page.
  • Authentication required: The site may require login credentials or cookies that your scraper lacks.
  • So in summary, error 1020 suggests your scraper cannot talk to the website - whether due to active blocking, technical issues, or missing credentials.

    5 Ways to Fix cURL Error 1020

    Luckily, there are a few approaches you can take to resolve error 1020:

    1. Check the URL

    Double check that the URL you are trying to scrape is valid. For example, if scraping a page on


    Correct any typos or protocol issues in the URL.

    2. Use a Browser User Agent

    Websites may detect and block common scraping user agents like curl. Specify a browser's user agent to appear like a normal visitor:

    curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36"

    3. Authenticate with Cookies

    If the site needs a login, you can pass saved cookies to authenticate:

    curl --cookie "session_id=1234; userId=5678"  

    4. Retry on Failure

    For intermittent issues, retry the request 2-3 times:

    while [ $RETRY -lt $MAX ]; do
      curl &> /dev/null
      if [ $? -eq 0 ]; then
      let RETRY=RETRY+1 

    This retries up to 3 times, waiting between each attempt.

    5. Use a Proxy or VPN

    If the site is actively blocking your server's IP range, you can route requests through a proxy or VPN to mask your origin.

    Proxies and VPNs provide an alternate IP to connect from. Just specify your proxy URL in cURL:

    curl --proxy http://1234.56.78.90:8080

    Wrap Up

    cURL error 1020 makes web scraping fail when the client cannot communicate with the website properly. Fixes like using browser agents, cookies, and proxies can help circumvent blocks to resolve the issue.

    Carefully check for typos, authentication requirements, or usage limits when running into 1020 errors. With the right approach, you can troubleshoot connection issues and get your scraper working again.

    Browse by language:

    The easiest way to do Web Scraping

    Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

    Try ProxiesAPI for free

    curl ""

    <!doctype html>
        <title>Example Domain</title>
        <meta charset="utf-8" />
        <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1" />


    Don't leave just yet!

    Enter your email below to claim your free API key: