Fixing aiohttp UnicodeDecodeErrors

Mar 3, 2024 ยท 2 min read

When building asynchronous web applications and APIs with Python's aiohttp library, you may occasionally run into UnicodeDecodeErrors when handling request or response bodies. These errors indicate there was an issue decoding bytes to text, often due to incorrect character encodings.

Here are some tips on fixing aiohttp UnicodeDecodeErrors:

Specify an Encoding

Many times the error occurs because no encoding was specified when converting bytes to text. Be explicit by passing the encoding to methods like response.text() and request.text():

response_text = await response.text(encoding='utf-8')
request_text = await request.text(encoding='utf-8')

'utf-8' is a safe encoding to try first.

Check the Actual Encoding

To handle cases where the encoding is unknown, check the response headers to determine the declared encoding first:

encoding = response.headers.get('Content-Type', 'utf-8')
text = await response.text(encoding=encoding)

This parses the value of the 'Content-Type' header to extract the encoding.

Decode Manually

For more control, manually decode the bytes using the known encoding:

data = await
text = data.decode('utf-8', errors='replace')

The errors='replace' parameter handles invalid UTF-8 characters gracefully.

Re-Encode the Text

If decoding fails, try re-encoding the text as UTF-8 first before decoding:

data = await
data = data.encode('UTF-8', 'replace').decode('UTF-8')

This replaces invalid UTF-8 sequences before decoding the bytes.

Carefully handling encodings and using the right decoding errors parameters is key to avoiding UnicodeDecodeErrors in aiohttp. Pay attention to character encodings on requests and responses to choose the correct decoding strategy.

Browse by tags:

Browse by language:

The easiest way to do Web Scraping

Get HTML from any page with a simple API call. We handle proxy rotation, browser identities, automatic retries, CAPTCHAs, JavaScript rendering, etc automatically for you

Try ProxiesAPI for free

curl ""

<!doctype html>
    <title>Example Domain</title>
    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />


Don't leave just yet!

Enter your email below to claim your free API key: