The aiohttp library is popular for making asynchronous HTTP requests in Python. Its async/await syntax helps you write non-blocking code that can handle many requests concurrently.
However, sometimes you need to integrate aiohttp with synchronous code or external libraries. This article covers different techniques to bridge the gap:
The run_in_executor() Method
The easiest way is using
import aiohttp
from concurrent.futures import ThreadPoolExecutor
async def fetch(url):
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.text()
def sync_fetch(url):
loop = asyncio.get_event_loop()
result = loop.run_until_complete(
aiohttp.ClientSession().run_in_executor(ThreadPoolExecutor(), fetch, url))
return result
This avoids blocking the event loop thread while running synchronous code. The downside is managing the thread/process pools yourself.
The asyncio.to_thread() Function
import asyncio
import aiohttp
async def fetch(url):
async with aiohttp.request('GET', url) as response:
return await response.text()
def sync_fetch(url):
return asyncio.get_event_loop().run_until_complete(
asyncio.to_thread(fetch, url))
Less code, but you have less control over the thread pool. Useful for simple cases.
Running an Event Loop in a Thread
You can run the entire asyncio event loop in a background thread:
import asyncio
import threading
async def main():
async with aiohttp.request('GET', url) as resp:
print(await resp.text())
thread = threading.Thread(target=asyncio.run, args=(main(),))
thread.start()
thread.join()
This keeps async code isolated and non-blocking. But uses more resources as the event loop runs separately.
The nest_asyncio Decorator (Python 3.7+)
The
import nest_asyncio
nest_asyncio.apply()
import aiohttp
def sync_fetch(url):
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
result = loop.run_until_complete(aiohttp.request('GET', url))
return result.text()
This allows running asyncio code synchronously by creating a new event loop. But has downsides around resource usage.
Key Takeaways
The ideal approach depends on your specific use case - whether you want to call async code from a sync program or vice versa. Measure resource usage and performance to pick the right method.
Related articles:
- Speed Up Your Asyncio Code with Thread Pools
- Making Asynchronous HTTP Requests in Python with aiohttp Connectors
- Fixing the "ImportError: No Module Named aiohttp" Error in Python
- Troubleshooting aiohttp ServerDisconnectedError
- Fetching Content with aiohttp in Python
- Making the Most of aiohttp's TCPConnector for Asynchronous HTTP Requests
- Efficiently Handling Data with aiohttp in Python
Browse by tags:
Browse by language:
Popular articles:
- Web Scraping in Python - The Complete Guide
- Working with Query Parameters in Python Requests
- How to Authenticate with Bearer Tokens in Python Requests
- Building a Simple Proxy Rotator with Kotlin and Jsoup
- The Complete BeautifulSoup Cheatsheet with Examples
- The Complete Playwright Cheatsheet
- Web Scraping using ChatGPT - Complete Guide with Examples