Skip to main content

Parallel Concurrent Requests in Python

You’d think that the fastest way to make parallel network requests would be to use asyncio, but it’s actually concurrent.futures.ThreadPoolExecutor. I’ve known ThreadPools before as I worked with them in Java 6+ months ago, but I couldn’t find something similar in Python until yesterday. There is also a ProcessPoolExecutor, but I’d only suggest you use that for CPU-bound tasks. ThreadPoolExecutor is better since requests are network bound (and also because I encountered a jumbled error when I tried to use ProcessPoolExecutor. Here is a quick proof of concept using ThreadPoolExecutor.

I hope that the comments in the code above were enough but if not, please let me know.