Tom Nguyen

How I Learned to Handle Concurrency in Python


When I first started working with Python, I wrote my code in a very sequential way – one task at a time, waiting for each function to finish before moving on to the next. It worked fine for small scripts, but as my projects grew, I realized that waiting for I/O operations, API calls, or database queries was slowing down my programs.


I had heard about concurrency and parallelism, but to be honest, it felt intimidating at first. What’s the difference between threads, multiprocessing, and async programming? Which one should I use? After a lot of trial and error, I finally learned how to make Python programs run faster and more efficiently using concurrency.


In this blog, I’ll share my journey of learning concurrency in Python, the mistakes I made, and the best practices that helped me improve performance in my projects.

Understanding Concurrency vs. Parallelism

At first, I thought concurrency and parallelism were the same thing, but they are actually different.

  • Concurrency means handling multiple tasks at the same time by switching between them. It doesn’t necessarily mean they are running at the exact same moment.
  • Parallelism means running multiple tasks at the same time, typically using multiple CPU cores.

Python provides three main ways to achieve concurrency:

1. Threads (threading module) – Useful for I/O-bound tasks (like downloading files).

2. Multiprocessing (multiprocessing module) – Used for CPU-bound tasks (like image processing).

3. Async programming (asyncio module) – Best for handling many tasks that wait for I/O (like web scraping).


Using Threads for I/O-bound Tasks


The first time I used threading was when I needed to download multiple files in parallel. Initially, I wrote a script that downloaded files one by one:

import requests

urls = ["https://example.com/file1", "https://example.com/file2"]

def download(url):
    response = requests.get(url)
    print(f"Downloaded {url}")

for url in urls:
    download(url)


This worked, but each file had to finish downloading before the next one started, making it very slow.


Using Threads to Speed It Up


I updated the script to use threading, allowing multiple downloads to happen at the same time:

import threading
import requests

urls = ["https://example.com/file1", "https://example.com/file2"]

def download(url):
    response = requests.get(url)
    print(f"Downloaded {url}")

threads = []
for url in urls:
    thread = threading.Thread(target=download, args=(url,))
    thread.start()
    threads.append(thread)

for thread in threads:
    thread.join()


This approach made downloads much faster because while one thread was waiting for a network response, another thread could start downloading.


However, Python’s Global Interpreter Lock (GIL) means that threads cannot run Python code at the exact same time, making them inefficient for CPU-intensive tasks.


Using Multiprocessing for CPU-bound Tasks


I first realized that threading was not enough when I had to process thousands of images in a project. The program was running too slowly, even with multiple threads. After researching, I learned that CPU-heavy tasks need multiprocessing instead of threading.


Why?

  • Python’s GIL allows only one thread to execute Python bytecode at a time.
  • The multiprocessing module creates separate processes, each with its own Python interpreter, allowing them to truly run in parallel using multiple CPU cores.


How I Used multiprocessing


I modified my image processing script to use multiple processes:

from multiprocessing import Pool
from PIL import Image

image_files = ["image1.jpg", "image2.jpg", "image3.jpg"]

def process_image(image_file):
    img = Image.open(image_file)
    img = img.resize((100, 100))
    img.save(f"resized_{image_file}")
    return f"Processed {image_file}"

if __name__ == "__main__":
    with Pool() as pool:
        results = pool.map(process_image, image_files)
    
    print(results)


With multiprocessing.Pool, the images were processed in parallel, and the performance improved dramatically.

When to Use Multiprocessing

  • If your task is CPU-bound (e.g., image processing, video encoding, heavy computations).
  • If you need to fully utilize multiple CPU cores.

However, creating multiple processes takes more memory than threads, so it’s not always the best solution.

Using Asyncio for High-Performance I/O


The real turning point in my understanding of concurrency came when I had to scrape thousands of web pages. At first, I used threading, but then I learned about async programming with asyncio, which made my scraper significantly faster.


How I Used asyncio for Web Scraping


At first, my scraper fetched pages one by one, waiting for each request to complete before starting the next one:

import requests

urls = ["https://example.com/page1", "https://example.com/page2"]

def fetch(url):
    response = requests.get(url)
    return response.text

for url in urls:
    html = fetch(url)
    print(f"Fetched {url}")


This approach was too slow, so I rewrote it using asyncio and aiohttp:

import asyncio
import aiohttp

urls = ["https://example.com/page1", "https://example.com/page2"]

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = [fetch(session, url) for url in urls]
        results = await asyncio.gather(*tasks)
    
    for url, html in zip(urls, results):
        print(f"Fetched {url}")

asyncio.run(main())


This version was much faster because it didn’t wait for each request to finish before starting the next one. Instead, it scheduled all requests asynchronously, allowing multiple downloads to happen in parallel.


When to Use asyncio

  • If your task is I/O-bound (e.g., web scraping, database queries, file operations).
  • If you need to handle thousands of tasks efficiently.


Lessons Learned

When I started learning concurrency, I tried using threads for everything, thinking it would make my programs faster. Over time, I realized that choosing the right concurrency model depends on the type of task:

  • Use threading for I/O-bound tasks with moderate concurrency (e.g., file downloads, web requests).
  • Use multiprocessing for CPU-bound tasks where performance depends on using multiple cores (e.g., image processing).
  • Use asyncio for high-performance I/O when handling thousands of tasks efficiently (e.g., web scraping, database queries).

After applying these techniques, my Python programs ran significantly faster and more efficiently. Concurrency in Python may seem complex at first, but once you understand the differences between threads, processes, and async programming, it becomes a powerful tool for optimizing performance.

Happy coding!