Concurrency vs. Parallelism: What You Need To Know For Effective Web Scraping

If you’re not a programmer, then understanding the difference between concurrency and parallelism might make very little since. After all, they both refer to multiple things running at exactly the same time.

Welcome to programming! We take words you know, like language, library, headless and give them new meanings 😉

If you are a seasoned programmer, then concurrency and parallelism should not be strangers to you. In that case, you can cut to the end, where we discuss parallel programming and concurrent programming in the context of web scraping.

Need Some Proxy Support?

Don’t get banned from using the same IP – use our range of quality proxies to scrape with ease!

Take A Look

But for those that are newer to all of this, let’s start from the beginning. Read on to get to the core – or multiple cores – of the problem…

A Note on Threads & Multiple Tasks

When talking about tasks, we should break them down into threads. A thread is the smallest element of a given task that can be handled on its own. In other words, it doesn’t have dependencies on other parts of the same task and can therefore be implemented. Any action or task is typically broken down into multiple instruction sequences like this.

We bring this up because, whether you’re working on parallel computing or concurrent systems, operating systems, programming languages and computers in general work on threads. We might refer to concurrency vs parallelism in the concept of handling a minimum of two tasks, but underneath there are multiple threads.

What is Concurrency in Programming?

Concurrent programming is the process of executing multiple threads by pausing and resuming each so that only one thread is ever active.

Put it another way, have you heard the argument that the human brain can’t multitask? What it can do is switch quickly from one task to another, so the end result is as if you were managing multiple tasks simultaneously. And this context switching is exactly how concurrent tasks work.

As a simple rule you can remember that one CPU core can do one thing at a time. Concurrent programming is when you make the most of modern CPU speeds to complete multiple tasks by shifting between them, one task at a time.

The end result is concurrent programming allows for managing threads in overlapping time periods, but it doesn’t mean those threads are being actively being worked on at the same time.

What is Parallelism in Programming?

Parallel tasks are much easier to understand. If you have more than one CPU core, then you can dedicate threads to each core. In other words, you can launch and execute tasks simultaneously, because you’re no longer switching between threads.

Of course, this requires multiple processing units and is therefore limited by your hardware capabilities. Modern computers feature multi core processors as standard, but its not infinite. Parallel programming, therefore, isn’t just a magic solution. You need to pair your multiple CPU cores with a big dose of common sense 😉

The benefits of Parallel Execution

Obviously, being able to do more, faster, is a huge advantage. But this also means that parallel processing simple scales well.

Need to two tasks? With just one central processing unit, you’re stuck with concurrent task execution. Parallel computing is, from a purely mathematical perspective, a great time saver and enabler.

The Difference Between Parallelism and Concurrency

Obviously, if you read the above definitions of concurrency and parallelism, you know the former only has one task active at any given point, where as parallel programming can have more than one task due to the use of multiple CPUs.

But since we don’t live by dictionary definitions, let’s look at the more technical details resulting from the use of concurrency and parallelism in your web scraping projects.

Time

While the end results might look the same from a “stuff got done, didn’t it?” perspective, the big difference between concurrency and parallelism is time. If you have multiple tasks, concurrent execution will manage the threads one at a time. The end time, then, is the total time of all tasks.

On the other hand, when managing multiple tasks via a parallel program, you can have two or more processes simultaneously.

So, take the total time of all your tasks, but this time split the threads into two piles, as even as possible. That’s the time difference between concurrent and parallel execution.

And needless to say, more cores lets you handle multiple requests, saving more time…

Yet, when running simple tasks, you’re probably saving incredibly small amounts of time. But when you’re doing repetitive tasks, such as large scale web scraping, for example, then resource utilization certainly becomes more important.

Multiple Cores vs Cost

Concurrent systems only need one CPU core, so therefore its often the easiest process to implement at the beginning. But once you start doing multiple tasks at large, you generally want multiple processors.

The problem with those multiple cores, however, is that it’s a hardware cost. You need to ensure you have the cores that you need. Whether you’re using a personal laptop or renting out server space, this is an additional cost to consider with parallel processing.

Using Parallelism and Concurrency Together

Now you know the advantages and drawbacks of concurrency vs parallelism, let’s talk about the obvious answer: using a bit of both.

We’re all doing multiple tasks at any given point, and web scraping is no different. By using concurrent and parallel programming together, you can still get the most simultaneous execution for your CPU buck.

As we’ve already mentioned, even with multiple CPUs in play, processing capacity is always going to be at a premium. Therefore, parallel programming across multiple cores is still effective, but you can further utilize context switching and concurrent programming on each.

Even with the increasing availability of multiple CPU cores, there will always be this upper limit, so efficient resource utilization is the name of the game. Breaking down processes into smaller sub tasks and putting different tasks through a combined parallel concurrent execution will likely always be the best way to get the most optimal results at scale.

Concurrent and Parallel Programming in Web Scraping

Now we’ve likely bored you with talk on multiple processors, multiple tasks, parallel programming and concurrent programs, let’s finally put it into the context of web scraping.

And since web scrapers always have more than one task on the go, concurrency vs parallelism is a very real challenge you’ll likely have to face. Maybe you’re simply scraping many pages at once, or maybe you need to better optimize multiple tasks related to data collection, data parsing and anything that even happens post-scrape.

We mention that last part because web scraping itself is part of a wider process. Let’s say, for example, that you’re providing some form of real time data analysis, such as brand monitoring, product reviews, price aggregation or a 101 other use cases we could talk about endlessly….

Parallel execution can be used to divide things as much as possible. Maybe you separate threads with scraping and parsing on one side, and analytics on the other? But what if your scraping tasks outweigh the analytics? Concurrent programming helps to further finetune your processes to ensure you’re getting the best of both – and as close to simultaneous execution as possible.

Don’t Forget Proxies!

Parallel & concurrent execution both help, but you need reliable proxies to scrape effectively.

View Our Proxies

That being said, we can also give you a few more pointers on how to finetune your task execution.

• Generally, I/O bound tasks are best left to concurrent programs, since they rely on requests. This way, you can have start with smaller sub tasks on only one CPU core.
• Parallel computation, on the other hand, can be useful for CPU-bound tasks that often occur after the data is extracted. This is because it uses much more of your own CPU potential.
• If you reach the limits of your hardware capabilities, this is where balancing concurrency and parallelism can help maximize your results. In other words, this is where you start to finetune the importance of sending multiple requests vs processing and utilizing the information previously gathered.

Which Programming Language Do You Use?

Concurrency and parallelism are essentials in software development, web scraping and other common coding purposes.

In Python – one of the most popular programming languages for web scraping – the terminology is quite clear. Threading will help ensure your threads are executed concurrently and, if you have separate processors or a multi core CPU, multiprocessing will help you develop a parallel program.

Below, we’ll provide some examples, but it’s worth nothing that we’re not just sticking to concurrency and parallelism. There are other tools available for managing multiple tasks and large workloads, so some of you have probably been screaming at us about load balancing and asynchronous operations.

We’ll be using both of those here – in fact, we’re no strangers to asynchronous web scraping 😉 – and we recommend using the following libraries and features in your web scraping endeavours.

Concurrency and Threading in Python

To better explain these key concepts, here is an example of how to use concurrent processes in Python:

import requests

from concurrent.futures import ThreadPoolExecutor

# List of URLs to scrape

urls = [

    "https://httpbin.org/get",

    "https://jsonplaceholder.typicode.com/todos/1",

    "https://jsonplaceholder.typicode.com/posts/1"

]

def fetch_url(url):

    response = requests.get(url)

    return f"URL: {url}, Status: {response.status_code}"

# Using ThreadPoolExecutor for concurrent execution

with ThreadPoolExecutor(max_workers=3) as executor:

    results = executor.map(fetch_url, urls)

# Print results

for result in results:

    print(result)

This particular example uses the ThreadPoolExecutor to run multiple requests. Remember, however, that concurrency focuses on multithreading tasks. They may have overlapping time periods, but will not run at the exactly the same time.

Parallelism and Multiprocessing in Python

Alternatively, when you have a multi core CPU or even separate cores at your leisure, we can turn to multiprocessing. This is fairy simple, but you should start by both knowing how many cores you have, and likewise informing Python of how many cores you intend to use.

You can start by checking your system’s CPU count.

import multiprocessing

print(f"Number of CPU cores available: {multiprocessing.cpu_count()}")

We can then use multiprocessing for simultaneous execution. In this example, we’re using fetch requests and have the option of limiting the number of CPU cores.

import requests

import multiprocessing

# List of URLs to scrape

urls = [

    "https://httpbin.org/get",

    "https://jsonplaceholder.typicode.com/todos/1",

    "https://jsonplaceholder.typicode.com/posts/1",

    "https://jsonplaceholder.typicode.com/posts/2",

    "https://jsonplaceholder.typicode.com/posts/3"

]

def fetch_url(url):

    """ Fetch a webpage and return the status code. """

    response = requests.get(url)

    return f"URL: {url}, Status: {response.status_code}"

if name == "__main__":

    with multiprocessing.Pool(processes=4) as pool:  # Use 4 CPU cores

        results = pool.map(fetch_url, urls)  # Run the function in parallel

    

    for result in results:

        print(result)

The key takeaways here are the use of multiprocessing.Pool and pool.map – the former creates a pool of worker processes, running on seperate cores. In our example, we implemented 4 processes, which are running on 4 different cores.

The latter – pool.map – distributes the URLs evenly across these processes. Together, these provide distributed systems for effective parallel scraping.

Alternatively, if you just want to hit the ground running, you can configure python to use all available cores, rather than having to set a specific count.

if name == "__main__":

    num_cores = multiprocessing.cpu_count()  # Get CPU core count

    with multiprocessing.Pool(processes=num_cores) as pool:

        results = pool.map(fetch_url, urls)

    for result in results:

        print(result)

Combining Concurrency and Parallelism in Python

Now let’s combine concurrent and parallel programming. Here, we’ve also going to use asyncio and aiohttp (as we want to fetch multiple pages without getting blocked), so make sure you also have these dependencies installed.

First, let’s implement a very simple asynchronous web scraper

import aiohttp
import asyncio

async def fetch_url(session, url):
    """Fetch a URL asynchronously."""
    async with session.get(url) as response:
        return f"URL: {url}, Status: {response.status}"

async def scrape_urls(url_list):
    """Handle multiple async requests."""
    async with aiohttp.ClientSession() as session:
        tasks = [fetch_url(session, url) for url in url_list]
        return await asyncio.gather(*tasks)  # Run all tasks concurrently

Next, we an use multiprocessing to launch multiple scrapers in parallel.

import multiprocessing

# Define multiple sets of URLs
url_batches = [
    ["https://httpbin.org/get", "https://jsonplaceholder.typicode.com/todos/1"],
    ["https://jsonplaceholder.typicode.com/posts/1", "https://jsonplaceholder.typicode.com/posts/2"],
    ["https://jsonplaceholder.typicode.com/posts/3", "https://httpbin.org/ip"]
]

def run_scraper(url_list):
    """Runs an async scraper in a separate process."""
    results = asyncio.run(scrape_urls(url_list))  # Run the async function
    for result in results:
        print(result)

if __name__ == "__main__":
    num_cores = min(multiprocessing.cpu_count(), len(url_batches))  # Use available CPU cores efficiently

    with multiprocessing.Pool(processes=num_cores) as pool:
        pool.map(run_scraper, url_batches)  # Run multiple scrapers in parallel

Here we’re using asyncio and aiohttp to to perform multiple tasks concurrently, whilst multiprocessing ensures different instants are running across different CPU cores. We’re likewise using url_batches to ensure each process gets its own set of URLs to scrape – this is a basic approach to ensure we don’t duplicate unnecessary tasks.

This is a basic AF example, but hopefully you’re seeing how it works, and can already apply the same concepts to your own particular projects.

When executed, you’ll see each scraper running and returning requests:

URL: https://httpbin.org/get, Status: 200
URL: https://jsonplaceholder.typicode.com/todos/1, Status: 200
URL: https://jsonplaceholder.typicode.com/posts/1, Status: 200
URL: https://jsonplaceholder.typicode.com/posts/2, Status: 200
...

The TL;DR on Concurrency vs Parallelism

For any programmer, resource utilization and allocation is always a challenge. Web scraping is a perfect example: you might have multiple CPUs, but obviously never enough to provide simultaneous execution across the entire board.

For large scale web scraping projects, you’ll likely need to combine concurrency and parallelism to get the best results for the resources at hand. You’re never going to have the perfect hardware, so making the most of multi core systems and separate processers is key to efficient web scraping.

Know when to multithread, be sure to favor parallel systems for CPU bound tasks, and maximize your web scraping potential!

Need Some Proxy Support?

Don’t get banned from using the same IP – use our range of quality proxies to scrape with ease!

Take A Look

The information contained within this article, including information posted by official staff, guest-submitted material, message board postings, or other third-party material is presented solely for the purposes of education and furtherance of the knowledge of the reader. All trademarks used in this publication are hereby acknowledged as the property of their respective owners.