A Complete Guide for Web Crawlers Using Python

Web crawling isn’t new: it’s been around since 1994 when Brian Pinkerton developed the first web crawler to improve search engine performance. Known as “WebCrawler,” this full-text crawler-based Web search engine was an early milestone for search engines. Today, it is a dependable technology for extracting data from websites.

Web crawlers are an invaluable tool for collecting data from the internet and are popular among researchers, businesses, and developers. With crawlers, you can easily gather large amounts of data and use it for various purposes, like analytics, website automation and content evaluation.

As a result, web crawlers are becoming increasingly important for businesses that want to stay competitive in the digital age. Nonetheless, it can be challenging to set up a web crawler, especially for users who are unfamiliar with coding. With Python, you can easily use and create web crawlers with a few lines of code.

This guide will provide an introduction to web crawlers and crawling the web with Python. You’ll learn about the different components of crawler systems and the key crawler web python libraries to use. Finally, you’ll learn how to use crawlers to extract valuable data from a website.

Web Crawling vs. Web Scraping

While web crawling is often used interchangeably with web scraping, they are actually two distinct processes. Web crawlers are used to explore websites by automatically visiting web pages and extracting links from the source code so that other pages can be discovered. This is often done to build a complete website map or collect data from multiple pages.

On the other hand, web scraping involves extracting specific data from a page and collecting it in an organized manner. Web scrapers are designed to identify and extract data based on a set of instructions. When compared, crawlers explore websites, while scrapers target specific pages and collect data from them.

How web crawlers work

The basic workflow of a web crawler is relatively straightforward and typically consists of the following four steps:

• Send an HTTP request to a web server to retrieve the page’s source code.
• Parse the HTML and extract all links from the page.
• Queue the links for further crawling or scrape the page for data.
• Store the scraped data and repeat the process with each new link.

Crawling the Web With Python

Python is one of the most popular programming languages for web crawlers due to its simplicity and rich ecosystem. Crawling the web with Python is easy. You just need to define the Python data crawler’s behavior and structure, set up a crawler object and launch the crawler. You can also use crawlers to automate website tasks like collecting data or checking for broken links.

Python crawlers are typically built using libraries like Scrapy, BeautifulSoup, and Selenium. All of these libraries are free to use and offer comprehensive crawler development features. Let’s discuss these libraries in detail so you can decide which one is right for your crawler project.

Scrapy

Scrapy is an open-source Python crawler framework that was designed for web scraping. Scrapy is easy to use and can be set up quickly with just a few lines of code. It is ideal for larger crawler projects and offers features like:

• Crawler scheduling
• Data extraction
• Crawler monitoring
• Crawler optimization

Pros

• It’s open source
• Scalable crawler framework
• Built-in crawler features
• Easy to debug and maintain crawlers

Cons

• Doesn’t support crawling with JavaScript (use Selenium instead)
• It is not ideal for small crawler projects because it contains heavier code
• Complex and hard-to-understand documentation for beginners

BeautifulSoup

BeautifulSoup is another widely used crawler web Python library. It is designed to parse HTML and XML documents and make crawler development more efficient. It can also be used to scrape data from web pages and extract specific information. Key features of BeautifulSoup include:

• Crawler management
• Data extraction
• HTTP authentication
• Crawler optimization

Pros

• Easy to learn and use
• Rich crawler features
• Lightweight crawler framework
• Accessible and easy-to-understand documentation for beginners

Cons

• It is not suitable for large crawler projects
• It cannot automatically execute crawlers

Selenium

Selenium is an open-source crawler web Python library designed for browser automation. It can be used to control a web browser and execute crawler scripts. Selenium is ideal for crawlers that require complex interactions with web pages. Its features include:

• Browser control
• Script execution
• Data extraction
• Crawler optimization

Pros

• It supports crawlers with JavaScript
• Accessible API for crawler development
• It can be used to automate complex interactions with web pages

Cons

• It can be difficult to learn and use
• It is not ideal for crawlers with large amounts of data

Environmental Preparation for Web Crawling

Before building crawlers using Python, you must set up the crawler’s environment. This involves the following steps:

• Creating a Virtual Environment: A virtual environment is a safe sandbox where crawlers can be tested and debugged without affecting other applications on the same computer. Creating a virtual crawler environment is easy and only takes a few minutes. All you need to do is set up a crawler folder, activate the environment and install crawler web python libraries.
• Install a Browser: Google Chrome is the most widely used and supports crawler automation. You can also use other browsers like Firefox, Edge or Safari.
• Installing Python: Next, you must install Python on your computer. Fortunately, installing Python is easy and only takes a few minutes. You can either install the latest version of Python or use an existing version on your computer.

Building a Web Crawler With Python and Scrapy

Scrapy is the most popular crawler web Python framework. With over 45K stars on GitHub, it is the go-to crawler library for many developers. One of its greatest features is its ability to schedule requests asynchronously. Scrapy crawlers can fetch many pages in parallel, and its throttling settings keep the load on each web server in check. Moreover, Scrapy crawlers are highly modular and easily customized depending on your crawler project’s requirements.

Here’s a step-by-step guide to creating crawlers with Scrapy. The project for this crawler web Python tutorial will crawl data from a news website.

1. Install Scrapy

Before you can start creating crawlers, you’ll need to install the Scrapy library. You can do so with a simple command as follows:

1

pip install scrapy

2. Create a crawler project

Once you’ve installed Scrapy, you can create crawler projects. Scrapy crawlers are built using spider classes and require only a few lines of code to get started. To set up a crawler project, type the following command in the terminal:

1

scrapy startproject <project_name>

In this case, you should type:

1

scrapy startproject newswebsite

This will create a newswebsite directory with this layout:

1
2
3
4
5
6
7
8

newswebsite/
    scrapy.cfg
    newswebsite/
        items.py
        middlewares.py
        pipelines.py
        settings.py
        spiders/

In the spiders directory, create a file called newswebsite.py, where your crawler code will reside, and open it in a code editor.

3. Write crawler code

When using Python to crawl websites, you must define a spider class that crawls data from the news website. To create a spider, you’ll need to use the crawler web python libraries for Scrapy.

In this example, you’ll create a crawler that crawls the headlines from a news website. You can define your crawler as follows:

1
2
3
4
5
6
7
8
9
10

import scrapy


class NewsWebsiteSpider(scrapy.Spider):
    name = 'news_website'
    start_urls = ['http://example.com/']

    def parse(self, response):
        for headline in response.css('h1::text').getall():
            yield {'headline': headline.strip()}

This crawler collects the headlines from the example.com home page. While this crawler is ready to run, it’s not yet finished: it only reads the pages listed in start_urls. To crawl a whole section of the site, make it a CrawlSpider. allowed_domains keeps it on one site, and a rule tells it which links to follow. Replace the spider with this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule


class NewsWebsiteSpider(CrawlSpider):
    name = 'news_website'
    allowed_domains = ['example.com']
    start_urls = ['http://example.com/news/']

    # Follow every link under /news/ and pass each page to parse_item
    rules = (Rule(LinkExtractor(allow=r'/news/'), callback='parse_item', follow=True),)

    def parse_item(self, response):
        for headline in response.css('h1::text').getall():
            yield {'headline': headline.strip(), 'url': response.url}

This crawler now follows every link under example.com/news/ and collects the headlines of each page it finds. A CrawlSpider uses the parse method itself, which is why the callback has its own name, parse_item.

4. Add a proxy

A proxy keeps your own IP address out of the logs of every site your Python crawler visits. It also keeps the crawler running, because crawlers get blocked when one address requests the same web page too often. To prevent this, crawlers should use a proxy server that can provide them with a new IP address each time they access the same web page.

With so many types of proxies available, you’ll need to select one suitable for web Python crawlers. You can use a rotating ISP proxy for this crawler web Python tutorial. Rotating ISP proxies suit web Python crawlers because they provide a new IP address on each request.

Rayobyte’s rotating ISP proxies suit crawler projects like this one. Each request goes out from a new IP address, and your crawler signs in with a username and password. For crawls that need city or ZIP targeting, use Premium Residential.

To send a spider’s requests through the proxy, set it on each request. Scrapy’s built-in HttpProxyMiddleware reads it, credentials included:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

import scrapy

# Your proxy endpoint and credentials, from the Rayobyte dashboard
PROXY = 'http://USERNAME:PASSWORD@PROXY_HOST:PORT'


class NewsWebsiteSpider(scrapy.Spider):
    name = 'news_website'
    start_urls = ['http://example.com/news/']

    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={'proxy': PROXY})

    def parse(self, response):
        for headline in response.css('h1::text').getall():
            yield {'headline': headline.strip()}

A CrawlSpider also makes requests of its own for the links it follows. To route every request through the proxy, set the proxy in the environment before you run the crawler instead:

1
2

export http_proxy='http://USERNAME:PASSWORD@PROXY_HOST:PORT'
export https_proxy="$http_proxy"

5. Run the crawler

After setting up your crawler web python spider, you can now run it. To do so, type the following command in the terminal:

1

scrapy crawl <spider_name> -o <output_file>.csv

In this case, run it from the newswebsite project directory:

1

scrapy crawl news_website -o headlines.csv

This crawler web Python crawler will now start crawling data from the example.com website and store the scraped data in headlines.csv.

6. Implementing the web crawler

For instance, you are crawling a news website such as BBC for a specific author name such as ‘John Doe.’ You can implement the web crawler as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

import scrapy


class BbcNewsSpider(scrapy.Spider):
    name = 'bbc_news'
    allowed_domains = ['bbc.com']
    start_urls = ['https://www.bbc.com/news']

    def parse(self, response):
        for link in response.xpath('//a/@href').getall():
            # Author pages carry the author's name in the URL
            if 'author' in link and 'john-doe' in link:
                yield response.follow(link, callback=self.parse_author)

    def parse_author(self, response):
        for article in response.xpath('//h3/a/text()').getall():
            yield {'title': article.strip()}

This crawler will crawl the home page of BBC news and collect all the links related to author ‘John Doe.’ It will then request each link and parse the response for article titles. Finally, it will store the scraped data in a list and yield it to the output.

Building a Web Crawler With Python and Beautiful Soup

BeautifulSoup is another widely used crawler web Python library. It’s an HTML parser that can be used to parse HTML and XML documents. Unlike Scrapy, Beautiful Soup is designed for small crawlers and doesn’t offer as many features. Nevertheless, it can be used to build basic web crawlers quickly.

Here’s a step-by-step guide to building a web crawler with Python using Beautiful Soup. The crawler project for this tutorial will be a simple data-crawling Python crawler that extracts price information from a website such as Amazon.

1. Install BeautifulSoup and requests

Before you can start building crawlers, you’ll need to install the Beautiful Soup library. You can do so with a few simple commands as follows:

1

pip install beautifulsoup4

You’ll also need the requests library. To install it, type the following command in the terminal:

1

pip install requests

2. Write crawler code

Once you’ve installed the necessary libraries, you can write crawler web python code. You can start by importing all the necessary libraries as follows:

1
2
3

import requests

from bs4 import BeautifulSoup

You’ll also need to define a few variables for the crawler web Python project. For this tutorial, you’ll need to set a target URL and a selector element.

The target URL is the web page that you want to crawl, while the selector element identifies the data that you want to extract from the web page. For this tutorial, you’ll set the target URL as https://www.amazon.com/ and use the selector span.a-price span.a-offscreen, where Amazon puts the price text. Amazon often answers plain scripts with a CAPTCHA page, so the code sends a browser User-Agent; for more than a few requests, route them through a proxy as shown at the end of this guide.

Now, you can write your crawler web Python code as follows:

1
2
3
4
5
6
7
8
9

url = 'https://www.amazon.com/'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0 Safari/537.36'}

response = requests.get(url, headers=headers, timeout=30)
soup = BeautifulSoup(response.text, 'html.parser')
prices = soup.select('span.a-price span.a-offscreen')

for price in prices:
    print(price.text)

This crawler web Python code downloads the HTML source code from https://www.amazon.com and extracts its price information. It then displays the extracted data in the terminal.

3. Implementing the web crawler

Once you’ve created your crawler, you can use it in a crawler web Python project. For instance, if you want to crawl Amazon for price information for red sneakers, you can implement the crawler as follows:

1
2
3
4
5
6
7
8
9
10
11
12

import requests
from bs4 import BeautifulSoup

url = 'https://www.amazon.com/s?k=red+sneakers'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0 Safari/537.36'}

response = requests.get(url, headers=headers, timeout=30)
soup = BeautifulSoup(response.text, 'html.parser')
prices = soup.select('span.a-price span.a-offscreen')

for price in prices:
    print(price.text)

This crawler web Python code downloads the HTML source code from https://www.amazon.com/s?k=red+sneakers and extracts the price information from it. It then prints out the extracted data in the terminal.

Because the code prints price.text, you get the text alone, such as $100.00. If you print the element itself (print(price)), you get its HTML instead: <span class="a-offscreen">$100.00</span>. To strip the tags from an HTML string like that, use a regular expression or let Beautiful Soup do it.

You can use a regular expression pattern as follows:

1
2
3
4
5

import re

html = '<span class="a-offscreen">$100.00</span>'
price = re.sub(r'<[^>]+>', '', html)
print(price)  # $100.00

Or let Beautiful Soup strip them with get_text():

1
2
3
4
5

from bs4 import BeautifulSoup

html = '<span class="a-offscreen">$100.00</span>'
price = BeautifulSoup(html, 'html.parser').get_text(strip=True)
print(price)  # $100.00

Building a Python Web Crawler With JavaScript Support Using Selenium

Crawling JavaScript webpages can be difficult because data on these pages load dynamically. Moreover, these pages require users to perform actions like page scrolling, form filling, and button clicking to access the data. This can be difficult to do with traditional web python crawlers. Selenium is a crawler web Python library that can automate these tasks and allow you to crawl JavaScript webpages. This is because Selenium let crawlers control web browsers and perform any action a user can.

Here’s a step-by-step guide on how to build a Python web crawler with JavaScript support using Selenium. The crawler project for this crawler web Python tutorial will be a crawler that crawls data from Quora. Quora is an excellent example of a website that may require a Python web crawler with JavaScript support.

1. Install Selenium and WebDriver Manager

Installing Selenium is easy and only takes a few minutes. All you need to do is type the following command in the terminal:

1

pip install selenium

You’ll also need to install WebDriver Manager, which is a crawler web python library for managing web drivers. You can install it with the following command:

1

pip install webdriver-manager

2. Import modules

You’ll need to import the necessary modules for crawler web python crawlers. To do so, type the following code in your crawler web python project:

1
2
3
4
5
6
7

import time

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from webdriver_manager.chrome import ChromeDriverManager

You’ll also need to add a few crawler optimization settings. For instance, to initialize a headless crawler, you can use the following code:

1
2
3
4

options = webdriver.ChromeOptions()
options.add_argument('--headless=new')

driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)

3. Write crawler code

Once you’ve imported all the necessary modules and crawler settings, you can write crawler web python code. You can start by setting the target URL for crawlers as follows:

1
2
3

url = 'https://www.quora.com/'

driver.get(url)

You’ll also need to define the crawler web python crawlers’ selector element. For this crawler project, you’ll use the selector class .answer to crawl data from Quora.

Now, you can write crawler web python code to crawl data from the Quora website. To do this, type the following code in your crawler web python project:

1
2
3
4

answers = driver.find_elements(By.CLASS_NAME, 'answer')

for answer in answers:
    print(answer.text)

This crawler web python crawler will now crawl data from the Quora website and print it out in the terminal.

4. Implementing the web crawler

Now, let’s look at the actual webpage you need to crawl; in this case, it’s a Javascript-rendered page. For example, if you want to crawl Quora for celebrity questions, you can add the following crawler web python code to your crawler web python project:

1
2
3
4
5
6
7
8
9
10

query = driver.find_element(By.NAME, 'q')
query.send_keys('celebrities')
query.send_keys(Keys.RETURN)
time.sleep(5)

answers = driver.find_elements(By.CLASS_NAME, 'answer')
for answer in answers:
    print(answer.text)

driver.quit()

Web Crawling With Python at Scale

When it comes to web crawling with Python, scaling is an important factor. Why? Because large websites like Google and others have millions of web pages. Plus, it’s impossible to crawl all of them in a reasonable amount of time while still being gentle and responsible on the website to avoid being blocked. You may need to adjust your crawler settings and use advanced scaling techniques to do this. Here are a few scaling settings you can apply when using Python to crawl websites:

• Regulate the number of concurrent requests.

By default, Scrapy sends up to 16 requests at once, and up to 8 to any one domain. That can still slow a small website down or make it unresponsive. To lower the load, set a smaller limit in your project’s settings.py:

1

CONCURRENT_REQUESTS_PER_DOMAIN = 4

• Set a crawler delay time.

Crawler delay time is the time a crawler web python spider waits between requests. This setting is important for scaling since it prevents websites from being overloaded with too many requests. To wait about one second between requests to the same site, add this setting:

1

DOWNLOAD_DELAY = 1  # seconds; Scrapy varies it between 0.5x and 1.5x by default

To let Scrapy adjust the delay to how fast the site responds, turn on AutoThrottle too:

1

AUTOTHROTTLE_ENABLED = True

• Specify the user agent.

The user agent tells websites what is making the request. Some websites block requests that send no user agent or a default library one. To name your crawler and give site owners a way to reach you, add this setting:

1

USER_AGENT = 'newswebsite (+https://www.example.com/contact)'

Build Python Web Crawler With Authentication With Rayobyte

While data crawling is a powerful tool for extracting data from websites, many websites have anti-scraping measures to prevent crawlers from accessing their data. Because of this, data-crawling Python spiders can be detected by websites and blocked or limited. You may, therefore, need to use proxies to prevent your crawlers from being blocked.

Rayobyte provides residential, ISP and datacenter proxies for web crawling with Python. You sign in to each one with a username and password, so it works from any HTTP library. With requests, it looks like this:

1
2
3
4
5
6
7
8

import requests

# Your proxy endpoint and credentials, from the Rayobyte dashboard
proxy = 'http://USERNAME:PASSWORD@PROXY_HOST:PORT'
proxies = {'http': proxy, 'https': proxy}

response = requests.get('https://checkip.amazonaws.com', proxies=proxies, timeout=30)
print(response.status_code)

In Scrapy, pass the same URL as each request’s proxy, as in step 4 above. Ready to get started? Start a free trial.

The information in this article is presented solely for the education of the reader. All trademarks used in this publication are hereby acknowledged as the property of their respective owners.