Using Scrapy Middlewares

What are Scrapy Middlewares?

Scrapy Middlewares are hooks into the Scrapy processing pipeline, allowing developers to inspect, modify, or create custom logic for requests and responses. They act as a layer between the engine and the spiders, providing the ability to alter data before it reaches the spider or after it’s returned. Middlewares can handle essential tasks such as:

• Rotating Proxies: Ensures you don’t get blocked by changing IPs for each request.
• Error Handling: Manages failed requests, retries, or logs them for later review.
• Custom Header Modifications: Modifies request headers to mimic different user-agents, making your requests appear more like those of real users.

Built-In Scrapy Middlewares

Scrapy provides several built-in middlewares that simplify common tasks:

• RetryMiddleware: Automatically retries requests that failed due to connection issues or response codes like 500. This helps maintain a higher success rate for your scraping.
• UserAgentMiddleware: Sets or rotates custom User-Agent headers, allowing your scraper to mimic different browsers and avoid detection.
• RobotsTxtMiddleware: Ensures compliance with robots.txt files on target sites, respecting their rules on web scraping.

Using these built-in middlewares, Scrapy can automatically manage many scraping tasks that would otherwise require manual handling.

Writing Custom Middlewares

Sometimes built-in middlewares aren’t enough, and you need to add your own. Custom middlewares let you implement functionality specific to your project. Here’s an example of a middleware that rotates User-Agent headers, helping avoid blocks by simulating various browser profiles.

import random

class RotateUserAgentMiddleware:
    def __init__(self, user_agents):
        self.user_agents = user_agents

    @classmethod
    def from_crawler(cls, crawler):
        return cls(user_agents=crawler.settings.get('USER_AGENTS'))

    def process_request(self, request, spider):
        request.headers['User-Agent'] = random.choice(self.user_agents)

Explanation:

• from_crawler: Initializes the middleware, retrieving a list of User-Agent headers from settings.py.
• process_request: Chooses a random User-Agent header for each request, making your scraper less predictable and helping avoid detection.

To use this middleware, add a USER_AGENTS list in your settings.py:

USER_AGENTS = [
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36",
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15",
    "Mozilla/5.0 (iPhone; CPU iPhone OS 14_0 like Mac OS X) AppleWebKit/605.1.15"
]

Inserting Middlewares in Scrapy

Once you’ve defined your custom middleware, you’ll need to activate it by adding it to the DOWNLOADER_MIDDLEWARES or SPIDER_MIDDLEWARES settings in settings.py. You can also set priorities to control the processing order.

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.RotateUserAgentMiddleware': 400,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
}

Explanation:

• Priority: The number (e.g., 400) defines the middleware’s priority. Lower numbers are processed first, allowing you to prioritize custom processing over built-in functionality.

Use Cases for Middlewares

Middlewares are highly versatile and can adapt to specific scraping challenges:

1. Rotating Proxies: To avoid IP-based blocking, you can set up a middleware that assigns a new proxy for each request:

class ProxyMiddleware:
    def process_request(self, request, spider):
        proxy = "http://proxy_ip:proxy_port"
        request.meta['proxy'] = proxy

2. Error Handling: You might use a middleware to log and retry requests that fail, improving your scraper’s resilience. Here’s a simple example:

class RetryOnFailureMiddleware:
    def process_response(self, request, response, spider):
        if response.status != 200:
            spider.logger.warning(f"Request failed with status {response.status}")
            return request
        return response

This middleware logs failed requests and retries them, which is particularly useful for handling temporary server issues.

Debugging and Testing Middlewares

Debugging custom middlewares is essential for ensuring they work as expected. Add logging statements to verify that each middleware step functions properly, or use Scrapy’s shell to test requests manually.

class DebuggingMiddleware:
    def process_request(self, request, spider):
        spider.logger.info(f"Request Headers: {request.headers}")
        return None

This middleware logs the headers of each request, which is helpful for verifying that headers are set correctly.

‍