How to Build a Reddit Scraper using Puppeteer and Nodejs

Reddit, known as ‘the front page of the internet,’ hosts millions of user-generated posts, comments, and votes across various topics and communities. In this tutorial, we’ll explore web scraping Reddit using JavaScript. You’ll learn how to extract post titles, comment threads, and upvote counts, allowing you to analyze trends, track discussions, or perform sentiment analysis across subreddits. This guide will provide the source code and tools you need to get started with web scraping Reddit efficiently.

You can find the complete source code for this project on my GitHub: GitHub Repo

• Table of Content

  • Introduction
  • What is Puppeteer
  • Prerequisites
  • Setting Up Puppeteer
  • Navigating and Executing Search Query
  • Getting More Data by Scrolling
  • Extracting all results links
  • Navigating to each post page
  • Extracting Relevant Post Data
  • Extracting Top Comments
  • Saving Data to a JSON file
  • Wrapping Things Up
  • Conclusion

Introduction

Reddit is a treasure trove of information, hosting discussions on every topic imaginable. Whether you’re looking to track the latest trends, monitor product reviews, or simply gather data for research, Reddit offers vast amounts of user-generated content. However, manually collecting this data can take time and effort. This is where web scraping comes in.

In this article, I’ll guide you through scraping Reddit using Puppeteer, a powerful browser automation tool. By the end, you’ll know how to navigate Reddit, execute search queries, extract post and comment data, and save the results in a structured format, all with minimal effort. Whether new to Puppeteer or looking to enhance your web scraping skills, this guide will equip you with the tools needed to scrape Reddit effectively.

What is Puppeteer

Puppeteer is a powerful Node.js library created by Google, designed to control Chrome or Chromium browsers through the DevTools Protocol. It provides developers with a high-level API for browser automation, making it ideal for tasks like web scraping, automated testing, and various browser-based workflows.

With Puppeteer, you can automate complex browser actions—such as navigating websites, interacting with elements, filling forms, and extracting data—without needing manual intervention. One of its standout features is its ability to run in headless mode, meaning it operates without a graphical interface, making it faster and more efficient for tasks like scraping.

Key Features of Puppeteer:

• Headless browser automation: Perform tasks without the browser UI for faster execution.
• Accurate page rendering: Scrape websites as they appear in an actual browser, ensuring access to up-to-date, rendered content.
• Flexible navigation control: Simulate user interactions, manage redirects, and navigate through dynamic web pages effortlessly.
• Network interception: Modify network requests and responses, handle API interactions, and capture server-side data.

Puppeteer’s versatility and reliability make it an excellent tool for scraping dynamic websites like Reddit, where content updates frequently and JavaScript is heavily involved.

Prerequisites

Before we dive into the code, let’s make sure you have everything set up. Here’s what you’ll need:

• Node.js: Puppeteer runs on Node.js, so make sure it’s installed on your machine. You can download it from here.
• Puppeteer: Puppeteer is a Node.js library that provides a high-level API for controlling Chrome or Chromium.

Installing Dependencies

First, let’s set up our project and install the required dependencies.

npm init -y

npm install puppeteer

Now that we have our environment ready, let’s move on to setting up Puppeteer.

With Puppeteer installed, let’s create the main script that will handle all our web scraping logic. 

In the root of your project, create a file named index.js.

Setting Up Puppeteer

To get started, we first need to require the Puppeteer library in our project:

const puppeteer = require('puppeteer')

Next, we’ll create an asynchronous function where all the scraping logic will live. This function will take a searchQuery as a parameter, which we’ll use to search for posts on Reddit:

async function run(searchQuery) {
   // All of our logic goes here!
}
run("Video Games")

Let’s bring the browser to life by using Puppeteer’s launch function to create a new browser instance:

const browser = await puppeteer.launch({
        headless: false // We want to see what's happening!
        defaultViewport: false
    })

The headless: false option keeps the browser UI visible so you can watch Puppeteer in action. It’s super helpful when you’re just getting started.

Next, we’ll create a new page where all our browser actions will take place:

const page = await browser.newPage()

Now your code should look something like this:

const puppeteer = require('puppeteer')
async function run(searchQuery) {
    const browser = await puppeteer.launch({
        headless: false,
        defaultViewport: false
    })
    const page = await browser.newPage()
}


run("Video Games")

You’ve set up everything, let’s see it in action! Run the project by typing the following command in your terminal: node  index.js

Once your project is up and running, it’s time to navigate to Reddit and start scraping some data. We’ll use Puppeteer to navigate to Reddit’s search page and execute the search query in the next section!

Navigating and Executing Seach Query

After setting up Puppeteer, one of the most important steps in web scraping is navigating to the correct page with the desired search results. For websites like Reddit, search functionality is usually handled via query parameters, which are additional pieces of information appended to the URL.

Let’s explore how to build the correct search URL for Reddit and perform the query dynamically based on user input.

What Are Query Parameters?

Query parameters are key-value pairs that follow a ? in a URL, and they allow us to pass data to the server as part of the request. For example, on Reddit, when you search for something, the URL might look like this:

https://www.reddit.com/search/?q=Video+Games

Here, the q parameter represents the search query, and the value of that parameter is Video Games. This is the string we want to manipulate programmatically.

How to Dynamically Add a Query Parameter

In our Puppeteer script, we want to take a search term (like “Video Games”) from the user and transform it into a valid query parameter that Reddit can understand. A common issue is that spaces in the search term need to be URL-encoded, meaning they should be replaced with + signs (or %20, but + is more common for search queries).

In the code, we handle this transformation as follows:

const searchQueryUrl = searchQuery.replace(' ', '+')

This line replaces all spaces in the searchQuery string with +, ensuring the URL is correctly formatted for Reddit’s search engine.

Building the Search URL

Now that we have the formatted search query, we can construct the full Reddit search URL by appending it as the q parameter in the URL:

const page = await browser.newPage()
    await page.goto('https://www.reddit.com/search/?q=' + searchQueryUrl, {
        waitUntil: "domcontentloaded"
    })

This does a few things:

1. Navigate to the URL: Puppeteer’s page.goto() method navigates the browser to the URL you specify.
2. Use the Query Parameter: The ?q= part of the URL is the query parameter that Reddit uses to search. We append our searchQueryUrl string to this, making the full URL something like https://www.reddit.com/search/?q=Nike+Air+Jordan.
3. Wait Until Content Loads: The waitUntil: “domcontentloaded” option tells Puppeteer to wait until the page’s DOM (Document Object Model) is fully loaded before continuing. This ensures that you don’t start scraping until the search results are visible.

Example Code in Action

Here’s a complete example of how we can search Reddit for posts using a dynamic query:

const puppeteer = require('puppeteer');

async function run(searchQuery) {
    // Launch browser
    const browser = await puppeteer.launch({
        headless: false, // Visible UI for debugging
        defaultViewport: false
    });
   
    // Create a new page
    const page = await browser.newPage();
   
    // Convert search query to URL-friendly format
    const searchQueryUrl = searchQuery.replace(' ', '+');
   
    // Navigate to Reddit search with the query
    await page.goto('https://www.reddit.com/search/?q=' + searchQueryUrl, {
        waitUntil: "domcontentloaded"
    });

    // Scraping logic goes here...
    await browser.close();
}

run("Video Games")

Why This Matters for Scraping? By properly understanding how query parameters work, you can easily scrape search results for any term you want. Instead of hardcoding the search URL, you can dynamically pass any keyword into the run() function, and Puppeteer will handle the rest.

Getting More Data by Scrolling

Reddit’s search results are dynamically loaded as you scroll down the page. To scrape more data, we need to simulate scrolling so that additional posts are loaded. Puppeteer allows us to automate this using the page.evaluate() method, which executes JavaScript in the browser context.

In the code, we’ve created a helper function called scrollTimes, which will scroll down the page multiple times to load more posts:

const scrollTimes = async (times) => {
        for (let i = 0; i < times; i++) {
            await page.evaluate(async () => {
                const distance = window.innerHeight; // Scroll by the viewport height
                window.scrollBy(0, distance);
            });


            await new Promise(resolve => setTimeout(resolve, 500)); // Wait for content to load
        }
    };


    await scrollTimes(100);

This function scrolls the page by the height of the viewport and waits briefly after each scroll to give the page time to load new content and the bigger the number I pass to it the bigger the number of posts that can be scraped.

Extracting All Results Links

When scraping a website, one of the key challenges is correctly selecting the elements you want to extract data from. In Puppeteer, selectors are used to identify and access HTML elements on the page. These selectors, such as CSS selectors, allow you to target specific content. Let’s explore how we extract the links to Reddit posts from the search results and how to work with selectors effectively.

First, we need to open our local browser and navigate to the eBay page we’ve been working with. To interact with the page elements, we’ll need to use Chrome’s DevTools, which you can easily open by pressing F12.

With DevTools open, select the inspector tool (the little arrow icon at the top left of the DevTools window) and click on the search button on the page. This will highlight the element in the DOM tree.

Now right-click on the highlighted element and press copy > copy selector

Now to extract the posts link dynamically Puppeteer provides methods like $$eval that allow you to work with multiple elements on a page. In our case, we want to extract all the links from the search results on Reddit. These links are the individual posts that match the search query.

Here’s how we do it using the $$eval method:

const links = await page.$$eval('#main-content > div > reddit-feed > faceplate-tracker > post-consume-tracker > div > faceplate-tracker:nth-child(1) > h2 > a', allAnchors => allAnchors.map(anchor => anchor.href))

This will return an array of URLs, each pointing to a post that matches the search query. 

Let’s break this down step by step:

1. Targeting Elements with a CSS Selector:
   • '#main-content > div > reddit-feed > faceplate-tracker > post-consume-tracker > div > faceplate-tracker:nth-child(1) > h2 > a'  is the CSS selector we just copied from the dev tool, and it is used to pinpoint where the links are located within Reddit’s search results.
   • This selector drills down through the HTML structure, navigating from #main-content (the main container of the page) down to the <a> tags that contain the URLs of the posts.
2. Extracting Multiple Elements with $$eval:
   • The $$eval method is used to select multiple elements that match the given CSS selector.
   • In this case, it grabs all the <a> tags that represent the links to individual posts on Reddit.
3. Mapping Data from Elements:
   • Once the anchors (<a> tags) are selected, we use the .map() function to loop over each element and extract the href attribute (the URL). The result is an array of links, each corresponding to a Reddit post.
4. Why Use Complex Selectors?
   • Websites like Reddit often use deeply nested HTML structures or dynamically generated content, which requires precise selectors to locate specific elements. By inspecting the page and carefully selecting the right path, you can efficiently extract the data you need.
   • If you open Reddit’s search results page and inspect the structure, you’ll see that the links are buried deep within several layers of HTML tags. Puppeteer allows us to traverse this hierarchy using CSS selectors.

Navigating to Each Post Page

Once we have the list of post URLs, the next step is to visit each post individually. Using a for loop, we iterate over each link and navigate to the respective post page:

for (let link of links) {
        await page.goto(link, { waitUntil: 'domcontentloaded' })

Again, we use the waitUntil: ‘domcontentloaded’ option to ensure the page is fully loaded before proceeding with data extraction.

Extracting Relevant Post Data

On each post page, we extract relevant data such as the title, post body, score, comments, subreddit name, and author. This is done using the page.$eval() method, which selects a single element and returns its text or attributes.

Here’s how the title and body of each post are extracted:

const title = await page.$eval('h1[slot="title"]', (el) => el?.innerText)
let body
const bodyEl = await page.$('div.text-neutral-content p')
if (bodyEl) {
     body = await bodyEl.evaluate((el) => el?.innerText)
     console.log(body);
}

If the post contains a body, we store it; otherwise, only the title is captured.

We also extract additional post metadata, such as score, number of comments, subreddit, and author:

const post = await page.$eval('shreddit-post', (el) => {
       return {
               score: el.score,
               comments: el.commentCount,
               subreddit: el.subredditPrefixedName,
               author: el.getAttribute('author'),
               link: el.contentHref
       }
})

Extracting Top Comments

In addition to the post’s data, we can scrape the top comments on each post. Reddit structures comments as nested elements, so we can use $$eval to retrieve multiple comment elements at once:

const comments = await page.$$eval('shreddit-comment', (els) => els.map(el => {
      const commentBody = el.querySelector('div[slot="comment"] p')?.innerText
      return {
                commentBody,
                author: el.getAttribute('author'),
                score: el.score,
                commentLink: el.getAttribute('reload-url'),
                depth: el.getAttribute('depth')
      }
}))

Here, we gather the comment body, author, score, and other relevant metadata for each comment.

Saving the data into a JSON file

Now that we’ve collected all the data in our appData array, it’s time to store it in a format we can easily use later. JSON (JavaScript Object Notation) is a lightweight format for storing and transporting data, making it perfect for saving the information we’ve scraped.

Here’s how we can do it:

1- Import the fs module: 

First, we need to bring in Node.js’s built-in fs (File System) module, which allows us to interact with the file system.

2- Convert the data to JSON: 

We then need to convert our appData array into a JSON string. This can be done easily using JSON.stringify().

3- Save the data: 

Finally, we write this JSON string to a file using fs.writeFileSync(). This method will create a new file named data.json in our project directory and store the data there.

const stringifiedData = JSON.stringify(data)
fs.writeFileSync('data.json', stringifiedData)

Here is a sample of the output data.json file

{
    "title": "What are some fun games to play with friends?",
    "body": "We are looking for recommendations on fun games...",
    "post": {
      "score": 125,
      "comments": 50,
      "subreddit": "r/gaming",
      "author": "gamer123",
      "link": "https://www.reddit.com/r/gaming/post/123456"
    },
    "comments": [
      {
        "commentBody": "Try playing Among Us!",
        "author": "player1",
        "score": 23,
        "commentLink": "/r/gaming/comment/1234",
        "depth": 0
      },
      {
        "commentBody": "Minecraft is always fun!",
        "author": "blockbuilder",
        "score": 15,
        "commentLink": "/r/gaming/comment/5678",
        "depth": 1
      }
    ]
  }

Wrapping Things Up

The last thing we need to do to wrap things up we would need to close the browser after the scraping is done

await browser.close()

Tip: after the project is done and tested now we can switch the headless mode to true to be faster and avoid any issues or bugs

const browser = await puppeteer.launch({
        headless: true,
        defaultViewport: false
    })

Conclusion

Congratulations! You’ve successfully built a web scraper that extracts Reddit post data using Puppeteer. Throughout this tutorial, you’ve learned how to:

• Set up Puppeteer and navigate dynamic websites like Reddit.
• Execute search queries and handle infinite scrolling to load more results.
• Extract key post data such as titles, links, and comments.
• Save the scraped data to a JSON file for further analysis.

This project is a great foundation for any web scraping work, and you can easily extend it by scraping more details, such as post scores or user information, or even scraping from different subreddits.

You can find the complete source code on my GitHub Repo

Video: Web Scraping Reddit

Feel free to explore different search queries, try scraping other types of data, or even integrate this scraper with a database or API to build more advanced applications!