• General Web Scraping

  How to scrape search results using a DuckDuckGo proxy with JavaScript?

  Posted by Raza Kenya on 12/10/2024 at 9:33 am

  Scraping search results through a DuckDuckGo proxy can be a powerful way to gather information without revealing your identity. JavaScript with Puppeteer is an excellent tool for such tasks, allowing you to automate a browser and send requests through a proxy server. Start by setting up a proxy in Puppeteer to route your traffic securely. Then, navigate to the DuckDuckGo search page, perform a search query, and extract the desired data like titles, URLs, and snippets. Managing request headers and delays ensures that your scraper mimics human behavior and avoids detection.Here’s an example using Puppeteer to scrape search results through a proxy:

  const puppeteer = require('puppeteer');
  
  (async () => {
      const browser = await puppeteer.launch({
          headless: true,
          args: ['--proxy-server=http://your-proxy-server:port']
      });
      const page = await browser.newPage();
      await page.goto('https://duckduckgo.com/');
      // Perform a search query
      await page.type('input[name="q"]', 'web scraping tools');
      await page.keyboard.press('Enter');
      await page.waitForSelector('.result');
      const results = await page.evaluate(() => {
          return Array.from(document.querySelectorAll('.result')).map(result => ({
              title: result.querySelector('.result__title')?.innerText.trim(),
              link: result.querySelector('.result__url')?.href,
              snippet: result.querySelector('.result__snippet')?.innerText.trim(),
          }));
      });
      console.log(results);
      await browser.close();
  })();
  

  Using a proxy helps bypass geographic restrictions and avoid rate limiting, especially for repeated or automated searches. Managing dynamic content loading ensures you get all the results effectively. How do you handle websites with strict anti-scraping measures like DuckDuckGo?

  Raza Kenya replied 1 week, 6 days ago 1 Member · 0 Replies
• 0 Replies

Log In to Reply