How to scrape search results using a DuckDuckGo proxy with JavaScript?

[Lena Celsa]

Scraping DuckDuckGo search results through a proxy is a great way to gather data while maintaining anonymity. While many opt for Puppeteer (a headless browser automation tool), it can be resource-intensive. A more lightweight and efficient approach is using Python’s requests library with a proxy, combined with BeautifulSoup for parsing the HTML.
Why Use a Proxy?
Avoid IP blocks – DuckDuckGo may limit repeated queries from the same IP.
Bypass geographic restrictions – Useful if you want results from different regions.
Improve anonymity – Keeps your real IP hidden.
A Python Approach with requests and BeautifulSoup
Instead of using a headless browser, you can send requests directly to DuckDuckGo’s search page and parse the results. Here’s how:

python
Copy
Edit
import requests
from bs4 import BeautifulSoup
# Define the search query
query = "web scraping tools"
duckduckgo_url = f"https://html.duckduckgo.com/html/?q={query}"
# Set up a proxy
proxies = {
"http": "http://your-proxy-server:port",
"https": "http://your-proxy-server:port",
}
# Custom headers to mimic a real browser
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
}
# Send a request via the proxy
response = requests.get(duckduckgo_url, headers=headers, proxies=proxies)
# Parse the response using BeautifulSoup
soup = BeautifulSoup(response.text, "html.parser")
# Extract search results
results = []
for result in soup.select(".result"):
title = result.select_one(".result__title")
link = result.select_one(".result__url")
snippet = result.select_one(".result__snippet")
if title and link and snippet:
results.append({
"title": title.text.strip(),
"link": f"https://duckduckgo.com{link.get('href')}",
"snippet": snippet.text.strip(),
})
# Print extracted results
for r in results:
print(r)

Why Use This Approach Instead of Puppeteer?
Faster Execution – No need to load an entire browser.
Lower Resource Usage – Uses simple HTTP requests instead of launching a Chromium instance.
Less Detectable – Looks more like a real user than a headless browser bot.
Handling Anti-Scraping Measures
DuckDuckGo is relatively scraper-friendly, but for tougher sites, consider:
Rotating User-Agents – Change headers with different browsers.
Using Residential Proxies – More trustworthy than data center IPs.
Introducing Random Delays – Mimic human behavior to avoid rate limiting.

• This reply was modified 1 month, 2 weeks ago by  Lena Celsa.

[Ketut Hippolytos]

How to Scrape Search Results Using a DuckDuckGo Proxy with Java?

Web scraping is a crucial technique for extracting information from search engines, and DuckDuckGo is a popular choice due to its privacy-focused approach. By using a proxy, developers can bypass restrictions, prevent IP bans, and maintain anonymity while scraping search results. This guide will explore how to scrape DuckDuckGo search results using Java and a proxy. We will also cover database integration for storing the scraped data efficiently.

Why Scrape Search Results from DuckDuckGo?

DuckDuckGo is widely used for privacy-preserving searches, making it a valuable search engine for research, competitive analysis, and SEO monitoring. Unlike Google, DuckDuckGo does not track users, making it an attractive alternative for gathering unbiased search data.

Advantages of Scraping DuckDuckGo

Scraping DuckDuckGo offers several benefits, including:

• Privacy-Friendly Searches: Since DuckDuckGo does not track user queries, data collection is less likely to be biased.
• Unfiltered Search Results: The search results are not influenced by a user’s previous search history.
• Less Scraping Restrictions: DuckDuckGo has fewer scraping protections than Google, reducing the risk of getting blocked.
• Data Collection for SEO: Businesses can track keyword performance, analyze competitors, and optimize their SEO strategies.
• Academic Research: Researchers can gather data for linguistic studies, sentiment analysis, and trend monitoring.

Setting Up Java for Web Scraping

To scrape search results from DuckDuckGo using Java, we need a few dependencies:

• JSoup: A Java library for parsing and extracting data from HTML.
• Apache HttpClient: A library for making HTTP requests.
• Proxy Configuration: To route requests through a proxy for anonymity.

First, install the required dependencies using Maven:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

<dependencies>

<dependency>

<groupId>org.jsoup</groupId>

<artifactId>jsoup</artifactId>

<version>1.15.3</version>

</dependency>

<dependency>

<groupId>org.apache.httpcomponents.client5</groupId>

<artifactId>httpclient5</artifactId>

<version>5.2</version>

</dependency>

</dependencies>

<dependencies> <dependency> <groupId>org.jsoup</groupId> <artifactId>jsoup</artifactId> <version>1.15.3</version> </dependency> <dependency> <groupId>org.apache.httpcomponents.client5</groupId> <artifactId>httpclient5</artifactId> <version>5.2</version> </dependency> </dependencies>

<dependencies>
    <dependency>
        <groupId>org.jsoup</groupId>
        <artifactId>jsoup</artifactId>
        <version>1.15.3</version>
    </dependency>
    <dependency>
        <groupId>org.apache.httpcomponents.client5</groupId>
        <artifactId>httpclient5</artifactId>
        <version>5.2</version>
    </dependency>
</dependencies>

Using a Proxy for Scraping DuckDuckGo

To avoid detection and IP bans, we use a proxy server. DuckDuckGo does not impose heavy restrictions, but it is still best practice to scrape using a rotating proxy.

Configuring a Proxy in Java

We configure a proxy in Java using Apache HttpClient:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

import java.io.IOException;

import java.net.*;

import org.jsoup.Jsoup;

import org.jsoup.nodes.Document;

public class DuckDuckGoScraper {

public static void main(String[] args) {

String searchQuery = "web scraping with Java";

String proxyHost = "your.proxy.server";

int proxyPort = 8080;

try {

Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort));

URL url = new URL("https://duckduckgo.com/html/?q=" + URLEncoder.encode(searchQuery, "UTF-8"));

HttpURLConnection connection = (HttpURLConnection) url.openConnection(proxy);

connection.setRequestMethod("GET");

connection.setRequestProperty("User-Agent", "Mozilla/5.0");

Document doc = Jsoup.parse(connection.getInputStream(), "UTF-8", url.toString());

doc.select(".result__title a").forEach(element ->

System.out.println("Title: " + element.text() + " | URL: " + element.attr("href"))

);

} catch (IOException e) {

e.printStackTrace();

}

}

}

import java.io.IOException; import java.net.*; import org.jsoup.Jsoup; import org.jsoup.nodes.Document; public class DuckDuckGoScraper { public static void main(String[] args) { String searchQuery = "web scraping with Java"; String proxyHost = "your.proxy.server"; int proxyPort = 8080; try { Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort)); URL url = new URL("https://duckduckgo.com/html/?q=" + URLEncoder.encode(searchQuery, "UTF-8")); HttpURLConnection connection = (HttpURLConnection) url.openConnection(proxy); connection.setRequestMethod("GET"); connection.setRequestProperty("User-Agent", "Mozilla/5.0"); Document doc = Jsoup.parse(connection.getInputStream(), "UTF-8", url.toString()); doc.select(".result__title a").forEach(element -> System.out.println("Title: " + element.text() + " | URL: " + element.attr("href")) ); } catch (IOException e) { e.printStackTrace(); } } }

import java.io.IOException;
import java.net.*;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

public class DuckDuckGoScraper {
    public static void main(String[] args) {
        String searchQuery = "web scraping with Java";
        String proxyHost = "your.proxy.server";
        int proxyPort = 8080;

        try {
            Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(proxyHost, proxyPort));
            URL url = new URL("https://duckduckgo.com/html/?q=" + URLEncoder.encode(searchQuery, "UTF-8"));
            HttpURLConnection connection = (HttpURLConnection) url.openConnection(proxy);
            connection.setRequestMethod("GET");
            connection.setRequestProperty("User-Agent", "Mozilla/5.0");

            Document doc = Jsoup.parse(connection.getInputStream(), "UTF-8", url.toString());
            doc.select(".result__title a").forEach(element ->
                System.out.println("Title: " + element.text() + " | URL: " + element.attr("href"))
            );

        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Extracting Search Results

The above script extracts search result titles and URLs from DuckDuckGo’s HTML response. The results are identified using CSS selectors:

• `.result__title a`: Selects the search result titles and URLs.
• `.result__snippet`: Extracts the search result snippets.

Storing Scraped Data in a Database

To store the extracted search results, we use a MySQL database. First, create a database and table:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

CREATE DATABASE SearchResultsDB;

USE SearchResultsDB;

CREATE TABLE results (

id INT AUTO_INCREMENT PRIMARY KEY,

title VARCHAR(255),

url TEXT,

snippet TEXT,

timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP

);

CREATE DATABASE SearchResultsDB; USE SearchResultsDB; CREATE TABLE results ( id INT AUTO_INCREMENT PRIMARY KEY, title VARCHAR(255), url TEXT, snippet TEXT, timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP );

CREATE DATABASE SearchResultsDB;

USE SearchResultsDB;

CREATE TABLE results (
    id INT AUTO_INCREMENT PRIMARY KEY,
    title VARCHAR(255),
    url TEXT,
    snippet TEXT,
    timestamp TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Inserting Data into MySQL

We use JDBC to insert scraped data into the database:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

import java.sql.Connection;

import java.sql.DriverManager;

import java.sql.PreparedStatement;

import java.util.List;

import org.jsoup.nodes.Document;

import org.jsoup.nodes.Element;

import org.jsoup.select.Elements;

public class DatabaseHandler {

private static final String DB_URL = "jdbc:mysql://localhost:3306/SearchResultsDB";

private static final String USER = "root";

private static final String PASSWORD = "password";

public static void saveResults(Document doc) {

String sql = "INSERT INTO results (title, url, snippet) VALUES (?, ?, ?)";

try (Connection conn = DriverManager.getConnection(DB_URL, USER, PASSWORD);

PreparedStatement stmt = conn.prepareStatement(sql)) {

Elements results = doc.select(".result__title a");

for (Element result : results) {

String title = result.text();

String url = result.attr("href");

String snippet = result.parent().nextElementSibling().text();

stmt.setString(1, title);

stmt.setString(2, url);

stmt.setString(3, snippet);

stmt.executeUpdate();

}

System.out.println("Data successfully stored in the database.");

} catch (Exception e) {

e.printStackTrace();

}

}

}

import java.sql.Connection; import java.sql.DriverManager; import java.sql.PreparedStatement; import java.util.List; import org.jsoup.nodes.Document; import org.jsoup.nodes.Element; import org.jsoup.select.Elements; public class DatabaseHandler { private static final String DB_URL = "jdbc:mysql://localhost:3306/SearchResultsDB"; private static final String USER = "root"; private static final String PASSWORD = "password"; public static void saveResults(Document doc) { String sql = "INSERT INTO results (title, url, snippet) VALUES (?, ?, ?)"; try (Connection conn = DriverManager.getConnection(DB_URL, USER, PASSWORD); PreparedStatement stmt = conn.prepareStatement(sql)) { Elements results = doc.select(".result__title a"); for (Element result : results) { String title = result.text(); String url = result.attr("href"); String snippet = result.parent().nextElementSibling().text(); stmt.setString(1, title); stmt.setString(2, url); stmt.setString(3, snippet); stmt.executeUpdate(); } System.out.println("Data successfully stored in the database."); } catch (Exception e) { e.printStackTrace(); } } }

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.util.List;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;

public class DatabaseHandler {
    private static final String DB_URL = "jdbc:mysql://localhost:3306/SearchResultsDB";
    private static final String USER = "root";
    private static final String PASSWORD = "password";

    public static void saveResults(Document doc) {
        String sql = "INSERT INTO results (title, url, snippet) VALUES (?, ?, ?)";

        try (Connection conn = DriverManager.getConnection(DB_URL, USER, PASSWORD);
             PreparedStatement stmt = conn.prepareStatement(sql)) {

            Elements results = doc.select(".result__title a");
            for (Element result : results) {
                String title = result.text();
                String url = result.attr("href");
                String snippet = result.parent().nextElementSibling().text();

                stmt.setString(1, title);
                stmt.setString(2, url);
                stmt.setString(3, snippet);
                stmt.executeUpdate();
            }

            System.out.println("Data successfully stored in the database.");

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Handling Anti-Scraping Mechanisms

Although DuckDuckGo is relatively lenient, websites often implement anti-scraping mechanisms. Here are some best practices to avoid detection:

• Use Rotating Proxies: Change IP addresses to avoid being blocked.
• Set User-Agent Headers: Mimic a real web browser.
• Introduce Delays: Add random delays between requests to avoid rate-limiting.
• Use Headless Browsers: If JavaScript rendering is needed, tools like Selenium can be useful.

Conclusion

Scraping search results from DuckDuckGo using Java and a proxy is a powerful technique for data collection. By leveraging JSoup for parsing, Apache HttpClient for requests, and MySQL for storage, we can efficiently extract and manage search engine data. Implementing best practices such as using rotating proxies and setting headers ensures a smooth scraping experience. Whether for SEO analysis, market research, or academic studies, web scraping from DuckDuckGo provides valuable insights while respecting user privacy.