How to scrape product prices from Newegg.com using Python?

Umeda Domenica · 2024-12-20T11:26:44+00:00

Scraping product prices from Newegg.com using Python is a straightforward way to gather data about electronics and tech products. Python’s requests library can retrieve page content, and BeautifulSoup can parse the HTML to extract product names, prices, and availability. The process involves sending a GET request to Newegg’s product listing page and targeting the appropriate elements for extraction. Below is a sample script for scraping data from Newegg.import requestsfrom bs4 import BeautifulSoup# Target URL for Newegg productsurl "https://www.newegg.com/p/pl?dgraphics+cards"headers { "User-Agent": "Mozilla/5.0"}response requests.get(url, headersheaders)if response.status_code 200: soup BeautifulSoup(response.content, "html.parser") products soup.find_all("div", class_"item-container") for product in products: name product.find("a", class_"item-title").text.strip() if product.find("a", class_"item-title") else "Name not available" price product.find("li", class_"price-current").text.strip() if product.find("li", class_"price-current") else "Price not available" print(f"Name: {name}, Price: {price}")else: print("Failed to fetch Newegg page.")This script fetches the Newegg product listing page, parses the HTML using BeautifulSoup, and extracts the names and prices of graphics cards. Handling pagination allows scraping additional pages, ensuring a more comprehensive dataset. Adding delays between requests helps reduce the risk of detection by anti-scraping mechanisms.

General Web Scraping

How to scrape product prices from Newegg.com using Python?

Posted by Umeda Domenica on 12/20/2024 at 11:26 am
Scraping product prices from Newegg.com using Python is a straightforward way to gather data about electronics and tech products. Python’s requests library can retrieve page content, and BeautifulSoup can parse the HTML to extract product names, prices, and availability. The process involves sending a GET request to Newegg’s product listing page and targeting the appropriate elements for extraction. Below is a sample script for scraping data from Newegg.
```
import requests
from bs4 import BeautifulSoup
# Target URL for Newegg products
url = "https://www.newegg.com/p/pl?d=graphics+cards"
headers = {
    "User-Agent": "Mozilla/5.0"
}
response = requests.get(url, headers=headers)
if response.status_code == 200:
    soup = BeautifulSoup(response.content, "html.parser")
    products = soup.find_all("div", class_="item-container")
    for product in products:
        name = product.find("a", class_="item-title").text.strip() if product.find("a", class_="item-title") else "Name not available"
        price = product.find("li", class_="price-current").text.strip() if product.find("li", class_="price-current") else "Price not available"
        print(f"Name: {name}, Price: {price}")
else:
    print("Failed to fetch Newegg page.")
```
This script fetches the Newegg product listing page, parses the HTML using BeautifulSoup, and extracts the names and prices of graphics cards. Handling pagination allows scraping additional pages, ensuring a more comprehensive dataset. Adding delays between requests helps reduce the risk of detection by anti-scraping mechanisms.
Umeda Domenica replied 2 days, 12 hours ago 1 Member · 0 Replies
0 Replies

Sorry, there were no replies found.