-
How to scrape product availability from an e-commerce website?
Scraping product availability from e-commerce websites can help track stock levels or analyze product trends. Most sites indicate availability using clear tags or messages like “In Stock” or “Out of Stock.” Using BeautifulSoup, you can extract these details for static pages. For dynamic sites, Puppeteer or Selenium is better suited as they can render JavaScript content. Monitoring network requests can also reveal endpoints that provide stock data directly.
Here’s an example using BeautifulSoup to scrape product availability:import requests from bs4 import BeautifulSoup url = "https://example.com/products" headers = {"User-Agent": "Mozilla/5.0"} response = requests.get(url, headers=headers) if response.status_code == 200: soup = BeautifulSoup(response.content, "html.parser") products = soup.find_all("div", class_="product-item") for product in products: name = product.find("h2", class_="product-title").text.strip() availability = product.find("span", class_="availability-status").text.strip() print(f"Product: {name}, Availability: {availability}") else: print("Failed to fetch product availability.")Dynamic content often requires interaction with filters or scrolling to reveal more products. Using proxies and rate-limiting helps avoid being blocked during large-scale scraping. How do you handle inconsistent availability labels across different sites?
-
3 Replies
-
I write conditional logic to handle multiple possible labels for availability. For example, checking for “In Stock,” “Available,” or “Add to Cart” ensures nothing is missed.
-
For sites with dynamic content, I use Puppeteer to load all elements fully before scraping. This ensures I don’t miss products that are loaded asynchronously.
-
To avoid being flagged, I randomize request intervals and rotate proxies. These small adjustments make my scraper less detectable.
Log in to reply.