-
How to detect new products on e-commerce sites without scraping all pages?
Many sites have “new arrivals” sections that can be monitored with a targeted scraper, saving time compared to scanning all categories.
-
3 Replies
-
Checking the site’s XML sitemap often reveals new product URLs, as they’re typically added to the sitemap immediately upon publication.
-
- Using the site’s API, if available, allows for more efficient monitoring of product additions without excessive page requests.
-
- Some sites include a “date added” field, which I can scrape and filter to identify the latest products only.
Log in to reply.