-
Ace Arash became a registered member a year ago
-
Kasih Berezi replied to the discussion How can I scrape product prices accurately without triggering anti-bot measures? in the forum General Web Scraping a year ago
Reply to How can I scrape product prices accurately without triggering anti-bot measures?
Using random intervals between requests reduces my chances of being flagged for repetitive price checks.
-
Kasih Berezi started the discussion What’s the best way to avoid triggering anti-bot mechanisms on large e-commerce in the forum General Web Scraping a year ago
What’s the best way to avoid triggering anti-bot mechanisms on large e-commerce
Using a combination of rotating residential proxies and random headers has helped me avoid detection. E-commerce sites often have strict limits, so I change my proxies every few requests.
-
Kasih Berezi changed their photo a year ago
-
Kasih Berezi became a registered member a year ago
-
Raul Marduk replied to the discussion How can I scrape websites with infinite scroll without losing data? in the forum General Web Scraping a year ago
Reply to How can I scrape websites with infinite scroll without losing data?
Check if the site loads new items through an AJAX request. If so, you can intercept the AJAX call URL in your browser’s dev tools and hit it directly. This way, you can get the data in JSON format without scrolling.
-
Raul Marduk replied to the discussion How can I scrape mobile-only websites from a desktop environment? in the forum General Web Scraping a year ago
Reply to How can I scrape mobile-only websites from a desktop environment?
If the site has a mobile API, consider using it directly. Many mobile sites use lighter APIs that are easier to parse and can be faster than scraping HTML.
-
Raul Marduk replied to the discussion What are the most reliable ways to detect website blocks before scraping? in the forum General Web Scraping a year ago
Reply to What are the most reliable ways to detect website blocks before scraping?
I often include honeypot URL detection in my scrapers. Some sites place these hidden links to catch bots, so if my scraper lands on one, I stop the session immediately.
-
Raul Marduk replied to the discussion How can I optimize my scraping code for faster performance? in the forum General Web Scraping a year ago
Reply to How can I optimize my scraping code for faster performance?
Batch requests when possible. By batching requests with connection pooling, I cut down the time spent on individual connections and speed up overall data retrieval.
-
Raul Marduk started the discussion How do I approach scraping flash sale or limited-time discount pages? in the forum General Web Scraping a year ago
How do I approach scraping flash sale or limited-time discount pages?
I schedule my scraper to start just before the sale begins, which helps capture initial stock levels and prices accurately.
- Load More
