Khloe Walther
-
Khloe Walther replied to the discussion What are the most reliable ways to detect website blocks before scraping? in the forum General Web Scraping 9 months ago
Reply to What are the most reliable ways to detect website blocks before scraping?
Also, watch for CAPTCHA pages or unexpected redirects. If I start getting CAPTCHA requests too often, I back off or switch proxies. Tools like Scrapy have middleware that can auto-detect CAPTCHA pages and respond accordingly.
-
Khloe Walther replied to the discussion How do I identify hidden APIs that might be easier to scrape? in the forum General Web Scraping 9 months ago
Reply to How do I identify hidden APIs that might be easier to scrape?
Many sites use GraphQL APIs, so I look for POST requests with query bodies in the network tab.
-
Khloe Walther replied to the discussion What are the best methods for scraping data from dynamically-loaded websites? in the forum General Web Scraping 9 months ago
Reply to What are the best methods for scraping data from dynamically-loaded websites?
Scrapy Splash is another option for Python users. It can render JavaScript within Scrapy pipelines, allowing you to handle dynamic content without switching libraries.
-
Khloe Walther started the discussion What’s the best way to scrape e-commerce sites for product specifications? in the forum General Web Scraping 9 months ago
What’s the best way to scrape e-commerce sites for product specifications?
I set up scripts to navigate product categories first, as this reduces redundant scraping of main product listings.
-
Khloe Walther changed their photo 9 months ago
-
Khloe Walther became a registered member 9 months ago
