Khloe Walther
-
Khloe Walther replied to the discussion What are the most reliable ways to detect website blocks before scraping? in the forum General Web Scraping 10 months ago
Reply to What are the most reliable ways to detect website blocks before scraping?
Also, watch for CAPTCHA pages or unexpected redirects. If I start getting CAPTCHA requests too often, I back off or switch proxies. Tools like Scrapy have middleware that can auto-detect CAPTCHA pages and respond accordingly.
-
Khloe Walther replied to the discussion How do I identify hidden APIs that might be easier to scrape? in the forum General Web Scraping 10 months ago
Reply to How do I identify hidden APIs that might be easier to scrape?
Many sites use GraphQL APIs, so I look for POST requests with query bodies in the network tab.
-
Khloe Walther replied to the discussion What are the best methods for scraping data from dynamically-loaded websites? in the forum General Web Scraping 10 months ago
Reply to What are the best methods for scraping data from dynamically-loaded websites?
Scrapy Splash is another option for Python users. It can render JavaScript within Scrapy pipelines, allowing you to handle dynamic content without switching libraries.
-
Khloe Walther started the discussion What’s the best way to scrape e-commerce sites for product specifications? in the forum General Web Scraping 10 months ago
What’s the best way to scrape e-commerce sites for product specifications?
I set up scripts to navigate product categories first, as this reduces redundant scraping of main product listings.
-
Khloe Walther changed their photo 10 months ago
-
Khloe Walther became a registered member 10 months ago
