-
Aridai Farzona became a registered member a year ago
-
Khloe Walther replied to the discussion What are the most reliable ways to detect website blocks before scraping? in the forum General Web Scraping a year ago
Reply to What are the most reliable ways to detect website blocks before scraping?
Also, watch for CAPTCHA pages or unexpected redirects. If I start getting CAPTCHA requests too often, I back off or switch proxies. Tools like Scrapy have middleware that can auto-detect CAPTCHA pages and respond accordingly.
-
Khloe Walther replied to the discussion How do I identify hidden APIs that might be easier to scrape? in the forum General Web Scraping a year ago
Reply to How do I identify hidden APIs that might be easier to scrape?
Many sites use GraphQL APIs, so I look for POST requests with query bodies in the network tab.
-
Khloe Walther replied to the discussion What are the best methods for scraping data from dynamically-loaded websites? in the forum General Web Scraping a year ago
Reply to What are the best methods for scraping data from dynamically-loaded websites?
Scrapy Splash is another option for Python users. It can render JavaScript within Scrapy pipelines, allowing you to handle dynamic content without switching libraries.
-
Khloe Walther started the discussion What’s the best way to scrape e-commerce sites for product specifications? in the forum General Web Scraping a year ago
What’s the best way to scrape e-commerce sites for product specifications?
I set up scripts to navigate product categories first, as this reduces redundant scraping of main product listings.
-
Khloe Walther changed their photo a year ago
-
Khloe Walther became a registered member a year ago
-
Ampelios Abhijit replied to the discussion How can I manage session-based scraping effectively? in the forum General Web Scraping a year ago
Reply to How can I manage session-based scraping effectively?
I’ve found headless browsers like Puppeteer useful for simulating session persistence in real-time, which is particularly helpful for complex logins or session-based navigation.
-
Ampelios Abhijit replied to the discussion How does Go’s performance compare to Node.js for building APIs? in the forum General Web Scraping a year ago
Reply to How does Go’s performance compare to Node.js for building APIs?
Both are great choices for building APIs, but Go has a slight edge in terms of raw speed and handling high concurrency with minimal overhead.
-
Ampelios Abhijit replied to the discussion What are the differences between learning Python and JavaScript for beginners? in the forum General Web Scraping a year ago
Reply to What are the differences between learning Python and JavaScript for beginners?
Both are great for beginners, but Python has a gentler learning curve due to its simplicity and extensive community support for beginners.
- Load More
