-
Yannig Avicenna replied to the discussion How can I scrape mobile-only websites from a desktop environment? in the forum General Web Scraping a year ago
Reply to How can I scrape mobile-only websites from a desktop environment?
Sometimes I use proxies with mobile IPs. This, combined with a mobile user-agent, gives the best result because the target server believes the request is truly from a mobile device.
-
Yannig Avicenna replied to the discussion How do I identify hidden APIs that might be easier to scrape? in the forum General Web Scraping a year ago
Reply to How do I identify hidden APIs that might be easier to scrape?
I also inspect any XHR or Fetch requests in the dev tools. They often lead to JSON endpoints, which are much simpler to parse than HTML.
-
Yannig Avicenna started the discussion How can I scrape product prices accurately without triggering anti-bot measures? in the forum General Web Scraping a year ago
How can I scrape product prices accurately without triggering anti-bot measures?
Rotating proxies with varying IPs ensures my scraper avoids detection, especially on sites with strict anti-bot protections around price data.
-
Yannig Avicenna changed their photo a year ago
-
Yannig Avicenna became a registered member a year ago
-
Daniel Teuku replied to the discussion Is Python really the best language for web scraping, or should I consider altern in the forum General Web Scraping a year ago
Reply to Is Python really the best language for web scraping, or should I consider altern
Python’s ease of use and broad library support make it accessible, especially for beginners. However, if performance is a priority, C# with HtmlAgilityPack is a strong contender, especially if you’re targeting Windows environments.
-
Daniel Teuku replied to the discussion How do I identify hidden APIs that might be easier to scrape? in the forum General Web Scraping a year ago
Reply to How do I identify hidden APIs that might be easier to scrape?
If the site is built with React or Vue, there’s a good chance data is loaded via API calls. I look for these in the page’s JavaScript code or in the dev tools.
-
Daniel Teuku replied to the discussion What are the best methods for scraping data from dynamically-loaded websites? in the forum General Web Scraping a year ago
Reply to What are the best methods for scraping data from dynamically-loaded websites?
If the data loads incrementally (like with infinite scroll), I use JavaScript event listeners to detect new content, then scrape the page once all content has loaded.
-
Daniel Teuku started the discussion What are the best methods to scrape product availability and stock status? in the forum General Web Scraping a year ago
What are the best methods to scrape product availability and stock status?
Product pages often have specific tags or indicators for availability, like “In Stock” or “Out of Stock.” I locate these using CSS selectors or XPath expressions.
-
Daniel Teuku changed their photo a year ago
- Load More
