-
Keith Marwin replied to the discussion How do I handle scraping for real-time data that updates frequently? in the forum General Web Scraping a year ago
Reply to How do I handle scraping for real-time data that updates frequently?
I prioritize only the most essential data fields, which speeds up each request and allows for more frequent updates.
-
Keith Marwin replied to the discussion How can I handle pagination when scraping JavaScript-heavy sites? in the forum General Web Scraping a year ago
Reply to How can I handle pagination when scraping JavaScript-heavy sites?
For sites with “Load More” buttons, I simulate clicks on the button until all items are loaded. This works well for e-commerce and content sites.
-
Keith Marwin started the discussion How can I monitor price trends on Lazada for specific product categories? in the forum General Web Scraping a year ago
How can I monitor price trends on Lazada for specific product categories?
Lazada has APIs in some countries, which makes data retrieval easier and more reliable for categories like electronics or home goods.
-
Keith Marwin changed their photo a year ago
-
Keith Marwin became a registered member a year ago
-
Suhaila Kiyoshi replied to the discussion How do I deal with scraped data that has inconsistent formatting? in the forum General Web Scraping a year ago
Reply to How do I deal with scraped data that has inconsistent formatting?
Pandas is incredibly useful for normalizing scraped data by filling in missing values and aligning data types.
-
Suhaila Kiyoshi replied to the discussion How do I scrape data from sites using custom fonts or icons? in the forum General Web Scraping a year ago
Reply to How do I scrape data from sites using custom fonts or icons?
Font Awesome icons and similar libraries can sometimes be scraped as their CSS classes directly without image recognition.
-
Suhaila Kiyoshi replied to the discussion How do I handle scraping for real-time data that updates frequently? in the forum General Web Scraping a year ago
Reply to How do I handle scraping for real-time data that updates frequently?
For high-frequency scrapes, rotating IPs and adding randomized delays are critical to avoid detection. Real-time scrapers can get flagged quickly.
-
Suhaila Kiyoshi replied to the discussion How can I handle pagination when scraping JavaScript-heavy sites? in the forum General Web Scraping a year ago
Reply to How can I handle pagination when scraping JavaScript-heavy sites?
Adding timeouts between page loads helps reduce detection and gives the page time to load all content, preventing skipped data.
-
Suhaila Kiyoshi started the discussion What’s the best way to scrape product listings and prices from Shopee? in the forum General Web Scraping a year ago
What’s the best way to scrape product listings and prices from Shopee?
Shopee provides public APIs in some regions, allowing for reliable and structured access to product listings and prices.
- Load More
