-
Wendel Vilko started the discussion How do I handle frequent website structure changes in scraping scripts? in the forum General Web Scraping a year ago
How do I handle frequent website structure changes in scraping scripts?
Use robust selectors like XPath or CSS that are less likely to break if the site’s structure changes slightly.
-
Wendel Vilko changed their photo a year ago
-
Wendel Vilko became a registered member a year ago
-
Joel Farzana replied to the discussion How to avoid IP bans while scraping heavily trafficked sites? in the forum General Web Scraping a year ago
Reply to How to avoid IP bans while scraping heavily trafficked sites?
Take regular breaks in scraping to mimic typical browsing behavior, which helps avoid drawing attention.
-
Joel Farzana started the discussion How to monitor price changes on e-commerce websites using scraping? in the forum General Web Scraping a year ago
How to monitor price changes on e-commerce websites using scraping?
Set up a scheduled scraper using Scrapy or BeautifulSoup to check product pages at regular intervals.
-
Joel Farzana changed their photo a year ago
-
Joel Farzana became a registered member a year ago
-
Uma Innes replied to the discussion How do I scrape data from an online form with hidden fields? in the forum General Web Scraping a year ago
Reply to How do I scrape data from an online form with hidden fields?
Hidden fields often contain CSRF tokens, so be sure to capture and use those correctly in your POST request.
-
Uma Innes replied to the discussion How to avoid IP bans while scraping heavily trafficked sites? in the forum General Web Scraping a year ago
Reply to How to avoid IP bans while scraping heavily trafficked sites?
Some people find that using IP geolocation that matches the site’s primary user base reduces the chance of being flagged.
-
Uma Innes started the discussion What’s the best way to scrape news websites and aggregate headlines? in the forum General Web Scraping a year ago
What’s the best way to scrape news websites and aggregate headlines?
Use Scrapy to set up a crawler that extracts headlines from multiple news websites in one go.
- Load More
