-
Tuva Shirley replied to the discussion How can I scrape data from complex multi-page forms? in the forum General Web Scraping a year ago
Reply to How can I scrape data from complex multi-page forms?
Sometimes, inspecting the network activity shows specific API endpoints that handle form data. Directly submitting to these endpoints can be quicker.
-
Tuva Shirley replied to the discussion What are the most common errors encountered in web scraping, and how can I troub in the forum General Web Scraping a year ago
Reply to What are the most common errors encountered in web scraping, and how can I troub
HTTP 403 or 404 errors often mean the server is blocking requests. Changing user agents or adding headers to mimic a real browser can sometimes solve this.
-
Tuva Shirley replied to the discussion How can I detect JavaScript traps in websites that prevent scraping? in the forum General Web Scraping a year ago
Reply to How can I detect JavaScript traps in websites that prevent scraping?
Tools like Ghostery can help reveal trackers that could be part of JavaScript traps.
-
Tuva Shirley replied to the discussion What are the best libraries for scraping data from non-English websites? in the forum General Web Scraping a year ago
Reply to What are the best libraries for scraping data from non-English websites?
I find Python’s translation libraries, like googletrans, helpful when I need to translate scraped data into English.
-
Tuva Shirley started the discussion How can I track price changes on an e-commerce site without getting blocked? in the forum General Web Scraping a year ago
How can I track price changes on an e-commerce site without getting blocked?
Setting up a rotating proxy pool lets me spread requests across multiple IPs, avoiding detection for frequent price checks.
-
Tuva Shirley changed their photo a year ago
-
Tuva Shirley became a registered member a year ago
-
Mhairi Virginie replied to the discussion How do I deal with rate limits on public APIs? in the forum General Web Scraping a year ago
Reply to How do I deal with rate limits on public APIs?
If the API rate limit is very strict, I distribute requests across multiple accounts with different API keys to maximize throughput.
-
Mhairi Virginie replied to the discussion What’s the best way to handle date-based scraping for historical data? in the forum General Web Scraping a year ago
Reply to What’s the best way to handle date-based scraping for historical data?
For sites with date filters, I automate date selection in the form or URL parameters to scrape data for specific periods.
-
Mhairi Virginie replied to the discussion How can I scrape data from complex multi-page forms? in the forum General Web Scraping a year ago
Reply to How can I scrape data from complex multi-page forms?
Capturing and storing form data as I go allows me to pick up where I left off if the script stops unexpectedly. This is crucial for long, multi-page forms.
- Load More
