-
Norbu Nata replied to the discussion What are some ways to handle redirects during scraping? in the forum General Web Scraping a year ago
Reply to What are some ways to handle redirects during scraping?
Analyzing the redirection chain in dev tools can reveal if it’s intended for bots. Sometimes, switching IPs can help avoid these traps.
-
Norbu Nata replied to the discussion How do I deal with rate limits on public APIs? in the forum General Web Scraping a year ago
Reply to How do I deal with rate limits on public APIs?
For batch processing, I queue requests and handle them in smaller chunks, which lets me stay within limits while gathering data continuously.
-
Norbu Nata replied to the discussion How can I scrape data from complex multi-page forms? in the forum General Web Scraping a year ago
Reply to How can I scrape data from complex multi-page forms?
Checking for CSRF tokens is important, as many multi-page forms use them for security. I include these tokens in each request to maintain sessions.
-
Norbu Nata started the discussion What’s the best way to scrape financial data from Yahoo Finance? in the forum General Web Scraping a year ago
What’s the best way to scrape financial data from Yahoo Finance?
Yahoo Finance has a public API that provides stock prices, historical data, and financial metrics, making it easier than scraping the site itself.
-
Norbu Nata changed their photo a year ago
-
Norbu Nata became a registered member a year ago
-
Phaenna Izan replied to the discussion What’s the best way to handle date-based scraping for historical data? in the forum General Web Scraping a year ago
Reply to What’s the best way to handle date-based scraping for historical data?
Automating navigation of pagination and date filters is helpful, especially on news or financial sites where archives are extensive.
-
Phaenna Izan replied to the discussion How can I scrape data from complex multi-page forms? in the forum General Web Scraping a year ago
Reply to How can I scrape data from complex multi-page forms?
For forms with required fields, I validate the input data beforehand to prevent submission errors. This minimizes issues during scraping.
-
Phaenna Izan replied to the discussion What are the most common errors encountered in web scraping, and how can I troub in the forum General Web Scraping a year ago
Reply to What are the most common errors encountered in web scraping, and how can I troub
If none of these work, I sometimes resort to scraping at off-peak hours. Sites often relax restrictions when there’s less user activity.
-
Phaenna Izan changed their photo a year ago
- Load More
