-
Norbu Nata started the discussion What’s the best way to scrape financial data from Yahoo Finance? in the forum General Web Scraping a year ago
What’s the best way to scrape financial data from Yahoo Finance?
Yahoo Finance has a public API that provides stock prices, historical data, and financial metrics, making it easier than scraping the site itself.
-
Norbu Nata changed their photo a year ago
-
Norbu Nata became a registered member a year ago
-
Phaenna Izan replied to the discussion What’s the best way to handle date-based scraping for historical data? in the forum General Web Scraping a year ago
Reply to What’s the best way to handle date-based scraping for historical data?
Automating navigation of pagination and date filters is helpful, especially on news or financial sites where archives are extensive.
-
Phaenna Izan replied to the discussion How can I scrape data from complex multi-page forms? in the forum General Web Scraping a year ago
Reply to How can I scrape data from complex multi-page forms?
For forms with required fields, I validate the input data beforehand to prevent submission errors. This minimizes issues during scraping.
-
Phaenna Izan replied to the discussion What are the most common errors encountered in web scraping, and how can I troub in the forum General Web Scraping a year ago
Reply to What are the most common errors encountered in web scraping, and how can I troub
If none of these work, I sometimes resort to scraping at off-peak hours. Sites often relax restrictions when there’s less user activity.
-
Phaenna Izan changed their photo a year ago
-
Phaenna Izan became a registered member a year ago
-
Tahvo Eulalia replied to the discussion What are some ways to handle redirects during scraping? in the forum General Web Scraping a year ago
Reply to What are some ways to handle redirects during scraping?
For sites with multiple redirection layers, I track URLs before and after redirects to identify any patterns and adjust my scraper accordingly.
-
Tahvo Eulalia replied to the discussion How do I deal with rate limits on public APIs? in the forum General Web Scraping a year ago
Reply to How do I deal with rate limits on public APIs?
I monitor error codes, like 429 (Too Many Requests), to detect rate limits in real time. My script pauses or retries after a delay when it sees this code.
- Load More
