-
Zusman Mimmi became a registered member a year ago
-
Gallus Maximilian replied to the discussion How can I scrape embedded data from audio or video content? in the forum General Web Scraping a year ago
Reply to How can I scrape embedded data from audio or video content?
YouTube offers closed captions (CC) on many videos, which can be downloaded using tools like youtube-dl for easier text extraction.
-
Gallus Maximilian replied to the discussion What are the best practices for scraping financial data from news or stock site? in the forum General Web Scraping a year ago
Reply to What are the best practices for scraping financial data from news or stock site?
Text parsing libraries like spaCy can extract financial terms and keywords, making it easier to analyze news sentiment on stocks.
-
Gallus Maximilian started the discussion How to scrape travel data from Skyscanner’s EU site using Rust? in the forum General Web Scraping a year ago
How to scrape travel data from Skyscanner’s EU site using Rust?
Rust’s Reqwest library is great for HTTP requests, handling Skyscanner’s static content efficiently for flight details and destination data.
-
Gallus Maximilian changed their photo a year ago
-
Gallus Maximilian became a registered member a year ago
-
Iraida Anicetus replied to the discussion What are the best practices for scraping e-commerce sites that allow it? in the forum General Web Scraping a year ago
Reply to What are the best practices for scraping e-commerce sites that allow it?
I set reasonable delays between requests, even if allowed, to minimize server load and maintain a good relationship with the website.
-
Iraida Anicetus replied to the discussion What’s the best way to scrape e-commerce sites for product specifications? in the forum General Web Scraping a year ago
Reply to What’s the best way to scrape e-commerce sites for product specifications?
Using BeautifulSoup for simple product pages with static HTML works well for grabbing titles, prices, and descriptions.
-
Iraida Anicetus replied to the discussion What are the best practices for scraping financial data from news or stock site? in the forum General Web Scraping a year ago
Reply to What are the best practices for scraping financial data from news or stock site?
Many financial sites offer RSS feeds with headline summaries. Parsing these feeds reduces the need to scrape individual pages directly.
-
Iraida Anicetus replied to the discussion How do I handle sites that block based on unusual request patterns? in the forum General Web Scraping a year ago
Reply to How do I handle sites that block based on unusual request patterns?
Using dynamic IPs from different locations is another way to vary patterns and reduce detection based on access frequency.
- Load More
