-
Jordan Gerasim replied to the discussion What strategies can I use to scrape websites with limited search functionality? in the forum General Web Scraping a year ago
Reply to What strategies can I use to scrape websites with limited search functionality?
I use pagination if available, or switch to scraping individual category pages rather than relying on the search function itself.
-
Jordan Gerasim replied to the discussion How can I handle pagination when scraping JavaScript-heavy sites? in the forum General Web Scraping a year ago
Reply to How can I handle pagination when scraping JavaScript-heavy sites?
Inspecting network requests can reveal the underlying AJAX calls. Directly calling these APIs is much faster than navigating through each page.
-
Jordan Gerasim replied to the discussion What’s the best way to handle date-based scraping for historical data? in the forum General Web Scraping a year ago
Reply to What’s the best way to handle date-based scraping for historical data?
When scraping extensive archives, it’s important to respect rate limits to avoid IP bans. Slowing down the scraper reduces the chance of getting blocked.
-
Jordan Gerasim started the discussion How can I collect flight delay & status data from FlightAware or FlightRadar24? in the forum General Web Scraping a year ago
How can I collect flight delay & status data from FlightAware or FlightRadar24?
FlightAware offers an API that provides flight tracking and delay data, making it easier to access real-time information legally.
-
Jordan Gerasim changed their photo a year ago
-
Jordan Gerasim became a registered member a year ago
-
Vieno Amenemhat replied to the discussion What’s the best approach to handling large datasets while scraping? in the forum General Web Scraping a year ago
Reply to What’s the best approach to handling large datasets while scraping?
Scheduling periodic scrapes rather than one large scrape can help maintain manageable data flows, especially for sites with frequent updates.
-
Vieno Amenemhat replied to the discussion How can I scrape data that’s only available after login? in the forum General Web Scraping a year ago
Reply to How can I scrape data that’s only available after login?
Logging in during off-peak hours reduces the risk of being flagged for unusual activity. Many sites monitor login frequency and usage patterns.
-
Vieno Amenemhat replied to the discussion How do I deal with rate limits on public APIs? in the forum General Web Scraping a year ago
Reply to How do I deal with rate limits on public APIs?
I sometimes find alternative APIs for the same data. Combining multiple sources reduces dependency on any one API and spreads out the request load.
-
Vieno Amenemhat replied to the discussion What’s the best way to handle date-based scraping for historical data? in the forum General Web Scraping a year ago
Reply to What’s the best way to handle date-based scraping for historical data?
Using Selenium’s calendar selector works well when scraping sites with graphical date pickers. I simulate clicks to pull data by day, month, or year.
- Load More
