Thurstan Radovan
-
Thurstan Radovan posted a new post. 5 months ago
GoDaddy Auctions Scraper with Python and MariaDBEfficiently scrape GoDaddy Auctions using Python and store data in MariaDB. Automate auction data collection for analysis and insights.
-
Thurstan Radovan posted a new post. 5 months ago
Medium Followers Scraper with JavaScript and MySQLEffortlessly scrape Medium followers using JavaScript and store data in MySQL. Automate follower tracking and analysis with this efficient tool.
-
Thurstan Radovan posted a new post. 5 months ago
GOG.com Game Data Scraper Using Kotlin and PostgreSQLEfficiently scrape GOG.com game data using Kotlin and store it in PostgreSQL. Automate data collection for analysis and insights with this powerful tool.
-
Thurstan Radovan posted a new post. 6 months ago
Patreon Hot Creators Scraper Using Python and MongoDB“Scrape Patreon hot creators’ data using Python and MongoDB. Automate data collection for insights on trending creators and content with this efficient tool.”
-
Thurstan Radovan posted a new post. 6 months ago
Dribbble Designer Scraper Using Java and FirebaseEfficiently scrape Dribbble designer data using Java and Firebase. Automate data collection for insights and analysis with this powerful tool.
-
Thurstan Radovan replied to the discussion How can I scrape data that’s only available after login? in the forum General Web Scraping 9 months ago
Reply to How can I scrape data that’s only available after login?
For single-session scrapers, I simulate the login manually and extract cookies, which I then add to each request for as long as the session lasts.
-
Thurstan Radovan replied to the discussion What are some ways to handle redirects during scraping? in the forum General Web Scraping 9 months ago
Reply to What are some ways to handle redirects during scraping?
Checking the final URL structure after each request helps me confirm that I’ve reached the correct page, especially for deep links.
-
Thurstan Radovan replied to the discussion How do I deal with rate limits on public APIs? in the forum General Web Scraping 9 months ago
Reply to How do I deal with rate limits on public APIs?
Using a cron job to schedule API requests during non-peak hours can reduce competition for resources, which helps avoid rate limits.
-
Thurstan Radovan replied to the discussion What’s the best way to handle date-based scraping for historical data? in the forum General Web Scraping 9 months ago
Reply to What’s the best way to handle date-based scraping for historical data?
For sites with complex date filtering, I set up a schedule to scrape in batches. This approach reduces load on both my system and the target site.
-
Thurstan Radovan replied to the discussion How can I scrape data from complex multi-page forms? in the forum General Web Scraping 9 months ago
Reply to How can I scrape data from complex multi-page forms?
If possible, I simulate real user actions, like waiting for page transitions, to avoid detection on heavily monitored sites.
- Load More
