Rayobyte Community | Mhairi Virginie | Activity https://rayobyte.com/community/members/mhairivirginie/activity/ Activity feed for Mhairi Virginie. Mon, 06 Apr 2026 06:44:03 +0000 https://buddypress.org/?v=2.6.80 en-US 30 hourly 2 e11ab0602fdbfb533e85df842637772b Mhairi Virginie replied to the discussion How do I deal with rate limits on public APIs? in the forum General Web Scraping https://rayobyte.com/community/discussion/how-do-i-deal-with-rate-limits-on-public-apis/#post-1742 Sat, 16 Nov 2024 06:59:28 +0000 Reply to How do I deal with rate limits on public APIs?

If the API rate limit is very strict, I distribute requests across multiple accounts with different API keys to maximize throughput.

]]> 4b7eec2d8450732333d422b50053e212 Mhairi Virginie replied to the discussion What’s the best way to handle date-based scraping for historical data? in the forum General Web Scraping https://rayobyte.com/community/discussion/whats-the-best-way-to-handle-date-based-scraping-for-historical-data/#post-1738 Sat, 16 Nov 2024 06:58:18 +0000 Reply to What’s the best way to handle date-based scraping for historical data?

For sites with date filters, I automate date selection in the form or URL parameters to scrape data for specific periods.

]]> 39b02bef693e365000a0ba00339bb63c Mhairi Virginie replied to the discussion How can I scrape data from complex multi-page forms? in the forum General Web Scraping https://rayobyte.com/community/discussion/how-can-i-scrape-data-from-complex-multi-page-forms/#post-1733 Sat, 16 Nov 2024 06:57:51 +0000 Reply to How can I scrape data from complex multi-page forms?

Capturing and storing form data as I go allows me to pick up where I left off if the script stops unexpectedly. This is crucial for long, multi-page forms.

]]> 4ab2ba8c4750005066a27a38430b3321 Mhairi Virginie replied to the discussion What are the most common errors encountered in web scraping, and how can I troub in the forum General Web Scraping https://rayobyte.com/community/discussion/what-are-the-most-common-errors-encountered-in-web-scraping-and-how-can-i-troub/#post-1775 Sat, 16 Nov 2024 06:57:14 +0000 Reply to What are the most common errors encountered in web scraping, and how can I troub

Encountering CAPTCHAs mid-scrape can stop the script. I’ve found that using services like 2Captcha or rotating proxies helps minimize this.

]]> 0d1872dcdc01a3c061c0048dd8d05f1d Mhairi Virginie replied to the discussion How can I detect JavaScript traps in websites that prevent scraping? in the forum General Web Scraping https://rayobyte.com/community/discussion/how-can-i-detect-javascript-traps-in-websites-that-prevent-scraping/#post-1723 Sat, 16 Nov 2024 06:56:17 +0000 Reply to How can I detect JavaScript traps in websites that prevent scraping?

I disable JavaScript initially to see if the content is still accessible. If it isn’t, then a headless browser is likely needed.

]]> 103c8bfe14eab95e9d12703274cc5687 Mhairi Virginie started the discussion What are efficient ways to scrape product images from an e-commerce site? in the forum General Web Scraping https://rayobyte.com/community/forums/general-web-scraping/ Sat, 16 Nov 2024 06:55:11 +0000 What are efficient ways to scrape product images from an e-commerce site?

Image URLs are often embedded in the page’s HTML, so I use BeautifulSoup to locate img tags and extract src attributes.

]]> e78d921ce6daf0286a99596693e2ec10 Mhairi Virginie changed their photo https://rayobyte.com/community/news-feed/p/1364/ Sat, 16 Nov 2024 06:53:30 +0000 0 b441a6ecfd666440e7ab3a537de1e058 Mhairi Virginie became a registered member https://rayobyte.com/community/news-feed/p/1363/ Sat, 16 Nov 2024 06:52:24 +0000 0