About the company
A global e-commerce group owned by a Fortune 500 leader approached Rayobyte with a request to collect pricing data from more than 100 competitor websites, with total expected volume exceeding 10 million scrapes per day.
As part of the initial proof of concept (POC), the client selected one of its largest target sites. The goal was to prove Rayobyte could collect publicly available data from that site at scale and within ethical limits.
The challenge
The client needed a vendor who could:
- Collect publicly available data reliably and ethically at scale
- Deliver redundancy and failover across multiple collection methods
- Meet strict latency requirements (P90 under 8 seconds)
- Scale operations from hundreds of thousands to millions of daily scrapes
Our solution
Within 48 hours, Rayobyte’s engineering team developed and validated three independent collection methods capable of operating at scale.
Our hybrid approach combined:
- Browser-based collection on our own Chromium browsers, for pages that render with JavaScript
- A second collection path for the same publicly available data
- Layered redundancy, so that if one method slowed down, the others carried the load
With three independent methods, the client always had a fallback.
All methods were built around ethical, publicly available data collection, with strict compliance at every stage.
The results
Rayobyte delivered tangible performance improvements on this project. After targeted optimizations, our system achieved a P90 response time of under eight seconds, meeting the client’s benchmark for production readiness.
The engagement showed that large-scale collection from a complex site could be done ethically, efficiently and cost-effectively, without relying on one method alone.
The impact
The same methods now support our large retail, finance and AI clients.