If you’re using Python, selenium has built-in support for scrolling. Just use a loop that scrolls down and checks if new data has loaded. I add a try-catch block in case loading fails.

Sometimes I use proxies with mobile IPs. This, combined with a mobile user-agent, gives the best result because the target server believes the request is truly from a mobile device.

I also inspect any XHR or Fetch requests in the dev tools. They often lead to JSON endpoints, which are much simpler to parse than HTML.

Rotating proxies with varying IPs ensures my scraper avoids detection, especially on sites with strict anti-bot protections around price data.