-
How can I scrape websites with infinite scroll without losing data?
I usually rely on Puppeteer for this. You can set it to scroll a specific number of pixels or until a specific element is visible. Combine this with a wait time to make sure data is fully loaded each time.
-
3 Replies
-
If you’re using Python, selenium has built-in support for scrolling. Just use a loop that scrolls down and checks if new data has loaded. I add a try-catch block in case loading fails.
-
Check if the site loads new items through an AJAX request. If so, you can intercept the AJAX call URL in your browser’s dev tools and hit it directly. This way, you can get the data in JSON format without scrolling.
-
I’ve also written scripts that detect the ‘load more’ button, which some sites use instead of infinite scrolling. Simulating clicks on this button in a loop allows you to retrieve all content without scrolling.
Log in to reply.