• Since Mercari uses AJAX for some dynamic content, I combine Nokogiri with Watir to handle content that loads after page interactions.

For dynamic content, like reviews and real-time stock information, Selenium with headless Chrome works well, especially when combined with randomized delays.

Axios or Fetch can be used for JSON data retrieval, especially if network requests expose structured product data.

Checking the site’s XML sitemap often reveals new product URLs, as they’re typically added to the sitemap immediately upon publication.

• For dynamically loaded reviews, I use PHP with a headless browser integration, like Panther, to render full pages before scraping.

• For Google Flights, I use automated notifications for specific routes and dates, reducing the need for constant scraping.