-
What data can be scraped from Yelp.com using Ruby?
Scraping Yelp.com using Ruby allows you to collect valuable business information such as names, ratings, locations, and reviews. Ruby’s open-uri for HTTP requests and nokogiri for parsing HTML provide a straightforward way to extract this data. By targeting Yelp’s search pages, you can gather data from multiple businesses in a specific category or location. Below is an example script for scraping business information from Yelp.
require 'open-uri' require 'nokogiri' # Target URL url = 'https://www.yelp.com/search?find_desc=restaurants&find_loc=New+York' html = URI.open(url).read # Parse HTML doc = Nokogiri::HTML(html) # Extract business details doc.css('.container__09f24__21w3G').each do |business| name = business.css('.css-1egxyvc').text.strip rescue 'Name not available' rating = business.css('.i-stars__09f24__1T6rz').attr('aria-label')&.value rescue 'Rating not available' address = business.css('.css-e81eai').text.strip rescue 'Address not available' puts "Name: #{name}, Rating: #{rating}, Address: #{address}" endThis script fetches a Yelp search results page, parses the HTML using Nokogiri, and extracts business names, ratings, and addresses. Handling pagination to navigate through multiple pages ensures a more complete dataset. Adding delays between requests helps avoid detection by Yelp’s anti-scraping mechanisms.
-
3 Replies
-
Handling pagination allows scraping data from multiple pages, ensuring a comprehensive dataset. Yelp displays limited results per page, and programmatically following the “Next” button helps collect all listings in a category. Random delays between requests make the scraper less likely to be detected. With pagination support, the scraper becomes more effective in gathering detailed data for analysis.
-
Adding error handling ensures the scraper doesn’t break if elements are missing or Yelp updates its structure. For instance, some businesses might not display ratings or full addresses. Wrapping the extraction logic in conditional checks or try-catch blocks prevents the script from crashing. Logging skipped businesses helps refine the script for better performance. This feature makes the scraper robust and reliable.
-
Using proxies and user-agent rotation helps avoid detection by Yelp’s anti-scraping mechanisms. Repeated requests from the same IP address or browser signature increase the likelihood of being flagged. Rotating these attributes and introducing random delays reduces this risk. These measures are essential for large-scale scraping projects.
Log in to reply.