Prerequisites<\/a>
Step 1: Setting Up the Yahoo Scraper<\/a>
Step 2: Parsing the HTML Content<\/a>
Step 3: Saving Data to CSV<\/a>
Step 4: Running the Scraper<\/a>
Expected Output<\/a>
Best Practices for Scraping<\/a>
Conclusion<\/a><\/p>\n
Introduction<\/h2>\n
Web scraping has become an essential skill for extracting valuable information from the web. Whether you’re collecting data for research, market analysis, or building a search engine aggregator, web scraping allows you to automate data extraction efficiently. In this tutorial, we\u2019ll walk you through creating a Yahoo scraper in Python<\/strong>. Using Python\u2019s robust libraries like requests<\/strong> and BeautifulSoup<\/strong>, we\u2019ll demonstrate how to fetch and parse search results from Yahoo.<\/p>\n Before diving in, ensure you have the following:<\/p>\n First, let\u2019s create a function to fetch search results from Yahoo. We’ll use the Code:<\/p>\n Next, we\u2019ll parse the HTML content using Code:<\/p>\n After extracting the data, we’ll save it to a CSV file using pandas for easy analysis and sharing.<\/p>\n Code:<\/p>\n Finally, we\u2019ll combine all the functions to fetch multiple pages of search results, parse them, and save the data.<\/p>\n Code:<\/p>\n Once you run the scraper, a CSV file named Websites often monitor the frequency of requests to prevent scraping. Adding small delays between actions can mimic human behavior and reduce the risk of getting blocked.<\/p>\n 2. Use Proxy Rotation<\/strong><\/p>\n For scraping a large number of reviews, especially across multiple businesses, proxy rotation is essential. It ensures your requests originate from different IPs, avoiding detection and blocking. Rayobyte<\/a> offers reliable proxy services.<\/p>\nPrerequisites<\/h2>\n
\n
pip install requests beautifulsoup4 pandas<\/pre>\n
\n
Step 1: Setting Up the Yahoo Scraper<\/h2>\n
requests<\/code> library to send an HTTP GET<\/strong> request and retrieve the HTML content of the search results page.<\/p>\nimport requests\nfrom bs4 import BeautifulSoup\nimport pandas as pd\nimport re\n\n\n# Function to fetch Yahoo search results\ndef fetch_yahoo_search_results(query, start=0):\n headers = {\n 'User-Agent': 'Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/114.0.0.0 Safari\/537.36'\n }\n base_url = \"https:\/\/search.yahoo.com\/search\"\n params = {\n 'p': query, # The search query\n 'b': start + 1 # Starting position of the results (1-based index)\n }\n\n response = requests.get(base_url, headers=headers, params=params)\n\n if response.status_code == 200:\n return response.text\n else:\n print(f\"Failed to fetch results. HTTP Status Code: {response.status_code}\")\n return None<\/pre>\nStep 2: Parsing the HTML Content<\/h2>\n
BeautifulSoup<\/code> to extract search results. The titles, links, and descriptions are typically located within specific HTML tags or classes.<\/p>\n# Function to parse the HTML content\ndef parse_search_results(html_content):\n soup = BeautifulSoup(html_content, 'html.parser')\n search_results = []\n\n for result in soup.select('.Sr'): # Yahoo search result container class\n title_tag = result.select_one('h3')\n link_tag = title_tag.a if title_tag else None\n description_tag = result.select_one('p')\n title = link_tag['aria-label'] if link_tag else 'N\/A'\n link = link_tag['href'] if link_tag else 'N\/A'\n description = description_tag.text if description_tag else 'N\/A'\n\n # Remove <b> and <\/b> tags from the title\n title = re.sub(r'<\/?b>', '', title)\n\n search_results.append({\n 'Title': title,\n 'Link': link,\n 'Description': description\n })\n\n return search_results<\/pre>\nStep 3: Saving Data to CSV<\/h2>\n
# Function to save data to a CSV file\ndef save_to_csv(data, filename=\"yahoo_search_results.csv\"):\n df = pd.DataFrame(data)\n df.to_csv(filename, index=False)\n print(f\"Data saved to {filename}\")<\/pre>\nStep 4: Running the Scraper<\/h2>\n
if __name__ == \"__main__\":\n query = \"Python\"\n num_pages = 5 # Number of pages to fetch\n\n all_search_results = []\n\n for page in range(num_pages):\n start = page * 10 # Assuming 10 results per page\n print(f\"Fetching search results for page {page + 1}...\")\n html_content = fetch_yahoo_search_results(query, start)\n\n if html_content:\n print(\"Parsing search results...\")\n search_results = parse_search_results(html_content)\n all_search_results.extend(search_results)\n else:\n print(\"Failed to scrape Yahoo search results.\")\n break\n\n print(\"Saving results to CSV...\")\n save_to_csv(all_search_results)\n\n print(\"Yahoo scraping completed successfully!\")<\/pre>\nExpected Output<\/h2>\n
yahoo_search_results.csv<\/code> will be created in your working directory. Since we set num_pages = 5<\/code>, you will get a total of 35 results (5 results per page across 5 pages). Here\u2019s an example of what the contents of the CSV file might look like:![\"yahoo](\"https:\/\/rayobyte.com\/community\/wp-content\/uploads\/2024\/12\/Screenshot-2024-12-22-190322.png\" "\\"\\"")
<\/p>\nBest Practices for Scraping<\/h2>\n
\n