{"id":834,"date":"2024-09-02T13:20:38","date_gmt":"2024-09-02T13:20:38","guid":{"rendered":"https:\/\/dev2.rayobyte.com\/community\/?post_type=scraping_project&p=834"},"modified":"2024-10-09T16:24:10","modified_gmt":"2024-10-09T16:24:10","slug":"building-a-dynamic-web-crawler-in-php-a-practical-tutorial-scraping-carousell-com-my-as-an-example","status":"publish","type":"scraping_project","link":"https:\/\/rayobyte.com\/community\/scraping-project\/building-a-dynamic-web-crawler-in-php-a-practical-tutorial-scraping-carousell-com-my-as-an-example\/","title":{"rendered":"Building a Dynamic Web Crawler in PHP: A Practical Tutorial Scraping Carousell.com.my As An Example"},"content":{"rendered":"

Web scraping is an invaluable skill for anyone looking to extract and analyze data from websites. Whether you’re gathering data for research, monitoring prices, or simply collecting information, a web crawler can help automate the process. In this tutorial, we’ll guide you through building a dynamic web crawler using PHP, with Carousell Malaysia (carousell.com.my<\/code>) as our example website.<\/p>\n

We’ll create a PHP script that starts from a given URL, dynamically discovers all relevant pages, and extracts specific data like the product title, featured image URL, description, price, and condition. Additionally, we’ll ensure that the script can operate through a proxy, which is useful for scraping sites that have access restrictions or rate limits.<\/p>\n

Introduction<\/h3>\n

PHP is a versatile and widely-used scripting language that makes it easy to build a basic web crawler. In this tutorial, we’ll focus on crawling a sample website\u2014Carousell Malaysia. We’ll show you how to set up a dynamic crawler that can navigate through pages, extract specific data, and save the results in a CSV file.<\/p>\n

This tutorial is ideal for developers, data analysts, and anyone interested in web scraping. By the end, you’ll have a fully functional PHP crawler that you can adapt to other websites as needed.<\/p>\n

Prerequisites<\/h3>\n

Before you begin, make sure you have:<\/p>\n

    \n
  • A working knowledge of PHP.<\/li>\n
  • A PHP environment set up on your local machine.<\/li>\n
  • A text editor or an integrated development environment (IDE).<\/li>\n
  • Access to a web server (like Apache or Nginx) with PHP support.<\/li>\n
  • If needed, proxy details (IP, port, username, and password) for sites that require proxy access.<\/li>\n<\/ul>\n

    Step 1: Set Up the PHP Crawler Script<\/h3>\n

    The first step is to create the PHP script that will perform the web crawling. Below is the PHP code you’ll use:<\/p>\n

    <?php\n\n\/\/ Define the base URL to start crawling\n$base_url = \"https:\/\/www.carousell.com.my\/\";  \/\/ Input the URL you want to start crawling from\n\n\/\/ Define your proxy details (if needed)\n$proxy = \"your_proxy_ip:your_proxy_port\";  \/\/ Input your proxy IP and port\n$proxy_auth = \"username:password\";  \/\/ Input your proxy username and password\n\n\/\/ Initialize an array to store crawled data and visited URLs\n$data = [];\n$visited_urls = [];\n\n\/\/ Function to get the HTML content of a URL using a proxy\nfunction get_html($url, $proxy = null, $proxy_auth = null) {\n    $options = [\n        'http' => [\n            'method' => \"GET\",\n            'header' => \"User-Agent: PHP-Crawler\/1.0rn\"\n        ]\n    ];\n\n    \/\/ Add proxy settings if provided\n    if ($proxy && $proxy_auth) {\n        $options['http']['header'] .= \"Proxy-Authorization: Basic \" . base64_encode($proxy_auth) . \"rn\";\n        $options['http']['proxy'] = \"tcp:\/\/$proxy\";\n        $options['http']['request_fulluri'] = true;\n    }\n\n    $context = stream_context_create($options);\n    return file_get_contents($url, false, $context);\n}\n\n\/\/ Function to extract product details from the HTML\nfunction extract_data($html, $url) {\n    $dom = new DOMDocument;\n    \n    \/\/ Suppress errors due to malformed HTML\n    @$dom->loadHTML($html);\n\n    $data = [];\n\n    \/\/ XPath to locate specific elements\n    $xpath = new DOMXPath($dom);\n\n    \/\/ Extract the title\n    $title_nodes = $xpath->query('\/\/meta[@property=\"og:title\"]');\n    $title = ($title_nodes->length > 0) ? $title_nodes->item(0)->getAttribute('content') : 'N\/A';\n\n    \/\/ Extract the featured image\n    $image_nodes = $xpath->query('\/\/meta[@property=\"og:image\"]');\n    $featured_image = ($image_nodes->length > 0) ? $image_nodes->item(0)->getAttribute('content') : 'N\/A';\n\n    \/\/ Extract description\n    $description_nodes = $xpath->query('\/\/meta[@property=\"og:description\"]');\n    $description = ($description_nodes->length > 0) ? $description_nodes->item(0)->getAttribute('content') : 'N\/A';\n\n    \/\/ Extract price\n    $price_nodes = $xpath->query('\/\/div[contains(@class, \"MMOxT3+_JqMFMZ4RTs4g\") and contains(@class, \"MMOxT3+_eJf5luTP5NQvc\")]');\n    $price = ($price_nodes->length > 0) ? trim($price_nodes->item(0)->nodeValue) : 'N\/A';\n\n    \/\/ Extract condition\n    $condition_nodes = $xpath->query('\/\/span[contains(@class, \"_2vJS6pPzGwaXAfZEqgKkZn\") and contains(text(), \"Condition\")]');\n    $condition = ($condition_nodes->length > 0) ? trim($condition_nodes->item(0)->nextSibling->nodeValue) : 'N\/A';\n\n    \/\/ Add the extracted data to the result array\n    $data = [\n        'title' => $title,\n        'image' => $featured_image,\n        'url' => $url,\n        'description' => $description,\n        'price' => $price,\n        'condition' => $condition\n    ];\n\n    return $data;\n}\n\n\/\/ Function to extract and crawl URLs from a page\nfunction extract_and_crawl_urls($html, $base_url, &$data, &$visited_urls, $proxy, $proxy_auth) {\n    $dom = new DOMDocument;\n    \n    \/\/ Suppress errors due to malformed HTML\n    @$dom->loadHTML($html);\n\n    \/\/ XPath to locate anchor elements\n    $xpath = new DOMXPath($dom);\n    $url_nodes = $xpath->query('\/\/a[@href]');\n\n    \/\/ Crawl each discovered URL\n    foreach ($url_nodes as $url_node) {\n        $relative_url = $url_node->getAttribute('href');\n        $absolute_url = filter_var($relative_url, FILTER_VALIDATE_URL) ? $relative_url : rtrim($base_url, '\/') . '\/' . ltrim($relative_url, '\/');\n        \n        \/\/ Filter and ensure we only crawl carousell.com.my pages and avoid already visited URLs\n        if (strpos($absolute_url, 'carousell.com.my') !== false && !isset($visited_urls[$absolute_url])) {\n            $visited_urls[$absolute_url] = true;\n            \n            \/\/ Crawl the page if it contains the structure of a product listing page\n            if (preg_match('\/\/p\/\/', $absolute_url)) {\n                $page_html = get_html($absolute_url, $proxy, $proxy_auth);\n                $page_data = extract_data($page_html, $absolute_url);\n                $data[] = $page_data;\n            } else {\n                \/\/ Recursively crawl other pages to discover more URLs\n                $page_html = get_html($absolute_url, $proxy, $proxy_auth);\n                extract_and_crawl_urls($page_html, $base_url, $data, $visited_urls, $proxy, $proxy_auth);\n            }\n        }\n    }\n}\n\n\/\/ Start crawling from the base URL\n$html = get_html($base_url, $proxy, $proxy_auth);\nextract_and_crawl_urls($html, $base_url, $data, $visited_urls, $proxy, $proxy_auth);\n\n\/\/ Output the results to a CSV file\n$csv_file = fopen(\"crawled_data.csv\", \"w\");\n\n\/\/ Write the headers to the CSV\nfputcsv($csv_file, ['Title', 'Featured Image URL', 'Page URL', 'Description', 'Price', 'Condition']);\n\n\/\/ Write each row of data\nforeach ($data as $row) {\n    fputcsv($csv_file, $row);\n}\n\nfclose($csv_file);\n\necho \"Crawling complete! Check crawled_data.csv for the results.n\";\n\n?>\n<\/pre>\n
    Step 2: Input Your Variables<\/h3>\n
    Before running the script, you’ll need to input specific details into the code:<\/p>\n
      \n
    1. Base URL to Crawl<\/strong>:\n
        \n
      • Line 4<\/strong>: Replace the $base_url<\/code> variable with the URL of the site you want to start crawling from. In this tutorial, we start with https:\/\/www.carousell.com.my\/<\/code>.<\/li>\n<\/ul>\n<\/li>\n
      • Proxy Settings<\/strong> (Optional):\n
          \n
        • Line 7<\/strong>: If you’re using a proxy, replace the $proxy<\/code> variable with your proxy’s IP and port (e.g., \"123.456.789.000:8080\"<\/code>).<\/li>\n
        • Line 8<\/strong>: Replace the $proxy_auth<\/code> variable with your proxy’s username and password (e.g., \"user:password\"<\/code>). If no proxy is needed, you can leave these variables as null<\/code>.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n
          Step 3: Running the Crawler<\/h3>\n
          With your variables set, you can now run the script:<\/p>\n
            \n
          1. Save the PHP Script<\/strong>: Save the provided PHP code as crawler.php<\/code> in your project directory.<\/li>\n
          2. Run the Script<\/strong>: Open your terminal or command prompt, navigate to the directory where crawler.php<\/code> is saved, and run:<\/li>\n<\/ol>\n
            php crawler.php<\/pre>\n
              \n
            1. Check the Output<\/strong>: After the script finishes running, you’ll find a crawled_data.csv<\/code> file in the same directory. This file contains all the data extracted by your crawler.<\/li>\n<\/ol>\n
              Step 4: Understanding the Output<\/h3>\n
              The generated CSV file will have the following columns:<\/p>\n
                \n
              • Title<\/strong>: The title of the product listing.<\/li>\n
              • Featured Image URL<\/strong>: The URL of the featured image associated with the product.<\/li>\n
              • Page URL<\/strong>: The URL of the product listing page.<\/li>\n
              • Description<\/strong>: The description provided in the product listing.<\/li>\n
              • Price<\/strong>: The listed price of the product.<\/li>\n
              • Condition<\/strong>: The condition of the product (e.g., “Brand New”, “Used”).<\/li>\n<\/ul>\n
                Here\u2019s an example of how the CSV might look:<\/p>\n\n\n\n\n\n
                Title<\/th>\nFeatured Image URL<\/th>\nPage URL<\/th>\nDescription<\/th>\nPrice<\/th>\nCondition<\/th>\n<\/tr>\n<\/thead>\n
                Clearance Dell Alienware X17 R2 X15 R2 M17 R5 M15 R7 M16 R1<\/td>\nImage URL<\/a><\/td>\nProduct URL<\/a><\/td>\nBrand new Alienware models available at clearance prices!<\/td>\nRM 9,999<\/td>\nBrand New<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
                Conclusion<\/h3>\n
                In this tutorial, we\u2019ve created a dynamic web crawler in PHP that can discover and extract data from pages on carousell.com.my<\/code>. The script is designed to start from a specified URL, dynamically navigate through the site, and gather relevant product information. By incorporating proxy support, you can ensure that your crawler operates effectively even in restricted or rate-limited environments.<\/p>\n
                This tutorial provides a practical example of using PHP for web scraping and can be adapted to other websites or more complex scraping tasks as needed. Whether you’re gathering data for analysis, research, or other purposes, this PHP crawler offers a solid foundation for your web scraping projects.<\/p>\n","protected":false},"excerpt":{"rendered":"
                Web scraping is an invaluable skill for anyone looking to extract and analyze data from websites. Whether you’re gathering data for research, monitoring prices, or…<\/p>\n","protected":false},"author":18,"featured_media":835,"comment_status":"open","ping_status":"closed","template":"","meta":{"rank_math_lock_modified_date":false},"categories":[],"class_list":["post-834","scraping_project","type-scraping_project","status-publish","has-post-thumbnail","hentry"],"_links":{"self":[{"href":"https:\/\/rayobyte.com\/community\/wp-json\/wp\/v2\/scraping_project\/834","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rayobyte.com\/community\/wp-json\/wp\/v2\/scraping_project"}],"about":[{"href":"https:\/\/rayobyte.com\/community\/wp-json\/wp\/v2\/types\/scraping_project"}],"author":[{"embeddable":true,"href":"https:\/\/rayobyte.com\/community\/wp-json\/wp\/v2\/users\/18"}],"replies":[{"embeddable":true,"href":"https:\/\/rayobyte.com\/community\/wp-json\/wp\/v2\/comments?post=834"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rayobyte.com\/community\/wp-json\/wp\/v2\/media\/835"}],"wp:attachment":[{"href":"https:\/\/rayobyte.com\/community\/wp-json\/wp\/v2\/media?parent=834"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rayobyte.com\/community\/wp-json\/wp\/v2\/categories?post=834"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}