How can I scrape product details from JD Central Thailand using Python n Scrapy?

[Osman Devaki]

One challenge of scraping Lazada Thailand is handling the rich media content such as images, JavaScript-loaded product details, and other dynamic elements. The static data can be scraped easily using BeautifulSoup, but if the site relies on JavaScript to load the products, you’ll need to find the API calls the site makes to load the data. This approach avoids loading the entire page and is much faster. Here’s an example of how you can extract the product titles and prices directly from the page’s HTML.

import requests
from bs4 import BeautifulSoup
url = 'https://www.lazada.co.th/catalog/?q=mobile'
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')
products = soup.find_all('div', class_='c2prKC')
for product in products:
    title = product.find('div', {'class': 'c16H9d'}).text.strip()
    price = product.find('span', {'class': 'c13VH6'}).text.strip()
    print(f'Product: {title}, Price: {price}')

[Lalitha Kreka]

To scrape product details from JD Central Thailand, you’ll need to target the HTML elements containing product names, prices, and availability. JD Central’s product listings typically contain structured data inside specific class attributes. Scrapy makes it easy to extract this data using CSS selectors. Handling pagination correctly is essential to scrape products across multiple pages, which can be done using Scrapy’s response.follow() method to navigate through links.

import scrapy
class JDSpider(scrapy.Spider):
    name = 'jd_spider'
    start_urls = ['https://www.jd.co.th/th/search?query=laptop']
    def parse(self, response):
        for product in response.xpath('//div[@class="product-item"]'):
            title = product.xpath('.//div[@class="product-name"]/text()').get()
            price = product.xpath('.//span[@class="product-price"]/text()').get()
            availability = product.xpath('.//span[@class="product-availability"]/text()').get()
            yield {
                'title': title.strip(),
                'price': price.strip(),
                'availability': availability.strip(),
            }
        next_page = response.xpath('//a[@class="next"]/@href').get()
        if next_page:
            yield response.follow(next_page, self.parse)

[Aston Martial]

JD Central Thailand utilizes JavaScript and dynamic content loading for product information, so it’s important to work with Scrapy’s middleware settings to handle AJAX requests. Scrapy is excellent for scraping static HTML content, but you may need to integrate it with other tools like Splash if dealing with pages that require JavaScript rendering. By using XPath selectors, you can easily extract the desired product data such as name, price, and availability from JD Central’s HTML structure.

import scrapy
class JDSpider(scrapy.Spider):
    name = 'jd_spider'
    start_urls = ['https://www.jd.co.th/th/products']
    def parse(self, response):
        for product in response.xpath('//div[@class="product-card"]'):
            name = product.xpath('.//h3[@class="product-name"]/text()').get()
            price = product.xpath('.//span[@class="price"]/text()').get()
            availability = product.xpath('.//span[@class="availability-status"]/text()').get()
            yield {
                'name': name.strip(),
                'price': price.strip(),
                'availability': availability.strip(),
            }
        next_page = response.xpath('//a[@class="next-page"]/@href').get()
        if next_page:
            yield response.follow(next_page, self.parse)