[Osman Devaki]

To handle responses, I use the -i flag to include headers in the output or redirect the response to a file for analysis. This helps debug issues with the request.

[Osman Devaki]

One challenge of scraping Lazada Thailand is handling the rich media content such as images, JavaScript-loaded product details, and other dynamic elements. The static data can be scraped easily using BeautifulSoup, but if the site relies on JavaScript to load the products, you’ll need to find the API calls the site makes to load the data. This approach avoids loading the entire page and is much faster. Here’s an example of how you can extract the product titles and prices directly from the page’s HTML.

import requests
from bs4 import BeautifulSoup
url = 'https://www.lazada.co.th/catalog/?q=mobile'
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')
products = soup.find_all('div', class_='c2prKC')
for product in products:
    title = product.find('div', {'class': 'c16H9d'}).text.strip()
    price = product.find('span', {'class': 'c13VH6'}).text.strip()
    print(f'Product: {title}, Price: {price}')