-
Raul Marduk changed their photo a year ago
-
Raul Marduk became a registered member a year ago
-
Straton Owain replied to the discussion Is Python really the best language for web scraping, or should I consider altern in the forum General Web Scraping a year ago
Reply to Is Python really the best language for web scraping, or should I consider altern
I primarily use Python because of its community support and tools, but for sites with complex anti-bot measures, I sometimes switch to R for specialized parsing functions or use Rust when speed is critical.
-
Straton Owain replied to the discussion How do I identify hidden APIs that might be easier to scrape? in the forum General Web Scraping a year ago
Reply to How do I identify hidden APIs that might be easier to scrape?
Sometimes, the API endpoint is hinted at in the page’s HTML source. A quick search for URLs or ‘endpoint’ keywords can reveal hidden paths.
-
Straton Owain replied to the discussion How can I optimize my scraping code for faster performance? in the forum General Web Scraping a year ago
Reply to How can I optimize my scraping code for faster performance?
Use efficient parsers. Lxml and BeautifulSoup’s lxml parser are faster than BeautifulSoup’s default parser. For large datasets, I opt for lxml for speed.
-
Straton Owain replied to the discussion How can I manage session-based scraping effectively? in the forum General Web Scraping a year ago
Reply to How can I manage session-based scraping effectively?
Another trick is to rotate through multiple user sessions. Each session can then have its own set of cookies, avoiding overloading a single session.
-
Straton Owain started the discussion What are the advantages of using e-commerce APIs over direct scraping? in the forum General Web Scraping a year ago
What are the advantages of using e-commerce APIs over direct scraping?
APIs typically return structured JSON data, which is easier to parse and doesn’t require HTML parsing, saving time on data cleaning.
-
Straton Owain changed their photo a year ago
-
Straton Owain became a registered member a year ago
-
Yannig Avicenna replied to the discussion How can I scrape websites with infinite scroll without losing data? in the forum General Web Scraping a year ago
Reply to How can I scrape websites with infinite scroll without losing data?
If you’re using Python, selenium has built-in support for scrolling. Just use a loop that scrolls down and checks if new data has loaded. I add a try-catch block in case loading fails.
- Load More
