FARHAN AHMED
-
FARHAN AHMED replied to the discussion Best practices for web scraping to avoid getting blocked by websites. in the forum General Web Scraping 4 days ago
Best practices for web scraping to avoid getting blocked by websites.
If a website doesn’t use JavaScript to load dynamic content, you can use Python’s
requests
library, passing headers and a proxy to avoid blocking and achieve a high success rate. If the site relies on JavaScript, however, you’ll need to use automation tools like Playwright or Selenium. From my testing, Selenium Stealth works best for avoiding… Read more -
FARHAN AHMED replied to the discussion Best techniques to rotate proxies when scraping websites. in the forum General Web Scraping 4 days ago
Best techniques to rotate proxies when scraping websites.
PADI proxies are better and more reliable than free proxies. If you have a list of URLs, you can use a for loop to iterate through them, applying a new proxy on each loop iteration. This technique helps you avoid blocks. Other factors also affect success, such as the type of automation browser you are using.
-
FARHAN AHMED changed their photo 7 weeks ago
-
FARHAN AHMED changed their photo 7 weeks ago
-
FARHAN AHMED changed their photo 7 weeks ago
-
FARHAN AHMED became a registered member 7 weeks ago