-
Thibaut Ron became a registered member a year ago
-
Zahir Xiu replied to the discussion What are the best methods for scraping data from dynamically-loaded websites? in the forum General Web Scraping a year ago
Reply to What are the best methods for scraping data from dynamically-loaded websites?
-
Sometimes, inspecting the network activity in dev tools can reveal a JSON API endpoint that loads the data, allowing you to directly query that endpoint instead of scraping rendered content.
-
-
Zahir Xiu started the discussion How can I optimize my scraping code for faster performance? in the forum General Web Scraping a year ago
How can I optimize my scraping code for faster performance?
Multithreading is a big help. Using libraries like Concurrent Futures or asyncio in Python allows me to run multiple requests simultaneously, which speeds up scraping significantly.
-
Zahir Xiu changed their photo a year ago
-
Gervasius Dagny replied to the discussion What are the benefits of using Elixir for building scalable web apps? in the forum General Web Scraping a year ago
Reply to What are the benefits of using Elixir for building scalable web apps?
Phoenix, Elixir’s web framework, is designed to handle real-time applications with high throughput, making it perfect for chat apps or streaming.
-
Gervasius Dagny replied to the discussion Should I learn Elm for frontend web development in 2024? in the forum General Web Scraping a year ago
Reply to Should I learn Elm for frontend web development in 2024?
If you enjoy functional programming, Elm might be worth learning, but its ecosystem is much smaller than JavaScript’s.
-
Gervasius Dagny replied to the discussion What benefits does C# offer over Python for desktop application dev environment? in the forum General Web Scraping a year ago
Reply to What benefits does C# offer over Python for desktop application dev environment?
For Windows development, C# is a better choice since it has better support for GUI frameworks like WPF and WinForms.
-
Gervasius Dagny replied to the discussion How do you handle memory management in C++ versus Rust? in the forum General Web Scraping a year ago
Reply to How do you handle memory management in C++ versus Rust?
Rust uses an ownership model with a strict borrow checker, which prevents memory leaks and dangling pointers at compile time.
-
Gervasius Dagny replied to the discussion How does functional programming in Haskell compare to Scala? in the forum General Web Scraping a year ago
Reply to How does functional programming in Haskell compare to Scala?
Haskell excels at academic or research projects where functional purity is key, while Scala is more suitable for enterprise-grade software.
-
Gervasius Dagny started the discussion What are the best methods for scraping data from dynamically-loaded websites? in the forum General Web Scraping a year ago
What are the best methods for scraping data from dynamically-loaded websites?
Headless browsers like Selenium or Puppeteer are ideal for handling dynamic sites. They can render JavaScript, so you can wait for content to load before scraping.
- Load More
