Keeping it Current: Web Scraping AJAX Pages, and Why it Helps

From stock market prices to fantasy football statistics, there’s no shortage of applications for web scraping. Web scraping is a powerful tool for businesses today, as the practice allows you to scout the web for desired data and extract it from each URL. The only problem is: some pages are harder to scrape than others.

Many pages have continuously updating feeds, changing multiple times per minute. These pages are called asynchronous Javascript and XML (AJAX). Their outputs are more dynamic than the average site. This means that web scrapers have to keep up with their constantly changing data to stay current. So how do you hit a moving target? Keep reading our how-to guide on web scraping AJAX pages to find out all you need to know.

What is AJAX Web Scraping?

Simply put, AJAX is an efficient way to create web pages with dynamic content. It works by trading smaller packets of data with the server on the backend, making it easier to update continuously without refreshing the page.

Some common examples of AJAX pages are: 

• Google Maps
• Gmail
• YouTube
• Stock market prices

Plenty of other websites use AJAX formatting to keep their pages current, and they must be scraped each time their data points change. Otherwise, the data that scrapers retrieve will quickly become outdated and will be useless to the decision-makers that rely on them.

How to Scrape AJAX Pages

For conventional static pages, web scraping entails the following general steps:

1. Identify the site you wish to scrape.
2. Gather the URLs of the pages you intend to scrape. 
3. Request the HTMLs of each page from their URLs.
4. Use selectors to locate the desired data point within each HTML.
5. Consolidate the data points into a single organized format like a JSON or CSV file.

The difference between scraping standard websites and scraping AJAX websites is that the latter requires more than manually identifying the HTML of the page you wish to scrape. Because the pages make a periodic server request by Javascript to update the given data point, scraping an AJAX page requires you to determine the format and destination of the server request to copy it, as well as the response so that you can extract it.

Tools for AJAX Web Scraping

Multiple tools exist to help you scrape an AJAX webpage. The most common AJAX web scraping modules are:

• Scrapy
• Requests
• Beautiful Soup
• Sky

Other tools like CSS selectors and robots.txt files can help you crawl and scrape the web, but Beautiful Soup has proven especially popular for AJAX web scraping. These may only retrieve the static portion of the HTML, though, when it is the dynamic portion — the part sent as a server request by Javascript — that is needed. 

Identifying the Request

The first step in scraping an AJAX site is to find the request for the data point you seek. Instead of looking simply for the HTML as you would for a static page, the data point will be the server request that is made by Javascript. This can be done using the Chrome developer tools by selecting “View”, followed by “Developer”, and then “Developer Tools”.

The “Developer Tools” box should appear on your screen with a “Network” tab and XHR subsection to follow. You may need to refresh the page for this subsection to populate.

Under the “Headers” tab, the field “Form Data” can be found. This contains the AJAX request. The amount of code here can be daunting, but the parameters designating the request and endpoint are all that are needed to form your web scraper.

Formatting the Response

Now that you have identified the server request, the next step is to see how your data point response is returned. This can be found under the “Response” tab. It will reveal the format in which your data is returned — likely a JSON format or something similar. With the output parameter and response format identified, you may configure your scrapers accordingly.

Creating Your Web Scraper

Once the server request parameter and the response format have been found, you are ready to write your web scraper. The contents of your scraper will depend greatly on the application.

If written in Python (other languages would work similarly), a general outline would look as follows:

1. Create a project.
2. Create a Virtualenv.
3. Install a requests library and create a Python file.
4. Open the Python file using your favorite text editor (Sublime, Atom, Vim, etc.).
5. Create a function that replicates the AJAX server request parameter. 
6. Create another function that parses the response — perhaps using Beautiful Soup or a CSS selector.
7. Create a function that repeats the process for each page you intend to scrape.
8. Designate a location for your newly scraped data to be stored.

When writing your scraper in other languages, consider using the Javascript engines “SpiderMonkey” for C++, “Rhino” for Java, or ICodeCompiler/CodeDOM for .Net — all of which are helpful for more advanced scraping use cases.

Rayobyte: Setting Your Scrapers up for Success!

Whether you’re web scraping AJAX web pages or static ones, your scraping system will still depend on one crucial element to thrive: residential proxies. Many sites have privacy mechanisms that prevent the use of web scrapers and may ban users if they’re found extracting their data. By providing alternative IP addresses, an effective residential proxy network aids in ban prevention, giving you access to the data you need to guide your decisions.

At Rayobyte, we’re committed to two things: the success of our clients, and ethically sourced proxy acquisition — and the two go hand-in-hand. We’re known for our custom solutions and personal attention to each client. Our own CEO even works one-on-one with some customers.

Part of that includes ethical proxy acquisition, which we’ve set the bar for. All of our partners can limit the terms of their residential proxy usage and opt out at any time. Contact us to learn more about our proxy acquisition policies, or get started today for the proxies you need for an effective AJAX scraping system.

The information contained within this article, including information posted by official staff, guest-submitted material, message board postings, or other third-party material is presented solely for the purposes of education and furtherance of the knowledge of the reader. All trademarks used in this publication are hereby acknowledged as the property of their respective owners.