Using News Proxies To Gather Information from News Sites

A news aggregator is a site that gathers the most relevant news content online from a wide number of sites and puts it all in one place for you to enjoy. This way, you can avoid the hassle of looking across the web for different sources and angles. Running an aggregator is simple and, generally, cost-effective. You typically won’t need to pay for the content you aggregate. If anything, you’ll only have to invest in technology to keep everything running smoothly.‌

However, news and other types of relevant content are scattered all over the world wide web and updated in real time. It’d be a repetitive, time-consuming task —and a highly unrealistic one, for that matter— to manually perform your own news compilation. Scraping the web for news is the most viable option to create an efficient news aggregator that your audience will love to use.‌

In order to extract the necessary data, you’ll need to use an effective news scraping tool or create one yourself. While both options are useful, coding your own news scraper allows for another level of customization that will better suit your specific needs. Now, you might be wondering, “What is a news scraper, and how do I create news aggregation?” Lucky for you, we gathered all the information you’ll need to code an effective web scraper to build a news aggregator.‌

What Is a News Aggregator?

A news aggregator often presents a summary of the most relevant events that happen daily across the globe. They gather reports, analyses, articles, and more from a wide range of sources all over the web, and present them to you neatly and in a digestible format. These types of sites save you valuable time and effort in searching for information on the topics you care about most. You can find news aggregators that focus, among several other categories on:

• Technology
• Marketing
• Business
• Politics
• Finance
• Design
• Sports
• Entertainment
• Health
• Lifestyle‌

‌But, is content aggregation a type of plagiarism? The short answer is no. While it’s true that content aggregation presents other authors’ work, it always gives proper credit and relevant links to the source. Plagiarism, on the other hand, means using this content without permission and credit and is a serious and punishable offense.

Incorporating plagiarized content into your aggregator — or any other type of site— will also harm your SEO efforts and make you rank lower on search engine result pages. This will ultimately prevent your aggregator from getting to the right eyes and reduce the earnings you can make from it.  In other words, steer clear!‌

Content aggregation, curation, and syndication

When trying to build a content aggregator, you must get familiar with some useful concepts. There’s a common misunderstanding surrounding aggregating, curating, and syndicating content. However, there are some very obvious differences between these three notions.

Content aggregation means collecting and grouping content automatically. As stated above, this content always focuses on a particular subject matter and comes from multiple sources.‌

Content curation, however, is more specific. It’s more about manually selecting the best content or the most valuable content for your audience. This method is more customized and often contains some commentary, context, or opinions.‌

Lastly, content syndication is all about republishing content from aggregators indicating the author and the source where it first appeared. It helps authors build credibility and increase their visibility.‌

Types of aggregators

Also known as news readers, feed readers, or RSS readers, news aggregators scrape the news and store them in a handy location. These types of content aggregators flood the web, and they’re highly useful sites that provide all the information you want to see gathered in one place. There are many different types of content aggregators available. The most common examples are:‌

Blog aggregators

These are often used to collect blog information such as URLs, author bio, titles, etc. A blog aggregator lets you gather information on the latest blogs for your audience to enjoy.‌

Social media aggregator

‌This type allows you to collect the data you need from all social media platforms. Social media aggregators are incredibly useful for digital marketers who need news on the audiences’ sentiment towards a specific service or product to improve marketing strategies and such.‌

‌eCommerce aggregator

These aggregators collect product information from numerous platforms. It’s useful for those running an online business and can help with price monitoring, competitor observation, reviews analysis, and more.‌

Benefits of News Aggregation

Constantly collecting valuable content will make your audience feel more interested and build a lasting relationship with your site. That’s where learning how to make your own news scraper comes in. It will give your news aggregation service fresh information to keep your followers engaged. Contrary to popular belief, aggregators do not typically compete with brands. Instead, they help them by providing additional exposure and reach among their target market.‌

Content aggregators offer a win-win situation to all parties involved. Users get convenience and immediacy, while publishers can:

• Reach a wider audience
• Produce more revenue
• Earn recognition for their work
• Gain powerful insights on consumer habits
• Learn more about content trends‌

Web scraping is the main pilar of news aggregation in that it allows you to collect news information easily and effectively. Scraping also facilitates the process of exporting the extracted information to a database, API, or even an Excel file — whatever’s more convenient to you. All this allows you to have the latest news conveniently updated with a certain frequency. Some other perks news aggregators offer are:

• Simplicity — Aggregators offer a centralized location for all information, which makes consumption a lot easier.
• Scope — Aggregated platforms collect data from numerous sites across the web, which provides them with a wider variety of information for their users to take advantage of.
• Personalization — By allowing users to select what content they want to be exposed to, they can reduce the aggregator’s scope and avoid being overwhelmed by topics they’re not interested in. Providing filtering is a must.
• Cost-effectiveness — Aggregators don’t need to pay writers or incur other costs simply because they do not generate original content. They also don’t need to pay for advertising.‌

What to Know Before Building Your Own News Aggregator

Before you learn how to build a news scraper, you need to come up with a strategy to develop a useful content aggregator your audience will actually want to use. Here are the main steps you must follow:‌

Define the type of content aggregator website you want to build

Perhaps the most important stage in content aggregator creation, deciding exactly what content you want to present your users with will pave the way to make better operational decisions. You’ll also want to determine how often you’ll update this content. As mentioned above, there are different categories of content aggregator platforms, and you’ll have to choose between becoming a:

• News aggregator – This will allow you to showcase the most recent and relevant events from the local scene or from across the globe.
• Review aggregator –This will let you collect reviews on products and services for users to easily compare and contrast.
• Social network aggregator – This allows you to gather existing social media posts and display them on a single feed. 
• Poll aggregator – It lets you track and collect polls from different sources to measure public sentiment. 
• Hybrid aggregator – This kind puts together different types of content from diverse platforms.‌

Select a good name

Once you’ve defined who you want to reach and the concept behind your aggregator, you can finally come up with the most appropriate name for it. Finding the right name for your news aggregator can make a world of difference between success or becoming a flop. After all, the domain name is your first chance at a good impression and will impact the way your audience sees you. Keep it short, catchy, and SEO-friendly so your audience can easily find you and remember you once they do.‌

Design a sleek site

Not only does your user interface need to be easy to use, but it also helps if it’s attractive to the user. Although appearances are not meant to be that important if you’re offering some killer content, the truth is that sleek, sophisticated design won’t hurt. Remember, less is more, so keeping your layout minimal will help avoid distractions and emphasize the content. Use colors and fonts that make reading much easier for your user.‌

You have two options when it comes to deciding how to create your site:‌

Using WordPress

This option saves you time and money by allowing you to skip wireframes, design, and testing altogether. WordPress also gives you a more cost-effective option by cutting down on web development expenses. However, this tool won’t allow you to create a unique product, and since it’s open-source, it’ll render your site more vulnerable to the work of hackers and other malicious actors.‌

From scratch

This alternative gives you the opportunity to build a one-of-a-kind platform that differs from others available in the market. You’ll be able to customize both UI and UX features and make your site feel tailor-made. Building your site from scratch gives you more design customization options and doesn’t limit you in functionality.‌

Equip your readers with strong filtering features

‌Aggregators can easily get out of hand. They keep gathering data all the time, and although you don’t want to skimp on the content you’re offering, you don’t want to overwhelm your users either. That’s why you must always provide readers with a solid filter that lets them find what they’re looking for as soon as possible. The most obvious option is offering a keyword filtering feature. Or, you could always sort out content in categories and subcategories.‌

Choose your preferred monetization method

This stage might be a tad daunting, but it’s not as hard as it seems. You can offer your readers membership options and provide premium features, like social sharing, hiding sponsored ads, Slack integration, and unlimited sources for an additional fee. You can also resort to advertising monetization, whether you want to use Google Adsense or integrate native ads.‌

Find the best content aggregator tools and technologies

News aggregators rely on third-party sources to retrieve information, so you must identify which technologies support this essential task. You can use a custom web scraper for news compilation, or you can resort to prebuilt web scraping software.‌

If you’re not the most tech-savvy individual, using software like Octoparse, Scrapinghub, Diffbot, and others will save you time and effort. However, these tools can only handle so much information, so they’re not recommended if you’re looking to scrape more complex sites.‌

Step-by-Step Instructions for How to Make Your Own News Scraper‌

Learning how to build a news scraper lets you use frameworks like BeautifulSoup, Selenium, Cheerio, or Scrapy to generate a scalable web crawler that will let you extract relevant data from all kinds of news sites. Here are the steps you’ll need to follow if you opt for this alternative:‌

Install and import packages

To create a basic news scraper in your preferred programming language, you’ll need to install some modules that will make your life much easier. Suppose you use Python, which is currently the most popular coding language for these purposes. In that case, you’ll need a Python framework.‌

BeautifulSoup, for example, is a library under the bs4 package that can parse HTML and XML docs into Python and access elements by identifying them with tags. It will provide functions to access particular elements and extract relevant information from a specific site.‌

Once you’ve chosen your favorite language and framework, you’ll need requests, which is a module that provides BeautifulSoup with the HTML code of any site you want to scrape. You’ll also need urllib, which is Python’s URL handling module. In short, it helps define functions and classes to open certain URLs. Urllib can also let you call libraries such as:

• time: which lets you call the sleep() function to delay or suspend execution
• sys: to get information about exceptions, like the type of error, error object, etc.‌

Lastly, to handle and visualize the data that results from your scraping, you’ll need to import pandas. This library will help you use DataFrame to store data in tabular rows and columns, and manipulate observations and variables.‌

Make simple requests

The request module will let you make simple “get” requests to store the HTML from a specific page and store it into the page variable. To use it, you’ll need the URL of the page you want to scrape.‌

Requests is a helpful library with handy features and methods to send requests through HTTP, which functions as a request-response system between the server (or the system that hosts the site you’re trying to access) and the client (which, in this case, would be your web browser).‌

Sending “get” request to an URL indicates that you are looking to obtain data from a resource on the web. In your code, the basic syntax would be “requests.get(url, params={key: value}, args),” where “url” is the URL of the site you’re looking to gather data from, “params” is used to send a query string, and “args” is optional and can be any argument like:

• auth — to enable an HTTP authentication
• timeout — to establish the number of seconds to wait for a connection with the client before sending a response
• cert — to mention a cert file or key
• headers — to send HTTP headers to the URL
• stream — to define if the response should be streamed or immediately downloaded‌

Inspect the response object

Once you’ve sent your requests, you must always see the response code that the server sent back. This will be useful to detect errors. If your request succeeds, you’ll get an HTTP OK success status response.‌

You can access the full response as text to get the HTML of the page in a large string in Unicode using “page.text” or “page.content” commands. The latter will return the response in bytes. Once you have the responses, look for a specific substring of text. To know the response’s content type, check if you got back CVS, JSON, XML, or HTML.‌

Delay request time

Use the time module to delay request time. If, for example, you want to delay sending requests to a web server by two seconds, use the sleep() with a value of two seconds.‌

Extract content from HTML

Once you’ve made your HTTP request and got back your HTML content, you can parse your results to extract the values you need. You have a couple of options here. You could use regular expressions to look up HTML content. This alternative is the least recommended, but it will still help you find specific string patterns such as phone numbers, email addresses, and prices. Alternatively, you can use BeautifulSoup’s object Soup.‌

To understand how to code this part, you’ll first need to inspect the webpage you’re trying to scrape. To do so, you must:

• Go to the URL you’re inspecting.
• Press ctrl+shift+I to open the inspect window.
• Press ctrl+shift+C to select and inspect an element on the page.‌

To familiarize yourself with the inspect window, try selecting random elements on the page and see the changes that each produces. Those are the attributes you’ll need to understand the “li” part of the above-mentioned command and the HTML tags. Now, you can continue coding. Use the following command to help inspect how many news articles there are on the site. This will be useful to understand what you’ll need for pagination.‌

Find useful elements and attributes

Look up all anchor tags on the page you’re scraping. This comes in incredibly handy if you’re building a crawler and need to identify what page to visit next. Using a command like links = soup.find_all(“a”) will throw a division tag that contains a specific attribute value, while using the text.strip() function will return all text contained within a tag and strip extra spaces from the text string. The strip() function will get all values organized so that you can better understand your output file.‌

You must also inspect the page you’re scraping to get the “date” attribute. You can use the text function since there is a string containing this attribute and do the same to retrieve the source, and so on. If your news aggregator requires a dataset that’s not altered and you cannot manually gather, you can use get() to fetch a specific text label.‌ You can put all concepts together and try to fetch a specific number of different attributes.‌

Visualize and download your dataset

In order for the information you scrape to be useful for your news aggregator, you need to be able to visualize it in a digestible format. Use panda’s DataFrame module to visualize your data on Jupiter. Then you can write a new CSV file and save it into your machine in the same directory where you saved your Python file. This way, when you run your file on the command shell, it will create a CSV file in your .py file directory.‌

Keep in mind that running the same code multiple times might throw an error if it has already generated a dataset using the same file writing method. Another way to convert your DataFrame into a CSV file is using the to_csv() command:‌‌

path = ‘C:\\Users\\Kajal\\Desktop\\KAJAL\\Project\\Datasets\\’‌

data.to_csv(path+’NEWS.csv’)‌

Or, avoid any ambiguities and allow code portability with:‌

import os‌

data.to_csv(os.path.join(path,r’NEWS.csv’))‌

Tips for Scraping Different Types of News Websites

Convenience is the best thing you could offer to your target audience, and news aggregators are all about providing a shortcut to the most relevant content available. Now that you know the principles behind building the best news aggregator for your public, you must make the following considerations.‌

Choose how you’ll present the information

Rather than publishing full posts, most content aggregators available today redirect readers to the original source. Some of them give the option of having a sneak peek by displaying the first few sentences of an article. Yet, once the user clicks on the header, they’ll end up in the original site that posted the content. You can choose whether you want to follow their lead or showcase full articles.‌

Choose between aggregation or curation

News aggregation will always be much easier than curation in that it needs no additional steps. However, curation provides extra value to your readers that might be the difference between becoming their go-to news platform or just another one in the bunch. Users are most likely to stay loyal to a site that goes the extra mile to bring them the information they want to see.‌

Respect intellectual property rights

Whether you choose to curate or syndicate your content, make sure to never violate the publisher’s rights. Although aggregators typically cite sources and give credit where credit is due (they never claim the content’s originally theirs), it’s vital to always ask for permission from the original source before reposting their work.‌

That’s also the reason why most aggregators don’t display the content in full. Following these rules will help you avoid any legal trouble.‌

Follow Google algorithm updates

Your news aggregator will handle loads of different data from multiple sources. That’s why understanding Google algorithms and monitoring their changes regularly is a must. This will help you keep penalizations for content duplication at bay.‌

Aggregate your content from trustworthy sites

Always make sure to aggregate content only from credible sources and double-check the data you’ve collected. Keep your information current and relevant, and take some time to verify all your links work. There’s just a handful of things that can be more frustrating to your audience than entering your platform and getting disappointed by what they find there.‌

Include a fresh perspective to keep your content relevant

Supplement the aggregated data by offering different angles and views. Add excerpts from authoritative sources to support the credibility of your platform. Curating the content you put out there and offering your point of view will show your readers you’re going out of your way for them. Keep the aggregated content aligned with your brand and values to avoid any contradictions that might throw off your audience.‌

News Data Scraping With Proxies

As mentioned above, web scraping is essential to create a functional news aggregation site. These platforms need a massive amount of data that’s impossible to gather any other way. However, you can’t use just any proxy to extract information from multiple sources. Looking for free proxies online might be counterproductive. These alternatives are typically shared with other users and come from questionable sources. Moreover, free poxy providers will rarely have your back if their servers are down or if you need support.‌

Public and free proxies can put your machine, your network, and your data at risk. Besides, you become more likely to be caught by those sites that frown upon web scraping and get blacklisted or banned. Free proxy sites seldom offer rotation alternatives. That’s why you must look for a provider that sells rotating residential proxies. These can be used for news proxies, as they will continuously get replaced, making you pretty much undetectable to the sites you’re scraping.

What’s more, using rotating residential IPs is an excellent way to minimize the risks of getting your IPs banned for scraping and avoid annoying captchas and other anti-scraping measures. Rotating residential proxies will make it seem like your requests are coming from different places rather than just one device, thus feeling more natural or humanlike to the sites you’re trying to scrape. When scraping the web, the less you look like a bot, the better. Keep in mind that most sites have stringent anti-bot bans that can affect your news scraping activity.‌

Another excellent idea if you want to make your news scraping process more efficient is using a proxy management app. It will let you handle retries and cooldown logic while supporting geo-targeting, detecting bans, and providing useful statistics. The good news is that some residential proxies already have a built-in proxy manager and are optimized for all kinds of web scraping activities. A tool that takes care of all your proxy concerns will make your life much easier and let you focus on other aspects of your aggregator.‌

Best Practices and Tactics for News Web Scraping

As mentioned above, web scraping is vital for the survival of your news aggregation site. However, it’s a risky practice. While it’s not strictly illegal to gather data from multiple sources online, most sites don’t condone it and have stringent anti-scraping measures in place. If you get caught scraping these sites, you might get an IP ban. Keep it ethical and make sure to follow these measures:

• Respect the site’s rules.
• Don’t scrape copyrighted content.
• Use rotating residential proxies for added protection.
• Avoid overwhelming the site’s server.
• Don’t follow repetitive crawling patterns.

Stay Top of Your News Scraping Game With the Right Proxies

The internet is a very busy place, and new sites emerge every day. It can get extremely overwhelming for our fellow internauts to find the content they’re looking for, especially when it comes to the news. Stories are often updated in real-time, and keeping up with the world’s most recent events every day can become an impossible task without a little help. That’s when news aggregators come in. They allow you to gather the most suitable information for a specific audience.‌

If you want to provide your readers with current and relevant information they can find within one platform, building a news aggregator is a good call. Offering them a rich news compilation coupled with convenience and value will keep them coming back for more, thus increasing your reach and revenue. However, news aggregation depends fully on web scraping. This guide on how to make a news aggregator contains all you need to know about news scraping and more. If you want to take your scraping experience to the next level, make sure to purchase the right proxies to keep everything running smoothly.

The information contained within this article, including information posted by official staff, guest-submitted material, message board postings, or other third-party material is presented solely for the purposes of education and furtherance of the knowledge of the reader. All trademarks used in this publication are hereby acknowledged as the property of their respective owners.