AIOHTTP Vs. HTTPX Vs. Requests: Everything You Need To Know

HTTP (Hypertext Transfer Protocol) is a client-server protocol that is the foundation of all web data exchanges. HTTPX, AIOHTTP, and Requests are widely-used Python clients that can simplify HTTP requests. Read more about httpx vs. requests.

However, there are a few differences between the three, and you’ll need an overview of HTTPX vs. Requests vs. AIOHTTP to better understand when and why you might want to use one over the other.

How Does HTTP Work?

Basic knowledge of HTTP can help you understand HTTPX vs. Requests vs. AIOHTTP functions. Clients, a device or program that initiates a request, send information to a server as a request. After processing the request, the server provides the requested information in the form of a response.

HTTP has been around since the early 1990s. The extensible application layer protocol is sent over a Transmission Control Protocol (TCP) or Transport Layer Security (TLS) connection. The flexibility of HTTP means you can adapt the protocol to fetch or post various forms of content, including HTTP documents, videos, and images.

What Are the Individual HTTP Components?

Breaking down the foundation of HTTP can help you appreciate the differences between HTTPX vs. Requests vs. AIOHTTP clients. While browsers are often used to send HTTP requests, they can be sent by any agent, including web proxies. Many web Rayobyte’s Web Scraping APIs rely on web proxies to extract information from web pages. How they function depends on your choice of HTTPX vs. Requests vs. AIOHTTP.

Remember that multiple entities are working to handle requests and responses. Here’s a look at the layers involved in HTTP functions.

Clients

A client, or user agent, functions on behalf of the user. They can be programs built by developers to debug applications or scrape data from web pages. For example, browsers display web pages by sending a request to pull an HTTP document outlining the page it needs. From there, the browser parses the file and sends additional requests that tie into any resources, scripts, or layout information contained within the document.

Next, the browser pulls the resources together and displays the complete web page document. Web pages contain links that robots can activate by clicking to retrieve a new web page. It’s how companies can set up Rayobyte’s Web Scraping APIs to navigate through web pages, whether they use HTTPX vs. Requests vs. AIOHTTP. Browsers translate the directions clients provide into HTTP requests to translate into a proper user response.

Other examples of clients include:

• cURL: A command-line tool that sends HTTP requests from a script or the command line.
• Node.js: An open-source runtime environment capable of sending HTTP requests from Node applications.
• Postman: An API development and testing tool that lets you send HTTP requests.

Web servers

Web servers can be a single machine or a collection of computers that share a load or software used to process requests and provide a response. How the server responds depends on the HTTPX vs. Requests vs. AIOHTTP client. In addition, you can have multiple software instances hosted on one machine that share the same IP address.

Proxies

Proxies are IP addresses that act as middlemen between someone sending a request and the server that receives the information. That’s different from a proxy server, a physical or virtual machine that houses one or more IP addresses. A proxy provides anonymity when making web requests, whether you go with HTTPX vs. Requests vs. AIOHTTP.

Sending HTTP Requests

Most developers have a preference when it comes to HTTPX vs. Requests vs. AIOHTTP and setting up HTTP requests. All three are compatible with Python, which is commonly used to build web scraping tools that collect data from web pages. Other libraries used for the same purpose with other programming languages include HTTPClient for .NET or the Fetch library for JavaScript.

After selecting your library, you’ll need to create an HTTP request object with a request method, which can include the following:

• GET: Requests information from a specific resource.
• PUT: Creates a new resource or replaces an existing resource with new information.
• POST: Sends data designed to create a new resource or update an existing one.
• DELETE: Removes a resource from a server.

You can use your chosen library, whether it’s HTTPX vs. Requests vs. AIOHTTP, to send your response. After processing the message, the client library returns a response containing information like headers, response body, and status code. You would extract all relevant fields from this response.

Regardless of whether you are using HTTPX vs. Requests vs. AIOHTTP, they’re all built on top of the HTTP protocol. These libraries simplify the HTTP request and response handling process, as most manual processes are covered. That lets developers work more efficiently when building tools and applications to retrieve web information.

Synchronous vs. Asynchronous HTTP Libraries

HTTP libraries can be synchronous or asynchronous. There are a few differences between the two types of libraries that are worth covering before further exploring HTTPX. That will also help you understand what separates HTTPX vs. Requests vs. AIOHTTP libraries.

Synchronous HTTP libraries

Synchronous libraries use a sequential or blocking execution model. When a client sends a request, program execution stops until it gets a response back from the server or receives an error. Any subsequent requests can only be worked on once the current request completes its processing. That can cause many delays, especially when the client needs to send multiple requests.

A synchronous library handles requests in receipt order. The simplicity of the design makes it easier to get applications like web scraping processes up and running quickly. Synchronous libraries are also more intuitive because they provide a predictable way for developers to work with requests and responses predictably. That can make a difference in whether you choose to work with HTTPX vs. Requests vs. AIOHTTP.

Asynchronous HTTP libraries

If you have processes that use concurrent operations, then it makes sense to choose an asynchronous library like HTTPX or AIOHTTP. Examples of processes include:

• Web servers handling multiple incoming HTTP requests from different clients
• Web browsers sending multiple HTTP requests simultaneously to retrieve different web page resources (images, stylesheets, APIs) for various clients
• APIs making parallel requests to pull information from different endpoints

Concurrent operations handle multiple requests without blocking execution flow. Applications become more scalable and optimize system resource usage without sacrificing performance. Below are some other benefits of using asynchronous libraries:

• Better performance: Asynchronous libraries leverage nonblocking I/O and parallelism to make your tools perform faster. They can handle scenarios where a request needs to wait for information from another external resource like an API or a database. Other tasks can continue executing, which makes for more efficient use of available processing time.
• Improved user experience: Applications built using libraries, whether you use HTTPX vs. Requests vs. AIOHTTP, are more responsive even during long-running processes. They don’t have to wait for a response or perform other blocking operations, so user interfaces can remain interactive and make the user experience more seamless.
• Efficient resource usage: Because asynchronous libraries switch to other tasks while other operations run, you can avoid idle time, keep applications efficient, and better use system resources.
• Higher concurrency: Asynchronous libraries enable more efficient handling of multiple connections within applications built for real-time operations, like chat or streaming. That allows them to provide real-time responsiveness without extensive delays.

It helps to dive deeper into the functions of HTTPX vs. Requests vs. AIOHTTP and understand how each protocol works. HTTPX and AIOHTTP are asynchronous, while Requests is synchronous but capable of supporting asynchronous operations.

What Is Requests?

Requests is a Python HTTP request library that simplifies sending HTTP requests and processing responses. The API lets users interact with various web services and handles HTTP requests by dealing with elements like cookies, headers, sessions, and SSL certificate verification behind a user interface.

The Requests library makes it easier to add values like parameters, headers, and other data to your HTTP requests, then retrieve the response in formats like text, JavaScript Object Notation (JSON), or binary. Many Python developers use Requests because it’s simple, versatile, and comes with extensive documentation.

Most developers use Requests to build web-scraping tools, communicate with HTTP-based APIs, and handle interactions within Python applications. When reviewing the differences between HTTPX vs. Requests vs. AIOHTTP, you should consider the following features available in Requests:

• International URLs and domains: Users can use Requests to handle URLs containing international domain names and convert them to ASCII-compatible encoding.
• HTTPS proxy support: This lets you specify a proxy server’s URL or IP address to include a user ID or password if required for authentication by a proxy.
• Chunked requests: Requests allows clients to send requests in smaller packets instead of one large stream of data, letting servers process each chunk as it arrives.
• Connection timeouts: Allows clients to establish a specific waiting period for a server response before raising an exception.
• Basic and digest authentication: Requests supports basic authentication, which typically requires a user ID and password. It also allows for Digest authentication, which requires the client to provide additional information before allowing access.
• Unicode response bodies: Requests decodes responses containing Unicode character encoding.
• Automatic decompression: Requests decompresses HTTP responses before passing them to the client.
• SSL verification: Requests verifies the security of SSL/TLS certificates provided by servers.

Implementing a request

Here are a few practical examples to give you an idea of how Requests work. Start by issuing the following command to import the Requests module:

import requests

Once you’ve done that, you can start working with the library to get more of an idea of the functionality between HTTPX vs. Requests vs. AIOHTTP. Here, we’ll run a simple request to retrieve a webpage. You can replace the below URL with one of your choosing:

ro = requests.get(‘http.samplepage.com’)

The above command sets up a Response object we’ve named ro, which can provide the necessary information. That’s all it takes to execute a GET request using Requests. You can send other requests like PUT and DELETE or add information to the headers.

ro = requests.put(‘http://testpage.com’, data= {‘key’: ‘value:})

ro = requests.head(‘http://testpage.com/get’)

ro = requests.options(‘http://testpage.com/get’)

ro = requests.delete(‘http://testpage.com/delete’)

When you want to pass parameters in a normal HTTP request, you’d need to provide a key/value pair in the URL after inserting a question mark, like http://testpage.com?key=val. With Requests, you can add arguments as a dictionary of strings using a params keyword argument.

load = {‘keyitem1’: ‘valueitem1’, ‘keyitem2’: ‘valueitem2’}

ro = requests.get(‘http://testpage.com/get, params=load)

When you print the URL, you’ll be able to see the information sent using the load value:

print(ro.url)

http://http://testpage.com/get?keyitem1=value1&keyitem2=value2

Requests automatically get decoded from the server, including Unicode charsets. The library guesses the encoding based on a request’s HTTP headers and uses that encoding when you access ro.text. If you need to change the encoding, you can access it using the ro.encoding property.

Changing the encoding causes Requests to use the new value when tapping into the ro.text property. That’s useful if you need to employ specialized logic to help work out the encoding of any content.

If you need to look at your response body as bytes, Responses lets you use the r.content property to create an image from binary data returned from a request:

from PIL import Image

from io import BytesIO

im = Image.open(BytesIO(r.content))

Requests also come with a built-in JSON decoder to work with JSON data. You get an exception thrown if there is a JSON decoding failure.

import requests

ro = requests.get(‘https://samplepage.com’)

ro.json()

If you forget to add headers to your HTTP request, you can pass a dict, a key-pair Python value, into the headers parameter as shown below:

url = ‘http://testapi.com/new/endpoint’

newheaders = {‘user-agent’: ‘my-scraping-app/0.0.1’}

ro = requests.get(url, headers = newheaders)

What Is HTTPX?

HTTPX is an alternate HTTP Python client library that extends the functionality of the basic HttpClient and Requests libraries. You can issue synchronous and asynchronous requests, which offers more flexibility than the Requests library. That’s something to consider as you continue evaluating the offerings of HTTPX vs. Requests vs. AIOHTTP.

Other key differences between HTTP and Requests include the following:

• Support for HTTP/2 protocol, which is not available in Requests
• Automatic decoding of standard formats like JSON
• Provides more optimal performance vs. Requests

You can install the HTTPX Python package using the pip utility:

$ python -m pip install httpx 

From there, you can issue the following command to send a request via HTTPX:

import httpx

ro = httpx.get(‘https://samplepage.org’)

ro

Then you’ll see the following:

<Response [200 OK>

You can also view the content type of a response object:

ro.headers[‘content-type’]

This returns:

‘text/html: charset=UTF-8

Below are other examples of requests you can make with HTTPX. Note that the syntax is quite similar to that used for Requests.

The following sends an HTTP PUT request to a specific URL:

ro = httpx.put(‘https://testpage.org/put’, data={‘key’: ‘value’})

Send a HEAD request to a specific URL:

ro = httpx.head(‘https://testpage.org/get’)

Send an OPTIONS request to a specific URL:

ro = httpx.options(‘https://testpage.org/get’)

Send a DELETE request to a specific URL:

ro = httpx.delete(‘https://testpage.org/delete’)

When it comes to the verbiage of HTTPX vs. Requests in Python, HTTPX is more verbose. However, this trade-off allows you to write asynchronous requests using the asyncio Python library, as shown below:

import httpx

import asyncio

async def main()

   async with httpx.AsyncClient() as newclient:

   aresponse = await client.get(http://samplepage.com)

   print(aresponse.text)

asyncio.run(main())

If a request requires additional information, you can pass parameters into URL requests with the params keyword:

params = {‘keyitem1’: ‘valueitem1’, ‘keyitem2’: ‘valueitem2’}

r = httpx.get(‘https://testpage.org/get’, params=params)

Or you can pass item lists as a value:

params = {‘keyitem1’: ‘valueitem1’, ‘keyitem2’: [‘valueitem2’, ‘valueitem3’]}

r = httpx.get(‘https://testpage.org/get’, params=params)

Another benefit of HTTPX in comparing HTTPX vs. Requests vs. AIOHTTP is how HTTPX automatically decodes response content into Unicode text. This is useful if you need to manipulate the text string. Some Python libraries are built to work with Unicode, so you may need to decode responses to ensure compatibility:

r = httpx.get(‘https://www.samplepage.org/’)

The above will produce the following when you look at the text property:

r.text

‘<!doctype html>\n<html>\n<head>\n<title>Sample Domain</title>…’

HTTPX also has support for Basic and Digest HTTP authentication. That’s something to keep in mind as you review the pros and cons of HTTPX vs. Requests vs. AIOHTTP. Some servers will require additional information before allowing you to access its resources. You can pass in Basic authentication credentials by using plain text str or bytes objects for your auth arguments:

httpx.get(“https://samplepage.com”, auth=(“auth_user”, “password45678”))

You must instantiate a DigestAuth object to provide Digest authentication credentials:

authObj = httpx.DigestAuth(“auth_user”, “password45678”)

httpx.get(“https://samplepage.com”, auth=authObj)

HTTPX provides exception-raising functions to handle issues that arise during sending a request or receiving a response. The exception classes most often relied upon by Python developers are RequestError and HTTPStatus Error. The RequestError class contains all exceptions when issuing an HTTP request.

try:

  response= httpx.get(“https://www.samplepage.com/”)

excep httpx.RequestError as except:

  print(f”An error occurred while requesting {except.request.url!ro}.”)+

The HTTPStatusError Class gets raised by when you call response.raise_for_status() on any response that does not provide a 2XX success code. Below is an example of an error for the .request and .response attributes:

response = httpx.get(“https://www.samplepage.com/”)

try:

  response.raise_for_status()

except for httpx.HTTPStatusError as excep:

  print(f”Error response {excep.response.status_code} while requesting {excep.request.url!ro}.”)

What Is AIOHTTP?

Asynchronous Input Output HTTP is another asynchronous HTTP client library and web server framework available from Python. The high-level API provides support for both HTTP/1.1 and HTTP/2 protocols. AIOHTTP also supports concurrency, which means having multiple connections open simultaneously. That’s an important consideration for companies have multiple web robots running to collect data.

Concurrency is vital in modern applications. Running programs in parallel lets you send more calls or handle many requests without consuming large amounts of resources. You can also build asynchronous applications that interact with different APIs and web services. It’s one reason that many developers choose AIOHTTP even after considering HTTPX vs. Requests. It also provides features like:

• Client and server-side cookies
• Client and server WebSockets
• Connection pooling
• Support for managing different consent types
• Session management

AIOHTTP offers middleware support and a flexible routing system. It integrates seamlessly with other popular asyncio-based libraries, which may factor into your decision to go with HTTPX vs. Requests vs. AIOHTTP.

Use the pip utility to download AIOHTTP and get started:

pip install aiohttp

It’s also a good idea to install the cchardet library as a replacement for charset-normalizer:

pip install cchardet

To get started, import both aiohttp and asyncio.

import aiohttp

import asyncio

Set up a main method that sends a client request and receives a response:

async def main():

  async with aiohttp.ClientSession() as newsession:

   async with snewession.get(‘http://mywebpage.org’) as response:

    html = await response.text()    

asyncio.run(main())

AIOHTTP code appears more verbose compared to the simplicity of Requests. It takes more than one step to perform a basic HTTP request because AIOHTTPS is asynchronous. The API does everything possible to make nonblocking network operations more efficient.

Everything that happens is entirely transparent. It might seem like a lot compared to HTTPX vs. Requests, but you can gain a lot in performance by typing a few more lines of code.

Now that we’ve seen the client in action, here’s an example of how you would write an AIOHTTP request:

from aiohttp import web

async def handle(request):

  callname = request.match_info.get(‘name’, “NewName”)

  resptext = “Hello, ” + callname

  return web.Response(text=resptext)

newapp = web.Application()

newapp.add_routes([web.get(‘/’, handle),

     web.get(‘/{name}’, handle)])

if __name__ == ‘__main__’:

  web.run_app(newapp)

Next, you instantiate your application, then add a web route. If the response matches the URL in the request, the application runs.

HTTPX vs. Requests vs. AIOHTTP: Summing It Up

The most significant difference between HTTPX and AIOHTTP is the library scope. HTTPX provides a rich HTTP client that you can use in a broader range of applications. AIOHTTP focuses more on giving developers an efficient way to make async HTTP requests. When it comes to HTTPX vs. Requests, you’ll find HTTPX to be more robust.

Another consideration for HTTPX vs. Requests is that it can handle streaming responses without loading the entire response body into memory. If you need to work with JSON, consider using HTTPX vs. Requests. Contact Rayobyte with any questions about your client and server needs or advice on choosing between HTTPX vs. Requests vs. AIOHTTP.

The information contained within this article, including information posted by official staff, guest-submitted material, message board postings, or other third-party material is presented solely for the purposes of education and furtherance of the knowledge of the reader. All trademarks used in this publication are hereby acknowledged as the property of their respective owners.