Python

Web Development

Requests Library Guide - Interactive Python Tutorial

Tech Lead, CTO AXI Technologies

Romain DE LA SOUCHÈRE

Published on 7 janvier 2025 · 17 min of reading

Table of content

Introduction to the requests library

The GET request

The response

Query string parameters

Other HTTP methods

The message body

Inspecting requests

Authentication

SSL certificate verification

Performance

Conclusion

Share with

In the world of web development, interaction with APIs is crucial for exchanging data between different platforms. For Python developers, the 'requests' library stands out as an essential tool for making HTTP requests simply and effectively. Whether you are a beginner or an expert, mastering 'requests' will allow you to navigate smoothly through the complexities of web communications, leveraging all the possibilities offered by this powerful and flexible module. Learn how to optimize your data exchanges and ensure the security of your transactions using 'requests'.

Introduction to the requests library

The requests library is one of the most popular libraries in Python for making HTTP requests simply and effectively. Whether you are a beginner or an experienced developer, requests provides an intuitive interface for interacting with APIs, downloading files, or sending data to a server. In this section, we will explore the basic features of this library and how it can facilitate your web development tasks.

Installing the library

Before you start using requests, you need to install it. This can be easily done with pip:

shell

Once installed, you are ready to import it into your Python scripts and use it to send HTTP requests.

Sending a GET request

The GET request is one of the most common HTTP operations. It is used to retrieve data from a specific resource. With requests, this is done in a single line of code:

python

The variable response contains the server's response, which can be manipulated to extract useful information.

Handling the response

Once you have the response, requests makes it easy to access the data. Here’s how you can access the content of the response, as well as other important attributes:

python

For JSON responses, requests offers a convenient method to convert them into Python objects:

python

Sending a POST request

To send data to a server, you will typically use a POST request. The requests library also simplifies this operation:

python

In this example, payload is a Python dictionary that contains the data to send to the server.

Exception handling

It is important to handle exceptions to prevent your program from crashing in case of connection issues or server errors. requests raises exceptions for these cases, which you can intercept with a try-except block:

python

With this introduction, you now have a basic understanding of the requests library and are ready to explore it further to meet your specific HTTP request needs.

The GET request

The GET request is one of the most fundamental operations of the HTTP protocol, and with the requests library, it becomes particularly easy to implement. In this section, we will delve into using the GET request to retrieve data from an API or a website.

Sending a GET request

Sending a GET request with requests is as simple as calling the get() function with the desired URL. Here’s a basic example:

python

In this example, response contains the server's response to the GET request sent to the specified URL.

Checking the response status

After making a GET request, it’s crucial to check if the request was successful. This is done by checking the HTTP status code:

python

A status code of 200 indicates that the request was successfully processed.

Extracting data

To extract data from the response, requests provides several options depending on the format of the returned data. For plain text:

python

If the response is in JSON format, you can easily convert it into a Python object:

python

Managing query parameters

Often, you will need to pass parameters to your GET request. requests simplifies this with the params parameter:

python

In this example, params is a dictionary that specifies the query parameters to include in the URL.

Error handling

As with any network operation, errors can occur when sending a GET request. You can handle them with a try-except block:

python

By mastering these concepts, you will be well-equipped to use the GET request effectively with the requests library and interact with various APIs and web services.

The response

Once a request has been successfully sent using the requests library, the server returns a response that you can analyze and manipulate according to your needs. This section explores how to extract and use the information contained in an HTTP request response.

Accessing response headers

HTTP headers contain important metadata about the response, such as content type or date. You can access this information using the headers attribute:

python

To extract a specific header, use the following syntax:

python

Parsing the response content

The content of the response is often the primary interest when working with HTTP requests. requests makes it easy to access this content in several forms:

Plain text:
Binary:
JSON: If the response is in JSON format, you can directly convert it into a Python object:

Checking the status code

The HTTP status code is essential for understanding the outcome of your request. A code of 200 indicates success, while other codes like 404 or 500 indicate errors. You can access the status code like this:

python

Managing redirections

HTTP redirections are common, and requests handles them automatically. You can disable this feature if necessary by using the allow_redirects parameter:

python

Response time

In applications where performance is critical, measuring response time can be useful:

python

By understanding these aspects of the HTTP response, you can not only extract the necessary data but also optimize your requests to better interact with web services.

Query string parameters

Query string parameters are key elements that allow you to customize HTTP requests by including specific data directly in the URL. The requests library simplifies adding and managing these parameters, which is particularly useful when interacting with APIs that require specific inputs.

Adding parameters to a GET request

To add query string parameters, you can use the params parameter of the get() function. This parameter accepts a Python dictionary where each key-value pair represents a parameter and its associated value:

python

In this example, the final URL sent will be https://api.example.com/search?query=python&page=2.

Automatic parameter encoding

One of the powerful features of requests is the automatic encoding of query string parameters. This means that special characters and spaces are properly encoded, preventing common errors related to manually constructing URLs:

python

Here, the space between 'web' and 'development' will be correctly encoded for inclusion in the URL.

Managing dynamic parameters

Query string parameters can be easily modified dynamically, which is useful in applications where users can enter search criteria or filters:

python

This code shows how a user can interact with an API by dynamically changing the request parameters.

Checking the final URL

To check the final URL that is generated with the parameters, you can access the url attribute of the response object:

python

This is particularly useful for debugging and ensuring that the parameters are correctly added.

These techniques allow you to fully leverage query string parameters, making your applications more flexible and powerful when interacting with web services.

Other HTTP methods

In addition to the GET request, the requests library supports several other essential HTTP methods for interacting with APIs and web services. These methods allow for more flexible data manipulation and transfer.

The POST method

The POST method is mainly used to send data to a server. With requests, sending a POST request is simple and requires only a few lines of code. Here’s how you can send data in the form of a form:

python

The data is sent in the body of the request, which is ideal for forms and sensitive data.

The PUT method

The PUT method is used to update existing resources on a server. It works similarly to POST but is generally used when you want to completely replace a resource:

python

This request updates the information of the specified user.

The DELETE method

The DELETE method allows you to remove a resource on a server. With requests, using this method is as simple as the others:

python

This request attempts to delete the user with ID 123.

The HEAD method

The HEAD method is similar to GET, but it only retrieves the headers of the response, without the body. This is useful for checking the availability of a resource or obtaining metadata:

python

The PATCH method

The PATCH method is used to make partial updates to a resource. Unlike PUT, it does not require a complete update:

python

Each HTTP method offers unique possibilities for effectively interacting with web services, and the requests library provides a simple interface to implement them in your Python applications.

The message body

The message body in an HTTP request is an essential component, especially when sending data to the server. The requests library makes it easy to send different types of content in the request body, depending on your application's needs.

Sending form data

To send data in the form of a form, you can use the data parameter in POST or PUT methods. The data is typically sent as a Python dictionary:

python

In this example, the dictionary data is encoded as application/x-www-form-urlencoded, the standard format for web forms.

Sending JSON data

When you need to send data in JSON format, the requests library offers a convenient method with the json parameter. This ensures that the data is properly encoded in JSON before being sent:

python

This approach is widely used for RESTful APIs that consume JSON.

Sending files

Sending files via an HTTP request is a common task, especially when uploading documents or images. With requests, you can use the files parameter:

python

This code sends the document.pdf file to the server, using the multipart/form-data content type.

Customizing headers

Sometimes, it is necessary to customize HTTP headers to specify content type or other metadata. This can be done using the headers parameter:

python

This flexibility ensures that your requests conform to the specifications of the API you are using, thereby guaranteeing effective communication with the server.

Inspecting requests

Inspecting requests is a crucial step for debugging and optimizing your interactions with web servers. The requests library provides several tools that allow you to examine the details of HTTP requests, helping to ensure that the requests are correctly formed and the responses are appropriate.

Examining the final URL

When building requests with dynamic parameters or many headers, it’s important to check the final URL to ensure it is correctly formed. You can access the final URL used in the request via the url attribute of the Response object:

python

This is particularly useful for ensuring that the query string parameters are correctly encoded and added.

Checking request headers

Request headers can influence how the server processes your request. With requests, you can inspect these headers to ensure they are properly configured:

python

This inspection allows you to verify that custom headers, such as authentication tokens or content types, are included as expected.

Analyzing redirections

requests automatically follows redirects, but it can be useful to see which redirections were followed to diagnose routing issues:

python

Each element of response.history is a Response object that represents a redirection.

Response time

Analyzing response time can help identify bottlenecks in network communication. requests provides the elapsed attribute to measure the total time of the request:

python

With these inspection techniques, you can optimize your HTTP requests for increased performance and reliability in your Python applications.

Authentication

Authentication is a crucial aspect when it comes to accessing protected resources or interacting with secure APIs. The requests library offers several simple authentication methods to implement, ensuring that your requests are authorized to access the necessary resources.

Basic authentication

Basic authentication is one of the simplest methods and is often used to protect resources with a username and password. With requests, you can easily add authentication information to your requests:

python

In this example, the HTTPBasicAuth object is used to include the authentication information in the request.

Token authentication

Many modern APIs use tokens for authentication, which provide enhanced security and flexibility. Tokens are typically included in the request headers:

python

The authentication token is here added to the Authorization header, using the Bearer scheme.

OAuth authentication

OAuth is a commonly used authorization protocol that allows access to resources without sharing credentials. Although requests does not natively handle OAuth, libraries like requests-oauthlib can be used to simplify this process:

python

This method is often used by platforms requiring a higher level of security, such as Twitter.

With these different authentication methods, requests allows you to easily secure your interactions with web services, ensuring that only authorized requests can access protected resources.

SSL certificate verification

SSL certificate verification is an essential practice to ensure the security of communications between your application and a web server. The requests library automatically verifies SSL certificates, but you can also customize this behavior according to your needs.

Default verification

By default, requests checks SSL certificates to ensure that the server you are connecting to is indeed what it claims to be. This prevents man-in-the-middle attacks that could intercept your data:

python

If there is an issue with the SSL certificate, requests will raise a SSLError exception.

Disabling SSL verification

In some development or testing environments, you may need to temporarily disable SSL verification. This can be done by passing verify=False in your request, although this is not recommended for production environments:

python

Note that disabling SSL verification exposes your application to security risks. It is best to use this option with caution.

Using a custom SSL certificate

If you are working with a server that uses a self-signed or non-standard SSL certificate, requests allows you to specify a path to a CA certificate file:

python

This allows requests to verify the server's certificate using the provided CA certificate.

Handling SSL exceptions

When an SSL verification fails, it is essential to handle exceptions to prevent your application from crashing. You can intercept SSL errors with a try-except block:

python

By understanding and correctly using SSL certificate verification, you can enhance the security of network communications in your Python applications.

Performance

Optimizing the performance of your HTTP requests can significantly impact the efficiency of your application, especially when frequently interacting with APIs or web services. The requests library offers several techniques to improve the performance of your requests.

Reusing connections with sessions

Using sessions is an effective method to reuse underlying HTTP connections, reducing request latency. Sessions allow you to retain configuration parameters and headers between multiple requests:

python

By using a session, you avoid recreating a new connection for each request, which can significantly reduce response time.

Enabling compression

Compressing HTTP responses can reduce the amount of data transferred, thus speeding up transmission. requests automatically handles gzip and deflate compression if the server supports it:

python

By ensuring that your servers support compression, you can improve loading times for end-users.

Managing timeouts

Configuring appropriate timeouts can prevent your requests from getting stuck indefinitely in case of network issues. You can specify timeouts for connections and reads:

python

Here, 3.05 seconds is the timeout for establishing a connection and 27 seconds for reading data.

Using asynchrony

For applications requiring a high degree of parallelism, considering asynchronous approaches can be beneficial. Although requests does not natively support asynchrony, libraries like aiohttp can be used to perform non-blocking requests:

python

By optimizing the use of requests with these techniques, you can significantly improve the performance and responsiveness of your web applications in Python.

Conclusion

In conclusion, the requests library proves to be a powerful and flexible tool for Python developers looking to perform HTTP operations. Whether you are building a complex application or simply interacting with an API, requests offers an intuitive interface that greatly simplifies these interactions.

Flexibility and simplicity

One of the main strengths of requests is its ease of use. With clear and concise syntax, even advanced HTTP operations become accessible. Whether sending GET or POST requests, managing authentication, or manipulating headers and message bodies, requests allows you to accomplish these tasks with minimal code.

Security and reliability

Security is a crucial aspect for any web application, and requests simplifies the management of SSL verifications and authentications. By supporting different authentication mechanisms and allowing fine control of SSL certificates, requests ensures that your communications are secure and reliable. Additionally, exception handling and timeout management enhance the robustness of your applications.

Performance and optimization

Optimizing the performance of your HTTP requests is essential for providing a good user experience. Using sessions to reuse connections, enabling data compression, and appropriately managing timeouts are techniques that can be easily implemented using requests. For applications requiring asynchronous request handling, integration with libraries like aiohttp offers additional flexibility.

A rich ecosystem

Finally, requests benefits from a vast ecosystem and an active community, meaning you can easily find extensions and resources to meet specific needs. Whether for handling advanced use cases or troubleshooting issues, the abundant documentation and numerous community contributions make requests a wise choice for Python developers.

Thus, by mastering the requests library, you are well-equipped to create robust, secure, and high-performing web applications, all while enjoying a simplified and pleasant development experience.

Want to go further?

This topic is part of our Become a Data Analyst course. Browse the full programme, or get it by email.

Share with

Romain DE LA SOUCHÈRE

Tech Lead, CTO AXI Technologies

Expert Data Engineering et Cloud, Romain affiche plus de 11 ans d'expérience, dont plusieurs années comme Lead Developer sur des solutions Smart Building haute performance. Il y a conçu et mis en production des moteurs de traitement capables d'absorber des centaines de milliers de données de capteurs par minute, ainsi que des bases clusterisées gérant plus de 10 millions de données dynamiques. Certifié Microsoft Azure DevOps Engineer Expert, il maîtrise aussi bien le développement back-end (Python, C#) que le DevOps (Docker, Kubernetes, Terraform) et les agents LLM. Formateur en Python, cloud, DevOps et IA générative appliquée, il forme avec une obsession : Amener chaque apprenant à concevoir et déployer des architectures réellement scalables en production.

» Learn More