Skip to main content
Taught by Tech Leads

Master pipelines, cloud & AI to become an operational Data Engineer.

DataScientist.fr
Image de Logging in Python with logging - Practical Tutorial
Python
DevOps

Logging in Python with logging - Practical Tutorial

Photo de Romain DE LA SOUCHÈRE

Tech Lead, CTO AXI Technologies

Published on 2 janvier 2025 · 5 min of reading

In today's digital world, where data flows at a dizzying speed, logging has become a crucial tool for developers and system administrators. It allows tracking, documenting, and analyzing events occurring within applications and infrastructures. This article guides you through the essential steps to get started with the logging module, helping you create custom loggers tailored to your specific needs. Discover how to improve the reliability and security of your systems through effective logging.

Getting started with the logging module

To get started with the logging module in Python, it is essential to understand its basic features that enable effective tracking and diagnosing of application behavior. Python offers a built-in module called logging, which provides flexibility and simplicity for recording logging messages. Here’s how you can start using this module.

Initializing logging

The first step is to import the logging module and set up a basic configuration. This configuration allows you to define the logging level, message format, and more. Here’s a simple example of initialization:
python
In this example, the logging level is set to DEBUG, meaning that all messages of this level and above (INFO, WARNING, ERROR, CRITICAL) will be recorded. The message format includes the time (asctime), the severity level (levelname), and the message itself (message).

Using logging levels

The logging module offers several logging levels that you can choose from based on the importance of the messages:
  • DEBUG: Detailed diagnostic information.
  • INFO: General informational messages confirming that everything is working as expected.
  • WARNING: Indications of a potential problem.
  • ERROR: Serious errors due to more severe issues.
  • CRITICAL: Very serious errors that may require immediate intervention.
Here’s how you can use these levels in your code:
python

Managing log files

To write logging messages to a file, you can adjust the basic configuration to include an output file:
python
With this configuration, all messages will be recorded in app.log instead of being displayed in the console.

Creating custom loggers

For more complex applications, it is often useful to create custom loggers for different modules or parts of your application:
python
By using these basic concepts, you can begin to implement an effective logging system in your Python projects, facilitating tracking and troubleshooting.

Creating a custom logger

Once you understand the basics of logging in Python, you can create custom loggers to meet specific needs of your application. This allows you to precisely control where and how logging messages are captured and displayed.

Creating a custom logger

To create a custom logger, start by obtaining a logger instance using logging.getLogger(). You can name your logger to distinguish it from others:
python
The name given to the logger, here 'MyLogger', allows it to be distinctly identified in a complex logging system.

Adding handlers

Handlers define where logging messages are sent. They can be directed to the console, a file, or even an external logging service. Let’s add a stream handler to display messages in the console:
python
Next, you can add this handler to your logger:
python

Defining formats

To customize the format of logging messages, use a Formatter. Here’s how to set up a simple format:
python
This format includes the time, logger name, logging level, and the message.

Logging messages

Once the logger and its handlers are configured, you can log messages as follows:
python
python

Using multiple handlers

To send logging messages to multiple destinations, you can add several handlers. For example, to also log messages to a file:
python
With this configuration, messages of level ERROR and higher will be logged in my_logger.log, while INFO and higher messages will appear in the console.
By configuring your loggers this way, you can create a sophisticated logging system that meets the specific needs of your application, ensuring effective traceability and monitoring.

Conclusion

In conclusion, logging in Python is an essential tool for developing robust and maintainable software. By using the logging module, you can not only track the behavior of your application but also efficiently identify and resolve issues as they arise.

Importance of good configuration

Good logging configuration is crucial. It allows you to filter and direct relevant messages where they are needed, whether on the console for rapid development or in files for post-mortem analysis. By setting appropriate logging levels, you can ensure that you don’t miss any critical information while avoiding unnecessary noise.

Flexibility and customization

As we have seen, the flexibility of the logging module allows you to customize every aspect of your logging system. The ability to create custom loggers and associate different handlers gives you complete control over the capture and distribution of logging messages. This is particularly useful in production environments where monitoring requirements may be complex and varied.

Best practices

To make the most of logging, here are some best practices to keep in mind:
  • Use descriptive names for your loggers: This makes it easier to identify the source of messages, especially in large applications.
  • Adjust logging levels: Modify levels according to context. For instance, use DEBUG during development and ERROR or CRITICAL in production.
  • Ensure log files are managed: Implement file rotation to avoid disk space issues.
  • Regularly test your logging configuration: Ensure that messages are being logged correctly and that alerts are triggered as expected.
By incorporating these practices into your workflow, you can significantly enhance the resilience and reliability of your Python applications. Logging is not only a tool for developers but also a vital component for IT operations management, enabling proactive incident response and better understanding of system behavior.
Ultimately, mastering logging provides you with a clearer and more detailed view of your software, allowing you to optimize its performance and long-term maintenance.

Want to go further?

This topic is part of our Become a Data Analyst course. Browse the full programme, or get it by email.

Share with

Photo de Romain DE LA SOUCHÈRE

Romain DE LA SOUCHÈRE

Tech Lead, CTO AXI Technologies

Expert Data Engineering et Cloud, Romain affiche plus de 11 ans d'expérience, dont plusieurs années comme Lead Developer sur des solutions Smart Building haute performance. Il y a conçu et mis en production des moteurs de traitement capables d'absorber des centaines de milliers de données de capteurs par minute, ainsi que des bases clusterisées gérant plus de 10 millions de données dynamiques. Certifié Microsoft Azure DevOps Engineer Expert, il maîtrise aussi bien le développement back-end (Python, C#) que le DevOps (Docker, Kubernetes, Terraform) et les agents LLM. Formateur en Python, cloud, DevOps et IA générative appliquée, il forme avec une obsession : Amener chaque apprenant à concevoir et déployer des architectures réellement scalables en production.

» Learn More

Associated trainings

All our trainings
Image de la formation Become a Data Analyst
Become a Data Analyst
6 months
Intermediate
Guarantee

Associated articles

See all our articles