15 Python Logging Best Practices for Developers

Let's say you own an application program. You wanted to expand its capability and integrated several microservices to do so. Everything works fine, and you will see significant growth in customer traffic.

Now, the problem slowly starts building up. You face some customer queries asking for certain fixes. You want to help as much as you can, but unfortunately, you don't know exactly where the problem has occurred. You will have to check the entire thing from the beginning.

It feels like doom, isn't it?

That is why logging is considered to be of the highest priority, especially when you have to deal with a lot of data coming from several different components or services.

In this blog, we will discuss tips to make your Python Logging journey easier and look at some of the best practices to adopt in this regard. We have already written a complete guide article on Python Logging. You might want to check it out first!

Table Of Contents:

• What is Python Logging?
• What to log in a Python Application?
• Why Should you log Python Applications?
• How to get started with Logging in Python?
• 15 Python Logging Best Practices:
  1. Descriptive Log Messages
  2. Set Log Levels Appropriately
  3. Configure Logging Globally
  4. Log Exceptions with Stack Traces
  5. Avoid Hardcoding Log Messages
  6. Use Structured Logging
  7. Rotate Log Files
  8. Use Logging Libraries
  9. Include Timestamps in Log Messages
  10. Secure Sensitive Data
  11. Use Contextual Logging
  12. Handle Log Configuration Dynamically
  13. Use Logger Hierarchies
  14. Monitor Log Messages
  15. Test Logging Configurations

What is Python Logging?

Python logging is a built-in module that provides a flexible framework for emitting log messages from Python programs. It allows developers to record events that occur during the execution of a program, which can be useful for debugging, monitoring, and auditing.

Key components of Python logging include:

• Loggers: Loggers are the entry points of the logging system. They are objects used to create log records. Loggers are named hierarchically, similar to Python packages, and facilitate organizing logging calls in a meaningful way.
• Handlers: Handlers are responsible for taking log records emitted by loggers and doing something with them, such as writing them to the console, a file, or sending them over the network.
• Formatters: Formatters specify the layout of log messages, including the format of the timestamp, log level, and any additional information. They allow developers to customize how log messages are presented.
• Levels: Logging levels define the severity of log messages. Python logging supports the following levels (in increasing order of severity): DEBUG, INFO, WARNING, ERROR, and CRITICAL. Developers can filter log messages based on their level to control verbosity.

By default, Python logging outputs messages to the console with a basic configuration. However, developers can configure logging to suit their specific needs by creating and configuring loggers, handlers, and formatters according to their requirements.

What to log in a Python Application?

What to log in a Python application depends on the nature of the application, its purpose, and the specific requirements of the development and operational teams. However, here are some common types of information that are often logged in Python applications:

• Errors and Exceptions
• Warnings
• Informational Messages
• Debugging Information
• User Actions
• Performance Metrics
• Security Related Events
• External Services, API Interactions

It's important to strike a balance between logging too much information, which can clutter logs and affect performance, and logging too little, which may make it difficult to diagnose issues. Developers should carefully consider the level of detail and the types of information that are most relevant for monitoring, debugging, and maintaining the application. Additionally, using log levels effectively can help filter and prioritize log messages based on their severity and importance.

Why Should you Log Python Applications?

There are certain key points why you must definitely log Python applications, which includes:

• Debugging - To identify and resolve issues within the application by analyzing log messages.
• Monitoring - Real-time monitoring of application behavior and performance in production environments.
• Troubleshooting - Diagnose user-reported issues more effectively by examining logs.
• Performance Analysis - Analyze performance metrics such as response times and resource utilization.
• Auditing and Compliance - To maintain a record of user actions and system events for auditing purposes.
• Security Monitoring - For detecting and responding to security incidents by logging security-related events.
• Predictive Maintenance - To proactively identify potential issues before they occur by analyzing historical logs.

How to get started with Logging in Python?

Import the logging module: Start by importing the logging module, which is part of Python's standard library.

import logging

Configure logging: You can configure logging to customize its behavior according to your requirements. This includes setting the logging level, specifying the output format, and defining where log messages should be sent (e.g., console, file).

logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')

In the above example:

• level=logging.DEBUG sets the logging level to DEBUG, which means all log messages will be captured.
• format='%(asctime)s - %(levelname)s - %(message)s' defines the format of log messages, including the timestamp, log level, and message content.

Start logging: You can start logging messages using various logging methods provided by the logging module, such as debug(), info(), warning(), error(), and critical(). Each method corresponds to a different log level.
(We will look at this point in more detail in the best practices section.)

Run your application: Once you’ve added logging statements to your application code, run your application.

Depending on your application’s need, you can customize the logging configuration further. This could involve adding additional handlers, specifying log levels, and adding additional components if required.

Here's a simple example of a Python script with logging:

import logging
# Configure logging
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')
def divide(x, y):
try:
result = x / y
except ZeroDivisionError:
logging.error("Division by zero!")
else:
logging.info(f"Division result: {result}")
# Example usage
divide(10, 2)
divide(10, 0)

This script sets up logging to output messages with timestamps, log levels, and message content. It defines a function divide() that performs division and logs relevant messages. Finally, it demonstrates calling the divide() function with different parameters to generate log messages.

15 Python Logging Best Practices

Let's take a detailed look into what practices you can include for improving your Python Logging Experience:

#1 Descriptive Log Messages

Whenever you are writing a code, bear in mind that someone is going to read it later. And when they do so, they must understand what you have written.

The same goes for log messages. If you have clear and concise log messages that are descriptive enough to let you know exactly what has happened, it becomes a great help. This would help in understanding the context of each logged message without the need to dig into the code.

When log messages provide informative descriptions of events, they aid in understanding the flow of execution and help diagnose issues.

For example:

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def process_data(data):
    logger.info(f"Processing data: {data}")
    # Processing logic...

This code sets up a basic logging configuration to log messages with the INFO level or higher to the console, then defines a logger named after the current module. Within the process_data function, it logs a message indicating the start of data processing along with the data being processed.

#2 Set Log Levels Appropriately

Log levels tell us the severity of the errors in an application. It allows us to segregate them according to their severity, thus reiterating their significance in the application program.

Logs are popularly categorized into five different stages: DEBUG, INFO, WARNING, ERROR, and CRITICAL. In Python, each of these log levels is indicated by a certain numerical value.

Debug: It is used to understand the flow of the program and diagnose issues.

Info: This message confirms that things are working as expected and provides information about significant program issues and events.

Warning: —This indicates that something might happen in the future. The messages at these levels require attention but do not necessarily indicate errors.

Error: —This indicates errors that have already occurred in the program and need to be addressed as soon as possible.

Critical: —They indicate serious errors in which programs are unable to run anymore. They represent severe failures that require immediate attention.

During development and debugging, it's common to set the log level to DEBUG to capture detailed information about the program's execution. This level helps developers trace through the code and identify issues.

logging.basicConfig(level=logging.DEBUG)

In production environments, it's advisable to set the log level to a higher value (e.g., INFO or above) to avoid flooding the logs with excessive debugging information. This ensures that only relevant and critical information is logged.

logging.basicConfig(level=logging.INFO)

In addition to setting the log level, you can use handlers and filters to further customize logging behavior based on specific criteria, such as the source of the log message or the severity level.

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

# Create a StreamHandler
handler = logging.StreamHandler()

# Set the level for the handler
handler.setLevel(logging.DEBUG)

# Add the handler to the logger
logger.addHandler(handler)

By setting the log levels appropriately, you can stay updated on the events occurring in your program and keep track of all the issues. One thing you must note is that you should have enough descriptive log messages to notice any changes in the usual pattern.

#3 Configure Logging Globally

Configure logging settings once at the beginning of the application to ensure consistency throughout the codebase.

This includes setting the log level, specifying log formatting, and defining log handlers such as file or console output.

import logging

logging.basicConfig(level=logging.INFO, filename="app.log")

#4 Log Exceptions with Stack Traces

Including stack traces along with logging is a good idea if you want to get more detailed information about the errors. They do not just pinpoint the errors; they also show the file, line number of the error and traceback details.

This makes it easier to get to the error and make a quick fix.

import logging

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

try:
    # Code that may raise an exception
    result = 10 / 0
except Exception as e:
    logger.error("An error occurred:", exc_info=True)

#5 Avoid Hardcoding Log Messages

Refrain from hardcoding log messages by using placeholders or formatting to include dynamic information in log entries. This allows us to have context-specific details in log messages. Altogether, this enhances readability and flexibility.

import logging

# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def process_data(data):
    logger.info("Processing data: %s", data)
    # Processing logic...

#6 Use Structured Logging

Earlier, we relied on text-based logging methods, which were difficult to read and time-consuming. But now, we have structured logging formats like JSON, which uses a standardized format to log messages. They make it easier to parse and analyze data.

Python logging libraries like python-json-logger, loguru, and structlog support JSON logging. Once you have installed and configured these joggers, you can use them to write structured logs in JSON format.

#7 Rotate Log Files

Using a logger accumulates tons of data. With every file, data gets piled up and uses up all our disk space and storage. What can we do in such a case?

We can rotate these log files!

Rotating log files means archiving or deleting the old files and making space for new ones. This not only saves space but also preserves log data, simplifies debugging, and improves overall performance.

You can set up rotation according to your need:
For example, you can set it to time-based rotation (daily or weekly), size-based rotation (when the current file reaches 10MB or so), or even hybrid rotation (combining both time and size-based rotation).

In the built-in Python logging module, you can use the RotateFileHandler class to rotate files. They can employ either time-based or size-based rotation practices.

# Create a logger
logger = logging.getLogger("my_logger")
logger.setLevel(logging.DEBUG)
# Create a rotating file handler
handler = logging.handlers.RotatingFileHandler(
    "my_log.log", maxBytes=10000, backupCount=5
)
# Set the formatter for the handler
formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
handler.setFormatter(formatter)
# Add the handler to the logger
logger.addHandler(handler)
# Test the logger
logger.debug("Debug message")

Explanation:

My_logger  = name of the logger

DEBUG = log level set to Debugging state

RotateFileHandler Size = 0.01 MB (when the file size reaches this amount, it will be rotated and archived)

backupCount = 5 (specifies the number of archived files going to be kept)

We then set up the formatter to include all these metrics with the log message, and thus, whenever a breach occurs, we will get timestamped data.

This is how it happens in a built-in logger module, but you won't have to do any of this if you have a third-party logging installed in your system.

Popular choices like Atatus have made logging so much easier for sysadmins and developers. They give complete visibility across your architecture, simplify troubleshooting errors and events, and provide live tails for your logs in real-time. (Trust me, I've not even started on their best features yet!)

Get complete details on Atatus Log Monitoring Features here.

#8 Use Logging Libraries

The built-in logging libraries in Python are easy to use. First, import the logging class, followed by some basic configuration changes using basicConfig() from the logging module.

After this, you invoke the getLogger() method and then apply all the methods to include corresponding log levels (the five log levels discussed above).

Python, by default, has set its lowest logging level to that of WARNING, which is why you won't be seeing INFO or DEBUG levels in the output. You can modify this using the setLevel() method of the logging module as follows:

import logging

logging.basicConfig()
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)  # <-
logger.debug("This a debug message")
logger.info("This is an info message")
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.critical("This is a critical message")

Output:

#9 Include Timestamps in Log Messages

Timestamps are of great help when it comes to identifying the exact time and period the error occurred. It allows us to provide temporal context and facilitate chronological analysis of log events.

Timestamps prove essential when we have several microservices integrated into a single application program. However, one thing to note here is that when several components are present, it is essential that they all use the same time format. It eases interoperability.

Adopting a time format may sometimes create chaos, so choosing a standard format like ISO-8601 is better.

2022-06-15T04:32:19.955Z

Formatting logger to include ISO-8601 time format:

LOGGING = {
    "formatters": {
        "json": {
            "format": "%(asctime)s %(levelname)s %(message)s",
            "datefmt": "%Y-%m-%dT%H:%M:%SZ",
            "class": "pythonjsonlogger.jsonlogger.JsonFormatter",
        }
    },
}
logging.config.dictConfig(LOGGING)

#10 Secure Sensitive Data

Developers and sysadmins use logs to troubleshoot and analyze errors. However, log messages may contain sensitive information, such as passwords, credit card numbers, or other private data, which can be compromised.

You might have heard of Log4j and others in this context.

So, what is a solution to this?

Tips to keep your data safe and secure:

• Do not store logs in plain text files or unencrypted files
• Try to minimize private data inside log messages
• Mask or redact data if it is sensitive but needs to be logged anyway
• Use asterisk or hashes for account numbers, passwords, etc.

How to filter sensitive data in log messages:

import logging
import re

class RedactingFilter(logging.Filter):
    def __init__(self, redact_patterns):
        super().__init__()
        self.redact_patterns = redact_patterns

    def filter(self, record):
        msg = record.msg
        for pattern in self.redact_patterns:
            msg = re.sub(pattern, "REDACTED", msg)
        record.msg = msg
        return True

def setup_logging():
    # Define sensitive data patterns to redact
    redact_patterns = [r"\d{4}-\d{4}-\d{4}-\d{4}"]  # Example: Social Security Numbers

    # Set up logging with the redacting filter
    logging.basicConfig(level=logging.INFO)
    logger = logging.getLogger(__name__)
    logger.addFilter(RedactingFilter(redact_patterns))
    return logger

if __name__ == "__main__":
    # Set up logging
    logger = setup_logging()

    # Log a message with sensitive data
    logger.info("User with SSN 123-45-6789 logged in")

    # Log another message with sensitive data
    logger.info("Credit card number 1234-5678-9012-3456 provided for payment")

    # Log a non-sensitive message
    logger.info("User logout")

Output:

#11 Use Contextual Logging

Including contextual information like request ID, user session, etc., in log messages would prove to be really good when you need to trace them again later. It also gives you the overall picture of your application's behavior.

For Example:

import logging

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def process_data(data):
    logger.info("Processing data: %s", data, extra={'context': 'data_processing'})
    # Processing logic...

#12 Handle Log Configuration Dynamically

Handling log configurations dynamically in Python allows you to adjust log settings (log level, format), output destination, etc, at runtime. This capability is extremely useful when you have to change logging behavior without restarting the whole application.

Let's see how we can go about this:

i.) Logger Configuration at runtime

Start by changing the logger objects directly. This way, the logging will also be changed dynamically. This approach allows you to change the logging setting programmatically.

import logging

# Get the root logger
logger = logging.getLogger()

# Set the log level to INFO
logger.setLevel(logging.INFO)

# Add a console handler
console_handler = logging.StreamHandler()

# Create a formatter and set it for the console handler
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
console_handler.setFormatter(formatter)

# Add the console handler to the root logger
logger.addHandler(console_handler)

ii.) Using logging.config module

Python's logging.config module provides a more structured way to configure logging settings from external configuration files or dictionaries. This method allows you to separate logging configuration from your code, making it easier to manage.

import logging
import logging.config

# Define logging configuration in a dictionary
LOGGING_CONFIG = {
    'version': 1,
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'level': 'DEBUG',
            'formatter': 'simple'
        }
    },
    'formatters': {
        'simple': {
            'format': '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
        }
    },
    'root': {
        'level': 'DEBUG',
        'handlers': ['console']
    }
}

# Configure logging using the dictionary
logging.config.dictConfig(LOGGING_CONFIG)

# Example usage
logger = logging.getLogger(__name__)
logger.debug("This is a debug message")
logger.info("This is an info message")
logger.warning("This is a warning message")
logger.error("This is an error message")
logger.critical("This is a critical message")

iii.) Using fileConfig or dictConfig

You can also configure logging using configuration files (e.g., .ini files) or dictionaries. This approach allows you to change logging settings without modifying the code.

import logging
import logging.config

# Configure logging from a configuration file
logging.config.fileConfig("logging.conf")
# Or configure logging from a dictionary
logging.config.dictConfig(LOGGING_CONFIG)

#13 Use Logger Hierarchies

Organizing loggers in a hierarchy makes our task easier. It helps us understand the application structure and allows us to exert fine-grained control over the logging behavior.

Logger hierarchies enable different log levels for specific components to be set.

#14 Monitor Log Messages

The importance of monitoring log messages cannot be downplayed. Log monitoring is a crucial step in understanding your system's activities.

Some common concerns regarding monitoring Logs?

• Can I get an easy log analyzer tool with more filter and insight options over here?
• Where are the Key Events?
• How do we correlate the data?
• Is there a log monitor that normalizes the data at the collection stage?

Keeping all this in mind, we developed Atatus Logs Monitoring. Atatus not only collects logs but goes a step further and helps with these things, too:

• Logs Explorer
• Live Tail
• Log Analytics
• Alerting
• Integrations

#15 Test Logging Configurations

Test logging configurations regularly to ensure that log messages are logged and formatted correctly under different scenarios. Testing regularly allows steps to be kept up-to-date, enhancing overall system credibility.

import unittest
import logging

class TestLogging(unittest.TestCase):
    def test_logging(self):
        logging.basicConfig(level=logging.INFO)
        logger = logging.getLogger(__name__)
        logger.info("Test log message")

if __name__ == "__main__":
    unittest.main()

For Example:

.
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK

• The dot (.) indicates that the test case test_logging passed successfully.
• The summary indicates that 1 test was run.
• It took 0.001 seconds to run the test.
• The "OK" at the end signifies that all tests passed without any errors.

Conclusion

By following the practices listed here, you can better understand your Python log messages, which will increase your system's performance and efficiency.

The best practices listed here would improve interpreting log messages, setting appropriate log levels and managing formats, rotating logs at required intervals, telling you how to use logging libraries properly, and explaining ways to secure sensitive data from malicious actors.

By prioritizing logging as a key element of your development journey, you'll improve your application's performance. You'll gain a deeper understanding of how your application works, the ability to diagnose problems more quickly, and a boost to its overall performance.

Atatus: Python Performance Monitoring

Atatus is an Application Performance Management (APM) solution that collects all requests to your Python applications without requiring you to change your source code. However, the tool does more than just keep track of your application's performance.

Monitor logs from all of your Python applications and systems into a centralized and easy-to-navigate user interface, allowing you to troubleshoot faster using Python monitoring.

We give a cost-effective, scalable method to centralized Python logging, so you can obtain total insight across your complex architecture. To cut through the noise and focus on the key events that matter, you can search the logs by hostname, service, source, messages, and more. When you can correlate log events with APM slow traces and errors, troubleshooting becomes easy.

Try your 14-day free trial of Atatus.