Docker Logging

As more organizations are moving to a cloud-native architecture, there is an ever-increasing need to monitor applications and services. Logging is a crucial part of this process, as it provides the insights and visibility to identify potential issues and track application performance.

When it comes to managing and monitoring applications, Docker logging is an essential part of the process. Logging is important to track the performance of your application, identify any problems, and ultimately gain insights into how your application or system is being used.

Docker logging helps you achieve this by providing you with the ability to monitor, store, and analyze all the relevant data of your Docker containers. Docker is a popular platform for applications, and it comes with its own set of logging methods.

In this blog post, we will discuss what Docker logging is, how it can help you better manage and monitor your applications, and the best practices for setting up Docker logging.

Table of Content

1. What is Docker Logging?
2. Why is Docker Logging different?
3. Docker Logging commands
4. Configuring the Logging Driver
5. Delivery mode of log messages from container to log driver
6. Why do you need to combine docker logging and delivery modes?

What is Docker Logging?

Docker is a popular platform for building, shipping, and running container applications. Containers are lightweight and portable, allowing developers to build, test, and deploy applications quickly and easily across different environments.

However, with the increasing number of containers running simultaneously, monitoring and managing container logs become essential to troubleshoot issues and ensure that the containers are running as expected.

Docker provides a logging mechanism that allows developers to capture and view container logs. First, the Docker logging system collects logs generated by container applications. Then, it sends them to various log drivers, which can store the logs in different locations, such as the local file system, remote syslog server, or a third-party log management service.

By default, Docker uses the JSON-file log driver to store container logs in JSON format on the local file system. However, developers can configure Docker to use other log drivers to store container logs in different formats and locations based on their needs.

Example of Docker Logging:

Let's say you have a Docker container running an application that generates log output. You can view the logs of this container using the docker logs command.

First, find the ID of the running container using the docker ps -a command:

$ docker ps -a

In this example, the container ID is a4664baab91e for the container named nsqd.

To view the logs of this container, you can run the following command:

$ docker logs a4664baab91e

This will display the logs generated by the container's application on the console.

You can also use other logging commands to filter and customize the output. For example, to follow the container logs in real-time, you can use the --follow or -f flag:

$ docker logs --follow a4664baab91e

This will display the logs as they are generated in real-time.

Why is Docker Logging different?

Docker logging is different because of the way containers are designed and managed. Containers are multi-leveled, meaning that they consist of multiple layers of file systems, making it difficult to capture and manage log messages consistently. Additionally, containers are temporary, meaning they can be created and destroyed rapidly, making it challenging to ensure that log messages are captured and retained.

To address these challenges, Docker provides a logging system that captures log messages from each container and sends them to a central location for storage and analysis. Docker accomplishes this by capturing the standard output and standard error streams of the processes running inside each container and redirecting them to a logging driver responsible for storing and managing the log data.

However, traditional methods that rely on system logging, such as syslog, may not work as effectively in a containerized environment because containers are isolated and have their file systems. So instead, Docker provides its logging drivers designed to work with containers and capture log data in a consistent and reliable way.

Docker logging is different because it is designed to address the unique challenges of containerized environments, including the multi-leveled and ephemeral nature of containers. In addition, Docker makes it easier to capture, manage, and analyze log data from containers by providing a centralized logging system and specific logging drivers.

Docker Logging commands

Docker provides various logging commands to view and manage the logs of Docker containers. Here are some commonly used Docker logging commands:

1. Display the logs of a running container:

docker logs <container_id>

Example:

docker logs af502d0c9422

2. To follow the logs of a running container in real-time:

docker logs --follow <container_id>

Example:

docker logs --follow af502d0c9422

3. To display the last 10 lines of logs of a running container:

docker logs --tail 10 <container_id>

Example:

docker logs --tail 10 af502d0c9422

4. To display the logs of a running container since the last 5 minutes:

docker logs --since 5m <container_id>

Example:

docker logs --since 5m af502d0c9422

5. To display the logs of a running container until the last 10 minutes:

// duration
docker logs --until 10m <container_id>

// timestamp
docker logs --until 2019-10-20T10:00:00 <container_id>

Example:

docker logs --until 2019-10-20T10:00:00 af502d0c9422

6. To display the logs of a running container with timestamps:

docker logs --timestamps <container_id>

Example:

docker logs --timestamps af502d0c9422

7. To display the logs of a running container with extra details:

docker logs --details <container_id>

Example:

docker logs --details af502d0c9422

8. To display the logs of a running container between two specified date times:

docker logs --since "2023-02-28T13:23:37Z" --until "2023-03-01T13:23:37Z" <container_id>

Example:

docker logs --since "2023-02-28T13:23:37Z" --until "2023-03-01T13:23:37Z" af502d0c9422

Note that you need to replace <container_id> in these examples with the actual ID of the container that you want to view the logs for.

Configuring the Logging Driver

To configure a Docker container to use a logging driver, you need to follow these general steps:

1. Choose a logging driver

Docker supports several logging drivers, including the default json-file, as well as syslog, journald, gelf, fluentd, awslogs, and others. Choose the logging driver that best suits your needs.

In this example, we'll use the syslog logging driver to send container logs to the syslog daemon on the host.

Log-driver and log-opt keys in the daemon.json file are configured to use the syslog driver by setting the appropriate values in /etc/docker/ on Linux hosts and C:/ProgramData/docker/config/daemon.json on Windows hosts.

{
  "log-driver": "syslog",
  "log-opts": {
    "syslog-address": "udp://1.2.3.4:1111"
  }
}

In the following example, syslog is set as the log driver and syslog-address is set as the address for the log. This example uses UDP as the syslog-address option and supports both TCP and UDP.

2. Start a container with the --log-driver option

When you start a container, you can use the --log-driver option to specify the logging driver to use. For example, to start a container with the syslog logging driver, you can use the following command:

docker run --log-driver=syslog image_name

Replace your-image-name with the name of the Docker image you want to run.

3. Configure logging driver options

Depending on the logging driver you choose, you may need to configure additional options. You can check the documentation for your chosen logging driver to learn about the required and optional configuration options.

If you use the awslogs logging driver, you will need to specify the AWS region and log group name using environment variables or the --log-opt option.

For the syslog logging driver, you can configure the following options:

• syslog-address: The address of the syslog daemon on the host. The default is unix:///dev/log, which means the syslog daemon is running on the host's local socket.
• syslog-facility: The syslog facility to use for the container logs. The default is daemon, which means the logs will be sent to the system log.

You can specify these options using the --log-opt option when you start the container. For example, to send container logs to a remote syslog server at IP address 192.168.1.100 and using the local0 facility, you can use the following command:

docker run \
	--log-driver=syslog \
    --log-opt syslog-address=udp://192.168.1.100:514 \
    --log-opt syslog-facility=local0 query_service

4. View container logs

Once you've started a container with a logging driver, you can view its logs using the docker logs command. For example, to view the logs of a container named your-container-name, you can use the following command:

docker logs query_service

By default, docker logs will show the logs in the JSON format. However, since we're using the syslog logging driver, you can also view the logs in the syslog format by using the --log-opt option:

docker logs --log-opt syslog-format=rfc3164 query_service

This will show the container logs in the traditional syslog format, which includes the timestamp, hostname, and program name.

Delivery mode of log messages from container to log driver

A delivery method is how training materials are available to support and facilitate learning. Multiple instructional options are available for mobile learning. SMS, USSD, mobile applications, and mobile web technologies are a few instances.

You can set up your container's logging in either a blocking or non-blocking delivery mode regardless of the logging driver you select. Depending on your chosen mode, the container will prioritize logging operations over all other duties.

The blocking and non-blocking delivery modes are available for writing logs in Docker instances. Depending on the mode you select, the container will prioritize logging operations about its other duties.

Blocking delivery mode

Docker's default behavior is blocking. Therefore, every time a communication for the driver needs to be delivered, it will pause the application.

Although it ensures that every communication is sent to the driver, it can cause your application's performance latency. This is because the container delays the application's other duties until the message has been delivered if the logging driver is busy.

The delay varies depending on the logging driver you use. Because it writes to the local filesystem, the default JSON file driver produces logs rapidly, making it unlikely to stop and increase latency. Nevertheless, log drivers that must establish a connection with a distant server may block for extended periods and produce observable latency.

Non-blocking delivery mode

A container first writes its logs to an in-memory ring buffer in non-blocking mode, where they are kept until the logging driver is ready to process them. The container can continue running the application even if the driver is busy by instantly passing the application output to the ring buffer.

This ensures that an application running in the container won't suffer from a large volume of logging activity.

The container sends logs to an in-memory ring buffer while operating in non-blocking mode. The ring buffer holds the logs until it is complete. The log is only delivered after that. The container sends logs to the ring buffer and keeps running the application even if the driver isn't accessible.

As a result, large-volume logging is guaranteed without affecting performance. However, there are drawbacks.

The logging driver might not record every occurrence, even in non-blocking mode. Buffered records will be deleted before being sent if the buffer is full. The max-buffer-size setting allows you to control how much RAM the ring buffer will use.

If you have more RAM available, you can raise the max-buffer-size parameter from its default value of 1 MB to improve the reliability of your container's logging.

The following example starts a container with log output in non-blocking mode:

docker run \
	--log-driver=syslog \
    --log-opt syslog-address=udp://192.168.1.100:514 \
    --log-opt syslog-facility=local0
    --log-opt mode=non-blocking query_service

Why do you need to combine Docker logging and Delivery modes?

Combining Docker logging and delivery modes can be helpful for a number of reasons:

1. Centralized Logging: Docker logging can capture logs from different containers on multiple hosts. By combining Docker logging with delivery modes such as ELK (Elasticsearch, Logstash, and Kibana), Splunk, or Fluentd, you can centralize all your logs in one place, making it easier to analyze and troubleshoot issues.
2. Scalability: Docker logging can be easily configured to send logs to a central logging system, which can be scaled up or down as needed. By combining Docker logging with a delivery mode that can handle high volumes of logs, you can ensure that your logging system can handle any level of scale required.
3. Real-time monitoring: By combining Docker logging with a delivery mode that provides real-time monitoring capabilities, you can quickly identify and respond to any issues. This can reduce downtime and ensure that your applications are always running smoothly.
4. Compliance: Many industries have specific compliance requirements around logging and data retention. By combining Docker logging with a delivery mode that meets these compliance requirements, you can ensure that you meet all necessary regulations.

Conclusion

Your decision regarding the log delivery method and Docker logging driver may have an appreciable impact on how well your containerized apps perform. Use the JSON file driver for accurate logging, consistent speed, and improved visibility.

Only if the data is analyzed can logging play a significant role in providing information about the status of your infrastructure. However, because there are so many journal records, doing it by hand would be like searching through a hay mountain, which is why you require a platform for log data processing. You can choose between open-source programs and paid software if you want to make the most of your Docker records.

Logging is undoubtedly one of the most crucial components to get right when developing containerized apps from a DevOps perspective. Log management enables DevOps teams to troubleshoot problems more quickly, recognize patterns, find bugs, and ensure they don't appear as problems.

Implying Atatus for Log Monitoring

With the Atatus Logs Monitoring platform's cost-effective, scalable approach to centralized logging, you can gain complete visibility across your entire architecture. All log data is processed in one place and can be visualized.

All high-priority incidents are reported immediately by endowing Atatus. Using error logs, custom queries, or trigger conditions, you can configure advanced alerts.

Depending on your log analysis, you may receive notifications of certain events. Logging events and slow traces can be correlated with APM errors and slow traces to simplify troubleshooting. Discover more and resolve problems faster.

Analyze, search, and filter your logs on the fly in your production environment to troubleshoot problems. From a single interface, tail across systems similar to "tail -f".

Atatus offers a free 14-day trial to streamline your log management!