Logging in Docker: Strategies and Best Practices

Docker provides a logging mechanism that captures your application's output and sends it to various destinations. Understanding how this mechanism works is crucial for optimizing the performance and the reliability of your logs.

In this guide, we'll delve into the world of Docker logging drivers, exploring how they influence your container's behavior and how to choose the right driver for your needs. We'll cover:

• How Docker captures and redirects container logs.
• How different drivers and configurations can impact your application's performance
• Ensuring your logs are captured consistently and without loss.
• Choosing the appropriate log delivery for your application logging.

By the end of this post, you'll be well equipped to make informed decisions about your Docker logging strategy, and ensure both optimal performance and reliable log capture.

Let's get started!

Prerequisites

• Basic command-line skills.
• A recent version of Docker installed on your system.

Logging to the standard output/error

The first step to understanding Docker logging is grasping the concept of logging to the standard output and standard error streams.

When an application running inside a container writes messages to stdout or stderr, Docker captures this output and makes it accessible through various mechanisms. This forms the basis of Docker's logging system.

Since the stdout and stderr are universal concepts in operating systems, this approach works seamlessly across different environments.

It's also a straightforward and standardized way for applications to generate logs, regardless of their programming language or framework.

By default, Docker captures the data sent to these streams from your containers and directs them to a logging driver, which determines where the logs are ultimately stored and accessed.

If the containerized service is configured to write logs to a file instead of the standard output or error streams, Docker's logging mechanisms won't work, and the logs will be lost when the container is removed.

To prevent this, you should configure the application to send its logs to stdout and stderr. If this isn't possible, you can create symbolic links from the log files to /dev/stdout or /dev/stderr:

# forward logs to Docker's log collector
RUN ln -sf /dev/stdout <log_file_path> \
    && ln -sf /dev/stderr <error_log_file_path>

This effectively tricks the application into writing to the standard streams, allowing Docker to capture the logs.

Choosing a logging driver

Docker's logging driver is what dictates how the logs captured from the standard output and error streams are processed and stored.

The default driver is json-file which converts the container logs into JSON format and stores them in files on the host machine.

You can check your Docker daemon's default logging driver with the command below:

docker info --format '{{.LoggingDriver}}'

You should see the following output:

json-file

To see the logging driver configured for a specific container, use:

docker inspect -f '{{.HostConfig.LogConfig.Type}}' <container>

json-file

You can also find out the log file location for a container with:

docker inspect -f '{{.LogPath}}' <container>

You will see a response that looks like this:

/var/lib/docker/containers/<container-id>/<container-id>-json.log

Some of the other supported logging drivers are:

• none: Disables logging altogether. Useful when logs are not required or handled externally.
• local: The recommended driver for most use cases. It offers better performance and efficient disk usage compared to json-file.
• syslog: Sends logs to the system's syslog daemon, integrating with your system-wide logging infrastructure.
• journald: Uses the journald logging system on Linux distributions.
• fluentd, splunk, awslogs, gcplogs, etc: Forward logs to logging tools or cloud platforms for centralized management and analysis.

Although json-file is the current default for backward compatibility, local is generally recommended due to its performance advantages and more appropriate defaults, which include zero allocations for logging and log rotation/compression enabled by default.

Let's look at how to configure it next.

Configuring the default logging driver

You can set the default logging driver by creating or editing the Docker daemon configuration file:

sudo nano /etc/docker/daemon.json

Populate it with the following contents:

{
  "log-driver": "local"
}

With the local driver, your container logs are automatically compressed and rotated. By default, it retains the latest 100MB of logs per container, divided into 5 compressed files of 20MB each.

If you'd like to modify the default options, you can specify the log-opts property as follows:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "20m",
    "max-file": "5",
    "compress": "true",
  }
}

Each property in the log-opts object must be a string (including boolean and numeric values, as seen above). The supported options for the local driver are:

• max-size: The maximum size of the log file before it is rotated (20MB). You may specify an integer plus the measuring unit (k for kilobytes, m for megabytes, and g for gigabytes).
• max-file: The maximum allowable number of log files for each container. When an excess file is created, the oldest one will be deleted. The max-size option must also be specified for this setting to take effect.
• compress: When set to true, the rotated log files will be compressed to save disk space.

After saving the file, you must restart the docker service to apply the changes to newly created containers:

sudo systemctl restart docker

To use a different logging driver or adjust logging options for a particular container, simply include the --log-driver and --log-opt flags in your docker container create or docker run command:

docker run --log-driver syslog --log-opt syslog-address=tcp://192.168.1.10:514 -d <image>

Please see the documentation for the supported log-opts for each logging driver.

Choosing a log delivery mode

The log delivery mode for a Docker container refers to how it prioritizes the delivery of incoming log messages to the configured driver. The following two modes are supported:

1. Blocking mode (default)

In blocking mode, log messages are delivered synchronously. This means your application waits for each log entry to be processed by the driver before continuing execution.

This approach guarantees that each log message will be delivered to the driver, but it can introduce performance overhead, especially with slow logging drivers.

Blocking mode is suitable for scenarios where log reliability is paramount and the logging driver is fast (such as the local or json-file drivers that write to the local disk).

2. Non-blocking mode

In non-blocking mode, log messages are processed asynchronously. Your application continues running without waiting for log delivery. Messages are buffered in memory and sent to the driver in the background.

This minimizes performance impact, even with high log volumes or slow drivers but at the risk of losing log messages if the buffer overflows before the driver can process them.

Non-blocking mode is ideal when application performance is critical and you're using a remote logging driver that might experience latency.

Configuring log delivery

You can configure the delivery mode in your Docker daemon configuration:

{
  "log-driver": "syslog",
  "log-opts": {
    "mode": "non-blocking"
  }
}

Or when running a container:

docker run --log-driver syslog --log-opt mode=non-blocking <image>

To improve reliability in non-blocking mode, you can increase the maximum buffer size from its default 1 MB to a more suitable size through the max-buffer-size property:

{
  "log-driver": "syslog",
  "log-opts": {
    "mode": "non-blocking",
    "max-buffer-size": "20m"
  }
}

This allows the buffer to hold more log messages before potentially overflowing, giving the driver more time to process them.

Centralizing your Docker logs

While Docker offers a variety of logging drivers and configurations, the local driver in blocking mode often provides the best balance of performance and reliability for most applications.

Writing logs to a local file is inherently fast, and blocking mode adds a layer of reliability by ensuring that every log message is captured and safely stored on disk. The local driver also provides control over log rotation and compression, allowing you to fine-tune disk usage and log retention.

If your application generates a high volume of logs and performs disk-intensive operations, you can also use the local driver in non-blocking mode to maintain application performance by buffering logs in memory before writing them to disk.

A log collector like Vector or the OpenTelemetry Collector can then be employed to stream the logs from the local files, apply any desired transformations, and forward them to a centralized log management solution such as Better Stack.

Deploying the log collector can be done in a dedicated logging container that collects and processes logs from all other containers in your Docker environment before forwarding them to a central location.

For more complex deployments, a sidecar approach pairs each application container with a dedicated logging container. This allows for customized logging strategies per application at the cost of increased resource consumption and complexity.

Final thoughts

Effectively managing Docker container logs is essential for maintaining application health, troubleshooting issues, and gaining valuable insights into your system's behavior.

By understanding the different logging drivers, configuring them appropriately, and implementing best practices like centralization and log rotation, you can ensure that your Docker logging strategy meets your needs.

Thanks for reading!

Article by

Ayooluwa Isaiah

Ayo is a technical content manager at Better Stack. His passion is simplifying and communicating complex technical ideas effectively. His work was featured on several esteemed publications including LWN.net, Digital Ocean, and CSS-Tricks. When he's not writing or coding, he loves to travel, bike, and play tennis.

Got an article suggestion? Let us know

Next article

Docker Container Logs: A Comprehensive Guide

This guide introduces Docker's built-in logging capabilities to help you easily monitor your containers' various operations.

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.