How to Change Log Levels Dynamically at Runtime

Logging plays a crucial role in troubleshooting and monitoring software systems, providing valuable insights into the behavior of your application at runtime. It allows you to track the flow of program execution, identify and address runtime errors, and analyze system performance.

One important aspect of logging is log levels, which categorize logged events based on severity. Besides organizing log entries, they also enable you to control the verbosity of log output and filter entries based on their significance.

In most production systems, the default minimum log level is set to INFO, meaning only records with a level higher than or equal to INFO will be logged. However, there are situations where you might need to adjust the log level to increase or decrease the verbosity of the logs.

Ideally, a well-designed application should allow you to modify the minimum log level without requiring a restart. After such change is made, it should also propagate seamlessly across all instances of the application. Therefore, this article will explore various methods to dynamically change log levels at runtime, eliminating the need for application restarts.

1. Creating an API endpoint to change the log level

One of the most straightforward ways to change the log level of your application without restarting is to create an API endpoint that accepts a payload specifying the desired log level. For security reasons, this endpoint should be protected and accessible only to authorized users.

Once a request is sent to the endpoint, you'll need to invoke a function that updates the log level setting of your chosen logging library or framework. If your application runs on multiple servers, you'll also need to propagate the change to all running instances. Depending on your application's architecture, this may involve implementing a mechanism for inter-process communication or using a messaging system, but such a set up is outside the scope of this article.

Here's an example that implements the above steps in a Django application that uses the Loguru framework for logging. The exact method of setting up the endpoint and changing the log level will differ in other languages or frameworks, but the idea remains the same. Please refer to the relevant documentation to learn more.

First, you need to define a URL pattern in your project's urls.py file to map the API endpoint to a view function:

from django.urls import path
from api.views import change_log_level

urlpatterns = [
    # Define an API endpoint
    path('api/change-log-level/', change_log_level, name='change-log-level'),
]

Next, you need to determine the format of the JSON payload that will be sent to the endpoint. In this example, we'll use a JSON payload with the log_level field:

{
  "log_level": "DEBUG"
}

Go ahead and implement the view function in the app's views.py file:

from django.http import JsonResponse

def change_log_level(request):
    # implement the logic here (see next snippet)
    return JsonResponse({"message": "Log level updated successfully."})

At this point, you must extract the log_level field from the request payload, validate it, and then change the log level on the logger:

from django.http import JsonResponse
import json

def change_log_level(request):
    if request.method != "POST":
        return JsonResponse({"error": "Invalid request method."}, status=405)

    payload = json.loads(request.body)
    log_level = payload.get("log_level")

    # Ensure that the log level is valid
    if log_level in [
        "TRACE",
        "DEBUG",
        "INFO",
        "SUCCESS",
        "WARNING",
        "ERROR",
        "CRITICAL",
    ]:
        # Then update the log level configuration
        logger.remove()
        logger.add(
            sys.stderr,
            format="{time:MMMM D, YYYY > HH:mm:ss!UTC} | {level} | {message} | {extra}",
            level=log_level,
            backtrace=False,
            diagnose=False,
        )

        # Log your changes so you know when it was changed (and ideally by whom)
        logger.info("log level updated to: {level}", level=log_level)

        return JsonResponse(
            {"message": f"Log level updated to {log_level} successfully."}
        )

    return JsonResponse(
        {"error": "The provided Log level field is invalid."}, status=400
    )

Once you send a request to this endpoint, you should observe that the change takes effect in your application. For example, assuming the default level is INFO, you can temporarily switch to the DEBUG level by sending the following request:

curl -X POST '<your_server_url>/api/change-log-level/' \
  -H 'Content-Type: application/json' \
  -d '{"log_level": "DEBUG"}'

You should observe the following response, confirming that the log level was updated successfully:

{"message": "Log level updated to DEBUG successfully."}

A basic way to propagate this change to all application instances is to create a separate program that iterates through the IP address of each server and sends the log level change request to each one:

import requests
import json

def changing_log_level(ip_addresses, log_level):
    for ip_address in ip_addresses:
        try:
            url = f"http://{ip_address}/api/change-log-level/"
            headers = {"Content-Type": "application/json"}
            response = requests.post(
                url, data=json.dumps({"log_level": log_level}), headers=headers
            )
            response.raise_for_status()
            print(f"Log level change successfully propagated on instance: {ip_address}")
        except requests.exceptions.RequestException as error:
            print(f"Error propagating log level change on instance: {ip_address}")
            print(error.response.text)

ip_addresses = ['192.168.1.101', '192.168.1.102', '192.168.1.103']
changing_log_level(ip_addresses, 'DEBUG')

Don't forget to set up authentication for the route so that only authorized personnel can make such requests to your server.

2. Sending UNIX signals to your program

Signals are a standardized way to communicate and control processes on UNIX-like operating systems. You can listen for a specific signal in your application and then take the appropriate action. For example, you can listen for SIGTERM (the signal for program termination), and perform a graceful shutdown before exiting the program.

The first step is to identify what signals to listen for. On Linux, there are two specially designated signals that are not associated with any predefined behavior. As a result, applications often use them to define custom actions. These are SIGUSR1 and SIGUSR2. For example, you can use the first signal to increase the log level, and the second to decrease it. This works best when your logging framework associates the log level with an integer.

Here's a Node.js example (using Pino) that listens for both signals and increments or decrements the log level as needed:

const express = require('express');
const pino = require('pino');

const app = express();

const logger = pino()

function updateLogLevel(signal) {
  let currentLevel = logger.levelVal;

  if (signal === 'SIGUSR1') {
    currentLevel += 10;
  } else {
    currentLevel -= 10;
  }

  if (`${currentLevel}` in logger.levels.labels) {
    logger.level = logger.levels.labels[currentLevel];
    console.log('The log level has been successfully changed to', logger.level);

    return;
  }

  console.log('Invalid operation. Current level is', logger.level);
}

process.on('SIGUSR1', updateLogLevel);
process.on('SIGUSR2', updateLogLevel);

app.listen('3000', () => {
  console.log('server is listening on port 3000. Process ID is', process.pid);
});

Here, the updateLogLevel() function is invoked when the SIGUSR1 and SIGUSR2 signals are received. The log level is subsequently incremented or decremented as appropriate. A message is printed to the screen once the bounds are exceeded in either direction.

Here's how to send either signal to a running application on Linux:

kill -SIGUSR1 <process_id>

kill -SIGUSR2 <process_id>

3. Utilizing framework-specific configuration files

Some logging frameworks attempt to make changing log levels at runtime easier by automatically scanning a configuration file for changes. For example, Logback, a popular logging framework for Java applications, allows you to reload its configurations upon file modification when the scan attribute of the element is set to true.

<configuration scan="true">
. . .
</configuration>

By default, the file will be scanned for changes once every minute, but you can also specify a custom scanning period:

<configuration scan="true" scanPeriod="30 seconds">
. . .
</configuration>

As an example, here is a simple Java program that logs some messages every 10 seconds using the Logback framework with SLF4J:

package com.example;

import org.slf4j.LoggerFactory;
import org.slf4j.Logger;

public class App {

    private static final Logger logger = LoggerFactory.getLogger(App.class);

    public static void main(String[] args) {

        while (true) {
            logger.trace("Entering method foo()");
            logger.debug("Received request from 198.12.34.56");
            logger.info("User logged in: john");
            logger.warn("Connection to server lost. Retrying...");
            logger.error("Failed to write data to file: myFile.txt");

            // Sleep for 10 seconds
            try {
                Thread.sleep(10000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
    }
}

Here's the corresponding Logback configuration for the above program. Notice that its minimum log level is set to INFO, and the scan period is set to 10 seconds:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration>

<configuration scan="true" scanPeriod="10 seconds">
    <import class="ch.qos.logback.classic.encoder.PatternLayoutEncoder" />
    <import class="ch.qos.logback.core.ConsoleAppender" />

    <appender name="STDOUT" class="ConsoleAppender">
        <encoder class="PatternLayoutEncoder">
            <pattern>%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} -- %msg.%n</pattern>
        </encoder>
    </appender>

    <root level="info">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

When you run the program, you will observe that only INFO, WARN, and ERROR logs are printed:

Keep the application running, and edit the configuration file. Change the log level into DEBUG as shown below:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE configuration>

<configuration scan="true" scanPeriod="10 seconds">
    . . .

    <root level="debug">
        <appender-ref ref="STDOUT" />
    </root>
</configuration>

Wait for about 10 seconds after saving the file. You should observe that DEBUG-level messages are now being included in the output without having to restart the program:

If you're using Log4J2, you can use the monitorInterval attribute instead to achieve the same results:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration monitorInterval="10">
. . .
</Configuration>

Likewise, if your logging framework of choice supports configuration files, do investigate further to see if it provides a way to watch the file for changes and reload its configuration.

Final thoughts

In this article, we explored various approaches to dynamically change log levels in production without restarting your application. We discussed creating APIs, sending signals, and modifying configuration files. Each of these methods provide varying levels of flexibility and can be adapted to different programming languages, so you should definitely find a solution that meets your requirements.

To learn more about log levels, we recommend reading our comprehensive tutorial on log levels and their usage or logging in microservices. It will provide you with a deeper understanding of log levels and their significance in logging. You can also check out our other logging guides here.

Thanks for reading, and happy logging!

Got an article suggestion? Let us know

Next article

Logging in Microservices: 5 Best Practices

Learn more about the best practices for effective logging in a microservice architecture

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.