How to Get Started with Logging in MongoDB

MongoDB is a popular open-source NoSQL database that stores data in Binary JSON format. MongoDB is designed to be flexible and scalable, making it suitable for a wide range of applications, including web and mobile applications, content management systems, and data analytics platforms.

Logging is an essential component of any database management system, including MongoDB, as it allows administrators and developers to track system behavior, monitor performance, and diagnose issues quickly and efficiently if something happens.

In this article, we will investigate how to configure logging options and how to view and analyze log messages in MongoDB.

Prerequisites

Before proceeding with this article, make sure:

• You are using Ubuntu 22.04 LTS and can log on as a user with root privileges.
• You have experience using command-line utilities.
• You understand some basic concepts in logging such as log levels and log rotation

Install MongoDB

To get started, you must install the latest version of the MongoDB server (currently V6.0) on your machine.

First, import the MongoDB public GPG Key with the following command:

wget -qO - https://www.mongodb.org/static/pgp/server-6.0.asc | sudo apt-key add -

Create a list file for MongoDB:

echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-6.0.list

Update the package database:

sudo apt-get update

And finally, install the latest version of MongoDB:

sudo apt-get install -y mongodb-org

After the installation process is done, you may start MongoDB with the following command:

sudo systemctl start mongod

Verify the status of MongoDB:

sudo systemctl status mongod

You may access MongoDB by starting a new mongosh session:

mongosh

MongoDB configurations

First of all, there are some logging-related configurations we must discuss. The configuration file is located at /etc/mongod.conf, and you may open it with the following command:

sudo nano /etc/mongod.conf

In this article, we are only concerned with the systemLog section.

. . .
# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
. . .

The destination option defines where the logs will be sent, either file or syslog. If file is specified, you must also define a path pointing to the log file. By default, the logs will be sent to /var/log/mongodb/mongod.log.

Besides destination and path, there are many other possible options for systemLog, as shown in the list below:

• verbosity: Defines the minimum verbosity level (also known as severity level or log level) a log entry has to have to be logged by MongoDB. The verbosity levels in MongoDB are slightly more complicated than other technologies, and we'll discuss this in detail later.
• quiet: When set to true, MongoDB will run in the quiet mode, limiting the number of logs pushed to the destination. This is not recommended in a production environment.
• traceAllExceptions: When set to true, MongoDB will print verbose information in the log messages, providing additional information for debugging purposes.
• syslogFacility: This setting is related to your operating system's implementation of syslog. Common facilities include kernel, user, mail, system, network, etc. By default, syslogFacility is set to the user.
• logAppend: Determines if mongos and mongod would append new log entries to the end of the existing log file when they restart. If set to false, MongoDB will back up the old log file and create a new one.
• logRotate: Specifies MongoDB's behavior when log rotation occurs. Two acceptable values are rename and reopen. rename will rename the old log file by appending a timestamp to the file name and then open a new log file. reopen closes the original log file and then reopens it.
• timeStampFormat: Determines the format of the timestamp, either iso8601-utc (1970-01-01T00:00:00.000Z) or iso8601-local (1969-12-31T19:00:00.000-05:00).
• component.<component>.verbosity: This configuration option has many variants, allowing you to define different verbosity levels for individual MongoDB components.

  systemLog:
   component:
      accessControl:
         verbosity: <int>
      command:
         verbosity: <int>
      . . .

After making changes to the configuration file, you need to restart MongoDB for the changes to take effect.

sudo systemctl restart mongod

The configuration settings mentioned above also have corresponding command-line options as shown in the linked article. However, the command-line utility does not change the configuration file, which means the next time MongoDB starts, the settings will be reset.

Additional MongoDB server parameters

MongoDB also provides a set of configuration parameters besides the ones listed above. These parameters may be defined under the setParameter option in the mongod.conf file so that they are set during start-up:

. . .
setParameter:
   <parameter1>: <value1>
   <parameter2>: <value2>

Or you may set a temporary value in the MongoDB shell (mongosh) using db.adminCommand() function:

db.adminCommand( { setParameter: 1, <parameter>: <value> } )

These parameters are used to customize MongoDB's behavior further. Some logging-related parameters include:

• logLevel: Equivalent of systemLog.verbosity, allowing you to set verbosity level using the following command:

  db.adminCommand( { setParameter: 1, logLevel: 2 } )

  { was: 0, ok: 1 }

• logComponentVerbosity: Equivalent of systemLog.component.xxx.verbosity, setting verbosity level for individual components.
• maxLogSizeKB: defines the maximum size of a single log entry's attribute field. If the log entry exceeds this limit, it will be truncated.

View log messages

By default, MongoDB logs are stored at /var/log/mongodb/mongod.log. Execute the following command to check its content:

sudo cat /var/log/mongodb/mongod.log

. . .
{"t":{"$date":"2023-03-16T14:31:51.393-04:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn15","msg":"client metadata","attr":{"remote":"127.0.0.1:42480","client":"conn15","doc":{"driver":{"name":"nodejs|mongosh","version":"5.1.0"},"os":{"type":"Linux","name":"linux","architecture":"x64","version":"5.15.90.1-microsoft-standard-WSL2"},"platform":"Node.js v16.19.1, LE (unified)","version":"5.1.0|1.8.0","application":{"name":"mongosh 1.8.0"}}}}

These logs are in JSON format and can be formatted for improved readability by using a third-party command-line utility called jq, which can be installed using the following command:

sudo apt install -y jq

Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  . . .
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libjq1
The following NEW packages will be installed:
  jq libjq1
0 upgraded, 2 newly installed, 0 to remove and 3 not upgraded.

Next, use jq to reformat the log messages:

sudo cat /var/log/mongodb/mongod.log | jq

. . .
{
  "t": {
    "$date": "2023-03-16T14:33:25.117-04:00"
  },
  "s": "I",
  "c": "NETWORK",
  "id": 23016,
  "ctx": "listener",
  "msg": "Waiting for connections",
  "attr": {
    "port": 27017,
    "ssl": "off"
  }
}

Understand MongoDB log messages

Now let's take a closer look at the log messages. Each log entry is a JSON object with the following fields:

• "t": Records the timestamp of the log message in ISO-8601 format.
• "s": The severity level of the log message.
• "c": Specifies the component this log record belongs to.
• "id": A unique identifier.
• "ctx": Context information.
• "msg": The message body.
• "attr": Include additional information such as client data, file path, line number, etc.
• "tags": Optional tags.
• "truncated": Contains truncation information if the message is truncated.
• "size": The original size of the entry before truncation.

Verbosity levels

You may have noticed that we mentioned verbosity levels many times in this article. It is an essential part of logging in MongoDB, and in this section, we'll discuss MongoDB's verbosity levels in detail.

MongoDB provides the following verbosity levels:

• "F": Fatal messages.
• "E": Error messages.
• "W": Warning messages.
• "I": Informational messages. Corresponds to numeric value 0.
• "D1"-"D5": Debug messages, Corresponds to numeric value 1-5.

You may use the numeric values to define the minimum verbosity level a log entry must have to be logged by MongoDB. Notice that only informational and debug messages have numeric values, meaning fatal, error, and warning messages will always be logged.

MongoDB provides multiple ways to define minimum verbosity level. First, you may use the systemLog.verbosity configuration option. Its default value is 0, meaning only fatal, error, warning, and informational messages will be logged. You can give it a different numeric value, for example:

. . .
systemLog:
  verbosity: 1
. . .

Restart the mongod instance, and MongoDB will start logging D1 level debug messages.

You can check the current verbosity level setting inside the MongoDB shell (mongosh):

db.getLogComponents()

{
verbosity: 1,
accessControl: { verbosity: -1 },
assert: { verbosity: -1 },
command: { verbosity: -1 },
control: { verbosity: -1 },
executor: { verbosity: -1 },
geo: { verbosity: -1 },
index: { verbosity: -1 },
. . .
}

Notice the verbosity level is set to -1 for individual components, which means it will inherit the parent's value. You can change that value by specifying the systemLog.component.<component>.verbosity option like this:

systemLog:
  component:
    accessControl:
      verbosity: 0
    command:
      verbosity: 1

Restart mongod again, and this time, its verbosity level settings should be as follows:

{
  verbosity: 1,
  accessControl: { verbosity: 0 },
  assert: { verbosity: -1 },
  command: { verbosity: 1 },
  . . .
}

Besides editing the configuration file, MongoDB also allows you to modify the verbosity level directly in the MongoDB shell using the following command:

db.adminCommand( { setParameter: 1, logLevel: 2 } )

Or modify verbosity level for individual components:

db.adminCommand( {
   setParameter: 1,
   logComponentVerbosity: {
      verbosity: 1,
      command: { verbosity: 2 },
      . . .
   }
} )

However, these components will not change the configuration file, which means when MongoDB restarts, all verbosity settings will reset based on the configuration file.

Log rotation

Log rotation refers to the process of archiving and deleting old log files to save disk space and improve performance. In MongoDB, log rotation can be triggered in three different ways.

You can send a SIGUSR1 signal to the mongod or mongos process. For example, if a mongod instance has a process ID (PID) of 2000, the following command would trigger log rotation:

kill -SIGUSR1 2000

Or you can run MongoDB's logRotate command. Enter the MongoDB shell and run the following command:

db.adminCommand( { logRotate : "server" } )

However, both methods require you to execute commands manually. The best way to set up log rotation is by using Ubuntu's default logrotate utility, which will trigger log rotation periodically. Refer to the linked article to set up log rotation for a custom application.

To use logrotate, you must go to the configuration file and set systemLog.logRotate to reopen, which follows the typical Linux log rotate behavior, and also set systemLog.logAppend to true, which is required by systemLog.logRotate: true.

Analyze log messages

Lastly, let's discuss how to analyze log entries generated by MongoDB. Unfortunately, MongoDB doesn't offer such features, but there are third-party tools you can use.

For example, we briefly mentioned the jq command-line utility and used it to reformat log entries for improved readability. The jq utility can also parse and analyze structured log messages. For instance, the following command returns the top 10 log entries with unique "msg" values, sorted by frequency.

jq -r ".msg" /var/log/mongodb/mongod.log | sort | uniq -c | sort -rn | head -10

Please refer to MongoDB documentation or jq documentation for more details.

Besides jq, which is designed for parsing JSON messages, there are also tools such as mtools, which is explicitly designed for analyzing MongoDB logs. The mtools allows you to create filters, gather information regarding a log file, and even visualize log entries by creating plots. Again, you may refer to the linked documentation for details.

Integrate MongoDB with Logtail

However, both these methods require a lot of effort and are unsuitable for users needing advanced features. A better option is to use a dedicated log management platform such as Logtail.

Logtail is a cloud-based log processing and analytics platform that helps you monitor your applications and quickly troubleshoot them if issues occur. Logtail collects, parses, and analyzes your logs in real time, providing powerful features such as alerts, dashboards, and integrations.

To integrate MongoDB with Logtail, head over to Logtail and create a new account. After you are logged in, go to Sources, and connect a new source.

After creating a new source, you will be redirected to this page, displaying the source token, data retention period, etc.

Scroll down to the installation instructions section, and follow the instructions to connect your MongoDB.

You might need to run the commands with root privileges, and if everything goes correctly, you should get the following output.

Save the changes and go to Live tail, and you should start receiving MongoDB logs in Logtail.

Conclusion

This tutorial covers how to configure MongoDB's logging behavior, how to view log messages, how to set up verbosity levels, how to rotate logs, as well as how to analyze log messages using Logtail. This tutorial aims to help you understand and use MongoDB's logging capabilities.

However, this article only discusses the basics of logging in MongoDB, and you'll likely be facing many other issues in practice. For example, when your application starts to scale, logging on to different servers to check the logs will become a tedious task. In this case, you should consider aggregating the logs so that you may monitor them in one place. For additional resources head over to MonoDB monitoring tools.

We hope this article has been helpful to you, and thank you for reading!

Article by

Eric Hu

Eric is a technical writer with a passion for writing and coding, mainly in Python and PHP. He loves transforming complex technical concepts into accessible content, solidifying understanding while sharing his own perspective. He wishes his content can be of assistance to as many people as possible.

Got an article suggestion? Let us know

Next article

Logging in Docker: Everything You Need to Know

This comprehensive guide covers everything you need to know about collecting, managing, and centralizing Docker container and daemon logs in production

→

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.