When troubleshooting server issues or diagnosing problems with your services, logs are your most valuable resource.
In Linux systems using systemd, a
logging system called the journal is used to capture and centralize log
entries from the kernel, various systemd services, and other userland
processes.
This journal is implemented by the journald daemon, which collects and stores
log data from various sources in a structured, binary format for ease of
retrieval.
However, this centralized log can grow large and contain tens or even hundreds of thousands of entries. To efficiently find the information you need, you must master the art of filtering and querying the journal.
This is where the journalctl utility comes in. It allows you to query and
filter logs based on various criteria, such as time, service, or boot session,
and it can output log data in different formats, making it adaptable to various
analysis and visualization needs.
In this guide, you'll learn how to effectively navigate and filter the systemd
journal using journalctl, enabling you to:
Let's get started!
If you are troubleshooting more than one server, journalctl becomes slow because it is per host. Shipping the journal to Better Stack gives you one place to search, filter, and correlate logs across all machines.
Linux users can only view log entries generated by their processes and services
by default. If you attempt to view all system logs using journalctl, you might
encounter a message like this:
This message indicates that your current user lacks the necessary permissions to
access all log entries. To grant a user access to the complete system journal,
add them to a privileged group, such as adm or systemd-journal:
After adding the user to the group, they need to log out and log back in for the
changes to take effect. Once they log back in, they can view all system logs
using journalctl.
This section will guide you through accessing and navigating system logs using
the journalctl command. We'll start with basic queries and then explore ways
to customize the output.
To see all log entries collected by the journald daemon, run the journalctl
command without arguments:
This command lists all available journal entries in chronological order (from oldest to newest) and pipes the output through a pager for ease of navigation, as the log often contains tens or even hundreds of thousands of lines.
You can show the most recent logs first by adding the --reverse flag:
The output begins with a header showing the time range of the displayed logs:
Following the header, you'll find individual log entries, sorted from oldest to newest. Each entry follows this format:
Each entry starts with a timestamp, the machine's hostname, the program that generated the log entry, and its process id. The log message itself comes afterward. The format will be recognizable to anyone familiar with standard syslog logging.
If you want to process the journalctl output using tools like grep, awk,
or sed, or redirect it to a file, you can use the --no-pager option to
disable paging:
Since the output can be extensive, it's often necessary to limit the data to
make it more manageable. You can achieve this by using various filtering options
provided by journalctl, such as:
--since and --until to display logs within a
specific timeframe:--unit flag:These options allow you to efficiently narrow down the log data to suit your specific needs. Other relevant options to explore include:
--no-hostname: Suppresses the hostname in log entries.--no-full: Truncates long log fields in the output instead of displaying
them in full.-a/--all: Displays all fields, even those that are normally suppressed or
truncated.--truncate-newline: Removes newline characters from the MESSAGE field.-q/--quiet: Suppresses the header and metadata output of the journalctl
command to leave only the raw log content.In the following sections, we'll delve into more advanced filtering and output customization techniques to help you efficiently find the information you need.
When working with logs generated by journalctl, it's often beneficial to
customize the output format to fit various needs.
A simple modification to the output is configuring timestamps to be displayed in UTC instead of the system time:
The -o/--output option allows you to print the journal output in a variety of
formats. For example, you can print the entries in a structured log
format such as JSON with:
As you can see, this output is far more detailed than the default with a wealth of information in an easily parsable format.
For improved readability, you can use the json-pretty format:
This will present the JSON output in a more human-friendly format, although it will take up more space on your screen.
You'll notice that this JSON format contains many fields that were not present in the default output. You'll spot three different kinds of fields in the each entry:
Fields prefixed with __: These fields are journal-specific metadata
that are generated and managed internally by systemd-journald. They are not
directly related to the log message content but provide additional context
for managing and querying logs.
Fields prefixed with _: These fields describe system or
process-related metadata that systemd-journald collects from the
environment when the log entry is created. They are tied to the source of the
log.
Fields without a prefix: These fields represent log content or attributes explicitly set by the logging application or system. They usually contain the actual information being logged or its classification (e.g., priority, message content, syslog facility).
To see all the available fields that are present in the systemd journal, use the
--fields flag:
You can find out more about these fields by reading the systemd manual:
If you'd like to display specific fields alone, you can then use the
--output-fields option when using certain formats such as json,
json-pretty, verbose, export and others.
Here's the syntax:
For example, if you're only interested in the log message and its priority level, you can use:
Depending on the --output format, you'll notice that certain fields are
included regardless of the --output-fields option. These are fields typically
required to identify or understand the entry's source or scope.
Here are the other formats that can control the output produced by journalctl.
You can examine the
complete list here.
short: This is the default output format.cat: Includes the log message alone by default.json: JSON-formatted output containing all available fields per entry.json-pretty: Prettified json output for better readability.verbose: Displays the entire log entry with all available fields per entry.Now that you're familiar with how to customize the presentation of journalctl
output, let's explore techniques to refine your log searches and zero in on the
specific information you need.
When working with systems that undergo frequent reboots, filtering logs based on
specific boot sessions can be helpful. journalctl provides options to isolate
logs generated during a particular boot, allowing you to focus your analysis on
a specific time window.
To display logs from the current boot session, use the -b flag:
This will show all log entries recorded since the system last started, including low-level kernel messages related to the boot process.
To see a list of all recorded boot sessions, use the --list-boots option:
This command outputs a table with information about each boot session, including:
You can use either the offset number or the boot ID to filter logs for a specific boot session:
One of the most common ways to narrow down your log search is by filtering
entries based on their timestamps. journalctl offers a few options to specify
time ranges for your queries.
You can define a time window for your log search using the --since and
--until flags to specify the lower and upper bounds of the time range
respectively.
Both flags accept flexible timestamp formats, including:
YYYY-MM-DD HH:MM:SS (e.g., 2021-11-23 23:02:15)YYYY-MM-DD (e.g., 2021-05-04)HH:MM (e.g., 12:00)5 hour ago, 32 min agoyesterday, today, nowFor instance, to view logs from today onward, use:
Note that regardless of the filtering period, the header will continue to reference the period of the logs available in the journal.
The output may show a lot of records, but you'll observe that they were all recorded on the current day.
You can also filter logs that fall on a specific date or between specific dates with:
While time-based filtering is useful, it still leaves you with logs from various sources. You can combine time filtering with other filtering options to refine your search further. For instance, you can focus on specific applications or services, as we'll explore in the next section.
When you're troubleshooting a specific application or service, it's necessary to
isolate its logs from the rest of the system. journalctl allows you to do this
by filtering entries based on the systemd unit they belong to.
For example, to view logs generated by a specific service, use the -u/--unit
flag followed by the service name:
You'll see the log entries from the Docker service alone:
If there are no entries for the specified service, you'll see the following instead:
You can also filter for multiple services simultaneously by repeating the
--unit flag:
This shows logs from both rsyslog.service and nginx.service that occurred
from an hour ago.
The entries will be merged and displayed in chronological order, making it significantly easier to understand the sequence of events within your system.
Beyond filtering by time or service, journalctl allows log entries to be
filtered based on their associated metadata. This allows for precise queries
that target logs with particular characteristics.
You've already seen the available metadata fields, which can be retrieved by running:
There are many options, but you can see which of the fields are available on the
logs you're interested in through the json or verbose format as follows:
Once you've figured out what fields you're interested in, you can display all
possible values for that field with the -F/--field flag. For example, to see
all possible priority levels, use:
The numbers can be mapped to the standard syslog priority levels:
To filter for entries with a specific metadata value, use the field name followed by an equals sign (=) and the desired value.
For example, to show only logs with priority level "3", use:
You can also combine multiple metadata filters to refine your search further. For instance, to see error logs from the kernel, run:
For commonly used fields, you can use dedicated flags to reduce verbosity and
make your journalctl commands more concise. This includes:
-p/--priority: PRIORITY-f/--facility: SYSLOG_FACILITY-t/--identifier: SYSLOG_IDENTIFIER-u/--unit: _SYSTEMD_UNITFor example, instead of:
You can simply write:
Similar to using tail -f to monitor a file for new content, journalctl
provides a way to "tail" or follow journal entries in real time.
This is incredibly useful for diagnosing issues in real time, especially when troubleshooting intermittent problems or observing the effects of configuration changes.
To initiate real time log following, use the -f/--follow flag with
journalctl:
This command will display the most recent 10 log entries and continuously
display entries as they are written to the journal. You can configure how many
logs are initially displayed with the -n/--lines option:
You can also use the --no-tail option to show all lines even when in follow
mode:
The --follow flag can be combined with other journalctl filters to focus on
specific events. For example, to tail messages with error severity or higher,
use:
Or to tail logs from a specific service:
You can stop following the logs and return to the command prompt any time by
pressing Ctrl+C.
journalctl --follow is great on a single box. Live Tail gives you the same feel, but across all your servers, with instant search and one click context on each event.
While filtering allows you to narrow down log entries based on specific
criteria, journalctl also offers the -g/--grep flag to find entries
containing particular keywords or patterns:
You can combine it with any of the filtering options discussed above for more
precise results. For instance, the command below will display all journal
entries from the ssh service containing the phrase "Invalid user" within the
last one hour:
The --grep flag also supports regular expressions for more complex searches:
The search is case-insensitive by default, but you can make it case-sensitive
through the --case-sensitive flag:
The journalctl utility also offers several options to manage the size and
content of the system journal to prevent excessive disk usage.
You can see how much space is currently occupied by the journal through the
--disk-usage flag:
This will display the total size of the active and archived journals:
If the journal is taking up too much space, you can choose from the following vacuuming options to manually shrink it to a desired size:
--vacuum-size=<bytes>: Shrink the journal to a desired size.--vacuum-files=<int>: Reduce the number of journal files to <int>.--vacuum-time=<time>: Remove log entries older than a specified date.For example, you can reduce the journal size to 500 MB with:
You'll see the program's output appear on the screen:
As you can see, the journal was shrunk to 500 MB after log entries totalling 1.3 GB in size were deleted from the archive.
Instead of specifying a size, you can also delete logs based on their age using
the --vacuum-time option. The command below will delete any entries that were
recorded more than one month ago:
To automatically manage journal size, you can modify the
following options
in the /etc/systemd/journald.conf file:
SystemMaxUse and RuntimeMaxUse: Set the maximum amount of space that the
journal should take up in persistent storage (/var/log/journal) and volatile
storage (/run/log/journal) respectively.SystemKeepFree and RuntimeKeepFree: Defines the percentage of disk space
that should always be kept free for other uses.SystemMaxFileSize and RuntimeMaxFileSize: Controls how large journal
entries should grow before being rotated.SystemMaxFiles and RuntimeMaxFiles: Specifies the maximum number of
journal files to keep.To further reduce disk space usage, you can enable compression for the journal
through the Compress option. However, note that compression can slightly
impact performance when retrieving log data.
While journalctl is an excellent tool for accessing and analyzing system logs locally, it starts to fall short as soon as you manage more than one server. At that point, your logs are scattered across machines, and you end up repeating the same queries host by host.
A centralized log management platform like Better Stack brings those logs into one place, so you can search and filter across all servers at once, correlate related events, and move much faster when diagnosing incidents.
If you want the quickest path, this walkthrough shows how to set up the Better Stack collector and start shipping logs with sensible defaults:
With logs centralized, you can go beyond journalctl --follow. Better Stack Live Tail lets you watch events in real time across all machines, drill into context with a click, and search instantly while new logs are still coming in:
Once your journald logs are flowing, you can also move past “find the one line” troubleshooting and start spotting patterns over time. Better Stack lets you build dashboards from your log data so you can visualise trends, correlate spikes with deploys, and catch anomalies early.
Centralization also makes alerting practical. Instead of manually checking logs or keeping a terminal open, you can trigger notifications when important patterns appear, and route incidents to the right people right away.
Finally, retention becomes simpler. Journald logs live on each server and are constrained by local disk limits and rotation policies. In contrast, Better Stack provides scalable storage so you can retain logs for longer, search historical incidents, and keep data around for compliance when needed. Dashboards and visualizations help you spot trends you would otherwise miss in raw output, like rising SSH failures, recurring service restarts, or bursts of kernel errors.
Start taking control of your logs with Better Stack by creating a free account here.
In this article, we've thoroughly explored the systemd journal and uncovered
the powerful capabilities of journalctl.
From understanding the journal's purpose and structure to learning how to filter, search, and customize log output, you've gained essential skills for navigating and analyzing your system logs.
With the techniques learned in this guide, you can effectively troubleshoot issues, debug complex problems, and gain valuable insights into your system's behavior.
By making the most of journalctl, you'll be better equipped to maintain a
reliable and well-monitored system.
For more details, be sure to check out the official documentation or type
man journalctl in your terminal.
Thanks for reading, and happy logging!
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
We use cookies to authenticate users, improve the product user experience, and for personalized ads. Learn more.