This tutorial describes how you can configure a centralized logging service on
Linux with Rsyslog. Such a centralized configuration
is beneficial when administering logs from multiple servers since you won't have
to log into each server to review its logs, particularly if there's a huge
number of servers.
With a centralized logging solution, the host server will receive log entries
from all client servers so that they can be monitored, analyzed, and archived is
one place. Usually
cloud-hosted log management solutions are
favored for their easy set up, rich feature-set, and transparent costs, but
self-hosted solutions can be a more appropriate option in some uses.
Rsyslog is a good and popular choice for setting up centralized logging as it
offers stunning performance (up to one million log entries per second), message
delivery over TCP, SSL, and TLS, multi-threading, and highly configurable
filtering and output formats. It can also forward incoming log entries to other
destinations after certain transformations are applied to them.
By reading through this article, you will learn how to do the following:
Configure a central logging server that receives all log messages from
Configure Rsyslog on each client server that forwards all its logs to
the logging server.
To follow through with this tutorial, you must have access to two Linux systems.
One of them will act as the centralized logging server, while the other will
pose as the production service that is generating logs. A non-root user with
sudo access is required on both systems.
The commands shown in this guide pertain to Debian-based distributions (such as
Ubuntu), but the concepts remain the same regardless of your Linux distribution
of choice. Also note that the client and server do not have to be the same
distribution although they will be portrayed as the same in this guide.
To simplify this tutorial, the traffic between both systems won't be encrypted,
and everyone with access to the network will be able to read your logs.
Therefore, ensure to encrypt the traffic between the servers to prevent this
from happening in your production systems.
🔭 Want to centralize and monitor your Linux logs?
Head over to Logtail and start ingesting your logs in 5 minutes.
Step 1 — Finding the private IP address of the centralized server (optional)
First off, you need to find out the private IPv4 addresses of the host server if
you don't intend to transmit your logs over the internet. Otherwise, you can use
the public IP address of the server.
Go ahead and connect to the server through ssh. Replace the placeholders in
the command below as appropriate.
After connecting to the host server, you can determine its private IP address by
executing the hostname command as shown below:
You should get an output that looks like this:
The first value of the output (0.0.0.0) is the private IPv4 address of the
server. This may be the same as the public IP address on some systems.
You can now proceed to the next step where we'll verify that Rsyslog is
installed and enabled on both the client and server.
Step 2 — Installing and enabling the Rsyslog service
In this section, we will verify the status of the Rsyslog service on both the
client and host server. You can check if Rsyslog is installed on a system
through the command below:
You should get the following output if everything goes well:
rsyslogd 8.2001.0 (aka 2020.01) compiled with:
PLATFORM (lsb_release -d):
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /run/rsyslogd.pid
Number of Bits in RainerScript integers: 64
See https://www.rsyslog.com for more information.
Since Ubuntu already pre-installs rsyslog, you can go ahead and confirm its
active status through the command below:
systemctl status rsyslog
You should observe the following output:
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-02-07 12:41:41 UTC; 2 days ago
TriggeredBy: ● syslog.socket
Main PID: 638 (rsyslogd)
Tasks: 4 (limit: 2275)
└─638 /usr/sbin/rsyslogd -n -iNONE
If it's not active, go ahead and start the service through the command below:
sudo systemctl start rsyslog
Step 3 — Configuring the host server to receive logs
Now that we've confirmed that Rsyslog is installed and running on the host
server, go ahead and open its configuration file for editing using a text editor
such as nano:
sudo nano /etc/rsyslog.conf
The file has the following contents (truncated for brevity):
#### MODULES ####
module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
# provides TCP syslog reception
# provides kernel logging support and enable non-kernel klog messages
#### GLOBAL DIRECTIVES ####
. . .
The output above shows that the Rsyslog service is already listening at
Step 4 — Setting up the remote log storage location
By default, all logs received from TCP port 514 will be merged in the /var/log
directory with the system's log file. This is typically unwanted behavior
because it mixes all remote logs with the host server's local logs, making it
more difficult to search and filter later on. To prevent this behavior, you need
to edit the Rsyslog configuration.
Once again, open up the /etc/rsyslog.conf file with your text editor and add
the following lines at the beginning of the file:
The $template RemoteLogs directive instructs Rsyslog to store all incoming log
entries in the location that is defined by the third parameter. In our case, the
remote logs will continue to be stored in /var/log directory, but each client
will have its own subdirectory with a name equivalent to client hostname. This
subdirectory will store each log entry in a file that matches the client program
that generated it.
On the following line, the *.* ?RemoteLogs directive applies the RemoteLogs
configuration rule at all facilities with all priority levels (in other words,
to all logs). Finally, the & ~ directive defines that Rsyslog stops processing
log input after it is stored to a file defined in previous lines. The default
configuration will overwrite the previous rule without this line.
Save the file, then restart the rsyslog service with systemctl:
sudo systemctl restart rsyslog.service
At this point, your Rsyslog server is now fully configured to receive logs from
any number of remote clients. You should consider mounting the /var/log
directory in a separate partition from the one that the host system resides on
so that incoming logs do not fill up the storage of the host server.
Step 5 — Forwarding logs from an Rsyslog client
In this section, we will instruct our Rsyslog client to forward all its logs to
the central Rsyslog server. First off, you must connect to your client server
Once you're logged in successfully, edit the default Rsyslog configuration at
/etc/rsyslog.d/50-default.conf with a text editor (sudo is required for
sudo nano /etc/rsyslog.d/50-default.conf
At the *.* @@<your_rsyslog_server_ip_address>:514 directive on a new line at
the beginning of the file. Make sure to replace the placeholder with the actual
IPv4 address of the server that you retrieved in step 1.
# Default rules for rsyslog.
# For more information see rsyslog.conf(5) and /etc/rsyslog.conf
# First some standard log files. Log by facility.
The directive you just added above defines that the Rsyslog service should send
all facilities with all priority levels (in other words, all logs) to the IP
address (0.0.0.0 in the above example) of the centralized server at TCP
port 514. If you set up this directive using @ instead of @@, then it will
forward the logs to the UDP port.
Note that the *.* syntax determines that all log entries on the server should
be forwarded. If you want to forward only specific logs, you can specify the
service name instead of * such as cron.* @@0.0.0.0:514 or
apache2.* @@0.0.0.0:514. You can also forward logs to more than one server:
After saving your changes to the file, restart the rsyslog service to apply
the new configuration:
sudo systemctl restart rsyslog.service
At this point, your Rsyslog client is now fully configured to send its logs to
the centralized Rsyslog server.
Step 6 — Testing your configuration
Now that you've configured both the Rsyslog client and centralized server, let's
go ahead and verify that our changes have had the desired effect. Go ahead and
manually create a log entry on the client through the logger command:
logger 'test from client'
Afterward, confirm that the entry is present in the /var/log/syslog file
through the tail command:
sudo tail /var/log/syslog
You should see the following message near the end of the output:
Feb 9 22:17:42 ubuntu-20-04 root: test from client
Now, switch to the centralized Rsyslog server and view the contents of the
You should see a directory that corresponds to the hostname of the client
system. In this example, the hostname of the client server is ubuntu-20-04 and
you can observe a ubuntu-20-04 directory in the output below.
You should see the identical log entry observed on the client server:
2022-02-09T22:17:42+00:00 ubuntu-20-04 root: test from client
At this point, it is clear that your Rsyslog client-server configuration is
working as expected.
Conclusion and next steps
In this tutorial, you created a centralized logging service with Rsyslog. We
started by discussing the value that a centralized logging service provides,
then set up the host Rsyslog server that will store all the log entries of
individual clients. Afterward, we configured a client server to forward its logs
to the central server and tested our configuration to confirm that it was
You're now all set to forward logs to the centralized server from as many
clients as you want. To guarantee that the log entries from each client is
stored in its own directory, ensure that each client's hostname is unique. Also
don't forget to employ Log
rotation on the host
server to prevent the log files from growing too large and to automatically
delete the logs that are older than a certain number of days or weeks.