By reading through this article, you will learn how to do the following:
- Configure a centralized Rsyslog server that will receive all log messages.
- Configure Rsyslog on a separate server (the client) to forward all its logs to a centralized server.
Contents
- Prerequisites
- Step 1 — Finding the private IP address of the Rsyslog server (optional)
- Step 2 — Installing and enabling the Rsyslog service
- Step 3 — Configuring the host server to receive logs
- Step 4 — Setting up the remote log storage location
- Step 5 — Forwarding logs from an Rsyslog client
- Step 6 — Testing your configuration
- Conclusion and next steps
This tutorial describes how you can configure Linux as a centralized logging service with Rsyslog. Such a centralized Rsyslog configuration is beneficial when administrating logs from multiple servers. The host server will receive log entries from all client servers so that they can be monitored and archived in one place. This approach makes system administration much easier compared to sshing into each server to review its logs, particularly if there's a huge number of servers.
Prerequisites
To follow through with this tutorial, you must have access to two Linux systems.
One of them will act the centralized Rsyslog server, while the other acts as the
Rsyslog client. A non-root user with sudo access is required on both systems.
The traffic between both systems won't be encrypted, and everyone with access to the network will be able to read your logs. However, if your servers are on a private network, then you encapsulate this traffic from the rest of the internet.
The commands shown in this guide pertain to Debian-based distributions (such as Ubuntu), but the concepts remain the same regardless of your Linux distribution of choice. Also note that the client and server do not have to be the same distribution although they will be portrayed as the same in this guide.
Step 1 — Finding the private IP address of the Rsyslog server (optional)
First off, you need to find out the private IPv4 addresses of the host server if you don't intend to transmit your logs over the internet. Otherwise, you can use the public IP address of the server.
Go ahead and connect to the server through ssh. Replace the placeholders in
the command below as appropriate.
ssh <username>@<your_server_public_ip_address>
After connecting to the host server, you can determine its private IP address by
executing the hostname command as shown below:
hostname -I
You should get an output that looks like this:
Output
0.0.0.0 2a01:4f8:c010:5f07::1
The first value of the output (0.0.0.0) is the private IPv4 address of the
server. This may be the same as the public IP address on some systems.
You can now proceed to the next step where we'll verify that Rsyslog is installed and enabled on both the client and server.
Step 2 — Installing and enabling the Rsyslog service
In this section, we will verify the status of the Rsyslog service on both the client and host server. You can check if Rsyslog is installed on a system through the command below:
rsyslogd -v
You should get the following output if everything goes well:
Output
rsyslogd 8.2001.0 (aka 2020.01) compiled with:
PLATFORM: x86_64-pc-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /run/rsyslogd.pid
Number of Bits in RainerScript integers: 64
See https://www.rsyslog.com for more information.
Since Ubuntu already pre-installs rsyslog, you can go ahead and confirm its
active status through the command below:
systemctl status rsyslog
You should observe the following output:
Output
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2022-02-07 12:41:41 UTC; 2 days ago
TriggeredBy: ● syslog.socket
Docs: man:rsyslogd(8)
https://www.rsyslog.com/doc/
Main PID: 638 (rsyslogd)
Tasks: 4 (limit: 2275)
Memory: 180.4M
CGroup: /system.slice/rsyslog.service
└─638 /usr/sbin/rsyslogd -n -iNONE
If it's not active, go ahead and start the service through the command below:
sudo systemctl start rsyslog
Step 3 — Configuring the host server to receive logs
Now that we've confirmed that Rsyslog is installed and running on the host
server, go ahead and open its configuration file for editing using a text editor
such as nano:
sudo nano /etc/rsyslog.conf
The file has the following contents (truncated for brevity):
/etc/rsyslog.conf
#################
#### MODULES ####
#################
module(load="imuxsock") # provides support for local system logging
#module(load="immark") # provides --MARK-- message capability
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
# provides kernel logging support and enable non-kernel klog messages
module(load="imklog" permitnonkernelfacility="on")
###########################
#### GLOBAL DIRECTIVES ####
###########################
. . .
Find the following lines that are commented out:
/etc/rsyslog.conf
# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")
# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")
These lines load the imudp and imtcp modules for listening at the specified
UDP or TCP ports (both at the port 514). You can listen to both ports if you
wish or change the port number as you see fit.
For this tutorial, we will use a TCP connection alone. Enable it by uncommenting
the following two lines of code (remove the # prefix):
/etc/rsyslog.conf
module(load="imtcp")
input(type="imtcp" port="514")
Save changes in file /etc/rsyslog.conf and exit the editor.
If your system includes a firewall configuration, you must also allow the new
TCP service at the specified port with ufw. It requires sudo because it
changes the firewall rules.
sudo ufw allow 514/tcp
After running the command above, the firewall won't interfere with Rsyslog
traffic. Go ahead and restart the rsyslog service to apply the new
configuration:
sudo systemctl restart rsyslog.service
Finally, you can check that the TCP port 514 is opened with the ss utility.
sudo is required because this utility reads sockets:
sudo ss -tulnp | grep "rsyslog"
Here's a brief explanation of the ss parameters utilized above:
-u: display UDP sockets.-t: display TCP sockets.-l: display only listening sockets.-n: do not convert port number into human-readable service name.-p: display process that uses the socket.
The output of ss is filtered by grep, which prints only lines containing
rsyslog:
Output
tcp LISTEN 0 25 0.0.0.0:514 0.0.0.0:* users:(("rsyslogd",pid=440497,fd=5))
tcp LISTEN 0 25 [::]:514 [::]:* users:(("rsyslogd",pid=440497,fd=6))
The output above shows that the Rsyslog service is already listening at port 514.
Step 4 — Setting up the remote log storage location
By default, all logs received from TCP port 514 will be merged in the /var/log
directory with the system's log file. This is typically unwanted behavior
because it mixes all remote logs with the host server's local logs, making it
more difficult to search and filter later on. To prevent this behavior, you need
to edit the Rsyslog configuration.
Once again, open up the /etc/rsyslog.conf file with your text editor and add
the following lines at the beginning of the file:
/etc/rsyslog.conf
$template RemoteLogs,"/var/log/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
& ~
The $template RemoteLogs directive instructs Rsyslog to store all incoming log
entries in the location that is defined by the third parameter. In our case, the
remote logs will continue to be stored in /var/log directory, but each client
will have its own subdirectory with a name equivalent to client hostname. This
subdirectory will store each log entry in a file that matches the client program
that generated it.
On the following line, the *.* ?RemoteLogs directive applies the RemoteLogs
configuration rule at all facilities with all priority levels (in other words,
to all logs). Finally, the & ~ directive defines that Rsyslog stops processing
log input after it is stored to a file defined in previous lines. The default
configuration will overwrite the previous rule without this line.
Save the file, then restart the rsyslog service with systemctl:
sudo systemctl restart rsyslog.service
At this point, your Rsyslog server is now fully configured to receive logs from
any number of remote clients. You should consider mounting the /var/log
directory in a separate partition from the one that the host system resides on
so that incoming logs do not fill up the storage of the host server.
Step 5 — Forwarding logs from an Rsyslog client
In this section, we will instruct our Rsyslog client to forward all its logs to
the central Rsyslog server. First off, you must connect to your client server
through ssh:
ssh <username>@<your_server_ip_address>
Once you're logged in successfully, edit the default Rsyslog configuration at
/etc/rsyslog.d/50-default.conf with a text editor (sudo is required for
write permission):
sudo nano /etc/rsyslog.d/50-default.conf
At the *.* @@<your_rsyslog_server_ip_address>:514 directive on a new line at
the beginning of the file. Make sure to replace the placeholder with the actual
IPv4 address of the server that you retrieved in step 1.
/etc/rsyslog.d/50-default.conf
# Default rules for rsyslog.
#
# For more information see rsyslog.conf(5) and /etc/rsyslog.conf
*.* @@0.0.0.0:514
#
# First some standard log files. Log by facility.
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
#cron.* /var/log/cron.log
#daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
#lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
#user.* -/var/log/user.log
The directive you just added above defines that the Rsyslog service should send
all facilities with all priority levels (in other words, all logs) to the IP
address (0.0.0.0 in the above example) of the centralized server at TCP
port 514. If you set up this directive using @ instead of @@, then it will
forward the logs to the UDP port.
Note that the *.* syntax determines that all log entries on the server should
be forwarded. If you want to forward only specific logs, you can specify the
service name instead of * such as cron.* @@0.0.0.0:514 or
apache2.* @@0.0.0.0:514. You can also forward logs to more than one server:
/etc/rsyslog.d/50-default.conf
*.* @@0.0.0.0:514
*.* @@192.168.122.235
cron.* @@192.168.122.237:514
After saving your changes to the file, restart the rsyslog service to apply
the new configuration:
sudo systemctl restart rsyslog.service
At this point, your Rsyslog client is now fully configured to send its logs to the centralized Rsyslog server.
Step 6 — Testing your configuration
Now that you've configured both the Rsyslog client and centralized server, let's
go ahead and verify that our changes have had the desired effect. Go ahead and
manually create a log entry on the client through the logger command:
logger 'test from client'
Afterward, confirm that the entry is present in the /var/log/syslog file
through the tail command:
sudo tail /var/log/syslog
You should see the following message near the end of the output:
Output
Feb 9 22:17:42 ubuntu-20-04 root: test from client
Now, switch to the centralized Rsyslog server and view the contents of the
/var/log directory:
ls /var/log
You should see a directory that corresponds to the hostname of the client
system. In this example, the hostname of the client server is ubuntu-20-04 and
you can observe a ubuntu-20-04 directory in the output below.
Output
alternatives.log btmp dpkg.log lastlog ubuntu-2gb-nbg1-1/ wtmp
apache2/ cloud-init.log faillog private/ ubuntu-advantage.log
apt/ cloud-init-output.log journal/ syslog ubuntu-advantage-timer.log
auth.log dist-upgrade/ kern.log syslog.1 ufw.log
bootstrap.log dmesg landscape/ ubuntu-20-04/ unattended-upgrades/
Examine the last 10 lines of the root.log file in this subdirectory:
sudo tail /var/log/<your_client_hostname>/root.log
You should see the identical log entry observed on the client server:
Output
2022-02-09T22:17:42+00:00 ubuntu-20-04 root: test from client
At this point, it is clear that your Rsyslog client-server configuration is working as expected.
Conclusion and next steps
In this tutorial, you created a centralized logging service with Rsyslog. We started by discussing the value that a centralized logging service provides, then set up the host Rsyslog server that will store all the log entries of individual clients. Afterward, we configured a client server to forward its logs to the central server and tested our configuration to confirm that it was working correctly.
You're now all set to forward logs to the centralized server from as many clients as you want. To guarantee that the log entries from each client is stored in its own directory, ensure that each client's hostname is unique. Also don't forget to employ Log rotation on the host server to prevent the log files from growing too large and to automatically delete the logs that are older than a certain number of days or weeks.
Thanks for reading, and happy logging!
Centralize all your logs into one place.
Analyze, correlate and filter logs with SQL.
Create actionable
dashboards with Grafana.
dashboards with Grafana.
Share and comment with built-in collaboration.
Got an article suggestion?
Let us know
Next article
How to View and Configure Apache Access & Error Logs
Learn how to view and configure Apache access and error logs.
→
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Centralize all your logs into one place.
Analyze, correlate and filter logs with SQL.
Create actionable
dashboards with Grafana.
dashboards with Grafana.
Share and comment with built-in collaboration.