Setup rsyslog Client Forwarder on Raspberry Pi
What is syslog
syslog is a syslog over TCP (new) and syslog over UDP (old) standard for message logging. It allows you to separate the software that generates messages, the system that stores them, and the software that reports and analyzes them. syslog is commonly used in enterprise environments to track logins, errors, and all types of sensitive information. Normally syslog is local, and setup to dump to the /var partition to avoid filling up a small hard drive with log junk. This tutorial will show you how to setup syslog on your raspberry pi to send information to another syslog server (hopefully also with enough disk space and on /var).
Splunk and syslog
Splunk is a free event correlation tool (free for some usage). It allows you index and make sense of huge volumes of messages, alerts, etc. Syslog is one the formats it accepts. My splunk server is running on 192.168.1.14 and I am going to add events from my raspberry pi so it shows up in the splunk dashboard. The first step is enabling syslog forwarding on the Pi. The Pi is already logging to it’s own internal syslog file, but I want to look at all of my computers at once. In order to do this, I have to forward syslog info to the splunk server. I reference splunk often throughout the tutorial because that is my syslog correlator (you are free to use other correlators though).
Syslog on Windows vs Linux
sylog is native on linux and can usually be installed easily. Windows uses event logs, and is not compatible with syslog. However, you can use tools like [PRTG](http://www.kiwisyslog.com/>Kiwi or <a href=) to forward event logs over syslog to a remote server for correlations. We aren’t going to cover installing syslog on windows, but the software just mentioned can do that for you. Instead, let’s talk about installing syslog on ubuntu, specifically on a Raspberry Pi.
What Should You Monitor?
This list is not complete, but it is a great start of the various things worth monitoring:
/var/log/messages
Contains most system messages
/var/log/secure
Authentication messages
/var/log/cron
Logs Cron job activities
/var/log/maillog
Mail transactions
You want to log/monitor things related to user issues, hardware, security. Syslog can do all of this.
syslog Marks
In case there are no events to report, you will see – MARK – to let you know the service is still running. Don’t be alarmed.
Oct 22 08:31:22 someuser -- MARK --
Oct 22 08:51:22 someuser -- MARK --
Possible Splunk Diagram
In the rest of this tutorial, we will use the above diagram as a reference and setup syslog to forward from 192.168.1.12 to 192.168.1.14
Remove rsyslog
In the event you need to uninstall rsyslog, the command is very simple. If you want to remove configuration files for rsyslog, type:
pi@raspberrypi:~ $ sudo apt-get purge rsyslog
In case you don’t want to touch your old config files, but want to uninstall the binaries for rsyslog:
pi@raspberrypi:~ $ sudo apt-get purge rsyslog
Install rsyslog Raspberry Pi
pi@raspberrypi:~ $ sudo apt-get install rsyslog
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
rsyslog-mysql | rsyslog-pgsql rsyslog-mongodb rsyslog-doc rsyslog-gnutls rsyslog-gssapi rsyslog-relp
The following NEW packages will be installed:
rsyslog
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 606 kB of archives.
After this operation, 1,502 kB of additional disk space will be used.
Get:1 http://raspbian.mirrors.lucidnetworks.net/raspbian buster/main armhf rsyslog armhf 8.1901.0-1 [606 kB]
Fetched 606 kB in 1s (822 kB/s)
Selecting previously unselected package rsyslog.
(Reading database ... 155890 files and directories currently installed.)
Preparing to unpack .../rsyslog_8.1901.0-1_armhf.deb ...
Unpacking rsyslog (8.1901.0-1) ...
Setting up rsyslog (8.1901.0-1) ...
Created symlink /etc/systemd/system/syslog.service → /lib/systemd/system/rsyslog.service.
Created symlink /etc/systemd/system/multi-user.target.wants/rsyslog.service → /lib/systemd/system/rsyslog.service.
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for systemd (241-7~deb10u4+rpi1) ...
Edit rsyslog.conf
The syslog configuration files are located at /etc/rsyslog.conf. You will edit this file on the client for the client portion and on the server for the server portions. You can use any editor you like to edit your syslog.conf file, but I use vim. Other options are: nano, pico, vi. vim has a pretty steep learning curve, but is also one of the most powerful text editors and is usually available on any version of linux you touch. It’s worth learning vim!
pi@raspberrypi:~ $ sudo vim /etc/rsyslog.conf
You will want to replace the 192.168.1.14 with the IP that matches your syslog server. Your client will send info to that server.
pi@raspberrypi:~ $ head /etc/rsyslog.conf
# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
*.* @@192.168.1.14:514
Restart syslog Service
As sudo, restart the syslog process.
pi@raspberrypi:~ $ sudo service rsyslog restart
Verify rsyslog is Running
You can verify that rsyslog is running by looking at the status:
pi@raspberrypi:~ $ sudo service rsyslog status
● rsyslog.service - System Logging Service
Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-10-22 09:41:44 CDT; 3s ago
Docs: man:rsyslogd(8)
https://www.rsyslog.com/doc/
Main PID: 7256 (rsyslogd)
Tasks: 4 (limit: 4915)
CGroup: /system.slice/rsyslog.service
└─7256 /usr/sbin/rsyslogd -n -iNONE
Oct 22 09:41:44 raspberrypi systemd[1]: Starting System Logging Service...
Oct 22 09:41:44 raspberrypi rsyslogd[7256]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3)
Oct 22 09:41:44 raspberrypi rsyslogd[7256]: [origin software="rsyslogd" swVersion="8.1901.0" x-pid="7256" x-inf
Oct 22 09:41:44 raspberrypi systemd[1]: Started System Logging Service.
View Packets of syslog to Verify
You can verify that your client (192.168.1.12 in my example) is sending packets to the server (192.168.1.14 in my example), by using tcpdump.
If you need to install tcpdump on the raspberry pi (or any debian based flavor), type:
pi@raspberrypi:~ $ sudo apt install tcpdump
After it has been installed, you can view the syslog traffic:
pi@raspberrypi:~ $ sudo tcpdump -nnei any port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
09:43:20.749639 Out dc:a6:32:35:cd:fa ethertype IPv4 (0x0800), length 171: 192.168.1.12.42318 > 192.168.1.14.514: Flags [P.], seq 179828733:179828836, ack 4187142578, win 502, options [nop,nop,TS val 2106919937 ecr 2138505283], length 103
09:43:20.749741 In 00:26:b9:ea:ee:7f ethertype IPv4 (0x0800), length 68: 192.168.1.14.514 > 192.168.1.12.42318: Flags [.], ack 103, win 501, options [nop,nop,TS val 2138545395 ecr 2106919937], length 0
09:43:20.757510 Out dc:a6:32:35:cd:fa ethertype IPv4 (0x0800), length 174: 192.168.1.12.42318 > 192.168.1.14.514: Flags [P.], seq 103:209, ack 1, win 502, options [nop,nop,TS val 2106919945 ecr 2138545395], length 106
Manually Trigger a Log Event
Linux has a tool called “logger” that lets you dump information into the syslog (useful for testing log functionality).
pi@raspberrypi:~ $ logger -p daemon.emerg "DANGER WILL ROBINSON!!!"
This sent a syslog event to the remote server instead of to my logs on the Pi because I have already configured the Pi to send to the remote server!
Configure rsyslog Server
Of course, unless your server is listening for syslog, it won’t matter that your client is sending syslog info. If you install splunk, it will have an option to listen for TCP 514, and therefore will be able to collect and index your syslog traffic from all of the clients that it info using the splunk daemons. You can also configure the same /etc/rsyslog.conf file to listen (more info to come in the next article).
Considerations of Syslog
Sensitive Info
Syslog usually has information that could be considered sensitive and UDP syslog is not encrypted. Upgrade to syslog-ng and use tcp with tls or use syslog on a network that you 100% trust. Encrypted syslog messages are recommended.
Centralized Collection
For syslog in general, it's considered best practice to consolidate all events into a single, hardened host. A syslog server will be a prime target to be attacked. It is advisable to harden it thoroughly, maintain patching schedule and not run other services on the same box.
Use NTP!
Synchronize all servers to the same time zone so you can easily track event times.
Filling Up Partitions
You might have a small partition that could easily fill up with volumes of syslog data. It's best to use a partition dedicated to log files, like /var that won't crash your root file system in the event it completely fills up. A backup and export plan using the logrotate program is recommended.
Missing Information
If you configure syslog to go directly to another machine, without logging, you might miss log info if the network or server is down. Configure syslog TCP for buffering at least and try to write to a file then forward file information into a correlation tool, like splunk. Splunk has a listener daemon, but if you are sending UDP traffic, you might miss important events and log info.
Baseline Patterns
Using a tool like splunk, you can detect a baseline and deviations from that baseline. If you aren't reviewing your logs regularly, they are useless.
Faster Alerts
Many syslog messages are great for audits, but configure your correlation software to alert you on the high priority messages so you can take action.
Prevent DDOS of syslog
syslog can be DOS'd. You can configure your server to only listen to specific hosts and you can also break up logs by facility (daemon, mail, auth, etc) to help sort through noise. Prevent users without reason from using the logger script so they cannot flood the syslog.