rsyslog Client Forwarder on Raspberry Pi

What is syslog

syslog is a syslog over TCP (new) and syslog over UDP (old) standard for message logging. It allows you to separate the software that generates messages, the system that stores them, and the software that reports and analyzes them. syslog is commonly used in enterprise environments to track logins, errors, and all types of sensitive information. Normally syslog is local, and setup to dump to the /var partition to avoid filling up a small hard drive with log junk. This tutorial will show you how to setup syslog on your raspberry pi to send information to another syslog server (hopefully also with enough disk space and on /var).

Splunk and syslog

Splunk is a free event correlation tool (free for some usage). It allows you index and make sense of huge volumes of messages, alerts, etc. Syslog is one the formats it accepts. My splunk server is running on 192.168.1.14 and I am going to add events from my raspberry pi so it shows up in the splunk dashboard. The first step is enabling syslog forwarding on the Pi. The Pi is already logging to it’s own internal syslog file, but I want to look at all of my computers at once. In order to do this, I have to forward syslog info to the splunk server. I reference splunk often throughout the tutorial because that is my syslog correlator (you are free to use other correlators though).

Syslog on Windows vs Linux

sylog is native on linux and can usually be installed easily. Windows uses event logs, and is not compatible with syslog. However, you can use tools like [PRTG](http://www.kiwisyslog.com/>Kiwi or <a href=) to forward event logs over syslog to a remote server for correlations. We aren’t going to cover installing syslog on windows, but the software just mentioned can do that for you. Instead, let’s talk about installing syslog on ubuntu, specifically on a Raspberry Pi.

What Should You Monitor?

This list is not complete, but it is a great start of the various things worth monitoring:

You want to log/monitor things related to user issues, hardware, security. Syslog can do all of this.

syslog Marks

In case there are no events to report, you will see – MARK – to let you know the service is still running. Don’t be alarmed.

    Oct 22 08:31:22 someuser -- MARK --
    Oct 22 08:51:22 someuser -- MARK --

Possible Splunk Diagram

In the rest of this tutorial, we will use the above diagram as a reference and setup syslog to forward from 192.168.1.12 to 192.168.1.14

Remove rsyslog

In the event you need to uninstall rsyslog, the command is very simple. If you want to remove configuration files for rsyslog, type:

pi@raspberrypi:~ $ sudo apt-get purge rsyslog

In case you don’t want to touch your old config files, but want to uninstall the binaries for rsyslog:

pi@raspberrypi:~ $ sudo apt-get remove rsyslog

Install rsyslog Raspberry Pi

    pi@raspberrypi:~ $ sudo apt-get install rsyslog
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    Suggested packages:
      rsyslog-mysql | rsyslog-pgsql rsyslog-mongodb rsyslog-doc rsyslog-gnutls rsyslog-gssapi rsyslog-relp
    The following NEW packages will be installed:
      rsyslog
    0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
    Need to get 606 kB of archives.
    After this operation, 1,502 kB of additional disk space will be used.
    Get:1 http://raspbian.mirrors.lucidnetworks.net/raspbian buster/main armhf rsyslog armhf 8.1901.0-1 [606 kB]
    Fetched 606 kB in 1s (822 kB/s)
    Selecting previously unselected package rsyslog.
    (Reading database ... 155890 files and directories currently installed.)
    Preparing to unpack .../rsyslog_8.1901.0-1_armhf.deb ...
    Unpacking rsyslog (8.1901.0-1) ...
    Setting up rsyslog (8.1901.0-1) ...
    Created symlink /etc/systemd/system/syslog.service → /lib/systemd/system/rsyslog.service.
    Created symlink /etc/systemd/system/multi-user.target.wants/rsyslog.service → /lib/systemd/system/rsyslog.service.
    Processing triggers for man-db (2.8.5-2) ...
    Processing triggers for systemd (241-7~deb10u4+rpi1) ...

Edit rsyslog.conf

The syslog configuration files are located at /etc/rsyslog.conf. You will edit this file on the client for the client portion and on the server for the server portions. You can use any editor you like to edit your syslog.conf file, but I use vim. Other options are: nano, pico, vi. vim has a pretty steep learning curve, but is also one of the most powerful text editors and is usually available on any version of linux you touch. It’s worth learning vim!

pi@raspberrypi:~ $ sudo vim /etc/rsyslog.conf

You will want to replace the 192.168.1.14 with the IP that matches your syslog server. Your client will send info to that server.

pi@raspberrypi:~ $ head /etc/rsyslog.conf
# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
# @@ = tcp, @ = udp
*.* @@192.168.1.14:514

Restart syslog Service

As sudo, restart the syslog process.

pi@raspberrypi:~ $ sudo service rsyslog restart

Verify rsyslog is Running

You can verify that rsyslog is running by looking at the status:

pi@raspberrypi:~ $ sudo service rsyslog status
● rsyslog.service - System Logging Service
   Loaded: loaded (/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-10-22 09:41:44 CDT; 3s ago
     Docs: man:rsyslogd(8)
           https://www.rsyslog.com/doc/
 Main PID: 7256 (rsyslogd)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/rsyslog.service
           └─7256 /usr/sbin/rsyslogd -n -iNONE

Oct 22 09:41:44 raspberrypi systemd[1]: Starting System Logging Service...
Oct 22 09:41:44 raspberrypi rsyslogd[7256]: imuxsock: Acquired UNIX socket '/run/systemd/journal/syslog' (fd 3)
Oct 22 09:41:44 raspberrypi rsyslogd[7256]:  [origin software="rsyslogd" swVersion="8.1901.0" x-pid="7256" x-inf
Oct 22 09:41:44 raspberrypi systemd[1]: Started System Logging Service.

View Packets of syslog to Verify

You can verify that your client (192.168.1.12 in my example) is sending packets to the server (192.168.1.14 in my example), by using tcpdump.

If you need to install tcpdump on the raspberry pi (or any debian based flavor), type:

pi@raspberrypi:~ $ sudo apt install tcpdump

After it has been installed, you can view the syslog traffic:

pi@raspberrypi:~ $ sudo tcpdump -nnei any port 514
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
09:43:20.749639 Out dc:a6:32:35:cd:fa ethertype IPv4 (0x0800), length 171: 192.168.1.12.42318 > 192.168.1.14.514: Flags [P.], seq 179828733:179828836, ack 4187142578, win 502, options [nop,nop,TS val 2106919937 ecr 2138505283], length 103
09:43:20.749741  In 00:26:b9:ea:ee:7f ethertype IPv4 (0x0800), length 68: 192.168.1.14.514 > 192.168.1.12.42318: Flags [.], ack 103, win 501, options [nop,nop,TS val 2138545395 ecr 2106919937], length 0
09:43:20.757510 Out dc:a6:32:35:cd:fa ethertype IPv4 (0x0800), length 174: 192.168.1.12.42318 > 192.168.1.14.514: Flags [P.], seq 103:209, ack 1, win 502, options [nop,nop,TS val 2106919945 ecr 2138545395], length 106

Manually Trigger a Log Event

Linux has a tool called “logger” that lets you dump information into the syslog (useful for testing log functionality).

pi@raspberrypi:~ $ logger -p daemon.emerg "DANGER WILL ROBINSON!!!"

This sent a syslog event to the remote server instead of to my logs on the Pi because I have already configured the Pi to send to the remote server!

Configure rsyslog Server

Of course, unless your server is listening for syslog, it won’t matter that your client is sending syslog info. If you install splunk, it will have an option to listen for TCP 514, and therefore will be able to collect and index your syslog traffic from all of the clients that it info using the splunk daemons.

Considerations of Syslog

Sensitive Info

Syslog usually has information that could be considered sensitive and UDP syslog is not encrypted. Upgrade to syslog-ng and use tcp with tls or use syslog on a network that you 100% trust. Encrypted syslog messages are recommended.

Centralized Collection

For syslog in general, it’s considered best practice to consolidate all events into a single, hardened host. A syslog server will be a prime target to be attacked. It is advisable to harden it thoroughly, maintain patching schedule and not run other services on the same box.

Use NTP!

Synchronize all servers to the same time zone so you can easily track event times. Verify your NTP status on all machines (if debian), like so:

systemctl status systemd-timesyncd
timedatectl status

Filling Up Partitions

You might have a small partition that could easily fill up with volumes of syslog data. It’s best to use a partition dedicated to log files, like /var that won’t crash your root file system in the event it completely fills up. A backup and export plan using the logrotate program is recommended.

Missing Information

If you configure syslog to go directly to another machine, without logging, you might miss log info if the network or server is down. Configure syslog TCP for buffering at least and try to write to a file then forward file information into a correlation tool, like splunk. Splunk has a listener daemon, but if you are sending UDP traffic, you might miss important events and log info. We configured rsyslog above to send tcp (@@), and we are listening syslog on tcp in splunk.

Baseline Patterns

Using a tool like splunk, you can detect a baseline and deviations from that baseline. If you aren’t reviewing your logs regularly, they are useless.

Faster Alerts

Many syslog messages are great for audits, but configure your correlation software to alert you on the high priority messages so you can take action.

Prevent DDOS of syslog

syslog can be overwhelmed. You can configure your server to only listen to specific hosts and you can also break up logs by facility (daemon, mail, auth, etc) to help sort through noise. Prevent users without reason from using the logger script so they cannot flood the syslog.