by: Linux/Unix Raspberry Pi

INDEX

  • Journal Control
    • Used to view and manipulate Systemd logs.
  • Remote Syslog
    • Forwards log messages over an IP network.
  • Log Rotating
    • Allows automatic rotating, compressing, removing, and messaging of log files.
  • Log Watch
    • Periodically goes through your logs for a given period and makes a report via email or output file.
  • Log Check
    • Helps to spot problems and security violations in your log files automatically and sends the results via email.
  • Audit Daemon
    • Is a powerful audit framework to log events on Linux systems.

SYSLOG PRIORITY/SEVERITY LEVELS

  • 0 – Emergency
  • 1 – Alert
  • 2 – Critical
  • 3 – Error
  • 4 – Warning – Default
  • 5 – Notifications
  • 6 – Information
  • 7 – Debug

JOURNAL CONTROL

JournalCTL (aka Journal Control) is used to query the contents of the systemd journal (systemd-journald.service).

Note: by default, Journal users can only watch their own logs, unless they are root or in the adm (or systemd-journal) group (usermod -a -G adm userName).

  • journalctl
    • shows the full content of the journal, starting with the oldest entry collected.
  • journalctl -r -n 10
    • shows the content of the journal in reverse order, limited to 10 entries. The newest at the top.
  • sudo journalctl -u ssh.service -f
    • filters and outputs the logs for the unit: ssh, and follows for real-time entries.
  • sudo journalctl –since “2021-12-31 23:59:59”
    • filters the entries by a starting date and time.
  • sudo journalctl –since “yesterday” -p err
    • filters the entries by a relative time window with the priority:
      • since and/or until can contain strings like “today”, “1 hour ago”, “2 days ago”…
      • the priorities are directly related to the syslog levels and can be expressed in text or numerically (“emerg”/0, “alert”/1, “crit”/2, “err”/3, “warning”/4, “notice”/5, “info”/6, “debug”/7).
  • sudo journalctl SYSLOG_IDENTIFIER=sshd -S “13:30:00” -U “14:00:00”
    • filters the entries by an identifier, since, and until time.
  • sudo journalctl -k
    • shows kernel messages.
  • sudo journalctl -b
    • Shows the current boot messages.
  • sudo journalctl -b -2 –utc
    • Shows the messages from the second previous boot represented in UTC timezone.
  • sudo journalctl –list-boots
    • lists the available boot messages with ordinal number time stamping.

Journal Clean Up

sudo journalctl --verify
sudo journalctl --disk-usage
sudo journalctl --vacuum-size=200M
sudo journalctl --vacuum-files=2
sudo journalctl --vacuum-time=3d
sudo journalctl --vacuum-size=1G --vacuum-time=5d --vacuum-files=5

Customizing Configuration

sudo nano /etc/systemd/journald.conf
  • Storage=
    • One of “volatile”, “persistent”, “auto”, and “none”.
  • Compress=
    • Suffixes like K, M, and G can be used to specify larger units.
  • SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, SystemMaxFiles=, RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize=, RuntimeMaxFiles=
    • Use the variables above to define limits to disk utilization in case the Journal starts to affect or jeopardize the stability of the system.
  • MaxFileSec=, MaxRetentionSec=, SyncIntervalSec=
    • Define additional retention and synchronization parameters with the variables above.
  • ForwardToSyslog=, ForwardToKMsg=, ForwardToConsole=, ForwardToWall=, MaxLevelStore=, MaxLevelSyslog=, MaxLevelKMsg=, MaxLevelConsole=, MaxLevelWall=
    • Limit and define what and how to forward messages.

Finding Failures

journalctl --no-pager --since today --grep 'fail|error|fatal' --output json|jq '._EXE' | sort | uniq -c | sort --numeric --reverse --key 1

REMOTE SYSLOG

Rsyslog (aka “the Rocket-fast SYStem for LOG processing”) uses the standard BSD syslog protocol to send messages over the IP protocol, but supports many others via extensions. The format of relayed messages can be customized.

Locally, the rsyslog is responsible for reading the Journal and sorting the entries based on filters to appropriately store/output the information accordingly because the Journal is ephemeral and disappears after the reboot.

Server-Side

Edit the configuration file:

sudo nano /etc/rsyslog.conf

Uncomment the following lines to start listening over UDP (faster – recommended):

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

Or uncomment the following lines to start listening over TDP (more reliable):

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

Right below the uncommented lines, add:

$template RemoteLogs,"/var/log/RemoteHosts/%HOSTNAME%-%$now%.log"
if $fromhost-ip != '127.0.0.1' then -?RemoteLogs
& stop

Note: it will create a template called RemoteLogs that will store the logs based on the hostname of the client. This template will not be applied to localhost and will stop to prevent double logs. Another variable that is popularly used it %PROGRAMNAME%.

Restart the service, check if it’s listening on port 514, and start monitoring for incoming messages:

sudo systemctl restart rsyslog
sudo ss -tulpn
sudo tail -f /var/log/messages

Client-Side

Edit the configuration file:

sudo nano /etc/rsyslog.conf

Append the following lines:

*.* action(type="omfwd" target="10.10.10.10" port="514" protocol="udp")

OR

*.* action(type="omfwd" target="10.10.10.10" port="514" protocol="tcp")

Also, consider flags for mode reliability.

*.* action(type="omfwd" target="10.10.10.10" port="514" protocol="tcp" tcp_framing="octet-counted" keepalive="on")

If you prefer, the old-fashioned syntax is:

For UDP:

*.* @10.10.10.10:514

For TCP:

*.* @@10.10.10.10:514

Note: the expression *.* means AllFacilities and AllSeverityLevels, respectively. Replace the IP address with the IP or Hostname of the server.

Restart the service and test it:

sudo systemctl restart rsyslog
logger -t "TagName" Value
logger -t "LocalTime" $(date)

LOG ROTATING

Log rotation is an automated process in which log files are compressed, moved (archived), renamed, or deleted once they are too old or too big.

The files /etc/logrotate.conf contains the default configuration that applies to all log files unless there is a specific configuration that overwrites it.

Each specific configuration file must be located at /etc/logrotate.d. See the example that can be applied to the example of the rsyslog server above:

/var/log/RemoteHosts/*.log {
  weekly 
  rotate 26
  copytruncate
  compress
  delaycompress
  missingok
  notifempty
  create 600 root root
  maxsize 500M
  minsize 50M
}
  • weekly
    • Rotates file once a week. It could also be daily, monthly, or early.
  • rotate 26
    • Keeps 26 files.
  • copytruncate
    • Instructs to create a copy of the original file and truncates the original file to zero byte size.
  • compress
    • Use gzip compression to save disk space.
  • delaycompress
    • Do not compress the first rotation, but all the others.
  • missingok
    • Ignore if one file is missing.
  • notifempty
    • Do not rotate if the file is empty.
  • create 600 root root
    • Apply the desired permission to the created files.
  • maxsize 500M
    • Rotate immediately if the file gets bigger than specified. Possible to use K for kilobyte, M for megabyte, and G for gigabit.
  • minsize 50M
    • Do not rotate if the file is too small. Possible to use K for kilobyte, M for megabyte, and G for gigabit.

Additionally:

  • sharedscripts
    • This option will execute the prerotate and the postrotate scripts only once, even if there is more than one file to rotate, instead of executing for each file being rotated.
  • prerotate
    • The content of this option has to be written in bash and execute any needed procedure prior to the file being rotated.
    • Has to end with endscript.
  • postrotate
    • The content of this option has to be written in bash and execute any needed procedure after the file is rotated.
    • Has to end with endscript.

Note:

Logrotate is executed daily by the cron via the script /etc/cron.daily/logrotate and also via timers set on the System Control with the configuration /usr/lib/systemd/system/logrotate.timer and /usr/lib/systemd/system/logrotate.service (systemctl list-timers).

The reason why there is a combination of cron and systemd managing the execution of the logrotate is to allow the execution of the rotation to occur within a range of acceptable times that will look for the best time to reduce resource competition based on the load of the system, for example.

In case it is needed to execute the logrotate manually or at a frequency higher than daily, for example, every 6 hours (depending on the needs of the system and the amount of data stored), use the -f or -force.

/usr/sbin/logrotate -f /etc/logrotate.conf

Check the health of its timer.

systemctl status logrotate.timer

LOGWATCH

Logwatch is a log parser and analyzer designed to give a unified report of all activity on a server, which can be delivered through the command line or email.

sudo apt install logwatch -y
sudo cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/
sudo nano /etc/logwatch/conf/logwatch.conf

Basic configuration:

  • Output = stdout
    • By default, it will print to the standard output (console).
  • Output = mail
    • Alternatively, it can send an email whenever the analysis is completed.
    • See option mailer = “/usr/sbin/sendmail -t” for additional configuration.
    • Also, consider using an MSMTP [Link] to provide email capability to the system.
  • Format = text
    • The default is in text format (no encoding). Good enough for console reading.
  • Format = html
    • It is recommended to change the format to HTML if the report is sent via email.
  • MailTo = root
    • Defines the local account that will receive the report.
  • MailTo = [email protected]
    • For a complete email address, just follow the example.
  • MailFrom = Logwatch
    • Same for the sender address. Can be a local account or a complete email address.
  • Filename = /tmp/logwatch
    • Defines where the report will be saved and requires the Output variable to be set to file.
  • Archives = No
    • Uncomment the configuration above to disable the achieve feature. It is enabled by default.
  • Range = yesterday
    • Default range. Can also be set for Today or All.
  • Detail = Low
    • Defines the level of detail for the report. From 0 for low up to 10 for high.
  • Service = All
    • It is recommended to leave the as All for no filter.

Most popular commands and their attributes:

  • logwatch –help
  • logwatch –range “-5 days”
    • It defines the range to be analyzed as the whole history. The default range is the previous day.
  • logwatch –range “between -7 days and -3 days” –output mail
    • In addition to the range, it defines the output method to email.
  • logwatch –service all –detail high –mailto [email protected]
    • Requests a highly detailed report of all services to a specific email destination.

LOG CHECK

It checks the logs hourly and after a reboot for “interesting” lines, looking for clues of misuse, security violations, or problems.

sudo apt install logcheck -y
sudo nano /etc/logcheck/logcheck.conf

Popular configuration options:

  • REPORTLEVEL=”server”
    • Controls the level for the filtering, from “workstation” for the low state of alert, until “paranoid” for the highest state of alert.
  • SENDMAILTO=”logcheck”
    • Change it to the destination email address of choice to receive the warnings/alerts. E.g. SENDMAILTO=”[email protected]”

See the list of log files to be checked at:

sudo nano /etc/logcheck/logcheck.logfiles

AUDIT DAEMON

auditd is the userspace component to the Linux Auditing System responsible for writing audit records to the disk. ausearch and aureport are the tools to view the logs.

sudo apt install auditd -y
sudo nano /etc/audit/auditd.conf

Popular commands and temporary rules:

  • sudo auditctl -s
    • shows the status of the auditd.
  • sudo auditctl -e 1
    • sets the enable flag to ON.
  • sudo auditctl -w /etc/passwd -p wrsa -k passwd_watch_key
    • adds the file in the path to the “watched” under the permission access types: r=read, w=write, x=execute, a=attribute change. And give it an identification key.
  • sudo ausearch -k passwd_watch_key
    • shows the status of the watched file and lists the accesses.

Make the new rules persistent after reboot, and append them to the file /etc/audit/rules.d/audit.rules:

-w /etc/passwd -p wrsa -k passwd_watch_key

It will require restarting the service:

sudo systemctl restart auditd

TUI NAVIGATION

The File Navigator (sks LNAV) will be your best friend [Link]. This terminal user interface transforms the painful experience of diving into logs from a terminal by adding features such as colouring/highlighting, regular expressions for searching, filtering, merging multiple files into a single view, and more [Link].

Install with APT

sudo apt install lnav -y

Or Brew

brew install lnav

Common Usage

lnav /var/log/syslog
lnav /var/log/syslog /var/log/apache2
journalctl -f | lnav

Try out the key bindings:

  • f goes to the next file.
  • e goes to the next error.
  • shift e  goes to the previous error.
  • / to open the search prompt.
    • n goes to the next search hit.
    • shift+n goes back.
  • :takes you to the commands menu.
    • Then, enable the mouse mode.
    • Or apply a filter.

BONUS

Remote Syslog with syslog-ng [Link]. Current version 3.36.1 (2022-05-25) available at [Link].

Install syslog-ng on both sides, server and client:

sudo apt-get install syslog-ng -y

OR (on Ubuntu/Debian)

wget -qO - https://ose-repo.syslog-ng.com/apt/syslog-ng-ose-pub.asc | sudo apt-key add -
echo "deb https://ose-repo.syslog-ng.com/apt/ stable ubuntu-focal" | sudo tee -a /etc/apt/sources.list.d/syslog-ng-ose.list
apt-get update && apt-get install syslog-ng -y

On the server-side, create a new file called /etc/syslog-ng/syslog-ng.conf:

@version: 3.36
@include "scl.conf"
options {
    time-reap(30);
    mark-freq(10);
    keep-hostname(yes);
};
source s_local {
    system(); internal();
};
source s_network {
    syslog(transport(tcp));
};
destination d_logs {
    file(
        "/var/log/syslog-ng/${HOST}.txt"
        owner("root")
        group("root")
        perm(0755)
        );
    };
log {
    source(s_local); source(s_network); destination(d_logs);
};

Restart the service with sudo systemctl restart syslog-ng.

On the client-side, create a new file called /etc/syslog-ng/syslog-ng.conf:

@version: 3.36
@include "scl.conf"
source s_local {
    system(); internal();
};
destination d_syslog_tcp {
    syslog("10.10.10.10" transport("tcp") port(514));
};
log {
    source(s_local);destination(d_syslog_tcp);
};

Restart the service with sudo systemctl restart syslog-ng.

Remember to open the respective port on the firewall of the server to allow the inbound connection on the respective port.

Default ports:

  • UDP – port 514
  • TCP – port 601
  • TLS/TCP – port 6514

READ MORE

  • Centralized Log Server with GrayLog [Link].
  • Another open-source tools for centralizing logs are Zabbix [Link], Nagios [Link], and NXLog [Link]. Consider giving it a try before deciding what is the best tool for your needs.