by: Linux/Unix Raspberry Pi

INDEX

  • Journal Control
    • Used to view and manipulate Systemd logs.
  • Remote Syslog
    • Forwards log messages over an IP network.
  • Log Rotating
    • Allows automatic rotating, compressing, removing, and messaging of log files.
  • Log Watch
    • Periodically goes through your logs for a given period and makes a report via email or output file.
  • Log Check
    • Helps to spot problems and security violations in your log files automatically and sends the results via email.
  • Audit Daemon
    • Is a powerful audit framework to log events on Linux systems.

SYSLOG PRIORITY/SEVERITY LEVELS

  • 0 – Emergency
  • 1 – Alert
  • 2 – Critical
  • 3 – Error
  • 4 – Warning – Default
  • 5 – Notifications
  • 6 – Information
  • 7 – Debug

JOURNAL CONTROL

JournalCTL (aka Journal Control) is used to query the contents of the systemd journal (systemd-journald.service).

Note: by default, Journal users can only watch their own logs, unless they are root or in the adm (or systemd-journal) group (usermod -a -G adm userName).

  • journalctl
    • shows the full content of the journal, starting with the oldest entry collected.
  • journalctl -r -n 10
    • shows the content of the journal in reverse order limited to 10 entries. The newest at the top.
  • sudo journalctl -u ssh.service -f
    • filters and outputs the logs for the unit ssh and follows for real-time entries.
  • sudo journalctl –since “2021-12-31 23:59:59”
    • filters the entries by a starting date and time.
  • sudo journalctl –since “yesterday” -p err
    • filters the entries by a relative time window with the priority:
      • since and/or until can contain strings like “today”, “1 hour ago”, “2 days ago”…
      • the priorities are directly related to the syslog levels and can be expressed in text or numerically (“emerg”/0, “alert”/1, “crit”/2, “err”/3, “warning”/4, “notice”/5, “info”/6, “debug”/7).
  • sudo journalctl SYSLOG_IDENTIFIER=sshd -S “13:30:00” -U “14:00:00”
    • filters the entries by an identifier, since, and until time.
  • sudo journalctl -k
    • shows kernel messages.
  • sudo journalctl -b
    • Shows the current boot messages.
  • sudo journalctl -b -2 –utc
    • Shows the messages from the second previous boot represented in UTC timezone.
  • sudo journalctl –list-boots
    • lists the available boot messages with ordinal number time stamping.

Journal Clean Up

sudo journalctl --verify
sudo journalctl --disk-usage
sudo journalctl --vacuum-size=200M
sudo journalctl --vacuum-files=2
sudo journalctl --vacuum-time=3d
sudo journalctl --vacuum-size=1G --vacuum-time=5d --vacuum-files=5

Customizing Configuration

sudo nano /etc/systemd/journald.conf
  • Storage=
    • One of “volatile”, “persistent”, “auto” and “none”.
  • Compress=
    • Suffixes like K, M, and G can be used to specify larger units.
  • SystemMaxUse=, SystemKeepFree=, SystemMaxFileSize=, SystemMaxFiles=, RuntimeMaxUse=, RuntimeKeepFree=, RuntimeMaxFileSize=, RuntimeMaxFiles=
    • Use the variables above to define limits to disk utilization in case the Journal start to affect of jeopardize the stability of the system.
  • MaxFileSec=, MaxRetentionSec=, SyncIntervalSec=
    • Define additional retention and synchronization parameters with the variables above.
  • ForwardToSyslog=, ForwardToKMsg=, ForwardToConsole=, ForwardToWall=, MaxLevelStore=, MaxLevelSyslog=, MaxLevelKMsg=, MaxLevelConsole=, MaxLevelWall=
    • Limit and define what and how to forward messages.

Finding Failures

journalctl --no-pager --since today --grep 'fail|error|fatal' --output json|jq '._EXE' | sort | uniq -c | sort --numeric --reverse --key 1

REMOTE SYSLOG

Rsyslog (aka “”the Rocket-fast SYStem for LOG processing”) uses the standard BSD syslog protocol to send messages over IP protocol but supports many others via extensions. The format of relayed messages can be customized.

Locally, the rsyslog is responsible for reading the Journal ans sorting the entries based on filters to appropriately store/output the information accordingly because the Journal is ephemeral and disappears after the reboot.

Server-Side

Edit the configuration file:

sudo nano /etc/rsyslog.conf

Uncomment the following lines to start listening over UDP (faster – recommended):

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

Or uncomment the following lines to start listening over TDP (more reliable):

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

Right below the uncommented lines add:

$template RemoteLogs,"/var/log/RemoteHosts/%HOSTNAME%-%$now%.log"
if $fromhost-ip != '127.0.0.1' then -?RemoteLogs
& stop

Note: it will create a template called RemoteLogs that will store the logs based on the hostname of the client. This template will not be applied to localhost and will stop to prevent double logs. Another variable that is popularly used it %PROGRAMNAME%.

Restart the service, check if its is listening on port 514, and start monitoring for incoming messages:

sudo systemctl restart rsyslog
sudo ss -tulpn
sudo tail -f /var/log/messages

Client-Side

Edit the configuration file:

sudo nano /etc/rsyslog.conf

Append the following lines:

For UDP:

*.* @10.10.10.10:514

For TCP:

*.* @@10.10.10.10:514

Note: the expression *.* means AllFacilities and AllSeverityLevels respectively. Replace the IP address by the IP or Hostname of the server.

Restart the service and test it:

sudo systemctl restart rsyslog
logger -t "TagName" Value
logger -t "LocalTime" $(date)

LOG ROTATING

logrotation is an automated process which log files are compressed, moved (archived), renamed or deleted once they are too old or too big.

The files /etc/logrotate.conf contains the default configuration that applies to all log files unless there is an specific configuration that overwrites it.

Each specific configuration file must me located at /etc/logrotate.d. See example that can be applied to the example of the rsyslog server above:

/var/log/RemoteHosts/*.log {
  weekly 
  rotate 26
  copytruncate
  compress
  delaycompress
  missingok
  notifempty
  create 600 root root
  maxsize 500M
  minsize 50M
}
  • weekly
    • Rotates file once a week. Could also be daily, monthly, or early.
  • rotate 26
    • Keeps 26 files.
  • copytruncate
    • Instructs to create the copy of the original file and truncates the original file to zero byte size.
  • compress
    • Use gzip compression to save disk space.
  • delaycompress
    • Do not compless the first rotation, but all the others.
  • missingok
    • Ignore if one file is missing.
  • notifempty
    • Do not rotate if the file is empty.
  • create 600 root root
    • Apply the desared permission to the created files.
  • maxsize 500M
    • Rotate immediately if the file get bigger than specified. Possible to use K for kilobyte, M for megabyte, and G for gigabite.
  • minsize 50M
    • Do not rotate if the file is too small. Possible to use K for kilobyte, M for megabyte, and G for gigabite.

Additionally:

  • sharedscripts
    • This option will execute the prerotate and the postrotate scripts only once even if there are more than one file to rotate instead of execute the for each file being rotated.
  • prerotate
    • The content of this option has to be written in bash and execute any needed procedure prior to the file is rotated.
    • Has to end with endscript.
  • postrotate
    • The content of this option has to be written in bash and execute any needed procedure after the file is rotated.
    • Has to end with endscript.

Bonus:

Logrotate is executed daily by the cron via the script /etc/cron.daily/logrotate and also via timers set on the System Control with the configuration /usr/lib/systemd/system/logrotate.timer and /usr/lib/systemd/system/logrotate.service (systemctl list-timers).

The reason why there is a combination of cron and systemd managing the execution of the logrotate is to allow the execution of the rotation float within a range of acceptable time that will look for the best time to reduce resources competition based on the load of the system, for example.

In case it is needed to execute the logrotate manually or in a frequency higher than daily, for example, every 6 hours (depending on the needs of the system and the amount of data stored), use the -f or -force.

/usr/sbin/logrotate -f /etc/logrotate.conf

LOGWATCH

Logwatch is log parser and analyzer designed to give a unified report of all activity on a server, which can be delivered through the command line or email.

sudo apt install logwatch -y
sudo cp /usr/share/logwatch/default.conf/logwatch.conf /etc/logwatch/conf/
sudo nano /etc/logwatch/conf/logwatch.conf

Basic configuration:

  • Output = stdout
    • By default it will print to the standard output (console).
  • Output = mail
    • Alternatively it can send an email whenever the analyse is completed.
    • See option mailer = “/usr/sbin/sendmail -t” for additional configuration.
    • Also consider using a MSMTP [Link] to provide email capability to the system.
  • Format = text
    • The default is in text format (no encoding). Good enough for console reading.
  • Format = html
    • It is recommended to change the format to HTML if the report is sent via email.
  • MailTo = root
    • Defines the local account that will receive the report.
  • MailTo = [email protected]
    • For a complete email address just follow the example.
  • MailFrom = Logwatch
    • Same for the sender address. Can be a local account or a complete email address.
  • Filename = /tmp/logwatch
    • Defines where the report will be saved and requires the Output variable to be set to file.
  • Archives = No
    • Uncomment the configuration above to disable the achieve feature. It is enabled by default.
  • Range = yesterday
    • Default range. Can be also set for Today or All.
  • Detail = Low
    • Defines the level of detail for the report. From 0 for low up to 10 for high.
  • Service = All
    • It is recommended to leave the as All for no filter.

Most popular commands and its attributes:

  • logwatch –help
  • logwatch –range “-5 days”
    • It defines the range to be analyzed to the whole history. The default range is the previous day.
  • logwatch –range “between -7 days and -3 days” –output mail
    • Additionally to the range, it defines the output method to email.
  • logwatch –service all –detail high –mailto [email protected]
    • Requests a highly detailed report of all services to an specific email destination.

LOG CHECK

It checks the logs hourly and after a reboot for “interesting” lines, looking for clues of miss-usage, security violation, or problems.

sudo apt install logcheck -y
sudo nano /etc/logcheck/logcheck.conf

Popular configuration options:

  • REPORTLEVEL=”server”
    • Controls the level for the filtering. From “workstation” for the low state of alert, until “paranoid” for the highest state of alert.
  • SENDMAILTO=”logcheck”
    • Change it to the destination email address of choice to receive the warnings/alerts. E.g. SENDMAILTO=”[email protected]”

See the list of log files to be check at:

sudo nano /etc/logcheck/logcheck.logfiles

AUDIT DAEMON

auditd is the userspace component to the Linux Auditing System responsible for writing audit records to the disk. ausearch and aureport are the tools to view the logs.

sudo apt install auditd -y
sudo nano /etc/audit/auditd.conf

Popular commands and temporary rules:

  • sudo auditctl -s
    • shows the status of the auditd.
  • sudo auditctl -e 1
    • sets the enable flag to ON.
  • sudo auditctl -w /etc/passwd -p wrsa -k passwd_watch_key
    • adds the file in the path to the “watched” under the permission access types: r=read, w=write, x=execute, a=attribute change. And give it an identification key.
  • sudo ausearch -k passwd_watch_key
    • shows the status of the watched file and lists the accesses.

Make the new rules persistent after reboot append them to the file /etc/audit/rules.d/audit.rules:

-w /etc/passwd -p wrsa -k passwd_watch_key

It will require restarting the service:

sudo systemctl restart auditd

BONUS

Remote Syslog with syslog-ng [Link]. Current version 3.36.1 (2022-05-25) available at [Link].

Install syslog-ng on both sides, server and client:

sudo apt-get install syslog-ng -y

OR (on Ubuntu/Debian)

wget -qO - https://ose-repo.syslog-ng.com/apt/syslog-ng-ose-pub.asc | sudo apt-key add -
echo "deb https://ose-repo.syslog-ng.com/apt/ stable ubuntu-focal" | sudo tee -a /etc/apt/sources.list.d/syslog-ng-ose.list
apt-get update && apt-get install syslog-ng -y

On the server-side, create a new file called /etc/syslog-ng/syslog-ng.conf:

@version: 3.36
@include "scl.conf"
options {
    time-reap(30);
    mark-freq(10);
    keep-hostname(yes);
};
source s_local {
    system(); internal();
};
source s_network {
    syslog(transport(tcp));
};
destination d_logs {
    file(
        "/var/log/syslog-ng/${HOST}.txt"
        owner("root")
        group("root")
        perm(0755)
        );
    };
log {
    source(s_local); source(s_network); destination(d_logs);
};

Restart the service with sudo systemctl restart syslog-ng.

On the client-side, create a new file called /etc/syslog-ng/syslog-ng.conf:

@version: 3.36
@include "scl.conf"
source s_local {
    system(); internal();
};
destination d_syslog_tcp {
    syslog("10.10.10.10" transport("tcp") port(514));
};
log {
    source(s_local);destination(d_syslog_tcp);
};

Restart the service with sudo systemctl restart syslog-ng.

Remember to open the respective port on the firewall of the server to allow the inboud connection on the respective port.

Default ports:

  • UDP – port 514
  • TCP – port 601
  • TLS/TCP – port 6514

Read the post Centralized Log Server with GrayLog [Link].

Another opensource tools for centralize the logs are Zabbix [Link], Nagios [Link], and NXLog [Link]. Consider giving it a try before deciding what is the best tool for your need.