Linux Log Forensics: syslog, journald, wtmp and auditd

Logs are usually the first place an investigator looks and the first place an attacker cleans. On a modern Linux host the same event can be recorded in several independent stores: text files written by rsyslog, the binary systemd journal, the binary login accounting files, and the audit log. Correlating them is what turns a suspicion into a defensible finding, and a disagreement between them is often the best tampering indicator you will get.

All commands below assume you work on a read-only mounted image (see acquiring Linux evidence) mounted at /mnt/evidence. On a live system the same tools work, but every command you run can itself generate log entries, and log rotation may fire while you collect.

Where Linux logs live

Two distribution caveats matter. First, rsyslog is no longer installed by default on some modern installs (Debian 12 and recent Fedora, for example), so the text files may simply not exist and the journal is the only record. Second, some recent releases replace wtmp and lastlog with SQLite databases managed by wtmpdb and lastlog2 (for example /var/lib/wtmpdb/wtmp.db) to avoid the 2038 limit of the old record format. Check what the host actually runs before concluding anything from an absent file.

rsyslog text logs

sshd lines

The sshd messages in auth.log or secure are the backbone of most intrusion timelines. The patterns to know:

Sep 20 03:12:44 web-prod-03 sshd[4121]: Invalid user admin from 203.0.113.50 port 40022
Sep 20 03:12:46 web-prod-03 sshd[4121]: Failed password for invalid user admin from 203.0.113.50 port 40022 ssh2
Sep 20 03:13:58 web-prod-03 sshd[4188]: Failed password for deploy from 203.0.113.50 port 40310 ssh2
Sep 20 03:14:07 web-prod-03 sshd[4190]: Accepted password for deploy from 203.0.113.50 port 40318 ssh2
Sep 20 03:14:07 web-prod-03 sshd[4190]: pam_unix(sshd:session): session opened for user deploy(uid=1001) by (uid=0)
Sep 20 09:02:11 web-prod-03 sshd[5530]: Accepted publickey for deploy from 198.51.100.7 port 51522 ssh2: ED25519 SHA256:...

Invalid user means the account does not exist; Failed password for <user> against a real account after a burst of invalid names is the classic spray-then-hit pattern. Accepted publickey lines include the key fingerprint, which you can match against authorized_keys entries to identify which key was used. On recent OpenSSH releases the per-connection process may log as sshd-session rather than sshd, so do not grep only for sshd[.

grep -hE 'Accepted|Failed password|Invalid user' /mnt/evidence/var/log/auth.log* 2>/dev/null
zgrep -hE 'Accepted|Failed password|Invalid user' /mnt/evidence/var/log/auth.log.*.gz

sudo lines

Sep 20 03:15:30 web-prod-03 sudo:   deploy : TTY=pts/0 ; PWD=/home/deploy ; USER=root ; COMMAND=/usr/bin/bash

This proves which account invoked sudo, from which terminal and directory, and the target command. It does not record what was done inside the resulting root shell; for that you need shell history (see shell history and user activity) or auditd.

Rotation and retention

logrotate (/etc/logrotate.conf and /etc/logrotate.d/) renames logs on a schedule. Debian and Ubuntu produce auth.log.1, auth.log.2.gz and so on; the RHEL family uses date extensions such as secure-20260920. Always read the rotated and compressed generations, and note the retention period: an intrusion older than the rotation window may be gone from text logs while still present in the journal, or vice versa.

The systemd journal

The systemd journal is a structured binary store. Each entry carries trusted fields added by journald itself (_PID, _UID, _COMM, _EXE, _SYSTEMD_UNIT, _BOOT_ID) plus a microsecond realtime timestamp in UTC. Persistence depends on configuration: with Storage=auto, the upstream default before systemd 259, journald writes to /var/log/journal only if that directory exists, otherwise to the volatile /run/log/journal, which disappears at shutdown. systemd 259 made persistent the compiled-in default, and distributions can override it at build time. Check /etc/systemd/journald.conf and drop-ins under /etc/systemd/journald.conf.d/ for Storage=, SystemMaxUse= and MaxRetentionSec=.

Reading an offline journal

J=/mnt/evidence/var/log/journal

journalctl --directory="$J" --list-boots
journalctl --directory="$J" -b -1 --utc -o short-iso
journalctl --directory="$J" --since "2026-09-20 03:00:00" --until "2026-09-20 04:00:00" --utc
journalctl --directory="$J" _SYSTEMD_UNIT=ssh.service      # sshd.service on RHEL family
journalctl --directory="$J" _COMM=sudo -o json > sudo.json
journalctl --file="$J"/<machine-id>/user-1001.journal -o export > user1001.export

Useful points:

• --utc forces UTC display; without it journalctl renders in the analysis host's time zone.
• -o json or -o json-pretty exposes every field, including __REALTIME_TIMESTAMP and _BOOT_ID. -o export is a lossless serialisation suitable for archiving.
• --list-boots gives boot IDs with first and last timestamps, a quick way to spot unexpected reboots or a boot whose entries end abruptly.
• Archived files are named like system@<id>-<seqnum>-<timestamp>.journal. Files ending in .journal~ were renamed after an unclean shutdown or corruption; read them with --file.

Verifying integrity

journalctl --directory="$J" --verify

--verify checks internal consistency. Without Forward Secure Sealing (set up with journalctl --setup-keys, rarely enabled) it detects corruption, not a careful rewrite. A failure is a lead, not proof: power loss also corrupts journal files.

Login accounting: wtmp, btmp, utmp, lastlog

These binary files (see wtmp and btmp) record sessions independently of syslog.

last -F -i -x -f /mnt/evidence/var/log/wtmp
lastb -F -i -f /mnt/evidence/var/log/btmp
utmpdump /mnt/evidence/var/log/wtmp > wtmp.txt

last -x adds shutdown and runlevel records; -F prints full timestamps with the year. utmpdump renders every record including fields last hides, which is how you spot anomalies such as zeroed records or out-of-order times. The lastlog command reads the live /var/log/lastlog and has no dependable offline option across versions; parse the image copy with a dedicated parser or Plaso instead. Remember that utmpdump -r can convert text back to binary, so editing wtmp is trivial for an attacker with root.

auditd

When auditd is running, /var/log/audit/audit.log is often the most detailed source: it can record logins, authentication results, command execution and file access, depending on the rules in /etc/audit/rules.d/. Timestamps are epoch seconds inside msg=audit(1789874047.512:8812), so they carry no time zone ambiguity. See the auditd glossary entry.

A=/mnt/evidence/var/log/audit/audit.log
ausearch -if "$A" -m USER_LOGIN,USER_AUTH -i
ausearch -if "$A" -m EXECVE -i --start 09/20/2026 03:00:00 --end 09/20/2026 04:00:00
aureport -if "$A" --login --summary -i
aureport -if "$A" -au -i

-i interprets numeric UIDs and syscalls into names, using the analysis host's user database unless the log was written with enriched format, so double-check names against the image's /etc/passwd. The --start date format follows the analysis host's locale.

Timestamp and time zone pitfalls

• Traditional syslog timestamps (Sep 20 03:14:07) have no year and no time zone. They are written in the host's local time at the moment of logging. Recover the zone from /etc/localtime in the image and infer the year from file metadata and rotation order.
• rsyslog's built-in default is RFC 3339 timestamps with an offset (Debian 12+, Ubuntu 24.04); the RHEL family sets the traditional template explicitly. Check the template in /etc/rsyslog.conf rather than assuming.
• The journal and auditd store UTC epoch values; wtmp stores epoch seconds. Normalise everything to UTC before correlating, as in a super timeline.
• A wrong system clock skews every source equally; look for NTP or systemd-timesyncd adjustments in the journal.

Linux Log Parser does this normalisation in the browser: it reads the zone from the image's /etc/localtime, infers the year of yearless lines and merges auth.log, the journal, audit.log and wtmp into one UTC timeline with rebuilt login sessions, without uploading anything.

Signs of log tampering

• Gaps in otherwise continuous logs that do not match rotation times or reboots.
• Truncated or zero-length current logs while rotated generations are intact.
• sed-style edits: a sshd session opened in the journal with no matching line in auth.log, or a last session with no corresponding Accepted message.
• wtmp records with zeroed fields or non-monotonic timestamps in utmpdump output.
• Journal --verify failures concentrated around the incident window, or .journal~ files without a matching unclean shutdown.
• Commands such as shred, truncate, journalctl --vacuum-time or history -c in shell history or auditd EXECVE records.
• A stopped rsyslog or auditd service, visible as a unit stop in the journal.

Key takeaways

• Know the distribution: auth.log on Debian/Ubuntu, secure on the RHEL family, and possibly no text logs at all.
• Read the journal offline with journalctl --directory and always add --utc.
• Correlate at least three sources (text logs, journal, wtmp, auditd); agreement builds confidence, disagreement reveals tampering.
• Normalise timestamps to UTC and recover the host time zone before building a timeline.
• Missing logs are a question to answer, not proof of anti-forensics.