dmesg, kern.log and the Kernel Ring Buffer on Linux

What it is

The kernel writes its messages (printk) to an in-memory ring buffer. dmesg reads it through /dev/kmsg. User-space loggers copy it to disk: rsyslog writes the kern facility to /var/log/kern.log on Debian and Ubuntu and to /var/log/messages on RHEL and SUSE, and journald stores it with _TRANSPORT=kernel. After a panic, the last messages may survive in pstore.

Much of what an attacker does never reaches the kernel log, but the side effects do: an exploit that crashes a service, a cryptominer that exhausts memory, a sniffer that switches the NIC to promiscuous mode, an unsigned rootkit module, or an eBPF program using risky helpers.

Where it lives

What it proves

• Process crashes: segfault at ... ip ... error N in <lib> lines give the process name, PID and faulting library. Bursts of segfaults in a network service fit exploitation attempts.
• Resource abuse: Out of memory: Killed process 4302 (xmrig) with UID and memory figures.
• Sniffing: device eth0 entered promiscuous mode (older kernels) or eth0: entered promiscuous mode (newer) when tcpdump or a sniffer starts, and the matching "left" line when it stops.
• Kernel integrity: module verification failed: signature and/or required key missing - tainting kernel, loading out-of-tree module taints kernel, see kernel modules.
• eBPF misuse: the kernel prints a warning with comm and PID when a program uses bpf_probe_write_user, see eBPF programs.
• Hardware and devices: USB attach with vendor, product and serial, see udev and USB, plus disk errors and new block devices.
• Boot facts: the Linux version line (exact kernel build) and the Command line: line (boot parameters, including selinux=0, init=, module.sig_enforce).
• It does not record commands or file access.

Key fields

[ 1234.567890] nginx[4302]: segfault at 0 ip 00007f3a1c2d4e10 sp 00007ffd9a2b1c40 error 4 in libc.so.6[7f3a1c200000+195000] likely on CPU 2 (core 2, socket 0)
[ 2211.004512] Out of memory: Killed process 5120 (kdevtmpfsi) total-vm:2452220kB, anon-rss:2310452kB, file-rss:0kB, shmem-rss:0kB, UID:33 pgtables:4620kB oom_score_adj:0
[ 3302.118004] device ens5 entered promiscuous mode

Timestamps

dmesg shows seconds since boot. dmesg -T converts to wall-clock time using the current boot time, which drifts after suspend and resume, so do not rely on it for precise timelines. Better sources are the journal (true UTC microseconds for each kernel message) and the syslog copies (local time of receipt). To convert by hand, add the monotonic offset to the boot time from journalctl --list-boots or the btime line in /proc/stat on a live host.

Retention

The ring buffer overwrites itself; on a busy host early boot messages can be gone within hours, and everything is lost at reboot. dmesg -C (clear) empties it, a trivial anti-forensic step, but it does not touch copies already in kern.log, messages or the journal. Debian and RHEL rotate those files weekly and keep four generations by default; journal retention depends on SystemMaxUse.

Collection

# Live, first: volatile ring buffer, both raw and human-readable
dmesg --raw > /media/ir/dmesg_raw.txt
dmesg -x -T > /media/ir/dmesg_T.txt
cp -a /sys/fs/pstore /media/ir/pstore 2>/dev/null

# Dead box
tar -C /mnt/evidence -cpf /cases/2026-017/kern.tar var/log/kern.log* var/log/messages* \
  var/log/syslog* var/log/dmesg* var/lib/systemd/pstore var/log/journal 2>/dev/null

UAC runs dmesg in live response and collects /var/log in full. Velociraptor can collect the journal and log files, or run dmesg through its execve artifact.

Parsing

L=/cases/2026-017/var/log
zgrep -hE 'segfault at|general protection|Out of memory|oom-kill|promiscuous mode|taints kernel|bpf_probe_write_user|Command line:' \
  $L/kern.log* $L/messages* 2>/dev/null | sort
journalctl -D $L/journal _TRANSPORT=kernel -o short-iso-precise \
  -g 'segfault|Out of memory|promiscuous|taint|bpf'                  # all boots, UTC-capable
journalctl -D $L/journal --list-boots                                  # boot times for offset maths

Investigator tips

• Cluster segfaults by process and time. Dozens of segfault lines for sshd, nginx, php-fpm or exim within minutes, followed by quiet, is the classic pattern of a brute-forced memory corruption exploit that eventually worked.
• OOM kills of processes with random names or names imitating kernel threads (kworker, kdevtmpfsi, kthreaddi) usually mean a miner.
• A promiscuous-mode line with no matching admin activity deserves a search for capture files (*.pcap) and the capturing process in /proc.
• If the ring buffer on a live host starts long after boot while the journal has earlier kernel lines, either the buffer wrapped or someone ran dmesg -C. Check shell history for it.
• On Ubuntu, kernel.dmesg_restrict=1 means an unprivileged attacker could not read the buffer; a change to 0 in sysctl configuration is itself a lead.