Linux Crash Reports and Core Dumps: coredump, apport

What it is

When a process dies from a signal such as SIGSEGV or SIGABRT, the kernel can write a core dump: an ELF image of the process memory. What happens next depends on kernel.core_pattern. On most systemd distributions it pipes the core to systemd-coredump, which stores it compressed and logs rich metadata to the journal. Ubuntu pipes it to apport, which writes a text crash report in /var/crash. RHEL 7 and 8 could use ABRT (removed in RHEL 9). Whole-kernel crashes are captured by kdump as a vmcore.

Attackers crash things: a failed memory corruption exploit against a service, an unstable implant, a fuzzed parser, a dropped binary built for the wrong libc. A crash record timestamps the attempt and can preserve command lines, environment and even decrypted strings or keys that exist only in memory.

Where it lives

Ubuntu 24.04 (apport 2.28) can run with systemd-coredump installed instead of apport-core-dump-handler; apport then builds its /var/crash report from the systemd-coredump data, so both locations may hold the same crash.

What it proves

• The crashing executable, PID, UID/GID, signal and exact time.
• The command line, working directory, environment (apport ProcEnviron, ABRT environ) and, for systemd-coredump, cgroup and unit, which identifies the service.
• Memory contents at crash time: strings, URLs, decrypted configuration, shellcode remnants.
• For kdump, kernel state at a panic, including loaded modules; useful for rootkit or faulty driver cases.
• It does not prove malicious intent: most crashes are bugs. A crash of a network-facing service right when an exploit arrived in web logs or auth logs is the meaningful pattern.

Key fields

systemd-coredump journal fields and file extended attributes:

apport .crash report (RFC 822 style key/value, binary values base64):

ABRT problem directories hold one file per element: executable, cmdline, pid, uid, time (epoch), reason, pwd, environ, maps, backtrace, coredump, package.

Timestamps

• Journal entries: __REALTIME_TIMESTAMP in UTC microseconds. The core file name ends in the crash time as epoch microseconds.
• apport Date: uses asctime() in local time without zone (Sun Sep 20 03:14:09 2026).
• ABRT time is epoch seconds; kdump directory names use the local time of the dump.
• apport marks a report as "seen" by setting its access time newer than its modification time (it rewinds mtime by one second), so a .crash file with atime > mtime has been opened by a user or tool.

date -u -d @$((1789874049123456 / 1000000))

Retention

• systemd-coredump: systemd-tmpfiles deletes files in /var/lib/systemd/coredump after 3 days on systemd before 256 and 2 weeks from 256. MaxUse (10% of disk, max 4 GiB) and KeepFree also vacuum old cores. The journal entry survives longer than the core, and cores above ExternalSizeMax (32 GiB on 64-bit) are only logged.
• apport: a daily cron job deletes reports older than seven days, and empty files.
• ABRT: total size capped by MaxCrashReportsSize (5000 MiB default).
• kdump vmcore files stay until deleted.

Collection

UAC's live response records core_pattern, coredumpctl list, coredumpctl info per dump and the xattrs of /var/lib/systemd/coredump. Its memory_dump/coredump.yaml artifact copies the systemd-coredump cores, the ABRT directories and /var/crash, but it is not part of the ir_triage or full profiles, so add it explicitly.

E=/mnt/evidence
tar -C "$E" --xattrs -cpf /cases/2026-017/crash.tar var/lib/systemd/coredump var/crash var/spool/abrt \
  etc/systemd/coredump.conf etc/systemd/coredump.conf.d etc/sysctl.d etc/default/apport etc/kdump.conf 2>/dev/null
coredumpctl -D "$E/var/log/journal" list --no-pager

Use --xattrs (and a filesystem that keeps them) or the metadata on the core files is lost. Cores and vmcores can contain passwords, keys and personal data.

Parsing

# Offline list and details from the image's journal
coredumpctl -D journal/ list
coredumpctl -D journal/ info 4121
coredumpctl -D journal/ dump 4121 -o core.4121          # write the core out
getfattr -d core.php-fpm.33.*.zst                        # metadata without the journal
# apport report into its parts, then inspect the core
apport-unpack _usr_sbin_php-fpm8.3.33.crash unpacked/
gdb /path/to/executable unpacked/CoreDump -batch -ex 'bt' -ex 'info registers'
strings -n 8 core.4121 | grep -Ei 'http|/tmp/|/dev/shm|password|BEGIN'

Use the matching binary and libraries from the image for gdb. For RHEL kdump, crash with the matching vmlinux (kernel debuginfo) analyses the vmcore.

Investigator tips

• Clusters of crashes of one network service (sshd, a web server worker, a VPN daemon) within minutes are typical of exploit attempts or brute-forced memory corruption; correlate with connection logs.
• A crash of a binary in /tmp, /dev/shm or a home directory is a dropped tool failing; the core may be the only copy left after the attacker deleted the file.
• COREDUMP_CMDLINE and COREDUMP_ENVIRON reveal arguments and the full environment (including LD_PRELOAD) that shell history lacks. apport's ProcEnviron keeps only a short allow-list and masks the value of LD_PRELOAD, so there you only learn that it was set.
• A changed core_pattern, fs.suid_dumpable, a Storage=none or ProcessSizeMax=0 in coredump.conf, or enabled=0 in /etc/default/apport can be anti-forensics; compare with package defaults.
• An empty /var/lib/systemd/coredump does not mean no crashes: check the journal for MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 entries.
• kdump vmcore-dmesg.txt shows the last kernel messages before a panic; an out-of-tree kernel module in the oops is a lead.