Hunting Linux Persistence: Cron, systemd, SSH and More

Linux gives an intruder dozens of ways to survive a reboot or a password reset, and most of them are plain text files in predictable places. That is good news for responders: a methodical sweep of known locations, combined with package verification and timestamps, finds the large majority of persistence. This guide lists where to look, what normal looks like, and how each mechanism maps to MITRE ATT&CK.

Work from a read-only mount (here /mnt/evidence) or a triage collection. On a live host, userland tools such as systemctl or crontab -l can be subverted by a rootkit, so prefer reading files directly.

Scheduled execution

Cron

Cron reads several locations, and the RHEL and Debian families differ:

• /etc/crontab and /etc/cron.d/*: system tables with a user field before the command.
• /etc/cron.hourly, cron.daily, cron.weekly, cron.monthly: scripts run by run-parts (triggered from /etc/crontab on Debian, from /etc/cron.d/0hourly and anacron on RHEL).
• User crontabs: /var/spool/cron/crontabs/<user> on Debian/Ubuntu, /var/spool/cron/<user> on RHEL family. No user field; they run as the file owner.

ls -la /mnt/evidence/etc/cron.* /mnt/evidence/var/spool/cron/ /mnt/evidence/var/spool/cron/crontabs/ 2>/dev/null
grep -hvE '^\s*(#|$)' /mnt/evidence/etc/crontab /mnt/evidence/etc/cron.d/* \
  /mnt/evidence/var/spool/cron/crontabs/* /mnt/evidence/var/spool/cron/* 2>/dev/null

Suspicious entries typically run every minute, fetch remote content (curl, wget), decode base64, or execute from /tmp, /dev/shm or dot-directories. Cron executions appear as CRON[pid]: (root) CMD (...) in syslog/journal on Debian and in /var/log/cron on RHEL. See the log forensics guide for retrieval.

Anacron and at

/etc/anacrontab defines delayed periodic jobs; /var/spool/anacron/ holds last-run date stamps. One-shot at jobs are stored as shell scripts in /var/spool/cron/atjobs/ (Debian) or /var/spool/at/ (RHEL). Each job file contains the full environment and the command at the end, so read it to the bottom.

systemd services and timers

Systemd is now the most common place for stealthy persistence because a unit named dbus-helper.service blends in. System unit search paths, highest precedence first, include:

/etc/systemd/system/          # admin and attacker territory
/run/systemd/system/          # runtime, lost at reboot
/usr/local/lib/systemd/system/
/usr/lib/systemd/system/      # package-owned (also /lib/systemd/system on older Debian)

User units live in ~/.config/systemd/user/, /etc/systemd/user/ and /usr/lib/systemd/user/. They run only while the user has a session, unless lingering is enabled; a file in /var/lib/systemd/linger/<user> means that user's units start at boot.

Check three things beyond .service files themselves:

1. Enablement symlinks in *.wants/ directories (for example multi-user.target.wants/).
2. Drop-ins in <unit>.d/*.conf that override ExecStart= or add ExecStartPre= to a legitimate unit.
3. Generators in /etc/systemd/system-generators/ and /usr/lib/systemd/system-generators/, executables that create units at every boot.

find /mnt/evidence/etc/systemd /mnt/evidence/usr/lib/systemd /mnt/evidence/home/*/.config/systemd \
  /mnt/evidence/root/.config/systemd -type f -newer /mnt/evidence/etc/hostname -printf '%C+ %p\n' 2>/dev/null | sort
grep -rhE '^(ExecStart|ExecStartPre|Environment)=' /mnt/evidence/etc/systemd/system | sort | uniq -c

-newer against a file created at install time is a quick heuristic; adjust the reference to your incident window. Timers (.timer units with OnCalendar= or OnBootSec=) replace cron for many attackers. On a live host, systemctl list-timers --all shows next and last trigger times; offline, read the .timer files and the unit they activate (Unit= or the same basename).

Boot and login scripts

• /etc/rc.local: executed on systemd up to version 259 by rc-local.service (created by systemd-rc-local-generator) if the file exists and is executable. systemd 260 removed both upstream, so check the systemd version on the image before assuming it ran.
• /etc/init.d/ and /etc/rc?.d/: SysV scripts, wrapped as units at boot by systemd-sysv-generator up to systemd 259; systemd 260 removed SysV script support.
• Shell startup files: /etc/profile, /etc/profile.d/*.sh, /etc/bash.bashrc (Debian) or /etc/bashrc (RHEL), and per-user ~/.bashrc, ~/.bash_profile, ~/.profile, ~/.bash_logout, ~/.zshrc. Profile files run for login shells, the bashrc files for interactive shells (default profiles usually source ~/.bashrc too), ~/.zshrc for every interactive zsh and ~/.bash_logout at logout, so a single appended line runs in almost every session. Also look for alias sudo= or functions that wrap sudo to capture passwords.
• MOTD scripts: on Debian/Ubuntu, /etc/update-motd.d/* is executed as root by pam_motd at login. A new script here runs as root every time anyone logs in over SSH.
• XDG autostart (desktops): ~/.config/autostart/*.desktop and /etc/xdg/autostart/.

Access and authentication

SSH authorized keys. Check ~/.ssh/authorized_keys (and authorized_keys2) for every account including root and service accounts, plus AuthorizedKeysFile and AuthorizedKeysCommand in /etc/ssh/sshd_config and /etc/ssh/sshd_config.d/*.conf. Key options such as command= or from= change behaviour. Key comments are attacker-controlled and prove nothing.

Accounts. New users, extra UID 0 entries and service accounts with login shells:

awk -F: '$3 == 0 || $7 ~ /(ba|z|da)?sh$/ {print $1, $3, $6, $7}' /mnt/evidence/etc/passwd
diff /mnt/evidence/etc/passwd- /mnt/evidence/etc/passwd

PAM. Review /etc/pam.d/* for unexpected modules or pam_exec.so lines, and verify module files in /lib/x86_64-linux-gnu/security/ (Debian/Ubuntu) or /usr/lib64/security/ (RHEL) against the package database. A backdoored pam_unix.so accepts a hardcoded password and can log credentials.

User-level history and dotfile analysis is covered in shell history and user activity.

Library and kernel level

Dynamic linker hijacking. /etc/ld.so.preload forces a library into every dynamically linked process and is a classic userland rootkit technique; it is normally absent. Also search for LD_PRELOAD in /etc/environment, systemd Environment= lines and shell startup files. See LD_PRELOAD.

ls -la /mnt/evidence/etc/ld.so.preload 2>/dev/null && cat /mnt/evidence/etc/ld.so.preload
grep -rn 'LD_PRELOAD' /mnt/evidence/etc 2>/dev/null

Kernel modules. Modules listed in /etc/modules-load.d/*.conf (and /etc/modules on Debian) load at boot; install directives in /etc/modprobe.d/ can run arbitrary commands. A malicious module can hide itself from lsmod, so compare against a memory image with Volatility 3 linux.malware.check_modules.Check_modules and linux.malware.hidden_modules.Hidden_modules (see the memory forensics guide).

udev rules. RUN+= in /etc/udev/rules.d/*.rules executes a program when a matching device event occurs, which can be made to fire at every boot.

SUID binaries. A copied shell with the setuid bit is a simple root backdoor:

find /mnt/evidence -xdev -perm -4000 -type f -printf '%C+ %u %p\n' 2>/dev/null | sort

Anything outside standard binary directories, or not owned by a package, is suspect.

Technique map

Making the sweep efficient

Three filters cut noise dramatically:

1. Package ownership. Files in /usr/lib, /lib and /etc that no package owns, or that fail verification, deserve attention first. Use dpkg -S / rpm -qf and dpkg --verify / rpm -Va; details are in the package manager guide.
2. Timestamps. Sort candidates by ctime rather than mtime: touch can backdate mtime but not ctime.
3. Execution evidence. Tie each candidate to logs (cron CMD lines, systemd unit start messages, sshd key fingerprints) to prove it actually ran.

Triage tools collect most of these paths in one pass. UAC's ir_triage profile and the Velociraptor artifacts Linux.Sys.Crontab and Linux.Ssh.AuthorizedKeys cover the common cases.

Key takeaways

• Cron paths differ by distro: /var/spool/cron/crontabs on Debian/Ubuntu, /var/spool/cron on RHEL family.
• systemd persistence hides in drop-ins, *.wants symlinks, user units with lingering, timers and generators, not just new .service files.
• /etc/ld.so.preload, PAM modules, MOTD scripts and udev rules are high-value, low-noise checks.
• Check every account's authorized_keys and look for extra UID 0 entries.
• Filter by package ownership and ctime, then prove execution with logs.
• Confirm kernel-level findings against memory, since rootkits lie to userland tools.