systemd Unit Files: Linux Service Persistence

What it is

systemd starts and supervises almost everything on a modern Linux host. Each service, timer, socket, mount or path watcher is a unit file in INI format, and systemd merges units from a list of search directories, drop-in override files and units generated at boot. An attacker who writes one small .service file and a symlink gets code execution at every boot, often as root (MITRE ATT&CK T1543.002).

Because systemd applies a strict precedence order, the unit that actually runs may not be the one you find first. Collecting every search path is the only reliable approach.

Where it lives

System manager search paths, highest precedence first (from systemd.unit(5)):

User manager paths include ~/.config/systemd/user/, ~/.local/share/systemd/user/, /etc/systemd/user/, /etc/xdg/systemd/user/, /usr/local/lib/systemd/user/ and /usr/lib/systemd/user/.

Other locations that matter:

What it proves

• Which services, timers and sockets are defined and enabled, and exactly what they execute.
• That a package unit was modified or overridden (drop-in, or copy in /etc/systemd/system with the same name).
• When a unit started, stopped or failed, using journal events.
• That a user had services running without being logged in (lingering).
• Not proven: a unit file on disk may never have been loaded. Check the journal for start events, and note that disabled units only run if something else pulls them in.

Key fields

Journal fields for unit events: UNIT= (or USER_UNIT=), MESSAGE_ID=, _PID=, INVOCATION_ID=. Useful catalog IDs: 7d4958e842da4a758f6c1cdc7b36dcc5 (start job began), 39f53479d3a045ac8e11786248231fbf (start finished), be02cf6855d2428ba40df7e9d022f03d (start failed), 98e322203f7a4ed290d09fe03c09fe15 (unit process exited).

Timestamps

Unit files hold no internal timestamps; use file system times. A unit from a package normally has an mtime matching the package build and a ctime near install time; a hand-written unit has mtime and ctime close together. ext4 and XFS v5 also record a creation time (see ext4 timestamps). The symlink in *.wants/ has its own times, which date the enable. Journal events use microseconds since 1970-01-01 UTC.

journalctl -D /mnt/evidence/var/log/journal -u suspicious.service -o short-iso-precise

Retention

Unit files persist until removed; /run content and transient units vanish at reboot. Package removal deletes package units but not files an administrator or attacker created. Unit lifecycle messages survive only as long as the journal (or syslog) keeps them.

Collection

UAC's systemd artifact collects /etc/systemd, /lib/systemd/system, /usr/lib/systemd, /usr/local/lib/systemd, /run/systemd/system, transient units and per-user ~/.config/systemd and ~/.local/share/systemd. On a live host it also saves systemctl list-units, list-unit-files, list-timers --all and timer status. Velociraptor's Linux.Sys.Services parses systemctl list-units --type=service on the endpoint.

# live: what systemd actually loaded, including drop-ins
systemctl cat suspicious.service
systemctl list-unit-files --state=enabled
systemctl show suspicious.service -p FragmentPath,DropInPaths,ExecStart
# dead box
tar -C /mnt/evidence -czf systemd.tgz etc/systemd usr/lib/systemd lib/systemd \
  usr/local/lib/systemd var/lib/systemd home/*/.config/systemd root/.config/systemd 2>/dev/null

Parsing

R=/mnt/evidence
systemctl --root="$R" list-unit-files                 # enablement read from disk
find "$R"/etc/systemd "$R"/usr/lib/systemd "$R"/home/*/.config/systemd -xdev \
  \( -type f -o -type l \) -printf '%C+ %M %u %p -> %l\n' 2>/dev/null | sort
grep -rhoE '^(ExecStart[A-Za-z]*|Environment(File)?|User)=.*' "$R"/etc/systemd | sort | uniq -c | sort -n

systemd-analyze --root="$R" verify <unit> flags unknown directives and ExecStart= binaries that do not exist. Check package ownership with dpkg -S or rpm -qf (see package manager logs).

Investigator tips

• Attackers prefer names that mimic real services (dbus-org.freedesktop.helper.service). Judge the ExecStart= path, not the name.
• Review every *.d/override.conf: a drop-in that clears and resets ExecStart= on a trusted unit hides well.
• Check *.wants/ for symlinks pointing at files outside the unit paths, or to units no package owns.
• A generator in /etc/systemd/system-generators/ runs before units load at every boot, which is rare on servers.
• User units plus /var/lib/systemd/linger/<user> give non-root persistence that survives logout; common with cryptominers.
• Environment=LD_PRELOAD= in a unit is a targeted form of dynamic linker hijacking.
• systemd logs a reload message on every daemon-reload; one shortly before a new unit's first start helps date the installation.