Linux Live Response: Triage Through /proc and Volatile Data

Live response is the part of an investigation you only get once. When the host is powered off, running processes, open sockets, deleted-but-open files and unsaved kernel state are gone. This guide describes a disciplined way to collect that volatile data on Linux, with a focus on /proc, and explains what each source proves and where it can lie. It complements disk acquisition in acquiring Linux evidence and automated collection in triage with UAC and Velociraptor.

Principles before commands

Minimal footprint

Every command you run changes the system: it creates processes, touches atime, allocates memory and may overwrite freed disk blocks. Keep interaction short and planned. Write output to external media or stream it over the network, never to the suspect filesystem. If memory acquisition is in scope, do it first, before your own tools pollute RAM; see memory forensics with LiME and Volatility.

Trusted binaries

Assume ps, ss, ls and netstat on the host may be replaced. Bring statically linked tools (a static BusyBox build is common) on read-only media and call them by absolute path. Static linking also sidesteps LD_PRELOAD and /etc/ld.so.preload hooks, which only affect dynamically linked programs. It does not protect you from a kernel-level rootkit, which can filter what /proc returns to every process.

export PATH=/mnt/irkit/bin
export HISTFILE=/dev/null          # keep your commands out of the suspect's history
cat /etc/ld.so.preload 2>/dev/null # should normally not exist

Log your own actions

Your commands are part of the evidence record. script from util-linux captures the session, and ts from moreutils can prefix output lines with timestamps:

script -a -T /mnt/usb/web-prod-03/session.timing /mnt/usb/web-prod-03/session.log
date -u; hostname; id

Hash every output file with sha256sum when you finish and record the hashes in your notes.

Anchor the clock first

Every timestamp you collect is only as good as your knowledge of the host clock. Record it before anything else:

date -u; date
timedatectl                        # local time, time zone, NTP sync state
cat /proc/uptime                   # seconds since boot
grep btime /proc/stat              # boot time as epoch seconds

Compare the host's UTC time with a trusted reference and note the drift. A time zone other than UTC changes how every log you later read must be interpreted, and an unsynchronised clock may explain apparently impossible sequences.

Collection order

Follow the order of volatility: the faster something changes, the earlier you collect it.

Network and process listing

ss -tanp  > net_tcp.txt     # all TCP sockets with owning process
ss -uanp  > net_udp.txt
ss -xp    > net_unix.txt
lsof -nP -i > lsof_net.txt  # -n and -P avoid DNS and port-name lookups
ps -eo pid,ppid,user,lstart,etime,args --forest > ps_tree.txt

-n and -P matter: reverse DNS lookups generate outbound traffic the attacker may observe, and they slow collection. lstart gives the full process start time, which you can correlate with logs. Note that ps reads /proc, so everything below can be verified manually.

Process triage through /proc

Each running process has a directory /proc/<pid>/. The key entries:

A quick sweep for suspicious processes:

ls -l /proc/[0-9]*/exe 2>/dev/null | grep '(deleted)'
ls -l /proc/[0-9]*/cwd 2>/dev/null | grep -E '/tmp|/dev/shm|/var/tmp'
grep -l '(deleted)' /proc/[0-9]*/maps 2>/dev/null

Then inspect a candidate in depth. In this fictional example, PID 4127 on web-prod-03 runs as deploy:

PID=4127
tr '\0' ' ' < /proc/$PID/cmdline; echo
cat /proc/$PID/comm
tr '\0' '\n' < /proc/$PID/environ
ls -l /proc/$PID/cwd /proc/$PID/exe /proc/$PID/fd
grep -E '^(Name|State|PPid|Uid|Gid):' /proc/$PID/status

/proc/4127/cwd -> /dev/shm/.x
/proc/4127/exe -> /dev/shm/.x/kworkerd (deleted)

A process named like a kernel thread, with a userland executable in /dev/shm that has been deleted, is a strong indicator. Genuine kernel threads have no exe target and an empty cmdline.

Recovering a deleted binary

The kernel keeps the inode alive while the process runs, so the content can be copied directly:

cp /proc/4127/exe /mnt/usb/web-prod-03/pid4127_exe.bin
sha256sum /mnt/usb/web-prod-03/pid4127_exe.bin
cat /proc/4127/maps > /mnt/usb/web-prod-03/pid4127_maps.txt

The same works for deleted files held open: ls -l /proc/<pid>/fd shows the descriptor number, and copying /proc/<pid>/fd/<n> retrieves the content. Do this before killing anything. Killing the process releases the last reference, and on ext4 the blocks become very hard to recover, as explained in ext4 and XFS timestamps and deleted files.

Users, mounts and kernel state

who -a; w
mount; cat /proc/mounts
lsmod; ls /sys/module
cat /proc/sys/kernel/tainted

lsmod is just a formatter for /proc/modules, so it is not an independent check. Comparing it with /sys/module can reveal inconsistencies, though a careful rootkit hides from both. The taint value is a bitmask; a non-zero value is not malicious by itself, but specific bits deserve explanation:

A server with no third-party drivers that reports out-of-tree or unsigned module taint needs an explanation. Taint flags are sticky until reboot, so they persist even if the module is later unloaded. Cross-reference with /etc/modules-load.d and other locations from hunting Linux persistence.

Check mounts for tmpfs or bind mounts over directories such as /proc/<pid>, a known trick to hide a process from userland tools.

When the tools lie

Everything above runs in userland and trusts the kernel. A loadable kernel module rootkit can hide processes, files, sockets and itself from /proc, ss and lsmod alike. Signs worth noting: PIDs that respond to kill -0 but have no /proc directory, gaps between ss output and traffic seen on the network, or load averages that do not match visible processes. When you suspect kernel compromise, memory acquisition and offline analysis with Volatility 3 are the reliable path, and the live data becomes a comparison baseline rather than ground truth.

Key takeaways

• Capture memory first when possible, then network, processes, open files, users and kernel state.
• Use trusted static binaries by absolute path and write all output off-host.
• Record date -u, timedatectl and boot time before anything else.
• /proc/<pid>/exe and /proc/<pid>/fd/<n> let you recover deleted binaries and files while the process lives.
• comm, cmdline and even /proc itself can be manipulated; corroborate with memory and disk.
• Log and hash everything you run and collect.