Container Forensics: Docker, containerd and Podman on Linux

Containers change where evidence lives, not whether it exists. A compromised web application in a container still writes files, spawns processes and produces logs, but those artifacts sit in layered filesystems and runtime metadata on the host, and they disappear fast when an orchestrator recycles the workload. This guide explains the on-disk layout of Docker, containerd and Podman, how to extract what an attacker changed, and which settings indicate that a container was used to reach the host.

Preserve first

Containers are ephemeral by design. Before touching anything, stop automated cleanup: pause autoscaling or cordon the Kubernetes node, and do not run docker rm, docker system prune or restart the service. A running container's writable layer and its /proc entries are volatile evidence, so apply live response techniques to container processes from the host, where they are ordinary PIDs. Then acquire the host disk as usual (acquiring Linux evidence).

Docker on-disk layout

Docker keeps its state under /var/lib/docker unless data-root is changed in /etc/docker/daemon.json; always check that file first.

Container IDs are 64 hex characters, and the directory name is the full ID. Newer Docker Engine releases can instead use the containerd image store; in that case image and container layers are managed by containerd under /var/lib/containerd, while /var/lib/docker/containers still holds configuration and logs.

Reading metadata offline

docker inspect needs a running daemon. On an image, read the JSON directly with jq:

C=/mnt/evidence/var/lib/docker/containers
for d in "$C"/*/; do
  jq -r '[.ID[0:12], .Name, .Config.Image, .Created, .State.StartedAt,
          (.Config.Cmd // [] | join(" "))] | @tsv' "$d/config.v2.json"
done
jq '{Privileged, PidMode, NetworkMode, CapAdd, SecurityOpt, Binds}' \
  "$C"/<id>/hostconfig.json

Config.Env often contains credentials passed as environment variables; treat them as compromised if the container was.

Container logs

The json-file driver writes one object per line, with UTC timestamps in RFC 3339 format:

{"log":"GET /upload.php?cmd=id HTTP/1.1\" 200 12\n","stream":"stdout","time":"2026-09-14T10:22:31.402114Z"}

By default these logs are not size-limited, but many deployments set max-size and max-file in daemon.json, which rotates older content into <id>-json.log.1 and so on. With the local driver, logs are in a compressed binary format; with journald, query the host journal:

journalctl --directory=/mnt/evidence/var/log/journal CONTAINER_NAME=webapp -o json
journalctl --directory=/mnt/evidence/var/log/journal -u docker.service -u containerd.service

The daemon's own unit logs record container start, stop, OOM kills, image pull errors and API problems, which helps reconstruct a lifecycle when containers have been removed. See the systemd journal for offline querying.

overlay2: what changed inside the container

Docker's overlay2 driver builds each container filesystem with OverlayFS: read-only image layers (lowerdir), one writable layer (upperdir), and a unified merged view that exists only while the container is running.

MID=$(cat /mnt/evidence/var/lib/docker/image/overlay2/layerdb/mounts/<id>/mount-id)
ls -la /mnt/evidence/var/lib/docker/overlay2/$MID/
# diff/  link  lower  work/   (merged/ only while running)
find /mnt/evidence/var/lib/docker/overlay2/$MID/diff -type f -newermt '2026-09-14' -ls

The diff directory of the container layer is the single most valuable artifact: it contains only files written or modified since the container was created, such as a web shell, a downloaded miner or an edited config. Deleted files appear as character devices with device number 0/0 (whiteouts). Files keep their inode timestamps, so they can go straight into a super timeline; Plaso also ships a Docker JSON parser for container logs and configs.

On a live host, docker diff <id> summarises the upper layer:

C /var/www/html
A /var/www/html/uploads/.cache.php
A /tmp/kx
D /var/log/nginx/access.log

A is added, C changed, D deleted.

export and commit caveats

docker export produces a tarball of the merged filesystem, but it excludes volumes and discards layering, so you lose the ability to tell image content from attacker content. docker commit pauses the container by default and creates a new image on the suspect host, which modifies evidence. Prefer copying the upper directory and the container directory from a disk image, and use export only as a convenience copy with hashes recorded.

Timestamp pitfalls

Container metadata and json-file logs use UTC, while the application inside the container may log in whatever time zone its image defines, and the host's syslog files may use local time. Record each source's zone before merging. Also remember that the upper layer shows what was written, not who wrote it: a file in diff may have been created by the application during normal operation, so correlate it with log lines and process evidence before attributing it to the attacker.

containerd, CRI and Podman

containerd (used directly by Kubernetes and underneath Docker) stores persistent state in /var/lib/containerd: a bbolt metadata database at io.containerd.metadata.v1.bolt/meta.db and snapshot contents under io.containerd.snapshotter.v1.overlayfs/snapshots/<n>/fs. Runtime state lives in tmpfs under /run/containerd, including the OCI bundle config.json for each running task, so it only exists on a live system or in memory.

containerd isolates resources by namespace: Kubernetes uses k8s.io and Docker uses moby. On a live host:

ctr namespaces list
ctr -n k8s.io containers list
crictl ps -a                # includes exited containers
crictl inspect <container-id>
crictl logs <container-id>

Podman is daemonless. Rootful storage is /var/lib/containers/storage; rootless containers live in the user's home at ~/.local/share/containers/storage, with per-container metadata under overlay-containers/<id>/userdata/. Rootless containers are easy to miss because nothing under /var/lib references them, so sweep every home directory. Podman commonly logs container output and events to the journal, depending on its configuration.

Escape and host-compromise indicators

Most container escapes are misconfigurations rather than exploits. Review hostconfig.json (or the Kubernetes pod spec) for:

• "Privileged": true, or CapAdd including SYS_ADMIN, SYS_PTRACE or SYS_MODULE
• PidMode: "host" or NetworkMode: "host"
• SecurityOpt containing seccomp=unconfined or apparmor=unconfined
• Binds such as /:/host, /etc:/host-etc or /var/run/docker.sock:/var/run/docker.sock
• A restart policy of always on an unfamiliar container, which is a persistence mechanism in its own right

If any apply, extend the investigation to the host: new cron jobs, systemd units, SSH keys and kernel modules, as covered in hunting Linux persistence.

Kubernetes node notes

On a node, the kubelet writes container logs to /var/log/pods/<namespace>_<pod>_<uid>/<container>/0.log, and /var/log/containers/ holds symlinks to them with flat names. The format is the CRI log format:

2026-09-14T10:22:31.402114512Z stdout F uid=33(www-data) gid=33(www-data)

The kubelet itself logs to the journal (-u kubelet). The kubelet rotates these files by size, and deleted pods lose their log directory, so ship node logs centrally. API server audit logs, if enabled, are on the control plane, not on the worker.

Key takeaways

• Check daemon.json for a non-default data-root or log driver before hunting for files.
• config.v2.json and hostconfig.json answer who ran what, when, and with which privileges; read them with jq offline.
• The container's overlay2 diff directory isolates everything written at runtime.
• docker rm, --rm and pod rescheduling destroy evidence; preserve first.
• Privileged mode, host namespaces, host-root binds and a mounted Docker socket mean the host must be treated as compromised.
• Rootless Podman storage lives in home directories and is easy to miss.