Skip to content

ExecutionPersistenceLogs

Docker, containerd and Podman Artifacts on Linux Hosts

Container evidence on a Linux host: Docker config.v2.json and JSON logs, overlay2 upper layers, containerd snapshots, Podman storage and Kubernetes pod logs.

Location
/var/lib/docker/containers/<id>/
Proves
Which containers ran, from which image and command, with which privileges, what they printed and which files they changed
Timestamps
RFC 3339 UTC with nanoseconds in config and log JSON; filesystem times in overlay layers
Access
root (or docker group, which is root-equivalent); rootless Podman data is owned by the user
Retention
Until the container is removed (docker rm, pod deletion); logs unbounded unless max-size/max-file set
Collection
UAC, Velociraptor, tar, docker export

What it is

On a container host, the evidence of a compromised workload lives in the container engine's state directories: per-container configuration, captured stdout/stderr, and layered filesystems in which a container's writes are stored separately from its image. Docker, containerd (used by Kubernetes and underneath Docker) and Podman each keep their own layout. Orchestrators delete containers quickly, so these artifacts are among the first to disappear.

Where it lives

EnginePathContent
Docker/var/lib/docker/containers/<id>/config.v2.jsonName, image, command, environment, state, created/started/finished times
Docker/var/lib/docker/containers/<id>/hostconfig.jsonPrivileged, capabilities, binds, network and PID mode, restart policy
Docker/var/lib/docker/containers/<id>/<id>-json.logstdout/stderr with the default json-file driver (rotated as .1, .2 when limited)
Docker/var/lib/docker/image/overlay2/layerdb/mounts/<id>/mount-idMaps a container to its overlay2 directory
Docker/var/lib/docker/overlay2/<mount-id>/diffThe container's writable (upper) layer
Docker/etc/docker/daemon.jsondata-root, log-driver, log-opts overrides
containerd/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.dbbbolt metadata database (containers, images, snapshots per namespace)
containerd/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/<n>/fsSnapshot contents, including writable layers
containerd/run/containerd/Runtime state on tmpfs (live only)
Podman (rootful)/var/lib/containers/storage/Images, layers, overlay-containers/<id>/userdata/
Podman (rootless)~/.local/share/containers/storage/Same layout per user; easy to miss
Kubernetes node/var/log/pods/<ns>_<pod>_<uid>/<container>/0.log, symlinks in /var/log/containers/CRI-format container logs written by the kubelet

Paths are identical across distributions; check daemon.json for a relocated data-root. Newer Docker Engine releases may use the containerd image store, in which case layers live under /var/lib/containerd while /var/lib/docker/containers still holds configuration and logs.

What it proves

  • Which containers existed (including stopped ones not yet removed), from which image, with which entrypoint, command and environment (often including secrets).
  • Privileges that enable host compromise: Privileged, added capabilities, host PID/network namespaces, sensitive bind mounts such as /var/run/docker.sock or /.
  • What the application and any interactive shell printed, with per-line UTC timestamps.
  • Every file created, modified or deleted inside the container since it started, isolated in the upper layer (deletions appear as whiteout character devices).
  • Not proven: activity of containers already removed (their directories are deleted), or commands whose output was not written to stdout/stderr.

Key fields

FileFieldMeaning
config.v2.jsonCreated, State.StartedAt, State.FinishedAtLifecycle times
config.v2.jsonConfig.Image, Config.Cmd, Config.Entrypoint, Config.EnvWhat ran
hostconfig.jsonPrivileged, CapAdd, Binds, PidMode, NetworkMode, SecurityOpt, RestartPolicyIsolation and persistence
<id>-json.loglog, stream, timeOutput line, stdout/stderr, UTC time
CRI log line<time> <stream> <P/F> <message>Kubernetes format, P partial / F full line

Timestamps

Docker and CRI logs use RFC 3339 timestamps in UTC with nanosecond precision (2026-09-14T10:22:31.402114512Z). Lifecycle fields in config.v2.json use the same format. Files in overlay layers carry ordinary inode times of the host filesystem, so an upper-layer file's birth time tells you when the container wrote it.

Retention

  • Container directories and their logs are deleted with the container (docker rm, --rm containers on exit, pod eviction or deletion).
  • json-file logs grow without limit by default; max-size and max-file in daemon.json or per container rotate them away. The local driver stores compressed binary logs; the journald driver sends output to the systemd journal instead.
  • The kubelet rotates pod logs by size.
  • Runtime state under /run is lost at reboot.

Collection

On a live host, stop nothing until state is captured:

docker ps -a --no-trunc; docker inspect <id> > inspect.json; docker diff <id>
docker export <id> -o /media/ir/<id>.tar        # flattened container filesystem
crictl ps -a; crictl inspect <id>; ctr -n k8s.io containers list

Then collect /var/lib/docker/containers, the relevant overlay2 diff directories, /etc/docker, /var/lib/containerd metadata and /var/log/pods. UAC has artifacts for container engines and Velociraptor offers Docker inspection artifacts. For dead-box work, mount the image read-only and analyse offline.

Parsing

  • container-explorer reads containerd, Docker and Podman state from a mounted image without running a daemon: ce --image-root /mnt/evidence list containers, then mount, drift (changes versus the image) or export.
  • jq for Docker JSON: jq -r '[.Name,.Config.Image,.Created,(.Config.Cmd//[]|join(" "))]|@tsv' config.v2.json.
  • find <diff> -printf '%T+ %p\n' | sort for a quick timeline of the writable layer.

Investigator tips

  • Read hostconfig.json early: a privileged container or a mounted docker.sock means the host itself is in scope, so hunt for systemd units, cron jobs and SSH keys on the host.
  • Membership of the docker group is equivalent to root; check /etc/group for unexpected members.
  • A restart policy of always on an unfamiliar container is a persistence mechanism.
  • Sweep every home directory for rootless Podman storage; nothing under /var/lib points to it.
  • Container processes are visible from the host in /proc; /proc/<pid>/root leads into the container's filesystem while it runs.
  • Web shells in containerized apps show up in the upper layer and in web server logs written to stdout.

See also