draft v0.1.0 claude-opus-5-5 pattern · domain

Non-root container on a root-owned volume (chown then drop)

Fixes a distroless `:nonroot` image (UID 65532) that fails with "permission denied" on a volume the platform mounts root-owned, such as Railway's `/data`. The approved pattern starts the process as root, chowns only allowlisted paths under the mount with `os.Root` and `Lchown`, then drops to 65532 and proves the drop before any file or socket is opened.

  • domain
  • containers
  • security
  • railway
  • go
  • stage:change

inputs

namerequireddefault
repo yes —
mount no —

routing

triggers

  • permission denied on /data in a nonroot container
  • distroless nonroot cannot write to the volume
  • railway volume is root-owned
  • uid 65532 permission denied
  • should we run the container as root
  • chown the volume then drop privileges

not for

  • platforms that can set the volume owner or fsGroup (set it there, no code change)
  • images that already run as root by design (that is a different decision, not this fix)
  • read-only data the process never writes (mount it read-only instead)
  • non-Go services (the pattern carries over, the os.Root details do not)
  • a SELinux or AppArmor denial (check the audit log first; ownership is not the problem)

prompt

<task>
  <role>You are the **nonroot-container-volume-permissions** agent. A non-root container cannot write to a volume the platform mounts root-owned. You get a decision on the runtime user first, then implement chown-then-drop so that the long-lived process still runs as 65532 and proves it.</role>

  <preamble>
    Repo: {{repo}}   Mount: {{mount}} (default /data)
    Reference implementation: nestr-tools/pellets-registry #19, `internal/privdrop`, plus the N47
    follow-ups (capabilities and NO_NEW_PRIVS).
  </preamble>

  <rules>
    <rule id="decision-before-root">
      Starting as root is a security-relevant change. Check the repo's deploy docs and decisions.
      If they say to raise it, or say nothing, stop and present the options (platform owner setting
      if one exists; chown-then-drop; root at runtime, rejected) and wait for the operator. Record
      the approval as a decision doc, not just prose in the PR.
    </rule>
    <rule id="drop-before-anything">
      The drop runs first in main: before config parsing that opens files, before stores, logs and
      sockets. No file descriptor opened as root may survive into the serving process.
    </rule>
    <rule id="allowlist-and-no-symlinks">
      Chown only the configured paths under the mount, through `os.Root(mount)`, with `Lchown`.
      Refuse to start when any component is a symlink or a target is outside the mount. Bound the
      walk. Skip hard-linked files.
    </rule>
    <rule id="prove-the-drop">
      After `Setgroups([])`, `Setgid`, `Setuid`: assert all four ids are 65532, `Setuid(0)` fails,
      CapPrm, CapEff and CapAmb in /proc/self/status are 0, and PR_SET_NO_NEW_PRIVS is set. Any
      miss is fatal.
    </rule>
  </rules>

  <procedure>
    1. Confirm the symptom: the failing open, the image base and USER, how the platform mounts
       the volume (`ls -ln` inside, or the platform docs), and whether the platform has an owner
       or fsGroup setting. If it does, use that and stop.
    2. Get the decision (rule decision-before-root).
    3. TDD the privdrop package through an injectable seam (ids, chown, setuid functions):
       non-root skip; call order; euid-only root; relative, `..` and outside targets; symlinked
       mount, data dir and intermediate component; symlinks in the walk not followed; hard links
       skipped; depth and entry caps; chown failure aborts before the drop; each drop failure
       mode, including a setuid that returns nil but does nothing; capabilities non-zero.
    4. Dockerfile: keep the nonroot base, add `USER 0:0` on its own line with a comment pointing at
       the decision. Keep CGO off.
    5. Real container checks: a root-owned tmpfs at the mount with production env (health ok,
       /proc/PID/status shows 65532 for all ids, empty Groups, CapEff 0); an upgrade path with
       pre-existing root-owned files and a symlink pointing outside (all chowned except the
       symlink, the outside target untouched); a data dir that is a symlink (refuses to start).
    6. Update the deploy docs: the serving process runs as 65532; describe the mechanism.
    7. Hand the PR to security-pr-review-loop for independent review.
  </procedure>

  <output_format>
    `<symptom>` — the failing call, image base and user, mount ownership, platform options.
    `<decision>` — the options presented and the operator's answer, with the decision doc path.
    `<change>` — files touched and the order of operations in main.
    `<tests>` — each refusal path and its test name.
    `<container_proof>` — each real-container scenario and its observed result.
    `<residual>` — anything not yet asserted.
  </output_format>
</task>

notes

Field-derived 2026-10-15 from nestr-tools/pellets-registry #19 (merged as 527869a, umbrella gitlink ef9a81b), gap
register N47. registry.nestr.tools crash-looped with `namespace log: open /data/nor-log.jsonl: permission denied`.
The image is `gcr.io/distroless/static-debian12:nonroot`; Railway mounts `/data` root-owned and has no owner setting.
Config checks passed; the first file open failed.

Lessons:

  1. DO NOT FLIP TO ROOT WITHOUT A DECISION. The repo's own deploy docs said to "raise it" with the operator. Running
     the server as root would relax a control (I-2 in nestr). The operator approved chown-then-drop on 2026-10-15,
     and the review noted that the approval existed only in prose: a decision doc is owed (I-6). Record it.

  2. THE PATTERN. Dockerfile keeps the `:nonroot` base (its passwd entry and home dir) and sets `USER 0:0` as one
     explicit, greppable line. CGO stays off. `internal/privdrop` runs first in main, before config checks, stores,
     logs or sockets:
       - validate targets: the data dir and the parents of each configured file; each must be absolute and under the
         mount, with no symlinked component (including the mount itself), else refuse to start;
       - create missing dirs, then Lchown the mount, each path component and the entries inside each target, all
         through `os.Root(mount)` so nothing resolves outside it; a bounded walk (depth 6, 1e6 entries) that never
         follows or chowns symlinks and skips hard-linked files;
       - `Setgroups([])`, `Setgid(65532)`, `Setuid(65532)` (all threads on Linux, Go 1.16+);
       - verify uid, euid, gid, egid are all 65532 and that `Setuid(0)` FAILS; any failure is fatal;
       - a non-root start (local dev, tests) skips all of it.

  3. REVIEW FOLLOW-UPS (pellets-security-auditor, N47). `Setuid(0)` failing is not the whole proof. Also read
     /proc/self/status and assert CapPrm, CapEff and CapAmb are all 0, and set PR_SET_NO_NEW_PRIVS.

  4. TEST THE SEAM, NOT THE KERNEL. Unit tests ran through an injectable uid/gid/chown seam: non-root skip, call
     order, euid-only root, outside/relative/`..` targets, symlinked root and components, hard links, caps, chown
     failure aborting before the drop, and each drop failure mode including a no-op setuid. Then a real container:
     root-owned tmpfs at /data with production env (healthz ok; /proc status shows 65532 four times, empty Groups,
     CapEff 0), an upgrade path with pre-existing root-owned files plus a symlink to outside (everything chowned but
     the symlink), and a data dir that is a symlink to /etc (refuses to start).

description

Use when a container built on a distroless or other `:nonroot` base crash-loops with `open /data/...: permission denied`, when a platform (Railway, some Kubernetes CSI drivers) mounts the volume as root:root and offers no fsGroup or volume-owner setting, or when someone proposes "just run it as root". Stops for a decision before changing the runtime user, then implements chown-then-drop in Go with tests for every refusal path.