standalone: run containers without a daemon (DOCKER_STANDALONE=1) - #7324
Open
ericcurtin wants to merge 4 commits into
Open
ericcurtin wants to merge 4 commits into
ericcurtin wants to merge 4 commits into
Conversation
Adds github.com/containerd/containerd/v2 and its dependencies, which the
standalone (daemonless) backend added in a following commit uses to store
images, unpack them into snapshots and supervise containers:
github.com/containerd/containerd/api v1.12.0
github.com/containerd/containerd/v2 v2.4.0
github.com/containerd/fifo v1.1.0
github.com/containerd/go-cni v1.1.14
github.com/containerd/ttrpc v1.2.9
github.com/containerd/typeurl/v2 v2.3.0
github.com/opencontainers/runtime-spec v1.3.0
go.etcd.io/bbolt v1.5.0
containerd 2.4 requires go 1.26.6, so the 'go' directive is updated
accordingly; the Dockerfiles already build with go 1.26.8.
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
"docker image inspect --format" uses the raw API response so that templates
can refer to the JSON field names, and falls back to the typed value when no
raw response is available. The raw response is only produced by the HTTP
client, so with any other client.APIClient implementation (the standalone
backend added in a following commit, or a fake in tests) templates such as
"{{.Id}}" failed with a template error instead of using the field.
Marshal the typed response in that case, which is what the template expects.
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
Setting DOCKER_STANDALONE=1 makes the CLI run containers itself, with no
Docker Engine and no containerd daemon. The CLI embeds the containerd
libraries and drives an OCI runtime (runc or crun) directly, both rootful
and rootless.
The backend is an implementation of client.APIClient, injected with
command.WithInitializeClient, so every existing command, flag and output
format keeps working unchanged for the supported subset.
Without a daemon, the two jobs a daemon does are handled per container:
- State lives in a containerd metadata database (bbolt) next to the
content store, snapshots and volumes. A session is opened for the
duration of an operation, so consecutive commands can each take the
database lock. Sessions are reference-counted because containerd's shim
manager calls back into the container store while an operation holds
one, and the lock is not re-entrant.
- Supervision uses containerd's own per-container shim,
containerd-shim-runc-v2, started through core/runtime/v2. It daemonizes
itself, so containers survive the CLI exiting and later commands
reconnect to it. The manager that started a container is kept for
waiting on it: rediscovering shims goes through containerd's loader,
which reaps the shims of containers that already exited and discards
their exit status.
- Logs, attach and exit status are handled by a helper process per
container (this binary re-executed in a logging mode). It writes a
json-file log, serves attach clients, forwards their input to the
container's stdin and records the exit status from the shim's TaskExit
event. Container start waits for an attaching client to be registered,
so "docker run" never misses output, and stdin EOF is propagated with
the task's CloseIO because the shim holds its own writer on the FIFO.
Rootless mode re-executes the CLI in a new user namespace, mapping the
caller's /etc/subuid and /etc/subgid ranges with newuidmap/newgidmap, and in
a new mount namespace so image layers can be mounted. /run is replaced by a
tmpfs with the host's entries bound into it, with the engine's shim socket
directory bound at /run/containerd/s, which shims older than containerd 2.4
hardcode. Containers are networked with pasta (slirp4netns as a fallback),
and get a cgroup when the user's systemd instance can create transient
units. Rootful mode uses the CNI bridge when the plugins are installed.
Commands that cannot work without a daemon or a builder (build, swarm,
plugins, checkpoints, events) report that they are not supported, rather
than failing to connect.
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
Describes what DOCKER_STANDALONE=1 does, how state and supervision work without a daemon, the requirements, the rootful and rootless differences, the configuration environment variables, the supported commands, the limitations and the common failure modes. Signed-off-by: Eric Curtin <eric.curtin@docker.com>
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
|
|
||
| // killTask sends a signal to the container's init process (or all processes). | ||
| func killTask(ctx context.Context, task runtime.Task, sig int, all bool) error { | ||
| err := task.Kill(nsCtx(ctx), uint32(sig), all) |
docker-agent
left a comment
Contributor
There was a problem hiding this comment.
docker-agent to retry.
Contributor
Author
|
Treat this as an RFC, it's not as much code as it looks, some of it is vendoring, some advantages:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a daemonless mode to the CLI: with
DOCKER_STANDALONE=1,dockerruns containers itself, using the containerd libraries and an OCI runtime directly. Nodockerdand nocontainerddaemon are involved. Both rootful and rootless are supported.With the variable unset nothing changes: the CLI talks to the Engine exactly as before.
Why
The CLI already contains everything needed to describe containers; what it lacks is something to run them. containerd's libraries (content store, metadata store, snapshotters, image puller, OCI spec generation and the runtime v2 shim) are usable in-process, so a single static binary can be a complete container engine. That makes
dockeruseful where a daemon is unwanted or unavailable: CI runners, containers, minimal images, and rootless setups where no system service can be installed.How
The backend is an implementation of
client.APIClientinjected withcommand.WithInitializeClient, so every existing command, flag and output format keeps working unchanged for the supported subset. No command code was modified.Without a daemon, the two jobs a daemon does are handled per container:
containerd-shim-runc-v2, started throughcore/runtime/v2. It daemonizes itself, so containers survive the CLI exiting and later commands reconnect to it. The manager that started a container is remembered for waiting on it, because rediscovering shims goes through containerd's loader, which reaps shims of already-exited containers and discards their exit status.json-filelog, serves attach clients, forwards their input to the container's stdin, and records the exit status from the shim'sTaskExitevent. Container start waits until an attaching client is registered sodocker runnever drops output, and stdin EOF is propagated with the task'sCloseIObecause the shim holds its own writer on the stdin FIFO.Nothing runs when no container is running; a running container has two helper processes (shim + logger), plus a network helper when a netns is used.
Rootless
The CLI re-executes itself in a new user namespace, mapping the caller's
/etc/subuid//etc/subgidranges withnewuidmap/newgidmap, plus a new mount namespace so image layers can be mounted./runis replaced by a tmpfs with the host's entries bound into it and the engine's shim socket directory bound at/run/containerd/s, which shims older than containerd 2.4 hardcode. Containers are networked withpasta(slirp4netnsas a fallback) and get a cgroup when the user's systemd instance can create transient units, so--memory,--cpus,--pids-limitanddocker pausework in a systemd user session.Rootful uses the CNI bridge when the plugins are installed, else pasta, and overlayfs (native as a fallback).
Testing
A 49-case matrix covering output and exit codes, stdin/
-i/-t, env/workdir/user/hostname/entrypoint, read-only rootfs, cgroup limits, capabilities, DNS and egress, all network modes, bind mounts, named/anonymous volumes, tmpfs, the detached lifecycle (ps/logs/exec/top/inspect/pause/rename/stop/restart/kill/rm), images (ls/tag/inspect/history/rmi), volumes, networks andinfopasses 49/49 rootful and 49/49 rootless from fresh state, with no leaked helper processes.Unit tests cover the pure helpers (reference handling, registry auth decoding, spec/signal/name helpers, mount and volume resolution, port mapping, pasta arguments, resolv.conf/hosts generation, metadata round-trip, status formatting).
golangci-lint runis clean, the fullgo test ./...passes, andgo mod tidy && go mod vendoris idempotent.Notes for reviewers
vendor/); the new code is ~8.6k lines underinternal/standalone/, of which ~440 are generated "not supported" stubs for theclient.APIClientmethods that do not apply, and ~900 are tests.go 1.26.6, so thegodirective moved from1.26.0; the Dockerfiles already use 1.26.8.binary://log URI (so FIFO I/O is used), and it keeps its own writer on the stdin FIFO (so EOF needsCloseIO).docker image inspect --formatnow falls back to marshalling the typed response when a client provides no raw response (separate commit; it also fixes templates against fake clients in tests).docker buildis out of scope (it needs BuildKit). Swarm, plugins, checkpoints, events and user-defined networks report that they are not supported.cp/commit/export/save/loadand restart policies are not implemented yet; all limitations are listed indocs/standalone.md.