Key takeaway
- Docker BuildKit explained for platform leads: drivers, LLB, registry cache, secret mounts, rootless builds and Kubernetes, sourced from Docker docs.
Docker BuildKit is the builder backend that Docker uses to turn a Dockerfile into an image. According to the Docker docs, it is the default builder for Docker Desktop and Docker Engine users, and it improves build performance, storage management and extensibility compared with the legacy builder. The Docker Engine 23.0 release notes (1 February 2023) state that Buildx and BuildKit became the default builder on Linux. Windows container builds still use the legacy builder, and BuildKit support for Windows containers is marked experimental.
For CTOs, platform leads and solution architects, the useful questions are practical: which builder driver to run in CI, where to keep the build cache so every runner can reuse it, how to pass credentials without leaving them in an image, and whether builds can run without root. This guide answers those from the Docker and BuildKit documentation. For the runtime side, see our guide to Docker container architecture and monitoring.
What is Docker BuildKit and why does it matter?
BuildKit is a toolkit for converting source code into build artefacts efficiently, expressively and repeatably (moby/buildkit README). It is part of the Moby project. The Docker docs list these improvements over the legacy builder:
- Unused build stages are skipped and independent stages run in parallel.
- Only changed files in the build context are transferred between builds, and unused context files are skipped.
- A fully concurrent build graph solver drives the build.
- Build cache can be exported to a registry and pulled on demand by later builds on any host.
- Intermediate images and containers are not left behind as side effects, and build cache is prioritised for automatic pruning.
The project README also lists automatic garbage collection, extensible frontend formats, nested build invocations, distributable workers, multiple output formats, a pluggable architecture and execution without root privileges.
How is BuildKit structured?
BuildKit runs as a daemon, buildkitd, which clients such as buildctl or Docker Buildx talk to. The daemon supports two worker backends: OCI (runc, the default, with crun also possible) and containerd, which is enabled with --oci-worker=false --containerd-worker=true. Garbage collection is configured through buildkitd.toml.
What is LLB?
LLB (low-level build) is BuildKit's intermediate binary format. It defines a content-addressable dependency graph, marshalled as Protobuf. Caching is based on checksums of the build graph and mounted content, not on image comparison heuristics, which is why cache hits are predictable. It also supports features that Dockerfiles do not expose, such as direct data mounting and nested invocation. Most teams never write LLB directly.
What is a frontend?
A frontend converts a human-readable build format into LLB. The Dockerfile frontend is the common one: Dockerfiles are built using an external Dockerfile frontend that can be distributed as an image, so a team can pin a specific frontend version. The README notes that other languages also have LLB implementations, including Buildpacks, HLB, Earthfile and Nix.
Which builder driver should you use?
A builder is a BuildKit daemon that runs your builds. Docker Engine creates a builder named default that uses the BuildKit library bundled with the daemon. Buildx adds three more drivers. The table summarises the Docker docs on drivers.
| Driver | Runs BuildKit as | Multi-platform builds | Use it when |
|---|---|---|---|
docker | BuildKit library inside the Docker daemon | No (per the docs table) | You want simplicity and no advanced features |
docker-container | A dedicated BuildKit container | Yes | You need multi-platform builds, tarball output or a custom BuildKit configuration |
kubernetes | BuildKit pods in a Kubernetes cluster | Yes | Builds should run inside a Kubernetes cluster |
remote | A BuildKit daemon you manage yourself | Yes | You already operate your own BuildKit daemon |
With the three non-default drivers, results stay in the build cache unless you specify an output, so add --load to get the image into the local store. Buildx 0.14.0 and later also support a default-load=true driver option. Note that docker build is an alias for docker buildx build but always uses the default builder unless you pass --builder or set BUILDX_BUILDER.
How do you make builds faster with caching?
Start with the Dockerfile itself. The Docker docs advise placing expensive, rarely changing steps near the top and frequently changing steps near the bottom: copy dependency manifests first, install dependencies, then copy the rest of the source, so a source edit does not rebuild the dependency layer. Add a .dockerignore file to exclude files from the build context. Bind mounts (RUN --mount=type=bind) make files available to a step without adding them to the image or the cache, and cache mounts provide a persistent package cache so rebuilt layers download only new or changed packages.
Where should the cache live?
Caches must be exported with --cache-to and imported with --cache-from. The Docker docs list these backends.
| Backend | What it does | Status on the docs page |
|---|---|---|
inline | Embeds cache in the image; works only with the image exporter | No label |
registry | Stores cache as a separate image, for example type=registry,ref=<registry>/<cache-image> | No label |
local | Writes cache to a local directory | No label |
gha | Uses the GitHub Actions cache | Beta |
s3 | Uses an AWS S3 bucket | Listed as unreleased |
azblob | Uses Azure Blob Storage | Listed as unreleased |
Set mode on --cache-to: the default min caches only layers that end up in the image, while mode=max also caches intermediate layers, which usually gives more cache hits but a larger cache. It is supported by every backend except inline. The default docker driver supports inline, local, registry and gha only when the containerd image store is enabled. Confirm the current status of the s3 and azblob backends on the Docker page before relying on them.
How do you keep secrets out of images?
Build arguments and environment variables persist in the final image, so Docker advises against using them for passwords or tokens. Use a secret mount instead. Pass the secret from the client with docker build --secret id=aws,src=$HOME/.aws/credentials . (or env= to read an environment variable), and consume it in the Dockerfile with RUN --mount=type=secret,id=aws. By default the secret appears as a file at /run/secrets/<id>; target= changes the path and env= exposes it as an environment variable. The secret exists only for the duration of that build instruction.
For private Git repositories, use an SSH mount: docker buildx build --ssh default .. BuildKit also supports the predefined secrets GIT_AUTH_TOKEN and GIT_AUTH_HEADER for private remote Git contexts. These are not compatible with GitLab's CI_JOB_TOKEN, which needs Basic authentication with the username gitlab-ci-token. For wider pipeline controls, see our DevSecOps checklist.
Can BuildKit run without root, and what about Kubernetes?
Yes. The BuildKit rootless documentation describes running buildkitd under RootlessKit, and the project publishes a moby/buildkit:rootless image. Requirements and limits from that page:
- RootlessKit must be installed for both the OCI and containerd workers.
- The
overlayfssnapshotter needs kernel 5.11 or later (or an Ubuntu kernel). On kernel 4.18 or later BuildKit usesfuse-overlayfs; on older kernels it usesnative. - Network mode is always
network.host. - The host setting
max_user_namespacesmust be non-zero, and on Ubuntu 24.04 or laterkernel.apparmor_restrict_unprivileged_userns=0may be required. - The container image needs relaxed seccomp, AppArmor and systempaths settings, or
--privileged.
For Kubernetes, the repository provides manifests under examples/kubernetes. Those examples use --oci-worker-no-process-sandbox, which the rootless page discourages because build containers can then kill arbitrary processes in the daemon container. Treat that as a risk to review with your security team. Compared with mounting the Docker socket into a build pod, a dedicated, rootless BuildKit daemon, or the Buildx kubernetes driver, gives a narrower privilege surface, but you still need to test it against your cluster policy. Our guide to Kubernetes architecture and components covers the cluster side, and deploying PostgreSQL on Kubernetes shows a containerised workload built this way.
Where does BuildKit fit in an agentic delivery pipeline?
Build pipelines are increasingly triggered and tuned by software agents as well as people. Whatever acts on the pipeline needs the same controls as a person would: scoped secrets, an auditable build record and policy on what may be pushed. ElixirClaw is described on its site as an agentic OS platform to build, orchestrate and govern AI agents and automation workflows while keeping humans in control, and AgentGRC is described as keeping autonomous agent actions inside policy, with the audit trail risk teams need.
Which decisions should you take first?
| Decision | Suggested default | When not to |
|---|---|---|
| Builder driver | docker-container for CI that needs multi-platform output or cache export | The default docker driver is enough for simple local builds |
| Cache backend | registry with mode=max for shared runners | Check size and registry limits before using mode=max |
| Credentials | Secret and SSH mounts, never ARG or ENV | None; build arguments persist in the image |
| Privilege | Rootless daemon where kernel and cluster allow | Hosts that cannot meet the kernel and user namespace requirements |
| Windows containers | Legacy builder, or test the experimental support first | Production builds that cannot tolerate experimental features |
To map one build or delivery decision your agents should make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop. The ElixirData Context OS keeps governed context, versioned policy and decision traces in one decision layer above AI agents.
Frequently asked questions
Is BuildKit the default builder in Docker?
Yes. The Docker docs say BuildKit is the default builder for Docker Desktop and Docker Engine users, and the Engine 23.0 release notes set Buildx and BuildKit as the default on Linux. Windows container builds still use the legacy builder.
What is the difference between BuildKit and Buildx?
BuildKit is the build engine. Buildx is the Docker CLI plugin that manages builders and drivers and runs builds on them, including docker-container, kubernetes and remote builders.
How do I pass a secret to a Docker build safely?
Pass it with --secret and read it in a RUN --mount=type=secret instruction. It is available only during that instruction and is not stored in the image, unlike ARG and ENV values.
Can BuildKit share cache between CI runners?
Yes. Export the cache with --cache-to, for example to a registry, and import it with --cache-from on any host. Use mode=max to include intermediate layers.
Does BuildKit run without root?
Yes, through RootlessKit and the moby/buildkit:rootless image, subject to the kernel, user namespace and network limits listed on the BuildKit rootless page.
Follow XenonStack on LinkedIn: XenonStack on LinkedIn for new guides on agentic AI, data and platform engineering.
Explore the XenonStack platform family: ElixirData · ElixirClaw · ElixirHub · NexaStack · MetaSecure · Neural AI · AgentGRC · AgentSRE · AgentAnalyst · AgentInstruct · AgentRAI · AgentScrum · AgentSearch · AgentForce · AI Aviator · SAIF Aviator · CohortIQ · XenonStack AI · XenonStack UAE