Skip to main content
XenonStack

DevOps

XenonStack Blog Docker BuildKit: Architecture, Caching and Secrets

and | Published | Updated

Key takeaway

  • Docker BuildKit explained for platform leads: drivers, LLB, registry cache, secret mounts, rootless builds and Kubernetes, sourced from Docker docs.

Docker BuildKit is the builder backend that Docker uses to turn a Dockerfile into an image. According to the Docker docs, it is the default builder for Docker Desktop and Docker Engine users, and it improves build performance, storage management and extensibility compared with the legacy builder. The Docker Engine 23.0 release notes (1 February 2023) state that Buildx and BuildKit became the default builder on Linux. Windows container builds still use the legacy builder, and BuildKit support for Windows containers is marked experimental.

For CTOs, platform leads and solution architects, the useful questions are practical: which builder driver to run in CI, where to keep the build cache so every runner can reuse it, how to pass credentials without leaving them in an image, and whether builds can run without root. This guide answers those from the Docker and BuildKit documentation. For the runtime side, see our guide to Docker container architecture and monitoring.

What is Docker BuildKit and why does it matter?

BuildKit is a toolkit for converting source code into build artefacts efficiently, expressively and repeatably (moby/buildkit README). It is part of the Moby project. The Docker docs list these improvements over the legacy builder:

Two-column diagram: left lists slow rebuilds, credentials in images, daemon privilege and where to build; right lists registry cache, secret mounts, rootless mode and builder drivers.
  • Unused build stages are skipped and independent stages run in parallel.
  • Only changed files in the build context are transferred between builds, and unused context files are skipped.
  • A fully concurrent build graph solver drives the build.
  • Build cache can be exported to a registry and pulled on demand by later builds on any host.
  • Intermediate images and containers are not left behind as side effects, and build cache is prioritised for automatic pruning.

The project README also lists automatic garbage collection, extensible frontend formats, nested build invocations, distributable workers, multiple output formats, a pluggable architecture and execution without root privileges.

How is BuildKit structured?

BuildKit runs as a daemon, buildkitd, which clients such as buildctl or Docker Buildx talk to. The daemon supports two worker backends: OCI (runc, the default, with crun also possible) and containerd, which is enabled with --oci-worker=false --containerd-worker=true. Garbage collection is configured through buildkitd.toml.

What is LLB?

LLB (low-level build) is BuildKit's intermediate binary format. It defines a content-addressable dependency graph, marshalled as Protobuf. Caching is based on checksums of the build graph and mounted content, not on image comparison heuristics, which is why cache hits are predictable. It also supports features that Dockerfiles do not expose, such as direct data mounting and nested invocation. Most teams never write LLB directly.

What is a frontend?

A frontend converts a human-readable build format into LLB. The Dockerfile frontend is the common one: Dockerfiles are built using an external Dockerfile frontend that can be distributed as an image, so a team can pin a specific frontend version. The README notes that other languages also have LLB implementations, including Buildpacks, HLB, Earthfile and Nix.

Which builder driver should you use?

A builder is a BuildKit daemon that runs your builds. Docker Engine creates a builder named default that uses the BuildKit library bundled with the daemon. Buildx adds three more drivers. The table summarises the Docker docs on drivers.

DriverRuns BuildKit asMulti-platform buildsUse it when
dockerBuildKit library inside the Docker daemonNo (per the docs table)You want simplicity and no advanced features
docker-containerA dedicated BuildKit containerYesYou need multi-platform builds, tarball output or a custom BuildKit configuration
kubernetesBuildKit pods in a Kubernetes clusterYesBuilds should run inside a Kubernetes cluster
remoteA BuildKit daemon you manage yourselfYesYou already operate your own BuildKit daemon

With the three non-default drivers, results stay in the build cache unless you specify an output, so add --load to get the image into the local store. Buildx 0.14.0 and later also support a default-load=true driver option. Note that docker build is an alias for docker buildx build but always uses the default builder unless you pass --builder or set BUILDX_BUILDER.

How do you make builds faster with caching?

Start with the Dockerfile itself. The Docker docs advise placing expensive, rarely changing steps near the top and frequently changing steps near the bottom: copy dependency manifests first, install dependencies, then copy the rest of the source, so a source edit does not rebuild the dependency layer. Add a .dockerignore file to exclude files from the build context. Bind mounts (RUN --mount=type=bind) make files available to a step without adding them to the image or the cache, and cache mounts provide a persistent package cache so rebuilt layers download only new or changed packages.

Where should the cache live?

Caches must be exported with --cache-to and imported with --cache-from. The Docker docs list these backends.

BackendWhat it doesStatus on the docs page
inlineEmbeds cache in the image; works only with the image exporterNo label
registryStores cache as a separate image, for example type=registry,ref=<registry>/<cache-image>No label
localWrites cache to a local directoryNo label
ghaUses the GitHub Actions cacheBeta
s3Uses an AWS S3 bucketListed as unreleased
azblobUses Azure Blob StorageListed as unreleased

Set mode on --cache-to: the default min caches only layers that end up in the image, while mode=max also caches intermediate layers, which usually gives more cache hits but a larger cache. It is supported by every backend except inline. The default docker driver supports inline, local, registry and gha only when the containerd image store is enabled. Confirm the current status of the s3 and azblob backends on the Docker page before relying on them.

How do you keep secrets out of images?

Build arguments and environment variables persist in the final image, so Docker advises against using them for passwords or tokens. Use a secret mount instead. Pass the secret from the client with docker build --secret id=aws,src=$HOME/.aws/credentials . (or env= to read an environment variable), and consume it in the Dockerfile with RUN --mount=type=secret,id=aws. By default the secret appears as a file at /run/secrets/<id>; target= changes the path and env= exposes it as an environment variable. The secret exists only for the duration of that build instruction.

For private Git repositories, use an SSH mount: docker buildx build --ssh default .. BuildKit also supports the predefined secrets GIT_AUTH_TOKEN and GIT_AUTH_HEADER for private remote Git contexts. These are not compatible with GitLab's CI_JOB_TOKEN, which needs Basic authentication with the username gitlab-ci-token. For wider pipeline controls, see our DevSecOps checklist.

Can BuildKit run without root, and what about Kubernetes?

Yes. The BuildKit rootless documentation describes running buildkitd under RootlessKit, and the project publishes a moby/buildkit:rootless image. Requirements and limits from that page:

  • RootlessKit must be installed for both the OCI and containerd workers.
  • The overlayfs snapshotter needs kernel 5.11 or later (or an Ubuntu kernel). On kernel 4.18 or later BuildKit uses fuse-overlayfs; on older kernels it uses native.
  • Network mode is always network.host.
  • The host setting max_user_namespaces must be non-zero, and on Ubuntu 24.04 or later kernel.apparmor_restrict_unprivileged_userns=0 may be required.
  • The container image needs relaxed seccomp, AppArmor and systempaths settings, or --privileged.

For Kubernetes, the repository provides manifests under examples/kubernetes. Those examples use --oci-worker-no-process-sandbox, which the rootless page discourages because build containers can then kill arbitrary processes in the daemon container. Treat that as a risk to review with your security team. Compared with mounting the Docker socket into a build pod, a dedicated, rootless BuildKit daemon, or the Buildx kubernetes driver, gives a narrower privilege surface, but you still need to test it against your cluster policy. Our guide to Kubernetes architecture and components covers the cluster side, and deploying PostgreSQL on Kubernetes shows a containerised workload built this way.

Where does BuildKit fit in an agentic delivery pipeline?

Build pipelines are increasingly triggered and tuned by software agents as well as people. Whatever acts on the pipeline needs the same controls as a person would: scoped secrets, an auditable build record and policy on what may be pushed. ElixirClaw is described on its site as an agentic OS platform to build, orchestrate and govern AI agents and automation workflows while keeping humans in control, and AgentGRC is described as keeping autonomous agent actions inside policy, with the audit trail risk teams need.

Which decisions should you take first?

DecisionSuggested defaultWhen not to
Builder driverdocker-container for CI that needs multi-platform output or cache exportThe default docker driver is enough for simple local builds
Cache backendregistry with mode=max for shared runnersCheck size and registry limits before using mode=max
CredentialsSecret and SSH mounts, never ARG or ENVNone; build arguments persist in the image
PrivilegeRootless daemon where kernel and cluster allowHosts that cannot meet the kernel and user namespace requirements
Windows containersLegacy builder, or test the experimental support firstProduction builds that cannot tolerate experimental features

To map one build or delivery decision your agents should make, and the context and policy it depends on, book a Context Workshop. See what happens in the workshop. The ElixirData Context OS keeps governed context, versioned policy and decision traces in one decision layer above AI agents.

Frequently asked questions

Is BuildKit the default builder in Docker?

Yes. The Docker docs say BuildKit is the default builder for Docker Desktop and Docker Engine users, and the Engine 23.0 release notes set Buildx and BuildKit as the default on Linux. Windows container builds still use the legacy builder.

What is the difference between BuildKit and Buildx?

BuildKit is the build engine. Buildx is the Docker CLI plugin that manages builders and drivers and runs builds on them, including docker-container, kubernetes and remote builders.

How do I pass a secret to a Docker build safely?

Pass it with --secret and read it in a RUN --mount=type=secret instruction. It is available only during that instruction and is not stored in the image, unlike ARG and ENV values.

Can BuildKit share cache between CI runners?

Yes. Export the cache with --cache-to, for example to a registry, and import it with --cache-from on any host. Use mode=max to include intermediate layers.

Does BuildKit run without root?

Yes, through RootlessKit and the moby/buildkit:rootless image, subject to the kernel, user namespace and network limits listed on the BuildKit rootless page.

Follow XenonStack on LinkedIn: XenonStack on LinkedIn for new guides on agentic AI, data and platform engineering.

Table of Contents

navdeep-singh-gill

Navdeep Singh Gill

Global CEO and Founder of XenonStack

LinkedIn profile

Navdeep Singh Gill is serving as Chief Executive Officer and Product Architect at XenonStack. He holds expertise in building SaaS Platform for Decentralised Big Data management and Governance, AI Marketplace for Operationalising and Scaling. His incredible experience in AI Technologies and Big Data Engineering thrills him to write about different use cases and its approach to solutions.

Get the latest articles in your inbox

Subscribe Now

Interested in solving your challenges with the XenonStack team?
Get Started

Tell us about you and your priorities so we can prepare the right conversation. It takes about 2 minutes.

Please Select your Industry
Banking
Fintech
Payment Providers
Wealth Management
Discrete Manufacturing
Semiconductor
Machinery Manufacturing / Automation
Appliances / Electrical / Electronics
Elevator Manufacturing
Defense & Space Manufacturing
Computers & Electronics / Industrial Machinery
Motor Vehicle Manufacturing
Food and Beverages
Distillery & Wines
Beverages
Shipping
Logistics
Mobility (EV / Public Transport)
Energy & Utilities
Hospitality
Digital Gaming Platforms
SportsTech with AI
Public Safety - Explosives
Public Safety - Firefighting
Public Safety - Surveillance
Public Safety - Others
Media Platforms
City Operations
Airlines & Aviation
Defense Warfare & Drones
Robotics Engineering
Drones Manufacturing
AI Labs for Colleges
AI MSP / Quantum / AGI Institutes
Retail Apparel and Fashion

Continue