Docker Bake and BuildKit represent the next generation of container build technology, offering powerful features for creating efficient, customizable, and maintainable Docker images. In this guide, we'll explore advanced build techniques that go beyond basic Dockerfiles, allowing you to define complex build configurations for different environments, optimize your builds, and streamline your development workflow.
Table of Contents #
- Introduction to Docker Bake and BuildKit
- Understanding BuildKit
- Docker Bake Explained
- Bake Configuration File Formats
- Multiple Build Targets and Inheritance
- Parameterization and Variables
- Practical Examples
- Integrating with CI/CD Pipelines
- Comparing HCL, JSON, and YAML Formats
- Best Practices for Complex Builds
- Cleanup
- Conclusion
Introduction to Docker Bake and BuildKit #
Docker Bake is a high-level build command that leverages BuildKit to offer advanced features for creating Docker images. This guide is based on our Lab7 Bake Example from the Docker Practical Guide repository.
Traditional docker build commands can become unwieldy when managing multiple build targets, environments, or configuration options. Docker Bake addresses these challenges by providing:
- A declarative approach to defining build configurations
- Support for multiple targets in a single file
- Variable substitution and parameterization
- Build inheritance and extension
- Integration with existing Docker commands and workflows
Understanding BuildKit #
Before diving into Bake, it's important to understand BuildKit, the modern build engine that powers it:
┌───────────────────────────────────────────────────────────┐
│ BuildKit │
│ │
│ ┌─────────────┐ ┌───────────────────────┐ │
│ │ │ │ │ │
│ │ Concurrent │ │ Content-Addressable │ │
│ │ Builds │ │ Cache │ │
│ │ │ │ │ │
│ └─────────────┘ └───────────────────────┘ │
│ │
│ ┌─────────────┐ ┌───────────────────────┐ │
│ │ │ │ │ │
│ │ Multi-Stage │ │ Build Secrets & │ │
│ │ Efficiency │ │ SSH Mounts │ │
│ │ │ │ │ │
│ └─────────────┘ └───────────────────────┘ │
│ │
│ ┌─────────────┐ ┌───────────────────────┐ │
│ │ │ │ │ │
│ │ Custom │ │ Multi-Platform │ │
│ │ Exporters │ │ Builds │ │
│ │ │ │ │ │
│ └─────────────┘ └───────────────────────┘ │
│ │
└───────────────────────────────────────────────────────────┘
BuildKit offers several advantages over the legacy Docker build system:
- Enhanced Performance: Parallel execution of build steps
- Improved Caching: More efficient layer caching based on content, not just commands
- Build Secrets: Safely use sensitive information during builds without embedding in layers
- Multi-Platform Support: Build for multiple architectures (arm64, amd64, etc.) from one command
- Output Options: Export build results to different formats (image, registry, tar, etc.)
To enable BuildKit, set the following environment variables:
export DOCKER_BUILDKIT=1
export COMPOSE_DOCKER_CLI_BUILD=1Docker Bake Explained #
Docker Bake is a command provided by Docker Buildx that allows you to define build configurations in a declarative way. Think of it as "Docker Compose for builds" - it lets you define a set of build targets and their relationships in a single file.
┌───────────────────────────────────────────────────────────┐
│ Docker Bake Process │
│ │
│ ┌───────────────┐ ┌─────────────────┐ │
│ │ │ │ │ │
│ │ Bake File │──────►│ Buildx Bake │ │
│ │ (.hcl/.json/ │ │ Command │ │
│ │ .yaml) │ │ │ │
│ │ │ └────────┬────────┘ │
│ └───────────────┘ │ │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────┐ │
│ │ │ │
│ │ BuildKit Engine │ │
│ │ │ │
│ └─────────────────┬────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌────────────┐ ┌────────────┐ ┌────────────┐ │
│ │ │ │ │ │ │ │
│ │ Target A │ │ Target B │ │ Target C │ │
│ │ │ │ │ │ │ │
│ └────────────┘ └────────────┘ └────────────┘ │
│ │
└───────────────────────────────────────────────────────────┘
The basic workflow for using Docker Bake is:
- Define your build configuration in a Bake file (HCL, JSON, or YAML format)
- Use the
docker buildx bakecommand to execute your builds - Access the resulting images through Docker
Bake Configuration File Formats #
Docker Bake supports three configuration file formats, giving you flexibility based on your preferences and ecosystem:
HCL (HashiCorp Configuration Language) #
# docker-bake.hcl
variable "TAG" {
default = "latest"
}
group "default" {
targets = ["app-dev", "app-prod"]
}
target "app-dev" {
dockerfile = "Dockerfile"
tags = ["myapp:${TAG}-dev"]
target = "development"
}
target "app-prod" {
dockerfile = "Dockerfile"
tags = ["myapp:${TAG}-prod"]
target = "production"
}JSON Format #
{
"variable": {
"TAG": {
"default": "latest"
}
},
"group": {
"default": {
"targets": ["app-dev", "app-prod"]
}
},
"target": {
"app-dev": {
"dockerfile": "Dockerfile",
"tags": ["myapp:${TAG}-dev"],
"target": "development"
},
"app-prod": {
"dockerfile": "Dockerfile",
"tags": ["myapp:${TAG}-prod"],
"target": "production"
}
}
}YAML Format #
variable:
TAG:
default: latest
group:
default:
targets:
- app-dev
- app-prod
target:
app-dev:
dockerfile: Dockerfile
tags:
- myapp:${TAG}-dev
target: development
app-prod:
dockerfile: Dockerfile
tags:
- myapp:${TAG}-prod
target: productionMultiple Build Targets and Inheritance #
One of the most powerful features of Docker Bake is the ability to define multiple targets that inherit from each other:
# Base target with common settings
target "common" {
context = "."
dockerfile = "Dockerfile"
platforms = ["linux/amd64", "linux/arm64"]
}
# Development target extends common
target "app-dev" {
inherits = ["common"]
target = "development"
tags = ["myapp:dev"]
args = {
NODE_ENV = "development"
}
}
# Production target extends common
target "app-prod" {
inherits = ["common"]
target = "production"
tags = ["myapp:prod"]
args = {
NODE_ENV = "production"
}
}This approach keeps your configuration DRY (Don't Repeat Yourself) and makes it easier to maintain consistent settings across targets.
Parameterization and Variables #
Docker Bake supports variables that can be defined in the Bake file and overridden at build time:
variable "VERSION" {
default = "1.0.0"
}
variable "REGISTRY" {
default = "docker.io/mycompany"
}
target "app" {
tags = [
"${REGISTRY}/myapp:${VERSION}",
"${REGISTRY}/myapp:latest"
]
}You can override these variables when running the bake command:
docker buildx bake --set VERSION=2.0.0 --set REGISTRY=ghcr.io/myorg appThis flexibility allows for dynamic build configurations without modifying the Bake file.
Practical Examples #
Let's implement some practical examples based on our Lab7:
Example 1: Node.js Application with Development and Production Targets #
In this example, we have a Node.js application with different configurations for development and production:
# Dockerfile
FROM node:18-alpine AS base
WORKDIR /app
COPY package*.json ./
FROM base AS development
RUN npm install
COPY . .
ENV NODE_ENV=development
CMD ["npm", "run", "dev"]
FROM base AS build
RUN npm ci
COPY . .
RUN npm run build
FROM node:18-alpine AS production
WORKDIR /app
COPY /app/package*.json ./
RUN npm ci --only=production
COPY /app/dist ./dist
ENV NODE_ENV=production
CMD ["node", "dist/index.js"]The corresponding Bake file:
group "default" {
targets = ["app-dev", "app-prod"]
}
target "app-dev" {
dockerfile = "Dockerfile"
target = "development"
tags = ["myapp:dev"]
}
target "app-prod" {
dockerfile = "Dockerfile"
target = "production"
tags = ["myapp:prod", "myapp:latest"]
}To build both targets:
docker buildx bakeTo build just the development target:
docker buildx bake app-devExample 2: Multi-Platform Build #
target "multi-platform" {
platforms = [
"linux/amd64",
"linux/arm64"
]
dockerfile = "Dockerfile"
target = "production"
tags = ["myapp:multi"]
}Build for multiple platforms:
docker buildx bake multi-platformExample 3: Matrix Builds #
Matrix builds allow you to generate multiple targets from a single definition:
variable "VERSIONS" {
default = ["1.0", "2.0"]
}
variable "ENVIRONMENTS" {
default = ["staging", "production"]
}
target "matrix-build" {
name = "app-${item.version}-${item.env}"
matrix = {
item = [
for v in VERSIONS for e in ENVIRONMENTS : {
version = v
env = e
}
]
}
dockerfile = "Dockerfile"
tags = ["myapp:${item.version}-${item.env}"]
args = {
VERSION = "${item.version}"
ENVIRONMENT = "${item.env}"
}
}This generates four targets:
app-1.0-stagingapp-1.0-productionapp-2.0-stagingapp-2.0-production
Integrating with CI/CD Pipelines #
Docker Bake integrates seamlessly with CI/CD pipelines, making it easier to maintain consistent build configurations across environments:
GitHub Actions Example #
name: Build and Push
on:
push:
branches: [main]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to DockerHub
uses: docker/login-action@v2
with:
username: $
password: $
- name: Build and push
uses: docker/bake-action@v3
with:
files: |
./docker-bake.hcl
targets: app-prod
push: true
set: |
*.tags=myorg/myapp:$
*.tags=myorg/myapp:latestComparing HCL, JSON, and YAML Formats #
Each Bake file format has its strengths and ideal use cases:
- HCL: The most feature-rich format with excellent support for complex expressions and functions. Ideal for sophisticated build configurations.
- JSON: Best for programmatic generation or modification of build files. Works well with existing JSON tooling and APIs.
- YAML: Provides a human-readable alternative that's familiar to users of Docker Compose, Kubernetes, and other YAML-based systems.
Choose the format that best fits your team's workflow and existing infrastructure:
┌───────────────────────────────────────────────────────────┐
│ Bake File Format Comparison │
│ │
│ ┌────────────────┐ ┌────────────────┐ ┌──────────────┐ │
│ │ │ │ │ │ │ │
│ │ HCL Format │ │ JSON Format │ │ YAML Format │ │
│ │ │ │ │ │ │ │
│ │ - Most flexible│ │ - Great for │ │ - Human- │ │
│ │ - Native format│ │ programmatic │ │ readable │ │
│ │ - Functions and│ │ generation │ │ - Familiar to│ │
│ │ expressions │ │ - Compatible │ │ DevOps │ │
│ │ - Dynamic │ │ with many │ │ teams │ │
│ │ behavior │ │ tools │ │ - Simple │ │
│ │ │ │ │ │ syntax │ │
│ └────────────────┘ └────────────────┘ └──────────────┘ │
│ │
└───────────────────────────────────────────────────────────┘
Best Practices for Complex Builds #
To get the most out of Docker Bake for complex builds:
- Organize by Responsibility: Group targets by their function or environment
- Use Inheritance: Create base targets that others can inherit from
- Parameterize Everything: Use variables for values that might change
- Document Your Configuration: Add comments to explain complex settings
- Version Your Bake Files: Keep them in version control alongside your code
- Name Consistently: Use clear naming conventions for targets and groups
- Test Locally First: Verify your Bake configuration works locally before CI/CD
- Set Output Destinations: Use build contexts and output targets appropriately
Example of well-organized Bake file:
# Base definitions
target "common" {
context = "."
dockerfile = "Dockerfile"
platforms = ["linux/amd64", "linux/arm64"]
args = {
BASE_IMAGE = "node:18-alpine"
}
}
# Environment-specific targets
target "development" {
inherits = ["common"]
target = "development"
tags = ["myapp:dev"]
args = {
NODE_ENV = "development"
}
}
target "staging" {
inherits = ["common"]
target = "production"
tags = ["myapp:staging"]
args = {
NODE_ENV = "staging"
}
}
target "production" {
inherits = ["common"]
target = "production"
tags = ["myapp:production", "myapp:latest"]
args = {
NODE_ENV = "production"
}
}
# Logical groupings
group "dev" {
targets = ["development"]
}
group "deploy" {
targets = ["staging", "production"]
}
group "all" {
targets = ["development", "staging", "production"]
}Cleanup #
After experimenting with Docker Bake and BuildKit, it's important to maintain a clean environment. Bake operations can create multiple images and use significant build cache, which may require periodic cleanup.
Cleaning Up Images Built with Bake #
First, identify and remove the images you've created with your Bake commands:
# List images created by your Bake targets
docker images --format ":" | grep myapp
# Remove specific images
docker rmi myapp:dev myapp:staging myapp:production
# Remove all images matching a pattern
docker images --format ":" | grep myapp | xargs docker rmiManaging BuildKit Cache #
BuildKit maintains its own cache separate from the traditional Docker build cache:
# Prune BuildKit cache
docker builder prune
# Remove all BuildKit build cache
docker builder prune --all
# Remove only cache older than 24h
docker builder prune --filter until=24hCleaning Up After Matrix Builds #
Matrix builds can generate many variants, requiring targeted cleanup:
# For images generated through matrix builds
# Identify with a common prefix/pattern
docker images --format ":" | grep "app-.*-.*" | xargs docker rmiManaging Docker Contexts #
If you've created custom BuildKit builders or contexts:
# List builder instances
docker buildx ls
# Remove a specific builder
docker buildx rm mybuilder
# Reset to default builder
docker buildx use defaultCleanup for Lab Examples #
For the specific examples in our Lab7:
# Navigate to the lab directory
cd lab7_bake_example
# Run the cleanup script
./cleanup.sh
# Or manually clean up
docker rmi $(docker images --format ":" | grep myapp)
docker builder prune --forceDocker Registry Cleanup #
If you've pushed images to a registry during testing:
# Using a tool like skopeo to remove registry images
skopeo delete docker://registry.example.com/myapp:dev
skopeo delete docker://registry.example.com/myapp:stagingVerifying Cleanup #
Ensure your environment is properly cleaned:
# Check for remaining images from your bake targets
docker images | grep myapp
# Check BuildKit cache usage
docker builder prune --force --dry-runRegular cleanup after working with Docker Bake and BuildKit ensures efficient disk usage and prevents leftover images from cluttering your environment or registry.
Conclusion #
Docker Bake and BuildKit represent the next evolution in container build technology, offering powerful features that simplify complex build scenarios while improving performance and flexibility. By using Bake files to define your build configurations, you can create a more maintainable, parameterized, and efficient build process.
In this guide, we've explored:
- The fundamentals of Docker Bake and BuildKit
- Different Bake file formats (HCL, JSON, YAML)
- Creating multiple targets with inheritance
- Parameterizing builds with variables
- Practical examples for different scenarios
- CI/CD integration
- Best practices for complex build configurations
By adopting these advanced build techniques, you can streamline your development workflow, standardize build practices across your team, and create more efficient, flexible Docker images.
In the next article in our Docker Practical Guide series, we'll explore how to use Docker with LocalStack for local cloud development, allowing you to emulate AWS services locally for testing and development. Stay tuned!
How are you handling complex Docker build scenarios in your projects? Share your experiences or questions in the comments below!