Docker — Cheat Sheet

Docker is a platform for packaging applications and their dependencies into containers. Containers share the host OS kernel — unlike VMs, they don't need a full guest OS. VM: Hypervisor virtualizes hardware. Each VM runs a full OS (2–4 GB overhead). Strong isolation (separate kernel). Slow to start (minutes). Container: Docker uses Linux namespaces (isolation) and cgroups (resource limits) to create isolated processes on the same kernel. Lightweight (MBs). Starts in seconds. Weaker isolation than VMs but sufficient for most workloads. Best together: Run containers inside VMs in production (AWS EC2 + Docker, GKE nodes). VMs provide hardware isolation; containers provide application packaging and density.

Image: An immutable, read-only template containing the application code, runtime, libraries, and configuration. Built from a Dockerfile and stored in layers. Can be shared via a registry (Docker Hub, ECR). Container: A running instance of an image. Docker adds a thin read-write layer on top of the image layers. Multiple containers can run from the same image simultaneously, each with their own writable layer. Analogy: Image is like a class; container is like an object instance. Containers are ephemeral — the writable layer is lost when the container stops (unless you use volumes for persistence). Use docker commit to create a new image from a container's state (generally avoid — use Dockerfile instead).

Both define what runs when the container starts, but they behave differently when overridden. CMD: Default command or arguments. Completely replaced when you pass a command to docker run: docker run myimage echo hello (replaces CMD entirely). ENTRYPOINT: The fixed executable that always runs. CMD provides default arguments to it. You can override ENTRYPOINT with --entrypoint flag. Combination (best practice): Use ENTRYPOINT for the executable and CMD for default arguments: ENTRYPOINT ["java", "-jar", "app.jar"] CMD ["--spring.profiles.active=prod"] Running docker run myimage --spring.profiles.active=dev overrides only CMD, keeping the java -jar invocation.

Multi-stage builds use multiple FROM instructions in one Dockerfile to separate the build environment from the runtime environment. Problem: A Java build image needs JDK, Maven, and build tools — hundreds of MBs. The final runtime only needs the JRE and the JAR. Solution: FROM maven:3.9-openjdk-21 AS builder COPY . . RUN mvn package FROM eclipse-temurin:21-jre COPY --from=builder /target/app.jar /app.jar CMD ["java","-jar","/app.jar"] Benefits: • Dramatically smaller final image (from ~600MB to ~150MB) • No build tools, source code, or test dependencies in production • Smaller attack surface for security • Faster image pulls and container startup

Volumes: Managed by Docker, stored in /var/lib/docker/volumes/. Created with docker volume create or -v myvolume:/data. Portable across hosts (can use volume drivers for cloud storage). Recommended for production data persistence. Bind mounts: Mount a specific host directory into the container (-v /host/path:/container/path). The container sees the actual host filesystem. Used in development to sync source code live without rebuilds. tmpfs mounts: In-memory only, not persisted. Useful for secrets or temporary processing files. Key rule: Use volumes for production databases and persistent data. Use bind mounts for local development workflows. Never use bind mounts in production — they couple the container to the host filesystem structure.

Docker provides several network drivers: • bridge (default): Containers on the same bridge network can communicate by container name. Isolated from the host. • host: Container shares the host's network namespace — uses host IP directly. Maximum performance, no isolation. • none: No networking. Container has only loopback interface. • overlay: Multi-host networking for Docker Swarm. Containers on different hosts communicate transparently. • macvlan: Assign a MAC address to the container, making it appear as a physical device on the network. User-defined bridge networks (docker network create): Better than the default bridge — containers can communicate by name (built-in DNS), not just IP. Recommended for multi-container apps.

Strategies (in order of impact): 1. Multi-stage builds: Use a build stage; copy only the artifact to a minimal runtime base image 2. Use minimal base images: alpine (5MB) or distroless instead of ubuntu (77MB) 3. Minimize layers: Combine RUN commands with && to reduce layer count 4. .dockerignore file: Exclude node_modules, .git, test files from build context 5. Don't install dev dependencies: npm install --production / pip install --no-dev 6. Clean up in the same RUN layer: apt-get install ... && rm -rf /var/lib/apt/lists/* 7. Use specific versions: Avoid :latest — unpredictable and disables layer caching Tools: docker image history (see layer sizes), dive (layer-level analysis), trivy (security scan).

Docker builds images layer by layer. Each instruction (FROM, RUN, COPY, ADD) creates a new layer. Layers are cached by their instruction + input hash. If a layer's cache is valid (instruction unchanged, input files unchanged), Docker reuses the cached layer — dramatically speeding up builds. Cache invalidation: Any change invalidates all subsequent layers. Ordering matters: Place instructions that change infrequently at the top, frequently changing instructions at the bottom. Bad order (Java): COPY . . ← source changes every build, invalidates below RUN mvn package ← always re-runs even if pom.xml didn't change Good order: COPY pom.xml . ← only invalidated when dependencies change RUN mvn dependency:go-offline COPY src/ src/ RUN mvn package

Docker Compose defines and runs multi-container applications using a YAML file (compose.yml). A single command starts all services with configured networking, volumes, and environment. Key use cases: Local development (app + database + cache + message broker), CI/CD integration testing, running multi-service demos. Compose features: • Automatic network creation — services communicate by service name • Dependency ordering: depends_on with healthcheck conditions • Volume management and bind mounts • Environment variable injection (.env files) • Scale services: docker compose up --scale worker=3 Not for production: Compose is single-host. Use Docker Swarm or Kubernetes for multi-host orchestration. However, compose.yml files are often used as the basis for Kubernetes manifests.

Options in order of security: 1. Environment variables: Simple but exposed in docker inspect, process listings, and logs. Acceptable for non-critical config. 2. Docker Secrets (Swarm): Stored encrypted in the Raft consensus log. Mounted as files in /run/secrets/ inside containers. Not exposed in env or inspect. 3. Kubernetes Secrets: Base64-encoded in etcd (not encrypted by default). Use KMS integration (AWS KMS, GCP Cloud KMS) to encrypt at rest. Mounted as env vars or volumes. 4. HashiCorp Vault: External secrets management. Dynamic secrets (short-lived DB credentials), encryption-as-a-service, audit logging. Vault Agent sidecar handles secret injection. 5. Cloud-native: AWS Secrets Manager, GCP Secret Manager, Azure Key Vault with IAM-based access. Never bake secrets into images with ARG or ENV in the Dockerfile — they appear in image history.

A health check runs a command inside the container periodically to determine if the application is healthy (not just running). Dockerfile syntax: HEALTHCHECK --interval=30s --timeout=10s --retries=3 \ CMD curl -f http://localhost:8080/actuator/health || exit 1 States: starting → healthy / unhealthy Docker Compose: docker compose ps shows health status. depends_on: condition: service_healthy waits for healthy status before starting dependent services. Kubernetes equivalents (more powerful): • readinessProbe: Determines if the pod should receive traffic • livenessProbe: Determines if the pod should be restarted • startupProbe: Gives slow-starting apps time to initialize before other probes start Always define health checks for production containers to enable orchestrators to route traffic only to healthy instances.

The Docker daemon (dockerd) is a background process that manages Docker objects: images, containers, networks, and volumes. It exposes a REST API (unix:///var/run/docker.sock or TCP). The Docker CLI (docker command) communicates with dockerd via this API. Remotely, you can connect to a Docker daemon over TLS. Containerd: dockerd uses containerd as the container runtime. containerd manages container lifecycle. It uses runc (OCI runtime) to actually create containers using Linux namespaces and cgroups. Stack: Docker CLI → dockerd (REST API) → containerd → containerd-shim → runc → container process Security concern: The Docker socket (/var/run/docker.sock) mounted into a container gives that container root-level access to the host. Never mount it in production unless absolutely necessary (rootless Docker mitigates this).

Key security practices for production containers: 1. Run as non-root: USER appuser in Dockerfile. Add --read-only filesystem where possible. 2. Minimal base image: Use distroless or alpine — fewer packages = smaller attack surface 3. Immutable images: Never modify running containers; redeploy instead 4. No privileged mode: --privileged grants root-equivalent access to host. Avoid entirely. 5. Drop capabilities: --cap-drop ALL --cap-add only what's needed 6. Resource limits: --memory and --cpus prevent DoS from runaway containers 7. Read-only filesystem: --read-only with tmpfs for /tmp 8. Scan images: Use Trivy, Snyk, or Docker Scout to detect known CVEs 9. Sign images: Docker Content Trust (Notary) verifies image authenticity 10. Network policy: Restrict which containers can talk to each other (Kubernetes NetworkPolicy)

Containers should write logs to stdout/stderr (12-factor app principle). Docker captures these via logging drivers. Default: json-file driver stores logs as JSON in the host filesystem. Rotate with --log-opt max-size=10m --log-opt max-file=3. Logging drivers: json-file, syslog, journald, fluentd, awslogs (CloudWatch), gcplogs, splunk. Centralized logging architectures: 1. EFK Stack: Fluentd sidecar/DaemonSet collects logs → Elasticsearch → Kibana 2. ELK Stack: Logstash pipeline 3. Loki + Grafana: Lightweight, stores logs as streams (like Prometheus for logs) 4. CloudWatch/Stackdriver: Cloud-native managed logging Best practice: Use structured logging (JSON), include trace_id/request_id in every log line for correlation across services.

docker stop: Sends SIGTERM to the container's PID 1, allowing graceful shutdown. Waits 10 seconds (configurable with --time) for the process to exit. If it doesn't exit, sends SIGKILL. docker kill: Immediately sends SIGKILL (or a custom signal with --signal), forcing the process to terminate instantly without cleanup. For graceful shutdown: • Ensure your application handles SIGTERM: flush buffers, close DB connections, finish in-flight requests • In Java/Spring: server.shutdown=graceful in application.properties + spring.lifecycle.timeout-per-shutdown-phase=30s • In Docker Compose: stop_grace_period: 30s • In Kubernetes: preStop hook + terminationGracePeriodSeconds Graceful shutdown is critical for zero-downtime deployments and prevents data loss in stateful applications.

BuildKit: The next-generation Docker build engine, enabled by default since Docker 23.0. Replaces the legacy builder with a dramatically improved architecture. Key improvements: 1. Parallel stage execution: Multi-stage builds run independent stages in parallel. Legacy builder runs them sequentially. A build with 3 parallel stages runs in the time of the slowest, not the sum. 2. Better cache management: Fine-grained caching. Cache mounts (--mount=type=cache) persist package manager caches between builds without baking them into the image: ```dockerfile RUN --mount=type=cache,target=/root/.m2 mvn package RUN --mount=type=cache,target=/var/cache/apt apt-get install -y curl ``` 3. Secret mounting: Pass secrets at build time without including in image layers: ```dockerfile RUN --mount=type=secret,id=github_token \ GITHUB_TOKEN=$(cat /run/secrets/github_token) mvn deploy ``` Built with: docker build --secret id=github_token,src=.github_token 4. SSH forwarding: Forward SSH agent into build without embedding keys: ```dockerfile RUN --mount=type=ssh git clone git@github.com:private/repo.git ``` 5. Cache export/import: Export build cache to a registry or local directory. Share cache across CI runners: docker buildx build --cache-to type=registry,ref=myregistry/myapp:cache --cache-from type=registry,ref=myregistry/myapp:cache 6. Multi-platform builds: Build images for multiple architectures (amd64, arm64) in a single command with buildx. Enable: DOCKER_BUILDKIT=1 docker build or use docker buildx build (buildx always uses BuildKit).

Essential debugging techniques: 1. Execute a shell inside the container: ```bash docker exec -it <container_id> /bin/sh # for alpine docker exec -it <container_id> /bin/bash # for debian/ubuntu ``` 2. View logs: ```bash docker logs <container_id> # all logs docker logs -f <container_id> # follow (tail -f) docker logs --tail 100 <container_id> # last 100 lines ``` 3. Inspect container config and state: ```bash docker inspect <container_id> # full JSON: env vars, mounts, network, state docker inspect --format '{{.State.ExitCode}}' <container_id> ``` 4. View resource usage: ```bash docker stats <container_id> # live CPU, memory, network, disk I/O docker top <container_id> # running processes inside container ``` 5. Copy files out for inspection: ```bash docker cp <container_id>:/app/logs/error.log ./local-error.log ``` 6. Debug a crashed container: Container exits immediately — can't exec into it. Override entrypoint: ```bash docker run -it --entrypoint /bin/sh myimage ``` 7. Attach to container's PID namespace from host (advanced): ```bash nsenter -t <container_pid> -n ip addr # inspect container network from host ``` 8. Distroless containers: No shell available inside. Use kubectl debug (K8s) or add a debug stage in multi-stage build that includes shell tools.

Docker Swarm: Docker's built-in container orchestration system. Manages clusters of Docker hosts (nodes), deploys services across them, and handles load balancing and failover. Core concepts: • Manager nodes: Handle orchestration, maintain cluster state (Raft consensus), dispatch tasks to workers • Worker nodes: Execute containers (tasks) • Service: The desired state declaration (image, replicas, ports, update config) • Task: A container running on a specific node — the unit of work • Stack: Multi-service deployment defined in a Compose file Swarm commands: ```bash docker swarm init # initialize swarm docker service create --replicas 3 nginx # deploy 3 nginx replicas docker service scale web=5 # scale to 5 docker service update --image nginx:1.25 web # rolling update ``` Swarm vs Kubernetes: Simplicity: Swarm is dramatically simpler to set up — built into Docker, one command to initialize. Kubernetes requires significant expertise to configure and operate. Scaling: Kubernetes scales to thousands of nodes with hundreds of features. Swarm works well for smaller deployments. Ecosystem: Kubernetes has a vast ecosystem — Helm, ArgoCD, Prometheus Operator, Istio. Swarm has limited ecosystem. Auto-scaling: Kubernetes has HPA (Horizontal Pod Autoscaler). Swarm has no built-in autoscaling. Storage: Kubernetes has robust PVC/StorageClass system. Swarm storage is simpler but more limited. Adoption: Kubernetes dominates production use. Swarm is used for simpler setups or when Kubernetes overhead is not justified.

Docker labels: Key-value metadata attached to images, containers, volumes, or networks. Used for organization, filtering, automation, and tooling integration. Setting labels in Dockerfile: ```dockerfile LABEL maintainer="team@company.com" LABEL version="1.2.3" LABEL org.opencontainers.image.source="https://github.com/org/repo" LABEL org.opencontainers.image.created="2024-01-15" ``` Adding labels at run time: ```bash docker run --label env=production --label app=web myimage ``` Filtering by labels: ```bash docker ps --filter "label=env=production" docker images --filter "label=app=web" ``` OCI image spec labels (standardized by Open Container Initiative): • org.opencontainers.image.title • org.opencontainers.image.version • org.opencontainers.image.source (git repo URL) • org.opencontainers.image.revision (git commit SHA) • org.opencontainers.image.licenses Practical uses: • CI/CD: Label images with git commit SHA, branch, build number for traceability • Prometheus: Prometheus discovers targets by querying Docker API for containers with specific labels • Traefik: Configures routing rules via container labels (traefik.http.routers.web.rule=Host('example.com')) • Log routing: Log agents (Fluentd, Logstash) apply different pipelines based on container labels • Compliance: Label images with data classification, team ownership for governance auditing

Docker images are composed of read-only layers. Containers add a thin writable layer on top. The overlay filesystem merges all these layers into a single unified view for the container process. Overlay2 driver (default on modern Linux): Components: • lowerdir: One or more read-only image layers • upperdir: Container's writable layer • workdir: Temporary directory for atomic operations • merged: The union view mounted into the container How reads work: 1. Docker looks in upperdir first (container writes) 2. If not found, walks down through lowerdir layers (bottom = base image) 3. Returns the first match found Copy-on-Write (CoW): When a container modifies a file that exists in a lower layer: 1. The file is copied from lowerdir up into upperdir (first modification only) 2. Subsequent writes go directly to the upperdir copy 3. The lower layer file remains unchanged — other containers sharing that layer are unaffected This is why: • Multiple containers from the same image don't duplicate storage — they share read-only layers • A large file modified in a container gets duplicated into the container's layer — for large files, use volumes instead • Deleting a file in a Dockerfile RUN doesn't reclaim space if the file was added in a previous layer — combine ADD and DELETE in the same RUN instruction Layer storage location: /var/lib/docker/overlay2/ — each layer is a directory.

.dockerignore: A file placed at the root of the build context that specifies files and directories to exclude from the Docker build context sent to the daemon. Why it matters: 1. Build speed: Docker sends the entire build context to the daemon before building. Large contexts (node_modules, .git, test data) make every build slow even if the Dockerfile doesn't reference those files. 2. Image security: Prevents accidentally including sensitive files (.env files, private keys, credentials) in the image layer via COPY . . 3. Cache effectiveness: Fewer files in the context means fewer spurious cache invalidations. Example .dockerignore: ``` # Dependencies (rebuilt inside Docker) node_modules .npm # Version control .git .gitignore # Test and documentation **/*.test.js **/*.spec.ts docs/ README.md # Build artifacts (built inside Docker) target/ dist/ build/ # Environment and secrets .env .env.* *.pem *.key # IDE files .idea/ .vscode/ *.iml # OS files .DS_Store Thumbs.db # Docker files themselves (no need to copy) Dockerfile* docker-compose* ``` Syntax: Same as .gitignore. Supports wildcards (*, **), negation (!include_this), and directory matching. Check context size: docker build sends context — watch for "Sending build context to Docker daemon Xmb" — large sizes indicate missing .dockerignore entries.

Both set variables in Dockerfile, but they differ in scope and behavior. ARG (Build-time variable): • Available only during docker build, not in the running container • Set with --build-arg NAME=value at build time • Can have defaults: ARG VERSION=1.0 ENV (Environment variable): • Available both during build and in the running container • Persisted in the image and visible to all processes • Can be overridden at runtime: docker run -e ENV_VAR=value Usage patterns: ```dockerfile # ARG for build-time values ARG JAR_VERSION=2.1.0 RUN curl -O https://releases.example.com/app-${JAR_VERSION}.jar # ENV for runtime configuration ENV APP_PORT=8080 ENV LOG_LEVEL=INFO EXPOSE ${APP_PORT} # Using ARG to set ENV (makes it available at runtime) ARG ENVIRONMENT=production ENV SPRING_PROFILES_ACTIVE=${ENVIRONMENT} ``` Security warning: • ARG values used as passwords WILL appear in docker history — the layer cache stores them even after the ARG scope ends. Use BuildKit secret mounts instead for sensitive values. • ENV values are permanently in the image — docker inspect shows them. Don't put secrets in ENV. Caching impact: ARG instructions before FROM are not part of the build cache after FROM. ARG instructions after FROM invalidate the cache for all subsequent layers when the value changes.

Both copy files into the image, but ADD has additional capabilities that COPY doesn't. COPY: Simple file/directory copy from build context to image. ```dockerfile COPY src/ /app/src/ COPY package.json package-lock.json ./ ``` ADD: Everything COPY does, plus: 1. Auto-extraction of tar archives: ADD app.tar.gz /app/ — automatically decompresses tar, tar.gz, tar.xz 2. URL support: ADD https://example.com/config.json /app/config.json — downloads from URL (discouraged) Best practice: Use COPY by default. Only use ADD when you specifically need tar auto-extraction. Why prefer COPY: • Explicit and predictable — no magic behavior • COPY with tar file won't auto-extract (surprising with ADD) • URL support in ADD is discouraged: doesn't cache well, depends on remote availability, includes curl/wget overhead without the ability to verify checksums in the same layer For URL downloads: ```dockerfile # Better: Use RUN with curl/wget so you control caching and verification RUN curl -fsSL https://example.com/tool -o /usr/local/bin/tool \ && echo "expectedsha256 /usr/local/bin/tool" | sha256sum -c \ && chmod +x /usr/local/bin/tool ``` For tar extraction in multi-stage: ```dockerfile # In builder stage COPY app.tar.gz /tmp/ RUN tar -xzf /tmp/app.tar.gz -C /app ```

Docker enforces resource limits using Linux cgroups. Without limits, one container can consume all host resources and starve others. Memory limits: ```bash docker run --memory=512m myimage # hard limit: OOM kill at 512MB docker run --memory=512m --memory-swap=512m myimage # disable swap docker run --memory-reservation=256m myimage # soft limit: reclaimed under pressure ``` CPU limits: ```bash docker run --cpus=1.5 myimage # limit to 1.5 CPU cores docker run --cpu-shares=512 myimage # relative weight (default 1024) docker run --cpuset-cpus=0,2 myimage # pin to specific CPU cores ``` Docker Compose resource limits: ```yaml services: api: image: myapp deploy: resources: limits: cpus: '1.0' memory: 512M reservations: cpus: '0.5' memory: 256M ``` Why set limits: • Prevent noisy neighbor: One misbehaving container starving others • Force memory discipline: Java apps with unbounded heap will use all available RAM • Kubernetes compatibility: Resource requests/limits in K8s match this model — test with realistic limits locally JVM consideration: JVM respects container memory limits since Java 10+. Set -XX:MaxRAMPercentage=75 instead of -Xmx to allow JVM to calculate heap relative to container memory limit: ```dockerfile ENV JAVA_OPTS="-XX:MaxRAMPercentage=75.0 -XX:+UseContainerSupport" ```

Docker registry: A server that stores and distributes Docker images. Docker Hub is the public default registry. Private registries provide security, access control, and reduced latency for internal images. Cloud-managed registries (recommended): • AWS ECR (Elastic Container Registry): IAM-based auth, integrated with ECS/EKS, lifecycle policies • GCP Artifact Registry: Replaces GCR, supports Docker + other artifact types • Azure Container Registry: Azure AD integration, geo-replication, Tasks for building • GitHub Container Registry (ghcr.io): Integrated with GitHub Actions Self-hosted Registry (Docker Registry v2): ```yaml # docker-compose.yml for private registry services: registry: image: registry:2 ports: ["5000:5000"] volumes: - ./registry-data:/var/lib/registry - ./certs:/certs environment: REGISTRY_HTTP_TLS_CERTIFICATE: /certs/domain.crt REGISTRY_HTTP_TLS_KEY: /certs/domain.key REGISTRY_AUTH: htpasswd REGISTRY_AUTH_HTPASSWD_REALM: Registry Realm REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd ``` Using the registry: ```bash docker tag myapp localhost:5000/myapp:1.0 docker push localhost:5000/myapp:1.0 docker pull localhost:5000/myapp:1.0 ``` Harbor: Enterprise open-source registry with RBAC, image scanning (Trivy integration), content signing, replication, and audit logging. Most feature-rich self-hosted option. Image lifecycle policies: Set up automatic deletion of old images in ECR/GCR to control storage costs — keep last N versions or images older than N days.

WORKDIR: Sets the working directory for all subsequent RUN, CMD, ENTRYPOINT, COPY, and ADD instructions in the Dockerfile. If the directory doesn't exist, Docker creates it. ```dockerfile FROM eclipse-temurin:21-jre WORKDIR /app COPY target/myapp.jar . CMD ["java", "-jar", "myapp.jar"] ``` Equivalent without WORKDIR (discouraged): ```dockerfile RUN mkdir -p /app COPY target/myapp.jar /app/myapp.jar CMD ["java", "-jar", "/app/myapp.jar"] ``` Why use WORKDIR: 1. Clarity: Explicit working directory — no assumptions about where files land 2. Shorter paths: Subsequent instructions can use relative paths 3. Avoids RUN cd tricks: RUN cd /app && ... doesn't persist between RUN instructions. WORKDIR does. 4. docker exec default: When you exec into a container, you start in WORKDIR Multiple WORKDIR instructions: Can be used multiple times. Relative paths are relative to the previous WORKDIR: ```dockerfile WORKDIR /app WORKDIR src # Now at /app/src ``` Best practice: Set WORKDIR early and use relative paths throughout. Use /app as a conventional directory (not /, not /home). Never use RUN mkdir && cd as a substitute.

Restart policies: Determine whether Docker automatically restarts a container when it exits. Policies: • no (default): Never restart automatically • on-failure[:max-retries]: Restart if exit code is non-zero. Optional max retry count: --restart on-failure:5 • always: Always restart regardless of exit code. Also starts on Docker daemon startup. • unless-stopped: Like always, but doesn't start if the container was manually stopped before daemon restart Usage: ```bash docker run --restart always nginx docker run --restart on-failure:5 myapp ``` Docker Compose: ```yaml services: api: image: myapp restart: unless-stopped ``` When to use each: • on-failure: For containers that should retry on crash but not loop forever. Web apps where a crash indicates a bug. • always: For critical infrastructure containers (reverse proxy, monitoring agents) that must stay up • unless-stopped: Preferred over always for most services — lets you manually stop without automatic restart loop during maintenance Kubernetes equivalent: Kubernetes automatically restarts failed containers (CrashLoopBackOff with exponential backoff). Restart policy at pod level: Always, OnFailure, Never. No direct mapping to Docker's restart policies — K8s always has the controller managing desired state. Restart policy + health check: Health check failure does NOT trigger restart in Docker standalone. Only a crash (non-zero exit) triggers restart. In K8s, liveness probe failure causes restart.

Container immutability: Containers should never be modified after they are deployed. If a change is needed, build a new image and deploy new containers — don't log into a running container and make changes. Why immutability matters: • Reproducibility: Deploy the same image in dev, staging, and prod — identical behavior • Auditability: Git history of Dockerfile = full history of what's in production • Fast rollback: Roll back by deploying the previous image tag — takes seconds • Security: No configuration drift — production state is always derivable from version-controlled Dockerfiles • No snowflake servers: Cattle, not pets. Any instance can be replaced without loss. Implementing immutability: 1. Build from Dockerfile, never docker commit a running container 2. All configuration via environment variables or config maps (not files edited inside container) 3. Read-only filesystem where possible: docker run --read-only --tmpfs /tmp myimage 4. Use volumes for writable data — application code is read-only 5. Treat images as artifacts: Tag with git commit SHA, push to registry, deploy by tag 6. CI/CD pipeline: code commit → build → test → push → deploy. Never SSH into prod to fix things. Anti-patterns to avoid: • SSH into production container to edit config files • docker exec to apply hotfixes • Building different images for dev vs prod with different code • Using :latest tag (can't reproduce what was deployed) Config injection pattern: App reads config from environment variables. Different configs for dev/staging/prod are injected at runtime — the image itself is environment-neutral.

The goal: Same Docker image runs in dev, staging, and production — only configuration differs. Approach 1 — Environment variables (12-factor): The canonical approach. Application reads all config from env vars. ```bash docker run -e DATABASE_URL=postgres://prod/db -e LOG_LEVEL=WARN myapp ``` Docker Compose with .env files: ```yaml services: api: image: myapp env_file: - .env.local # local dev overrides ``` .env.local: DATABASE_URL=postgres://localhost/dev, LOG_LEVEL=DEBUG Approach 2 — Config file injection: Mount a config file via volume or ConfigMap (Kubernetes): ```bash docker run -v ./config/prod.yml:/app/config/app.yml myapp ``` App reads /app/config/app.yml at startup. Approach 3 — External config service: App fetches config from a central service (Spring Cloud Config, AWS AppConfig, Vault) at startup. Config is not in the image or env — it's fetched dynamically. Approach 4 — Build-time profiles (anti-pattern): Building different images per environment (--build-arg ENV=production). Reduces immutability — you can't be sure dev image is equivalent to prod image. Kubernetes: ConfigMaps for non-sensitive config (injected as env vars or volume-mounted files). Secrets for sensitive values. External Secrets Operator syncs from Vault/AWS Secrets Manager into K8s Secrets. Hierarchy: Default values in code → environment-specific .env → runtime env vars (highest priority).

Docker Content Trust (DCT): A security feature that ensures image integrity and publisher authenticity. When enabled, Docker verifies that pulled images are cryptographically signed by a trusted publisher — preventing tampered or impersonated images. How DCT works: • Image publishers sign images using a private key (Notary framework) • Signatures stored in a separate Notary server alongside the registry • When DCT is enabled and you pull an image, Docker verifies the signature • If no valid signature exists or the signature doesn't verify → pull refused Enabling DCT: ```bash export DOCKER_CONTENT_TRUST=1 docker pull nginx:latest # will only pull if signed docker push myapp:1.0 # automatically signs on push ``` Cosign (modern replacement): Developed by Sigstore project, widely adopted for supply chain security. Signs and verifies container images using short-lived certificates (keyless signing) tied to OIDC identity (GitHub Actions, GCP SA). ```bash cosign sign --key cosign.key myregistry/myapp:1.0 cosign verify --key cosign.pub myregistry/myapp:1.0 ``` Keyless signing (Sigstore): ```bash cosign sign myregistry/myapp:1.0 # uses OIDC identity — no key management ``` Signed attestation stored in transparency log (Rekor) — publicly verifiable. Kubernetes admission control: Use Policy Controller (Kyverno, OPA Gatekeeper) to enforce image signature verification before pods can start. Rejects unsigned or improperly signed images at admission. Supply chain security: Signing images, verifying SBOMs (Software Bill of Materials), and provenance attestations form the backbone of software supply chain security (SLSA framework).

Docker networking uses Linux networking primitives — no kernel modules or virtualization. Bridge network internals: 1. Docker creates a virtual bridge (docker0 by default, or custom bridges for user-defined networks) 2. For each container, Docker creates a veth pair (virtual ethernet cable): one end inside the container (eth0), one end on the host (vethXXXX) 3. The host-side veth is attached to the bridge 4. Bridge acts like a virtual switch — all containers attached to it can communicate at L2 Container-to-container communication: • Same bridge network: Traffic goes through the bridge — no host IP routing needed. Containers find each other by container name (user-defined bridges have embedded DNS) • Different networks: Traffic must go through host routing (iptables) Container port publishing (iptables DNAT): When you publish a port (-p 8080:80), Docker adds iptables rules: • PREROUTING: Incoming packets to host:8080 → DNAT to container-IP:80 • POSTROUTING: MASQUERADE (NAT) for outbound traffic ```bash iptables -t nat -L DOCKER # see Docker's NAT rules ``` Container outbound internet access: • Container sends packet to its default gateway (bridge IP) • Host routes it through iptables MASQUERADE → container's traffic appears to come from host IP DNS for user-defined networks: • Docker runs an embedded DNS server at 127.0.0.11 inside containers • Container name lookups resolve to container IP via this DNS • Default bridge (docker0) does NOT have embedded DNS — only user-defined bridges do Other networks: • Host mode: Container uses host network namespace directly — no veth, no bridge, host IP is container IP • None: Container gets only loopback, no external connectivity • Overlay (Swarm): VXLAN tunnels between hosts, containers communicate across hosts transparently

Multi-platform image: A single image tag that automatically serves the correct architecture-specific image to each platform. docker pull nginx:latest on an M1 Mac pulls the arm64 variant; on an x86 server it pulls amd64. Same tag, different binary. Why needed: Apple Silicon Macs (arm64), AWS Graviton (arm64), Raspberry Pi — all need arm64 images. Most production servers are amd64. Developers on Macs need images that work locally and in prod. How it works — Docker Manifest List: A manifest list is a meta-image pointing to platform-specific image manifests: ``` nginx:latest (manifest list) ├── nginx@sha256:abc... (linux/amd64) ├── nginx@sha256:def... (linux/arm64) └── nginx@sha256:ghi... (linux/arm/v7) ``` Building with docker buildx: ```bash # Create and use a multi-platform builder docker buildx create --use --name mybuilder # Build and push for multiple platforms docker buildx build \ --platform linux/amd64,linux/arm64 \ --tag myregistry/myapp:1.0 \ --push \ . ``` QEMU emulation: For architectures you don't have native hardware for, Docker uses QEMU to emulate. Much slower than native — use native CI runners (AWS Graviton) for arm64 builds in production pipelines. CI/CD multi-platform: ```yaml # GitHub Actions - uses: docker/setup-qemu-action@v3 - uses: docker/setup-buildx-action@v3 - uses: docker/build-push-action@v5 with: platforms: linux/amd64,linux/arm64 push: true tags: myregistry/myapp:latest ```

At scale, container logs must be collected centrally — local json-file logs on each host don't survive container restarts and can't be searched across thousands of containers. Logging architecture options: 1. EFK Stack (most common): • Fluentd/Fluent Bit DaemonSet: Runs on every Kubernetes node, reads container logs from /var/log/containers/, parses and ships to Elasticsearch • Elasticsearch: Indexes logs — stores and searches • Kibana: Query, visualize, and alert on logs • Fluent Bit preferred over Fluentd: Lower resource footprint, written in C 2. Loki + Grafana (Grafana Labs): • Loki: Stores log streams (labels + compressed text) — much cheaper than Elasticsearch (no full indexing) • Promtail/Alloy: Log collector agent • Grafana: Query logs with LogQL alongside metrics • Best for: Teams already using Grafana stack 3. Cloud-native managed: • AWS: CloudWatch Logs (awslogs driver), OpenSearch (Elasticsearch managed) • GCP: Cloud Logging (built-in GKE integration) • Azure: Log Analytics Workspace Log shipping Docker driver: ```yaml services: api: logging: driver: fluentd options: fluentd-address: fluentd:24224 tag: app.api ``` Structured logging: Always log JSON. Include trace_id, span_id, user_id, service name in every log line. Enables correlation across services. Log levels in production: INFO for business events, WARN for recoverable issues, ERROR for failures. DEBUG only temporarily — verbose logs are expensive at scale. Retention: Set log retention policies — 30 days hot, 1 year cold storage (S3). Logs are expensive to store and query.

EXPOSE: A documentation instruction in the Dockerfile. Declares which port the container application listens on. Does NOT actually open the port or make it accessible from outside the container. ```dockerfile EXPOSE 8080 EXPOSE 443 ``` Publishing ports (-p flag): Actually maps a host port to a container port — makes the port accessible from outside. ```bash docker run -p 8080:8080 myapp # host:8080 → container:8080 docker run -p 80:8080 myapp # host:80 → container:8080 (different ports) docker run -p 127.0.0.1:8080:8080 myapp # only accessible from localhost docker run -P myapp # publishes all EXPOSE'd ports to random host ports ``` When EXPOSE is useful: • Documentation: Other developers know which port to publish • docker run -P: Uses EXPOSE declarations to know which ports to publish to random host ports • Docker Compose: Services on the same Compose network can reach each other on EXPOSE'd ports without publishing — no -p needed for internal service communication Compose example: ```yaml services: api: image: myapp # No ports: section needed — db can reach api:8080 directly db: image: postgres # ports: ["5432:5432"] ← only add if you need host access to DB ``` Security: Only publish ports that external clients need. Internal services should communicate on the container network without publishing ports to the host.

Java/Spring Boot images have unique optimization opportunities compared to other languages. 1. Layered JAR (Spring Boot 2.3+): Spring Boot fat JARs include all dependencies in a single file — but dependencies rarely change. Extract into layers: ```dockerfile FROM eclipse-temurin:21-jre AS extractor WORKDIR /app COPY target/myapp.jar myapp.jar RUN java -Djarmode=layertools -jar myapp.jar extract FROM eclipse-temurin:21-jre WORKDIR /app COPY --from=extractor /app/dependencies/ ./ COPY --from=extractor /app/spring-boot-loader/ ./ COPY --from=extractor /app/snapshot-dependencies/ ./ COPY --from=extractor /app/application/ ./ ENTRYPOINT ["java", "org.springframework.boot.loader.launch.JarLauncher"] ``` Dependency layers don't change between builds → cached → only the application layer re-copies on code changes. 2. Native images with GraalVM: Compile Spring Boot to native binary — starts in milliseconds, uses fraction of heap. Tradeoff: build time is 5-10 minutes, reflection/dynamic proxies need configuration. 3. Use JRE, not JDK: Runtime image only needs JRE: eclipse-temurin:21-jre (vs JDK at ~400MB) 4. Container-aware JVM settings: ```dockerfile ENV JAVA_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 \ -XX:+UseG1GC -XX:+ExitOnOutOfMemoryError" ``` 5. Buildpacks (Paketo): Spring Boot plugin can build optimized layered images without writing a Dockerfile: ```bash ./mvnw spring-boot:build-image -Dspring-boot.build-image.imageName=myapp:latest ``` Produces layered, non-root, optimized images automatically.

Docker-in-Docker (DinD): Running Docker daemon inside a Docker container. Often used in CI/CD to build Docker images as part of a pipeline running inside a container. Why it's needed: CI runners (Jenkins, GitLab Runner) run inside containers. Building Docker images inside these containers requires a Docker daemon. DinD approach: ```yaml # privileged container running Docker daemon docker run --privileged docker:dind ``` Problems with DinD: 1. Privileged mode required: --privileged gives the container full access to the host kernel — effectively removes container isolation. Major security risk. 2. Storage driver conflicts: Inner Docker daemon uses storage driver incompatible with the outer container's filesystem. 3. Layer cache loss: Inner daemon starts fresh each run — no layer caching across builds → slower builds. 4. Complexity: Managing two nested Docker environments. Better alternative — Docker socket mounting: ```yaml # Mount host Docker socket into CI container docker run -v /var/run/docker.sock:/var/run/docker.sock docker:cli ``` CI container uses the host Docker daemon via the socket. No nesting, no privileged mode. Concern with socket mounting: Container with Docker socket access has root-equivalent access to the host — can inspect/modify all containers. Acceptable in trusted CI environments; unacceptable in multi-tenant setups. Best alternatives for security-sensitive environments: • Kaniko: Builds Docker images without Docker daemon — runs in unprivileged containers. Recommended in Kubernetes CI. • Buildah: Rootless image builds without daemon. • img: Rootless, daemonless image builder. • Podman: Rootless container runtime, Docker-compatible.

Systematic approach to diagnosing container startup failures: Step 1 — Check container status: ```bash docker ps -a # show all containers including stopped docker inspect <container_id> # check State.Status, State.ExitCode, State.Error ``` Step 2 — View logs: ```bash docker logs <container_id> # stdout/stderr from the container docker logs --details <id> # includes extra attributes ``` Step 3 — Common exit codes and meanings: • Exit 0: Clean exit (may be expected for one-shot tasks) • Exit 1: Application error (check logs for exception/error message) • Exit 125: Docker run itself failed (invalid flag or permission error) • Exit 126: Command found but not executable (permission denied) • Exit 127: Command not found (wrong path, missing binary in image) • Exit 137: SIGKILL — OOM kill (container hit memory limit) or explicit kill • Exit 139: Segmentation fault • Exit 143: SIGTERM — graceful shutdown (docker stop) Step 4 — Override entrypoint to debug interactively: ```bash docker run -it --entrypoint /bin/sh myimage # or for images without shell: docker run -it --entrypoint ls myimage /app ``` Step 5 — Check resource constraints: ```bash docker events --since 10m # shows OOM events, start/die events ``` Step 6 — Inspect image: ```bash docker inspect myimage # check CMD, ENTRYPOINT, ENV, WORKDIR docker history myimage # see all layers and their commands ``` Step 7 — Kubernetes context: ```bash kubectl describe pod <name> # Events section shows why pod failed kubectl logs <name> --previous # logs from crashed container ```

OCI: A set of open standards for container formats and runtimes, ensuring interoperability between different container tools. Created in 2015 by Docker, CoreOS, and other industry players to prevent vendor lock-in. Three OCI specifications: 1. OCI Image Spec: Defines the format for container images (layers, config JSON, manifest). Any tool that builds OCI-compliant images (Docker, Buildah, Podman, kaniko) produces images runnable by any OCI-compliant runtime. 2. OCI Runtime Spec (runc): Defines how to run a container (create namespaces, configure cgroups, execute the root process). runc is the reference implementation. Other runtimes: crun (faster, lower memory), kata-containers (VM-based isolation), gVisor (userspace kernel). 3. OCI Distribution Spec: Defines the HTTP API for pushing and pulling images from registries. All major registries (Docker Hub, ECR, GCR, ghcr.io) implement this spec — enabling tools to work with any registry. Practical impact: • docker build → produces OCI image • podman build → produces OCI image • buildah build → produces OCI image • All three images are interchangeable — push from Docker, pull with Podman Container runtime hierarchy: • High-level runtime (containerd, CRI-O): Manages image pulling, storage, network setup, calls low-level runtime • Low-level runtime (runc, crun): Creates the actual Linux namespaces and starts the container process CRI (Container Runtime Interface): Kubernetes interface that high-level runtimes implement. Kubernetes doesn't talk to Docker or containerd directly — it uses CRI. containerd and CRI-O are both CRI-compliant.

Docker is ideal for local database development — instant setup, isolated environments, disposable data. Running PostgreSQL locally: ```bash docker run -d \ --name dev-postgres \ -e POSTGRES_DB=myapp \ -e POSTGRES_USER=dev \ -e POSTGRES_PASSWORD=devpass \ -p 5432:5432 \ -v postgres-data:/var/lib/postgresql/data \ postgres:16-alpine ``` Connect: jdbc:postgresql://localhost:5432/myapp Docker Compose for full dev stack: ```yaml services: api: build: . ports: ["8080:8080"] environment: SPRING_DATASOURCE_URL: jdbc:postgresql://db:5432/myapp depends_on: db: condition: service_healthy db: image: postgres:16-alpine environment: POSTGRES_DB: myapp POSTGRES_USER: dev POSTGRES_PASSWORD: devpass volumes: [pgdata:/var/lib/postgresql/data] healthcheck: test: ["CMD", "pg_isready", "-U", "dev"] interval: 5s retries: 5 volumes: pgdata: ``` Fresh database for tests (no volume = ephemeral): ```bash docker run --rm -e POSTGRES_PASSWORD=test postgres:16-alpine # --rm: container deleted when stopped (data lost) # no volume: data in container layer (lost on stop) ``` Testcontainers: Java library that programmatically starts Docker containers in JUnit tests. Each test gets a fresh, isolated DB — no state leakage between tests. Data seeding: Use init scripts mounted to /docker-entrypoint-initdb.d/ — PostgreSQL/MySQL execute *.sql files in that directory on first startup. Multiple versions simultaneously: Run Postgres 15 on 5432 and Postgres 16 on 5433 — easily test migrations against different versions.

docker exec: Runs a new command inside an already-running container. Used for debugging, administration, and one-off tasks without stopping the container. Basic usage: ```bash docker exec <container_id> <command> docker exec -it myapp /bin/sh # interactive shell docker exec myapp env # list environment variables docker exec myapp cat /app/config.yml # read a file ``` Flags: • -i: Keep stdin open (for interactive input) • -t: Allocate a pseudo-TTY (makes output formatted properly) • -e: Set environment variables for the exec'd command • -u: Run as specific user: -u root • -w: Set working directory for the command Common use cases: 1. Database administration: ```bash docker exec -it postgres-container psql -U postgres docker exec -it mysql-container mysql -u root -p ``` 2. Check application state: ```bash docker exec myapp curl -s localhost:8080/actuator/health docker exec myapp ls -la /app/logs docker exec myapp ps aux ``` 3. Run migrations: ```bash docker exec myapp ./run-migrations.sh ``` 4. Debug networking: ```bash docker exec myapp ping database docker exec myapp nslookup redis-service ``` Vs docker attach: • docker exec: Runs a NEW process in the container. Clean, does not interfere with PID 1 • docker attach: Attaches to PID 1's stdin/stdout. Detach with Ctrl+P Ctrl+Q; pressing Ctrl+C may kill the container process Best practice: In production, avoid exec for regular operations — use proper APIs. Reserve exec for emergency debugging. Prefer structured log output over running ad-hoc queries inside containers.

SHELL: Overrides the default shell used for RUN, CMD, and ENTRYPOINT instructions in shell form. Default shells: • Linux: /bin/sh -c • Windows: cmd /S /C Using SHELL to change the default: ```dockerfile # Use bash for bash-specific features SHELL ["/bin/bash", "-c"] RUN echo "Using ${BASH_VERSINFO[0]} major bash version" # Use powershell on Windows SHELL ["powershell", "-command"] RUN Get-Process # Use fish shell SHELL ["/usr/bin/fish", "-c"] ``` Why use bash over sh: • Arrays, associative arrays • [[ ]] conditional syntax (more powerful than [ ]) • Process substitution <(command) • Here strings <<< • pipefail option: set -o pipefail — sh doesn't support this Pipefail is important for build correctness: ```dockerfile SHELL ["/bin/bash", "-o", "pipefail", "-c"] # Now pipeline failures are caught: RUN curl -fsSL https://example.com/install.sh | bash # Without pipefail: if curl fails, bash might still exit 0 # With pipefail: exit code reflects any pipe stage failure ``` Exec form vs Shell form: • Shell form: RUN apt-get install -y curl → becomes /bin/sh -c "apt-get install -y curl" • Exec form: RUN ["apt-get", "install", "-y", "curl"] → no shell involved, direct exec • SHELL only affects shell form instructions Windows containers: SHELL is essential for switching between cmd and PowerShell in Windows Dockerfiles.

Rootless Docker: Runs the Docker daemon and containers as a non-root user on the host. The daemon itself doesn't need root privileges — all container operations happen within the user's namespace. Traditional Docker security problem: • Docker daemon runs as root on the host • Docker socket (/var/run/docker.sock) owned by root • Any user in the docker group can effectively run root commands on the host: docker run -v /:/hostroot -it ubuntu chroot /hostroot — full host access • Container "root" user maps to host root — container breakout has maximum impact Rootless Docker: • Docker daemon runs as a regular user process • Uses user namespaces: container root maps to an unprivileged UID on the host • Even if a container escapes, the attacker gains only the limited privileges of the host user — not root Setup: ```bash dockerd-rootless-setuptool.sh install export DOCKER_HOST=unix:///run/user/1000/docker.sock docker run nginx # runs without root ``` Limitations of rootless Docker: • Some networking features require host network (overlay networks limited) • Cannot bind to privileged ports < 1024 (need net.ipv4.ip_unprivileged_port_start adjustment) • Performance overhead from user namespace UID mapping • Some storage drivers behave differently Kubernetes equivalent: • Run pods with securityContext.runAsNonRoot: true • User namespace support in K8s is available (alpha/beta) — maps pod UIDs to non-root host UIDs • Pod Security Standards enforce non-root and other security constraints Podman: Always rootless by default — no daemon, runs containers as the calling user.

Integration tests need real dependencies — databases, message brokers, caches. Docker Compose spins up the full stack for tests and tears it down cleanly after. Compose file for testing (compose.test.yml): ```yaml services: app: build: . environment: SPRING_DATASOURCE_URL: jdbc:postgresql://db:5432/testdb SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092 depends_on: db: condition: service_healthy kafka: condition: service_healthy db: image: postgres:16-alpine environment: POSTGRES_DB: testdb POSTGRES_USER: test POSTGRES_PASSWORD: test healthcheck: test: [CMD, pg_isready, -U, test] interval: 3s retries: 10 kafka: image: confluentinc/cp-kafka:7.5.0 environment: KAFKA_KRAFT_MODE: "true" CLUSTER_ID: "test-cluster" healthcheck: test: [CMD, kafka-topics, --bootstrap-server, localhost:9092, --list] interval: 5s retries: 10 ``` Run integration tests: ```bash docker compose -f compose.test.yml up --build --exit-code-from app docker compose -f compose.test.yml down -v # cleanup volumes ``` --exit-code-from app: Compose exits with the exit code of the app service (0 = tests passed, non-0 = failed). Testcontainers (Java/Go/Python): Programmatically spin up containers in test code: ```java @Container PostgreSQLContainer<?> db = new PostgreSQLContainer<>("postgres:16-alpine"); @Test void testOrderCreation() { // db is running and healthy before this test runs String jdbcUrl = db.getJdbcUrl(); } ``` Each test class gets a fresh container — complete isolation, no shared state.

Image tags identify specific versions. A poor tagging strategy makes deployments unpredictable and rollbacks difficult. Anti-pattern — only using :latest: • :latest is just a tag — not guaranteed to be newest, doesn't auto-update • docker pull may use a cached :latest — you might not get the newest image • Can't tell what version is running in production • Can't roll back to a specific version Recommended tagging strategy: 1. Git commit SHA (most precise): ``` myapp:a3f8b2c1 # immutable — always refers to exactly this build ``` 2. Semantic version: ``` myapp:1.2.3 # specific release myapp:1.2 # latest 1.2.x (mutable — updates with patches) myapp:1 # latest 1.x.x (mutable — updates with minor versions) ``` 3. Combined approach (recommended): ```bash # In CI/CD GIT_SHA=$(git rev-parse --short HEAD) docker build -t myapp:${GIT_SHA} -t myapp:${VERSION} -t myapp:latest . docker push myapp:${GIT_SHA} # always push immutable SHA tag docker push myapp:${VERSION} # push version tag ``` Deploy with SHA tag: ```yaml # Kubernetes deployment image: myregistry/myapp:a3f8b2c1 # pinned — GitOps knows exactly what's deployed ``` Kubernetes imagePullPolicy: • Always: Always pull from registry (even if cached) • IfNotPresent: Pull only if not in local cache — DO NOT use with mutable tags like :latest • Never: Only use cached image Branch-based tags for dev environments: ``` myapp:main-latest # latest build from main branch myapp:feature-auth # latest build from feature branch ```

Docker instructions (CMD, ENTRYPOINT, RUN) support two forms: shell form and exec form. Shell form (string): ```dockerfile CMD java -jar app.jar # Docker runs: /bin/sh -c "java -jar app.jar" ``` Process tree: sh(PID 1) → java(PID 2) Exec form (JSON array): ```dockerfile CMD ["java", "-jar", "app.jar"] # Docker runs: java -jar app.jar directly ``` Process tree: java(PID 1) Critical difference — signal handling: • Shell form: Signals sent to PID 1 (sh) are NOT forwarded to the child java process. SIGTERM from docker stop → sh exits, java killed with SIGKILL after timeout — NO graceful shutdown. • Exec form: java IS PID 1 and receives SIGTERM directly — can shut down gracefully. Always use exec form for CMD and ENTRYPOINT: ```dockerfile # WRONG — java won't receive SIGTERM CMD java -jar app.jar # CORRECT — java is PID 1, receives signals CMD ["java", "-jar", "app.jar"] # CORRECT — ENTRYPOINT + CMD combination ENTRYPOINT ["java", "-jar"] CMD ["app.jar"] ``` Shell form uses for RUN: RUN instructions don't need to handle signals. Shell form is often convenient for RUN because it allows shell features: ```dockerfile RUN apt-get update && apt-get install -y curl # shell && chaining works ``` tini init process: If your app doesn't handle SIGTERM well, use tini as PID 1: ```dockerfile RUN apt-get install -y tini ENTRYPOINT ["tini", "--"] CMD ["java", "-jar", "app.jar"] ``` tini properly forwards signals and reaps zombie processes.

CI/CD image builds must be fast (developer feedback loop) and efficient (cost, resource usage). GitHub Actions example: ```yaml name: Build and Push on: push: branches: [main] jobs: build: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: docker/setup-buildx-action@v3 - uses: docker/login-action@v3 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - uses: docker/build-push-action@v5 with: context: . push: true tags: | ghcr.io/myorg/myapp:${{ github.sha }} ghcr.io/myorg/myapp:latest cache-from: type=gha # use GitHub Actions cache cache-to: type=gha,mode=max # save all layers to cache platforms: linux/amd64,linux/arm64 ``` Caching strategies: • GitHub Actions cache (type=gha): Persists build cache in GitHub's cache storage. Free up to 10GB per repo. • Registry cache (type=registry): Store cache in registry alongside image. Works across different CI runners. • Inline cache: Embed cache metadata in the image itself (type=inline). Limited — only caches final stage. Optimize Dockerfile for CI cache: • Copy dependency files before source code — dependency layer cached until pom.xml/package.json changes • BuildKit cache mounts: --mount=type=cache,target=/root/.m2 persists Maven cache between builds PR vs main builds: • PRs: Build and test, don't push to registry (or push with PR tag for review) • Main: Build, test, push, deploy to staging • Tags (v1.2.3): Build, push, deploy to production

Container escape: An attacker exploits a vulnerability to break out of the container's isolation and gain access to the host system or other containers. Common escape vectors: 1. Privileged containers (--privileged): Grants all Linux capabilities and disables seccomp/AppArmor. Inside a privileged container: ```bash mount /dev/sda1 /mnt # mount host disk chroot /mnt # access entire host filesystem ``` Prevention: Never use --privileged. If specific capabilities needed, add only those: --cap-add NET_ADMIN. 2. Host namespace sharing: ```bash docker run --pid=host # see all host processes docker run --net=host # share host network stack docker run --ipc=host # share host IPC namespace ``` Prevention: Never share host namespaces unless absolutely necessary. 3. Docker socket mounting: docker run -v /var/run/docker.sock:/var/run/docker.sock → full Docker API access → start privileged containers. Prevention: Never mount Docker socket in production containers. 4. Kernel exploits: Containers share the host kernel. A kernel vulnerability (e.g., dirty pipe, runc CVEs) can allow escape. Prevention: Keep kernel and container runtime patched. Use gVisor or Kata Containers for untrusted workloads (separate kernel). Prevention checklist: • Run as non-root user inside containers • Use seccomp profiles to restrict syscalls • Use AppArmor/SELinux profiles • Drop unnecessary Linux capabilities (--cap-drop ALL, then add specifically) • Enable read-only filesystem (--read-only) • Scan images for vulnerabilities (Trivy) • Use minimal base images (fewer attack surfaces) • Enable Kubernetes Pod Security Standards (restricted profile) • Use admission controllers to enforce security policies

Database migrations in containerized environments require careful orchestration to ensure migrations run before the application starts, and run only once. Approach 1 — Application-managed migrations (Flyway/Liquibase): Spring Boot with Flyway runs migrations automatically on startup: ```yaml spring: flyway: enabled: true locations: classpath:db/migration validate-on-migrate: true ``` Problem: With multiple replicas, multiple pods may run migrations simultaneously → conflicts. Solution: Flyway/Liquibase use a lock table — only one instance runs migrations, others wait. Approach 2 — Init container (Kubernetes): ```yaml initContainers: - name: migrate image: myapp:latest command: [java, -jar, app.jar, --spring.profiles.active=migrate-only] # runs migrations, exits containers: - name: app image: myapp:latest # starts only after init container completes successfully ``` Approach 3 — Separate migration job: ```yaml # Kubernetes Job — runs migration to completion before deployment update apiVersion: batch/v1 kind: Job metadata: name: db-migrate-{{ .Release.Revision }} annotations: helm.sh/hook: pre-upgrade,pre-install helm.sh/hook-weight: "-5" ``` Approach 4 — Docker Compose (development): ```yaml services: migrate: image: flyway/flyway command: -url=jdbc:postgresql://db:5432/myapp -user=dev -password=dev migrate depends_on: db: condition: service_healthy app: depends_on: migrate: condition: service_completed_successfully ``` Zero-downtime migrations: Use expand-contract — additive schema changes first, deploy app, then remove old columns/tables in a separate migration.

Docker context: A configuration profile that defines how to connect to a Docker daemon. Allows switching between multiple Docker environments (local, remote server, cloud VM, Kubernetes) without changing environment variables. Default context: local Docker daemon via /var/run/docker.sock. Managing contexts: ```bash docker context ls # list all contexts docker context inspect default # details of default context # Create context for remote server docker context create production \ --docker "host=ssh://deploy@prod.example.com" # Switch context docker context use production # Run command in a specific context without switching docker --context production ps ``` Use cases: • Multiple environments: dev, staging, prod Docker hosts • Remote Docker hosts: Connect to a cloud VM's Docker daemon over SSH without installing Docker locally • Docker Desktop contexts: Docker Desktop manages contexts for Kubernetes integration • Team collaboration: Share context configs for team-standard environments Context over SSH: Docker communicates with the remote daemon via SSH tunnel — secure, no need to expose Docker daemon over TCP. ```bash # All Docker commands now run against the remote daemon docker context use production docker ps # shows containers on prod server docker images # shows images on prod server docker logs myapp # logs from prod container ``` Security: SSH-based contexts use your SSH keys — much safer than exposing the Docker TCP socket (which must be TLS-secured or it's unauthenticated). Never expose Docker daemon TCP without mTLS.

cgroups (Control Groups): A Linux kernel feature that limits, accounts for, and isolates resource usage (CPU, memory, disk I/O, network) of process groups. Docker uses cgroups to: • Limit memory: Prevent container from using more than allocated • Limit CPU: Control CPU shares, pinning, and quota • Limit I/O: Throttle disk read/write bandwidth and IOPS • Account: Measure actual resource consumption per container cgroups v1 vs v2: v1 (legacy): • Each resource type has a separate hierarchy: /sys/fs/cgroup/memory/, /sys/fs/cgroup/cpu/, etc. • No unified view of a process group's total resource usage • Complex to manage multiple controllers v2 (unified hierarchy, default in modern Linux kernels since ~2021): • Single unified hierarchy: /sys/fs/cgroup/<group>/ • All controllers managed in one place • Better delegation for rootless containers • Supports pressure-stall information for memory pressure notification • Required by newer Kubernetes versions for advanced features Docker cgroup hierarchy: Each container gets its own cgroup: /sys/fs/cgroup/docker/<container-id>/ Inspecting container cgroups: ```bash # See memory limit for a container cat /sys/fs/cgroup/memory/docker/<container-id>/memory.limit_in_bytes # See actual memory usage cat /sys/fs/cgroup/memory/docker/<container-id>/memory.usage_in_bytes # Or via Docker stats (easier) docker stats <container_id> ``` Kubernetes + cgroups: Kubelet creates cgroup hierarchy for nodes, pods, and containers. QoS classes (Guaranteed, Burstable, BestEffort) map to cgroup priorities.

Image scanning: Analyzes the software packages and libraries in a Docker image against known CVE (Common Vulnerabilities and Exposures) databases. Identifies security vulnerabilities before deployment. Trivy (recommended — open source by Aqua Security): ```bash # Scan an image trivy image myapp:latest # Scan with severity filter trivy image --severity HIGH,CRITICAL myapp:latest # Output as JSON for CI integration trivy image --format json --output results.json myapp:latest # Scan a local Dockerfile/filesystem trivy fs . # Fail CI if CRITICAL vulnerabilities found trivy image --exit-code 1 --severity CRITICAL myapp:latest ``` Trivy detects vulnerabilities in: • OS packages (apt, apk, yum) • Language packages (npm, pip, Maven, Go modules, Ruby gems, Cargo) • Configuration issues (Kubernetes YAML, Terraform) • Secrets (hardcoded passwords, API keys in files) Other scanners: • Snyk: SaaS with deeper remediation advice, IDE integration • Grype: Fast, open source, similar to Trivy • Docker Scout: Built into Docker CLI (docker scout cves myimage) • AWS ECR scanning: Automatic scanning on push using Inspector • GitHub Dependabot: Scans container base images in GitHub repos CI/CD integration (GitHub Actions): ```yaml - name: Scan image uses: aquasecurity/trivy-action@master with: image-ref: myapp:${{ github.sha }} format: sarif output: trivy-results.sarif severity: CRITICAL,HIGH exit-code: 1 - name: Upload SARIF uses: github/codeql-action/upload-sarif@v3 with: sarif_file: trivy-results.sarif # shows in Security tab ``` Remediating vulnerabilities: Update base image to patched version, update specific packages, or apply multi-stage build to exclude vulnerable tools that aren't needed at runtime.

Docker Compose creates a default network for each project. All services on the same network can communicate by service name — no IP addresses needed. Basic inter-service communication: ```yaml services: api: image: myapi environment: DB_URL: jdbc:postgresql://db:5432/myapp # "db" = service name REDIS_URL: redis://cache:6379 KAFKA_URL: kafka:9092 db: image: postgres:16-alpine # No ports needed — api accesses db:5432 on the compose network cache: image: redis:7-alpine kafka: image: confluentinc/cp-kafka:7.5.0 ``` Networks for isolation: ```yaml services: api: networks: [frontend, backend] # can talk to both web and db web: networks: [frontend] # can only reach api, not db db: networks: [backend] # can only be reached from api networks: frontend: backend: ``` External access: Only services with ports: exposed are accessible from the host. Internal services should not publish ports. Service aliases (alternative names on network): ```yaml services: api: networks: backend: aliases: [api-service, backend-api] # reachable by multiple names ``` Waithing for dependencies with health checks: ```yaml api: depends_on: db: condition: service_healthy kafka: condition: service_healthy ``` Without condition (just depends_on: [db]), Compose starts db before api but doesn't wait for it to be ready — the health check condition ensures actual readiness.

PID 1 has special responsibilities in Linux that most application processes are not designed to handle. Running your app as PID 1 without understanding this causes subtle production problems. PID 1 responsibilities: 1. Signal forwarding: PID 1 must explicitly handle and forward signals. Unlike non-PID-1 processes, the kernel does NOT deliver SIGTERM to PID 1 by default in some contexts — but Docker does send it. Your app must catch SIGTERM and shut down gracefully. 2. Zombie reaping: When any child process exits, it becomes a zombie until its parent calls wait(). If PID 1 doesn't reap zombies (call wait()), they accumulate and consume PID table entries. Eventually no new processes can be created. Problem example: Many apps spawn child processes (shell scripts, helper tools). If the app itself is PID 1 and doesn't reap, zombies accumulate in long-running containers. Solutions: 1. Use tini (init for containers): ```dockerfile RUN apt-get install -y tini ENTRYPOINT ["tini", "--"] CMD ["java", "-jar", "app.jar"] ``` tini runs as PID 1, properly reaps zombies, and forwards signals to your app. 2. Docker built-in init: ```bash docker run --init myimage # uses host tini as PID 1 ``` 3. Exec form CMD/ENTRYPOINT: Makes your app PID 1. Acceptable if your app handles SIGTERM and doesn't spawn children. 4. Shell form CMD (problematic): sh becomes PID 1. sh may not forward SIGTERM to your app → graceful shutdown never happens. Java-specific: JVM properly handles SIGTERM (triggers JVM shutdown hooks) when it is PID 1 with exec form. Spring Boot shutdown hooks close contexts and flush data.

Docker Compose profiles: A way to selectively start specific subsets of services in a Compose file. Services marked with a profile are not started by default — only when that profile is explicitly activated. Use case: A single compose.yml with all services, but different modes: start only the app and DB for development, add monitoring services for local testing, enable debug services when troubleshooting. Defining profiles: ```yaml services: # Core services (no profile = always started) api: image: myapp db: image: postgres:16 # Only started in dev profile adminer: image: adminer profiles: [dev] ports: ["8081:8080"] # Only started in monitoring profile prometheus: image: prom/prometheus profiles: [monitoring] grafana: image: grafana/grafana profiles: [monitoring] # Only started in debug profile jaeger: image: jaegertracing/all-in-one profiles: [debug] ``` Using profiles: ```bash # Start only core services (api + db) docker compose up # Start core + dev tools docker compose --profile dev up # Start core + monitoring docker compose --profile monitoring up # Start multiple profiles docker compose --profile dev --profile monitoring up # Environment variable alternative COMPOSE_PROFILES=dev,monitoring docker compose up ``` Service dependencies: A service with a profile can depend on core services. Core services don't need the profile to be active — they always start when any profile activates the composite set.

Monitoring strategy: Collect metrics at three levels — host, container, and application — then centralize and alert. Docker stats (basic, built-in): ```bash docker stats # live view: CPU, MEM, NET, BLOCK I/O docker stats --no-stream # snapshot (good for scripts) docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}" ``` Prometheus + cAdvisor (standard approach): • cAdvisor: Google's container advisor. Runs as a container, exposes per-container metrics (CPU, memory, network, disk I/O) on /metrics endpoint in Prometheus format. ```yaml services: cadvisor: image: gcr.io/cadvisor/cadvisor:v0.47.2 privileged: true volumes: - /:/rootfs:ro - /var/run:/var/run:ro - /sys:/sys:ro - /var/lib/docker:/var/lib/docker:ro ports: ["8080:8080"] ``` • Prometheus scrapes cAdvisor + application /metrics endpoints • Grafana dashboards: Docker dashboard (ID 193 or 14282 for cAdvisor) Alerts to configure: • Container CPU > 80% for 5 minutes • Container memory > 90% of limit • Container restart count increasing • Container not running (up{job="myapp"} == 0) Kubernetes monitoring (kube-state-metrics + metrics-server): • kube-state-metrics: Pod status, restart counts, resource limits vs requests • metrics-server: Real-time CPU/memory usage for kubectl top • Grafana Kubernetes dashboards: Node Exporter Full (ID 1860) Loki for logs: Correlate container logs with metrics in Grafana using shared labels (container name, pod name).

Docker volumes: Persist data beyond container lifecycle. When a container is deleted, its writable layer is gone — volumes survive. Volume operations: ```bash docker volume create mydata docker volume ls docker volume inspect mydata # shows mountpoint: /var/lib/docker/volumes/mydata/_data docker volume rm mydata docker volume prune # remove all unused volumes (careful!) ``` Using volumes: ```bash # Named volume (recommended) docker run -v mydata:/var/lib/postgresql/data postgres # Anonymous volume (harder to manage) docker run -v /var/lib/postgresql/data postgres # Compose services: db: image: postgres volumes: [pgdata:/var/lib/postgresql/data] volumes: pgdata: # defines a named volume ``` Backup a volume: ```bash # Create backup archive docker run --rm \ -v pgdata:/data \ -v $(pwd):/backup \ alpine tar czf /backup/pgdata-backup.tar.gz /data # Restore from backup docker run --rm \ -v pgdata:/data \ -v $(pwd):/backup \ alpine tar xzf /backup/pgdata-backup.tar.gz -C / ``` Database-specific backups: ```bash # PostgreSQL dump inside container docker exec postgres-container pg_dump -U postgres mydb > backup.sql # Restore cat backup.sql | docker exec -i postgres-container psql -U postgres mydb ``` Cloud volume drivers: • AWS EBS: docker volume create --driver rexray/ebs • AWS EFS: docker volume create --driver local --opt type=nfs • Kubernetes: PersistentVolumeClaim (PVC) abstracts the underlying storage — cloud-provider CSI driver provisions EBS, EFS, GCP PD automatically.

seccomp (Secure Computing Mode): A Linux kernel feature that restricts which system calls a process can make. Reduces the attack surface by preventing access to syscalls the application never needs. Docker default seccomp profile: Docker ships with a default seccomp profile that blocks ~44 dangerous syscalls (out of ~300+) including: • keyctl (kernel keyring — prevents credential theft) • ptrace (process tracing — prevents debugging/code injection) • reboot (prevent host reboot) • kexec_load (load another kernel) • mount (prevent arbitrary filesystem mounts) • Various clock setting syscalls Checking seccomp status: ```bash docker inspect <container_id> | jq '.[0].HostConfig.SecurityOpt' # Should show: "seccomp=...profile..." ``` Custom seccomp profile: ```json { "defaultAction": "SCMP_ACT_ERRNO", "syscalls": [ { "names": ["read", "write", "open", "close", "stat", "fstat", "mmap", "mprotect", "exit", "exit_group", "futex", "brk", "access", "execve", "arch_prctl", "munmap"], "action": "SCMP_ACT_ALLOW" } ] } ``` Applying: ```bash docker run --security-opt seccomp=./my-seccomp.json myimage # Disable seccomp (for privileged debugging only) docker run --security-opt seccomp=unconfined myimage ``` Kubernetes seccomp: ```yaml securityContext: seccompProfile: type: RuntimeDefault # use container runtime default profile # or type: Localhost, localhostProfile: profiles/custom.json ``` Seccomp + AppArmor + SELinux: Layered defense. seccomp blocks syscalls, AppArmor restricts file access and capabilities, SELinux enforces mandatory access controls. Use all three for defense-in-depth.

Distroless images: Container base images that contain only your application and its runtime dependencies — no package manager (apt, apk), no shell (/bin/sh), no system utilities (ls, curl, grep). Developed and maintained by Google. Available distroless bases (gcr.io/distroless/): • base: glibc, libssl, openssl, tzdata — for C/C++ apps • base-nossl: Without SSL libraries • java17: JRE 17 for Java apps • nodejs20: Node.js 20 runtime • python3: CPython 3 runtime • static: Fully static, no libc — for Go static binaries Example — Java distroless: ```dockerfile FROM eclipse-temurin:21-jdk AS builder WORKDIR /app COPY . . RUN mvn package -DskipTests FROM gcr.io/distroless/java21-debian12 WORKDIR /app COPY --from=builder /app/target/myapp.jar . CMD ["myapp.jar"] ``` Benefits: • Minimal attack surface: No shell = attacker can't drop to shell even after container escape. No package manager = can't install tools. • Smaller image: java21 distroless ~230MB vs eclipse-temurin:21-jre ~400MB • Fewer CVEs: Fewer packages = fewer vulnerabilities to patch Debugging challenge: No shell — docker exec -it container /bin/sh fails. Solutions: • Use multi-stage build with a debug stage that includes shell tools • Kubernetes: kubectl debug with an ephemeral container: kubectl debug -it <pod> --image=busybox • Docker: docker run -it --pid=container:<id> busybox (share PID namespace) When to use: Production images for security-sensitive services. Not for development (you need debugging tools locally).

Dependency caching: Separate dependency installation from source code copy. Dependency layers are cached until the dependency manifest changes — avoiding reinstallation on every code change. Java (Maven): ```dockerfile FROM maven:3.9-eclipse-temurin-21 AS builder WORKDIR /app COPY pom.xml . # copy manifest first RUN mvn dependency:go-offline -q # download all deps, cache this layer COPY src/ src/ RUN mvn package -DskipTests ``` Node.js: ```dockerfile FROM node:20-alpine WORKDIR /app COPY package.json package-lock.json ./ # copy manifests RUN npm ci # install deps — cached until lock file changes COPY . . # copy source (changes every commit) RUN npm run build ``` Python: ```dockerfile FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # cached until requirements.txt changes COPY . . ``` Go: ```dockerfile FROM golang:1.22-alpine AS builder WORKDIR /app COPY go.mod go.sum . # copy module files RUN go mod download # download all deps COPY . . RUN go build -o /app/server . ``` BuildKit cache mounts (better than layer caching for large caches): ```dockerfile # Maven with BuildKit cache mount — Maven cache persists between builds RUN --mount=type=cache,target=/root/.m2 mvn package -DskipTests # npm with cache mount RUN --mount=type=cache,target=/root/.npm npm ci # pip with cache mount RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt ``` Cache mounts don't add to image size (unlike caching via layers) and persist across ALL builds.

Image manifest: A JSON document that describes a Docker image — its config and the ordered list of layer blob digests needed to reconstruct the image. Manifest structure: ```json { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "digest": "sha256:abc...", // image config (ENV, CMD, labels, etc.) "size": 1234 }, "layers": [ {"digest": "sha256:111...", "size": 5000000}, // base image layer {"digest": "sha256:222...", "size": 200000}, // deps layer {"digest": "sha256:333...", "size": 50000} // app layer ] } ``` Manifest digest: The SHA256 hash of the manifest JSON. Pinning to a digest (myimage@sha256:abc...) is more secure than pinning to a tag — tags are mutable, digests are immutable. Manifest list (OCI Index): A manifest that points to platform-specific manifests. Enables multi-platform images. ```json { "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json", "manifests": [ {"digest": "sha256:amd...", "platform": {"os": "linux", "architecture": "amd64"}}, {"digest": "sha256:arm...", "platform": {"os": "linux", "architecture": "arm64"}} ] } ``` Practical commands: ```bash # Inspect an image manifest docker manifest inspect nginx:latest # Get the exact digest of the current tag docker inspect --format='{{index .RepoDigests 0}}' nginx:latest # → nginx@sha256:abc123... # Pull by digest (immutable) docker pull nginx@sha256:abc123... ``` Security: Deploy with digest pins in production. Tag can be silently replaced — digest cannot.

Network debugging requires checking connectivity, DNS resolution, and routing between containers. Step 1 — Verify containers are on the same network: ```bash docker network ls docker network inspect mynetwork # shows connected containers docker inspect container1 | jq '.[0].NetworkSettings.Networks' ``` Step 2 — Test connectivity (from inside a container): ```bash docker exec container1 ping container2 docker exec container1 curl -v http://container2:8080/health docker exec container1 nslookup container2 # DNS resolution docker exec container1 telnet container2 5432 # TCP connection test ``` Step 3 — Check if service is actually listening: ```bash docker exec container2 netstat -tlnp # listening ports docker exec container2 ss -tlnp ``` Step 4 — Tools container (if target has no debugging tools): ```bash # Use a debugging container on the same network docker run --rm --network mynetwork nicolaka/netshoot \ curl -v http://container2:8080/health # nicolaka/netshoot: packed with networking tools # (curl, tcpdump, dig, nmap, traceroute, ss, etc.) ``` Step 5 — Capture traffic: ```bash # tcpdump from inside a container docker exec container1 tcpdump -i eth0 host container2 # tcpdump from host targeting container's network interface CONTAINER_PID=$(docker inspect -f '{{.State.Pid}}' container1) nsenter -t $CONTAINER_PID -n tcpdump -i eth0 ``` Common issues: • Default bridge vs user-defined: Default bridge (docker0) doesn't support DNS by container name. User-defined bridge does. Fix: docker network create mynet; add both containers to mynet. • Port not published: Container A can reach container B by service name on container port — no -p needed for inter-container communication. • Firewall/iptables: Host firewall rules may interfere with container networking.

Container lifecycle: The states a container transitions through from creation to deletion. States: • Created: Container created (docker create) but not started. Resources allocated, filesystem ready, but no process running. • Running: Container is executing. PID 1 is alive. • Paused: Container processes frozen. cgroups freezer suspends all processes. Memory preserved. Resume with docker unpause. • Stopped/Exited: PID 1 has exited (normal or crash). Container filesystem still exists. Can be restarted. • Dead: Container in failed state that couldn't be cleaned up properly. • Removing: docker rm in progress. Lifecycle commands: ```bash docker create myimage # Created docker start container1 # Created → Running docker stop container1 # Running → Stopped (SIGTERM then SIGKILL) docker kill container1 # Running → Stopped (immediate SIGKILL) docker pause container1 # Running → Paused docker unpause container1 # Paused → Running docker restart container1 # Stopped/Running → Running (stop + start) docker rm container1 # Stopped → Deleted docker rm -f container1 # Running → Deleted (force kill + rm) ``` Shorthand: ```bash docker run = docker create + docker start docker run --rm = docker run + automatically docker rm on exit ``` Lifecycle events: ```bash docker events --filter event=die # watch for container deaths docker events --since 1h # events from last hour ``` Resource cleanup: ```bash docker container prune # remove all stopped containers docker system prune # remove stopped containers + dangling images + unused networks docker system prune -a # also remove unused images (not just dangling) ```

Certificate handling in containers requires careful management to avoid hardcoding secrets and to support rotation without image rebuilds. For outbound HTTPS (trusting internal CAs): If your service calls internal APIs signed by a corporate CA, add the CA cert to the image: ```dockerfile COPY internal-ca.crt /usr/local/share/ca-certificates/internal-ca.crt RUN update-ca-certificates ``` For inbound HTTPS (serving TLS): Never bake private keys into images. Options: 1. Mount certificate via volume/secret at runtime: ```bash docker run \ -v /etc/ssl/certs/tls.crt:/app/certs/tls.crt:ro \ -v /etc/ssl/private/tls.key:/app/certs/tls.key:ro \ myapp ``` 2. Kubernetes Secrets (recommended): ```yaml volumes: - name: tls-cert secret: secretName: myapp-tls containers: - name: myapp volumeMounts: - name: tls-cert mountPath: /app/certs readOnly: true ``` cert-manager automatically provisions and rotates Let's Encrypt certificates. 3. Terminate TLS at the ingress layer: Don't handle TLS in your application at all. Terminate at: • Nginx/Envoy sidecar • Kubernetes Ingress controller • Service mesh (Istio mTLS) • Load balancer (AWS ALB) Application only handles plain HTTP internally. 4. Vault PKI secrets engine: Application fetches short-lived TLS certificate from Vault on startup, rotates automatically before expiry. Vault Sidecar injector injects credentials into pods automatically. Certificate rotation: Use cert-manager with automatic certificate renewal. Mount as volumes — containers see updated certificates via volume refresh (no restart needed for apps that reload certs on SIGHUP).

COPY --chown: Copies files into the image and simultaneously sets ownership, without needing a separate RUN chown command (which would create an extra layer). Basic usage: ```dockerfile # Copy files owned by specific user COPY --chown=appuser:appgroup src/ /app/src/ # Use numeric UID/GID (for portability — no /etc/passwd lookup needed) COPY --chown=1001:1001 target/myapp.jar /app/myapp.jar ``` Full permission setup pattern: ```dockerfile FROM eclipse-temurin:21-jre # Create non-root user RUN groupadd --gid 1001 appgroup && \ useradd --uid 1001 --gid appgroup --no-create-home appuser WORKDIR /app # Copy with correct ownership (no extra layer needed) COPY --chown=appuser:appgroup target/myapp.jar . COPY --chown=appuser:appgroup config/ ./config/ # Switch to non-root user USER appuser EXPOSE 8080 CMD ["java", "-jar", "myapp.jar"] ``` Why not RUN chown: ```dockerfile # BAD: Creates an extra layer doubling storage for large files COPY target/myapp.jar /app/myapp.jar RUN chown appuser:appgroup /app/myapp.jar # extra layer! # GOOD: Single layer COPY --chown=appuser:appgroup target/myapp.jar /app/myapp.jar ``` ADD also supports --chown: ADD --chown=appuser:appgroup files.tar.gz /app/ Directory permissions: ```dockerfile RUN mkdir -p /app/logs /app/tmp && \ chown -R appuser:appgroup /app && \ chmod 755 /app && \ chmod 777 /app/logs # writable log directory ``` Minimal permissions principle: Read-only for everything except directories the app specifically needs to write to.

Blue-green deployment: Two identical environments (blue and green). One serves production traffic, the other is idle. Deploy new version to the idle environment, verify, then switch traffic. Instant rollback by switching back. With Nginx as load balancer: ```bash # Current state: blue is live docker run -d --name app-blue --network mynet myapp:1.0 # Deploy green (new version) docker run -d --name app-green --network mynet myapp:2.0 # Verify green is healthy curl http://app-green:8080/actuator/health # Switch traffic: update Nginx config cat > /etc/nginx/conf.d/app.conf <<EOF upstream backend { server app-green:8080; # switched from blue to green } EOF nginx -s reload # Verify traffic is flowing to green # Then remove blue docker rm -f app-blue ``` With Traefik (label-based routing): ```yaml # Blue: active services: app-blue: image: myapp:1.0 labels: - traefik.http.routers.app.rule=Host(`app.example.com`) - traefik.http.services.app.loadbalancer.server.port=8080 app-green: image: myapp:2.0 labels: - traefik.enable=false # not yet in rotation ``` Switch: Disable blue labels, enable green labels → Traefik auto-updates routing. With Kubernetes: ```yaml # Service selects by label spec: selector: version: blue # change to green to switch traffic ``` deploy-blue deployment + deploy-green deployment both exist. Change Service selector from blue → green. Advantage: Zero downtime. Green is fully tested before traffic switches. Rollback = re-point to blue. Disadvantage: Double infrastructure cost during deployment.

Over time, Docker accumulates unused images, stopped containers, dangling volumes, and unused networks — consuming disk space. Identify disk usage: ```bash docker system df # summary: images, containers, volumes, build cache docker system df -v # verbose: per-image and per-volume details ``` Cleanup commands: ```bash # Remove stopped containers docker container prune # Remove dangling images (untagged images from old builds) docker image prune # Remove all unused images (not just dangling — also removes tagged but unused) docker image prune -a # Remove unused networks docker network prune # Remove unused volumes (careful: contains data!) docker volume prune # Nuclear option: everything unused docker system prune docker system prune -a # + unused images docker system prune -a --volumes # + unused volumes (DATA LOSS RISK) ``` BuildKit cache: ```bash docker builder prune # remove build cache docker builder prune --keep-storage=5GB # keep only 5GB of most-recent cache ``` Dangling images: Images with no tag (shows as <none>) — left over when a tag is moved to a newer build. Safe to remove. Automating cleanup: ```bash # Cron job: daily cleanup of images older than 24h crontab -e 0 2 * * * docker system prune -af --filter "until=24h" >> /var/log/docker-cleanup.log 2>&1 ``` Registry cleanup: Registries (ECR, GCR) also accumulate images. Set lifecycle policies: • AWS ECR: Keep last 10 images, delete untagged images after 1 day • Docker Hub: Automated builds retention settings Production monitoring: Alert when Docker data root (/var/lib/docker) disk usage exceeds 80%.

FROM scratch: A special Dockerfile instruction that starts with an empty filesystem — no base image, no OS, no shell, no anything. You must provide literally every file your application needs. When to use FROM scratch: 1. Fully static binaries (Go, Rust, C): Languages that compile to fully static binaries with no runtime dependencies are perfect candidates: ```dockerfile FROM golang:1.22 AS builder WORKDIR /app COPY . . # Build fully static binary RUN CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-extldflags "-static"' -o server . FROM scratch COPY --from=builder /app/server /server # Copy any required files (CA certs, timezone data if needed) COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ COPY --from=builder /usr/share/zoneinfo /usr/share/zoneinfo EXPOSE 8080 CMD ["/server"] ``` Result: Minimal possible image — just your binary. Often < 10MB. Zero attack surface. 2. Custom base images: Build your own minimal base image for internal use: ```dockerfile FROM scratch ADD rootfs.tar.gz / # add a minimal OS filesystem ``` What you need to add manually: • CA certificates (for HTTPS calls to external services): /etc/ssl/certs/ca-certificates.crt • Timezone data (if your app uses time zones): /usr/share/zoneinfo • /etc/passwd and /etc/group (if running as non-root user) • Any shared libraries (for CGO-enabled binaries) Verify static binary: ```bash ldd ./server # should show: not a dynamic executable # or file ./server # should show: statically linked ``` Vs distroless: distroless provides a curated minimal environment (glibc, CA certs, timezone) without a shell. FROM scratch is more minimal but requires manual setup of everything.

Profiling containerized applications requires exposing profiling endpoints or using host-side tools that can reach into containers. Java profiling in containers: 1. JVM flags for profiling (add to JAVA_OPTS): ``` -XX:+FlightRecorder -XX:StartFlightRecording=duration=60s,filename=/tmp/recording.jfr -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9090 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false ``` 2. Async Profiler (low-overhead, production-safe): ```bash # Copy profiler into running container docker cp async-profiler/ myapp:/tmp/profiler # Run profile from inside container docker exec myapp /tmp/profiler/profiler.sh -d 30 -f /tmp/flame.html 1 # Copy results out docker cp myapp:/tmp/flame.html ./flame.html ``` 3. Heap dump: ```bash docker exec myapp jcmd 1 VM.heap_info docker exec myapp jmap -heap 1 docker exec myapp jcmd 1 GC.heap_dump /tmp/heap.hprof docker cp myapp:/tmp/heap.hprof ./heap.hprof # Analyze with Eclipse MAT or JProfiler ``` Memory analysis: ```bash # Container memory usage breakdown docker stats myapp --no-stream # Detailed cgroup memory info cat /sys/fs/cgroup/memory/docker/<id>/memory.usage_in_bytes cat /sys/fs/cgroup/memory/docker/<id>/memory.stat # OOM events dmesg | grep -i oom ``` Go profiling: ```go // Add pprof HTTP endpoint import _ "net/http/pprof" go http.ListenAndServe(":6060", nil) ``` ```bash # CPU profile docker exec myapp sh -c "curl -s localhost:6060/debug/pprof/profile?seconds=30 > /tmp/cpu.prof" docker cp myapp:/tmp/cpu.prof . && go tool pprof cpu.prof ```

Docker Engine: The core Docker daemon (dockerd) + CLI. Runs natively on Linux. No GUI. Open source (Moby project). Docker Desktop: A packaged application for Mac and Windows that includes Docker Engine (running in a lightweight Linux VM), Docker CLI, Docker Compose, Buildx, Kubernetes (optional), and a GUI dashboard. Why Docker Desktop on Mac/Windows: • Docker requires Linux kernel features (namespaces, cgroups). Mac and Windows don't have these. • Docker Desktop runs a lightweight Linux VM (HyperKit on Mac Intel, Apple Hypervisor on M1/M2, WSL2 on Windows). • The Docker CLI on your Mac/Windows communicates with the daemon running inside the VM. Performance consideration: • File sharing between host and VM adds overhead — bind mounts are slower than on Linux • Solution: Use volumes instead of bind mounts for data, use bind mounts only for development code sync • Docker Desktop 4.x has improved file sharing performance (VirtioFS on Mac) Docker Desktop alternatives (Mac/Windows): • Colima: Free, open-source. Lima VM + Docker Engine. Much lighter than Docker Desktop. • Podman Desktop: Rootless, daemonless, Docker-compatible CLI and GUI • Rancher Desktop: Includes K3s (Kubernetes) + dockerd or containerd • OrbStack: Fast, lightweight Docker + Linux on Mac (commercial) Licensing: Docker Desktop is free for personal use, education, and small businesses. Commercial use by companies > 250 employees or > $10M revenue requires a paid subscription. Kubernetes in Docker Desktop: One-click Kubernetes cluster for local development. Runs in the same VM as Docker. Alternative: kind (Kubernetes in Docker) or minikube.

Linux namespaces: A kernel feature that provides isolation by giving each process group its own view of specific system resources. Processes in different namespaces see different "universes" of resources. Docker uses 6 namespaces to isolate containers: 1. PID namespace: Container sees its own process tree starting at PID 1. Host processes are invisible. Container processes can't send signals to host processes. 2. NET namespace: Container has its own network interfaces, routing tables, iptables rules, and ports. Multiple containers can bind to port 8080 — they each have their own network stack. 3. MNT (Mount) namespace: Container has its own filesystem mount points. The root filesystem (/) is the container's image layers + writable layer. Host filesystem is not visible. 4. UTS (Unix Timesharing System) namespace: Container has its own hostname and domain name. docker run --hostname mycontainer sets the container's hostname independently of the host. 5. IPC namespace: Container has its own System V IPC objects and POSIX message queues. Prevents cross-container IPC by default. docker run --ipc=host shares the host IPC namespace. 6. User namespace (optional, rootless Docker): Maps container UIDs/GIDs to different UIDs/GIDs on the host. Container root (UID 0) maps to an unprivileged UID on the host. Enables rootless containers. NOT isolated by default (shared with host): • Time namespace: Containers share the host clock — can't change system time independently (though time namespace exists in newer kernels) • Cgroup namespace: Containers see a subset of cgroup hierarchy Inspecting namespaces: ```bash CONTAINER_PID=$(docker inspect -f '{{.State.Pid}}' mycontainer) ls -la /proc/$CONTAINER_PID/ns/ # see all namespaces for the container process ```

Services often depend on other services being fully ready — not just started. A database container may be running but not yet accepting connections. Basic depends_on (insufficient for readiness): ```yaml services: api: depends_on: - db # starts db first, but doesn't wait for it to be ready ``` With condition: service_healthy (requires health check): ```yaml services: api: image: myapp depends_on: db: condition: service_healthy kafka: condition: service_healthy redis: condition: service_healthy db: image: postgres:16-alpine environment: POSTGRES_USER: dev POSTGRES_PASSWORD: dev healthcheck: test: ["CMD-SHELL", "pg_isready -U dev"] interval: 5s timeout: 5s retries: 10 start_period: 10s # grace period before failures counted kafka: image: confluentinc/cp-kafka:7.5.0 healthcheck: test: ["CMD", "kafka-topics.sh", "--bootstrap-server", "localhost:9092", "--list"] interval: 10s retries: 10 start_period: 30s redis: image: redis:7-alpine healthcheck: test: ["CMD", "redis-cli", "ping"] interval: 3s retries: 5 ``` Condition types: • service_started: Container started (default — doesn't wait for readiness) • service_healthy: Container passes health check (requires healthcheck defined) • service_completed_successfully: For one-shot services (migrations, init tasks) — waits for exit code 0 Healthcheck parameters: • interval: How often to run the check • timeout: How long before a single check is considered failed • retries: How many consecutive failures before unhealthy • start_period: Time after start during which failures don't count toward retries

Containerd shim: A small process that sits between containerd and the container's runc process. Each container has its own shim process. Why the shim exists: 1. Daemon independence: Without the shim, restarting the Docker daemon or containerd would kill all running containers (since they'd be children of the daemon process). The shim allows containers to survive daemon restarts — the shim outlives its parent. 2. stdio handling: The shim holds open the container's stdin/stdout/stderr file descriptors after the container process starts. This prevents pipe closures when clients disconnect from docker attach. 3. Exit code reporting: After runc creates and executes the container and exits, the shim remains to capture and report the container's final exit code back to containerd. 4. Runtime abstraction: The shim implements the containerd shim API — different runtimes (runc, kata-containers, gVisor) implement the same shim protocol, allowing containerd to support multiple OCI runtimes. Process hierarchy: ``` systemd └── dockerd └── containerd ├── containerd-shim-runc-v2 (container1) │ └── runc (creates container) │ └── app-process (PID 1 in container) └── containerd-shim-runc-v2 (container2) └── app-process (PID 1 in container) ``` Runc exits after container creation. Shim persists until container exits. Daemon-less containers: This architecture (shim + runc) allows dockerd to restart without container disruption. In practice: systemctl restart docker — containers keep running.

Container data sharing: Containers have isolated filesystems by default. Several patterns enable data sharing. Pattern 1 — Named volumes: ```yaml services: processor: image: processor volumes: [shared-data:/data/input] consumer: image: consumer volumes: [shared-data:/data/input:ro] # read-only for consumer volumes: shared-data: ``` Both containers read/write the same volume. Simple but requires coordination (locking) to avoid conflicts. Pattern 2 — Sidecar container (log shipping): ```yaml services: api: image: myapp volumes: [log-vol:/app/logs] log-shipper: image: fluentd volumes: [log-vol:/fluentd/log:ro] volumes: log-vol: ``` Common pattern: App writes logs to volume, Fluentd sidecar reads and ships them. Pattern 3 — --volumes-from (legacy): ```bash # Data container pattern (old approach) docker run -d --name data-container -v /shared ubuntu docker run --volumes-from data-container myapp ``` Older pattern — named volumes are cleaner. Pattern 4 — Bind mount for host-shared data: ```bash docker run -v /host/shared-dir:/container/data myapp ``` All containers mounting the same host path share data. Host filesystem is the source of truth. Pattern 5 — Over the network (preferred for microservices): For actual microservices, services don't share filesystems — they share data via APIs, message queues, or shared databases. Direct filesystem sharing couples containers too tightly. Shared memory (IPC): ```bash docker run --ipc=shareable myapp1 docker run --ipc=container:myapp1 myapp2 # Both containers share the same IPC namespace # Enables shared memory (shm) between containers ```

Podman (Pod Manager): A daemonless, rootless container engine developed by Red Hat. OCI-compliant — runs the same container images as Docker. Key differences from Docker: 1. No daemon: Docker requires a running dockerd daemon. Podman directly calls the container runtime (runc/crun) — no background daemon process. More secure, simpler architecture. 2. Rootless by default: Podman containers run as the current user without root. User namespace maps container root to your unprivileged UID. 3. Pods: Native pod concept matching Kubernetes pods — multiple containers sharing network/IPC namespaces. Generate Kubernetes YAML from running pods. 4. Drop-in Docker replacement: alias docker=podman works for most commands. Docker CLI syntax is compatible. 5. Docker Compose: podman-compose works with Compose files. Or use podman generate kube to convert pods to Kubernetes manifests. Podman commands: ```bash podman run -d nginx podman build -t myapp . podman push myregistry/myapp:1.0 podman pod create --name mypod -p 8080:80 podman generate kube mypod > mypod.yaml # generate K8s manifest ``` Systemd integration: ```bash # Generate systemd service for auto-start podman generate systemd --new mycontainer > ~/.config/systemd/user/mycontainer.service systemctl --user enable mycontainer ``` When to choose Podman: • Security-first environments: Rootless by default, no privileged daemon • RHEL/CentOS/Fedora: Default container tool, well integrated with the OS • Kubernetes transition: Pod concept aids migration • CI without root: Works in environments where Docker socket is unavailable When to stick with Docker: Larger ecosystem, Docker Desktop GUI, wider tooling compatibility, established team familiarity.

Configuration management challenge: The same application image must run correctly in dev, staging, and prod with different config values (database URLs, feature flags, API keys). Approach 1 — Environment-specific Compose files: ```bash # Base config compose.yml # Environment overrides compose.dev.yml # local dev overrides compose.staging.yml # staging overrides compose.prod.yml # prod overrides # Merge files (later files override earlier) docker compose -f compose.yml -f compose.dev.yml up ``` compose.yml (base): ```yaml services: api: image: myapp:${IMAGE_TAG:-latest} environment: DB_HOST: ${DB_HOST} LOG_LEVEL: ${LOG_LEVEL:-INFO} ``` compose.dev.yml (override): ```yaml services: api: build: . # build locally instead of pulling image volumes: [.:/app] # bind mount for live reload environment: LOG_LEVEL: DEBUG ``` Approach 2 — .env files per environment: ```bash .env.dev → DB_HOST=localhost, LOG_LEVEL=DEBUG .env.staging → DB_HOST=staging-db, LOG_LEVEL=INFO .env.prod → DB_HOST=prod-db.internal, LOG_LEVEL=WARN docker compose --env-file .env.staging up ``` Approach 3 — Spring profiles in containers: ```bash docker run -e SPRING_PROFILES_ACTIVE=production myapp docker run -e SPRING_PROFILES_ACTIVE=staging myapp ``` Approach 4 — External config server: • HashiCorp Consul: KV store for config, service discovery • AWS SSM Parameter Store: Hierarchical config by environment path • Spring Cloud Config: Git-backed config server App fetches config from external store at startup — no env vars needed. Secrets: Never in Compose files or .env files in version control. Use Docker secrets, Vault, or cloud secrets managers.

docker commit: Creates a new image by capturing the current state of a running or stopped container's filesystem as a new layer. ```bash # Manual process example docker run -it ubuntu bash # Inside container: apt-get update && apt-get install -y nginx exit # Commit container state as new image docker commit <container_id> my-nginx:1.0 ``` Why docker commit is an anti-pattern: 1. Non-reproducible: No record of what commands were run inside the container. Can't recreate the image from scratch. If you need a slightly different version, you can't automate it. 2. No version control: What changed between my-nginx:1.0 and my-nginx:1.1? Impossible to know without diffing layers manually. 3. Includes garbage: The apt cache, temporary files, and any intermediate state from your manual work are baked into the image. 4. Breaks immutability: You're treating containers as pets (modifiable) instead of cattle (reproducible). 5. Security unknown: No way to audit what's inside — could include credentials entered interactively. When docker commit might be acceptable: • Creating a quick snapshot for debugging a specific production issue (not for permanent images) • Capturing state during interactive development exploration before writing a Dockerfile Dockerfile advantages: • Reproducible: Run `docker build` anywhere, get the same image • Version controlled: Dockerfile lives in git — full history • Reviewable: Code review catches security issues, bad practices • Layered caching: Only changed steps are re-executed • CI/CD compatible: Build triggered by code changes Always translate docker commit work into a Dockerfile before committing to version control.

Docker daemon configuration: Customize dockerd behavior through /etc/docker/daemon.json (Linux) or Docker Desktop settings (Mac/Windows). Common daemon configurations: ```json { "data-root": "/mnt/data/docker", "storage-driver": "overlay2", "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" }, "default-address-pools": [ {"base": "172.30.0.0/16", "size": 24} ], "registry-mirrors": ["https://mirror.example.com"], "insecure-registries": ["registry.local:5000"], "dns": ["8.8.8.8", "8.8.4.4"], "max-concurrent-downloads": 10, "max-concurrent-uploads": 5, "features": {"buildkit": true}, "experimental": false, "metrics-addr": "0.0.0.0:9323", "live-restore": true } ``` Key settings explained: • data-root: Change Docker's data directory (useful when /var is small) • log-driver + log-opts: Default logging for all containers (can override per container) • default-address-pools: Customize IP range for container networks (avoid conflicts with corporate networks) • registry-mirrors: Pull-through cache for Docker Hub — reduce rate limits and latency • insecure-registries: Allow HTTP (not HTTPS) for private registry — only for internal networks • live-restore: Keep containers running when daemon restarts (requires daemon restart, not container restart) • metrics-addr: Expose Prometheus metrics endpoint for monitoring dockerd itself Apply changes: ```bash sudo systemctl reload docker # reload without full restart (some settings) sudo systemctl restart docker # full restart (stops containers unless live-restore=true) ``` Validate config: ```bash docker info # shows current daemon configuration and storage driver ```

Build context: The set of files sent to the Docker daemon when you run docker build. It's the working directory for COPY and ADD instructions. What happens during docker build: 1. Docker CLI compresses and sends the build context to the daemon over the Docker API 2. Daemon receives the context (stored temporarily) 3. Dockerfile instructions run — COPY/ADD only work with files from this context 4. Context is deleted after build Performance impact: • Large context = slow upload even before build starts • Every build re-sends the full context to the daemon • Context size affects ALL builds, not just ones that COPY large files Checking context size: ```bash # See "Sending build context to Docker daemon Xmb" output docker build . # List what would be in context git ls-files # if repo is clean, this is your context ``` Reducing context: 1. .dockerignore: Most impactful. Exclude node_modules, .git, dist, test data. 2. Specific path: docker build ./src instead of docker build . — limits context to ./src directory 3. Build from stdin (no context): ```bash docker build - < Dockerfile # no context sent at all # Only works if Dockerfile has no COPY/ADD instructions ``` 4. Remote URL context: ```bash docker build https://github.com/org/repo.git#branch # clone and build docker build https://example.com/archive.tar.gz # download, extract, build ``` BuildKit improvement: BuildKit can skip transferring files that aren't referenced in the Dockerfile — smarter context handling. With BuildKit, context transfer is more efficient for large projects.

Canary deployment: Route a small percentage of traffic to the new version. If metrics are healthy, gradually increase the percentage. Roll back instantly by removing new version from rotation. Nginx weighted upstream: ```nginx upstream backend { server app-stable:8080 weight=9; # 90% traffic server app-canary:8080 weight=1; # 10% traffic } server { listen 80; location / { proxy_pass http://backend; } } ``` Compose setup: ```yaml services: nginx: image: nginx volumes: [./nginx.conf:/etc/nginx/nginx.conf] ports: ["80:80"] depends_on: [app-stable, app-canary] app-stable: image: myapp:1.0 deploy: replicas: 9 app-canary: image: myapp:2.0 deploy: replicas: 1 ``` Progressive rollout steps: 1. Deploy canary (1 instance): stable=9, canary=1 → 10% traffic 2. Monitor error rate, latency, business metrics for canary 3. If healthy, increase: stable=7, canary=3 → 30% traffic 4. Continue: stable=5, canary=5 → 50% 5. Fully promote: stable=0, canary=10 → 100% (rename canary to stable) Rollback: Set canary weight to 0, stable weight to 100 → instant recovery. Kubernetes canary: Use multiple Deployments + Service (split traffic by pod count) or use Ingress controller annotations (nginx-ingress: nginx.ingress.kubernetes.io/canary) or Argo Rollouts for automated progressive delivery with metric-based promotion. Monitoring canary: Compare P99 latency, error rate, and conversion rate between stable and canary cohorts. Automate promotion only if canary metrics are equal or better.

Linux capabilities: Fine-grained breakdown of root privileges. Instead of all-or-nothing root, capabilities split root permissions into ~40 distinct units. Docker drops many capabilities by default for security. Capabilities Docker drops by default (not available to containers): • SYS_ADMIN: Broad system administration (mounting, changing network namespaces) • NET_ADMIN: Network configuration (interface setup, iptables) • SYS_TIME: Set system clock • SYS_BOOT: Reboot • MAC_OVERRIDE: Bypass MAC (SELinux/AppArmor) • SYS_RAWIO: Raw device I/O Capabilities Docker keeps by default: • CHOWN: Change file ownership • DAC_OVERRIDE: Bypass file permission checks • FOWNER: Bypass permission checks for process owning file • NET_BIND_SERVICE: Bind to ports < 1024 • SETUID, SETGID: Set user/group IDs Managing capabilities: ```bash # Drop specific capabilities docker run --cap-drop DAC_OVERRIDE myimage # Drop ALL capabilities, then add only what's needed (principle of least privilege) docker run --cap-drop ALL --cap-add NET_BIND_SERVICE myimage # Add capabilities for special use cases docker run --cap-add SYS_PTRACE myimage # enable strace/gdb debugging docker run --cap-add NET_ADMIN myimage # network configuration tools ``` Kubernetes: ```yaml securityContext: capabilities: drop: [ALL] add: [NET_BIND_SERVICE] ``` Common capability needs: • Bind to port 80/443: NET_BIND_SERVICE (or better: run on high port, use iptables redirect or Kubernetes NodePort/LoadBalancer) • Packet capture (tcpdump): NET_RAW • Process tracing: SYS_PTRACE (debug only) • NFS mounts: SYS_ADMIN Best practice: Run with --cap-drop ALL and add only the specific capabilities required. Document why each capability is needed.

GitHub Actions provides hosted runners with Docker pre-installed. Use Docker to build, test, scan, and push images as part of your CI/CD pipeline. Complete CI/CD workflow: ```yaml name: CI/CD Pipeline on: push: branches: [main] pull_request: branches: [main] jobs: build-test-push: runs-on: ubuntu-latest permissions: contents: read packages: write # for GitHub Container Registry security-events: write # for SARIF upload steps: - uses: actions/checkout@v4 # Set up BuildKit - uses: docker/setup-buildx-action@v3 # Login to registry (only for main branch pushes) - uses: docker/login-action@v3 if: github.ref == 'refs/heads/main' with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} # Extract metadata for tags and labels - uses: docker/metadata-action@v5 id: meta with: images: ghcr.io/${{ github.repository }} tags: | type=sha # git sha type=semver,pattern={{version}} # v1.2.3 type=raw,value=latest,enable=${{ github.ref == 'refs/heads/main' }} # Build (and push if on main) - uses: docker/build-push-action@v5 with: context: . push: ${{ github.ref == 'refs/heads/main' }} tags: ${{ steps.meta.outputs.tags }} labels: ${{ steps.meta.outputs.labels }} cache-from: type=gha cache-to: type=gha,mode=max # Scan image for vulnerabilities - uses: aquasecurity/trivy-action@master with: image-ref: ghcr.io/${{ github.repository }}:sha-${{ github.sha }} severity: HIGH,CRITICAL exit-code: 1 ``` Optimizations: • type=gha cache: Persists BuildKit layer cache in GitHub Actions cache (10GB limit) • Conditional push: Only push on main branch, not on PRs • Matrix builds: Build multiple services in parallel using matrix strategy

Docker Compose Watch (introduced in Compose 2.22, Docker Desktop 4.24+): Automatically syncs file changes from your host into containers and triggers rebuilds — without manually stopping and restarting containers. Replaces the old "bind mount everything" approach for many use cases. Configuration in compose.yml: ```yaml services: api: build: . ports: ["8080:8080"] develop: watch: # Action: sync — copy changed files into container (fast) - path: ./src action: sync target: /app/src ignore: - src/**/*.test.ts # Action: rebuild — rebuild image and recreate container - path: ./pom.xml action: rebuild - path: ./Dockerfile action: rebuild # Action: sync+restart — sync file then restart container process - path: ./config/application.yml action: sync+restart target: /app/config/application.yml ``` Starting watch mode: ```bash docker compose up --watch # or docker compose watch ``` Actions: • sync: Hot-copies changed files into the container filesystem without restart. Works great for interpreted languages (Python, Node.js with hot reload enabled) or static files. • rebuild: Triggers a full docker compose build + container recreation. Use for dependency changes (pom.xml, package.json, requirements.txt). • sync+restart: Syncs file then sends restart signal to container. Use for config files the app reads on startup. Advantage over bind mounts: • More selective — only specific paths, not the entire project directory • Faster for large node_modules (not bind-mounted, stays in container) • Works with compiled languages (Java, Go) by syncing artifacts • No permission issues from UID mapping differences

DNS resolution in Docker: How containers look up other containers and external domains by name. Default bridge network (docker0) — LIMITED DNS: • No embedded DNS server • Containers only get the host's /etc/resolv.conf entries (external DNS only) • No container name resolution — must use container IP addresses or --link (deprecated) User-defined bridge networks — FULL DNS: • Docker runs an embedded DNS server at 127.0.0.11 inside every container • Configured in container's /etc/resolv.conf: nameserver 127.0.0.11 • Container names resolve to their container IP addresses • Network aliases also resolvable (set via docker run --network-alias or Compose aliases) Docker Compose: Always creates a user-defined network — service names are automatically resolvable: ```yaml services: api: image: myapp environment: DB_URL: jdbc:postgresql://db:5432/myapp # "db" resolves via embedded DNS db: image: postgres ``` DNS lookup process inside container: 1. App calls getaddrinfo("db") 2. glibc checks /etc/hosts first (container name + aliases) 3. Queries 127.0.0.11 (Docker embedded DNS) 4. Embedded DNS checks if "db" is a known service in the same network → returns container IP 5. If not found → forwards to external DNS (from /etc/resolv.conf options ndots:0) Custom DNS: ```bash docker run --dns 1.1.1.1 myimage # use specific DNS server docker run --dns-search example.com myimage # append search domain ``` Troubleshooting: ```bash docker exec mycontainer cat /etc/resolv.conf docker exec mycontainer nslookup db docker exec mycontainer dig db @127.0.0.11 ```

Layer sharing: Multiple containers and images share identical layers on disk. Docker identifies layers by content hash (SHA256 of compressed layer tarball). Same content = same hash = single copy on disk. How it works: • docker pull nginx:1.25 downloads layers and stores them by digest • docker pull nginx:1.26 — layers shared with 1.25 are NOT downloaded again • 100 containers running from nginx:1.25 — all share the same read-only layers • Each container only has its own thin writable layer (~KB to few MB overhead) Practical savings: ``` nginx:1.24 layers: base(50MB) + nginx-core(20MB) + config-1.24(5MB) nginx:1.25 layers: base(50MB) + nginx-core(20MB) + config-1.25(5MB) Disk usage: Without sharing: 150MB With sharing: 105MB (base + nginx-core shared, only configs duplicated) ``` Checking shared layers: ```bash docker image history nginx:1.24 # layers with sizes docker image history nginx:1.25 # compare digests docker system df -v # shows shared size vs unique size per image ``` Optimizing for layer sharing in your images: • Use the same base image across your microservices: FROM eclipse-temurin:21-jre • Put shared layers early: base OS → common runtime → app-specific layers • Common dependencies layer: Extract deps before copying source code • Internal base image: Build a custom base image with common deps, derive all services from it ```dockerfile # All company services share these layers FROM gcr.io/company/java-base:2024-01 # eclipse-temurin + company CA certs + monitoring agent COPY target/myservice.jar . ``` Multi-service Compose: All services using the same base image share its layers — zero storage overhead for the base.

Docker standalone doesn't include autoscaling — you need orchestration (Swarm or Kubernetes) or external tooling. Docker Swarm autoscaling (manual + tooling): Swarm has no built-in autoscaler. Solutions: 1. dockerd-exporter + Prometheus + custom scaler: ```bash # Scale based on custom metric docker service scale myapp=10 ``` 2. Swarmpit or Portainer: GUI tools that provide scaling controls. 3. Custom autoscaler script: ```bash #!/bin/bash # Query Prometheus for RPS RPS=$(curl -s "prometheus:9090/api/v1/query?query=rate(http_requests_total[1m])" | jq '.data.result[0].value[1]') if (( $(echo "$RPS > 1000" | bc) )); then docker service scale myapp=20 elif (( $(echo "$RPS < 200" | bc) )); then docker service scale myapp=5 fi ``` Kubernetes HPA (proper autoscaling): ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: myapp-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: myapp minReplicas: 2 maxReplicas: 50 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 - type: Pods pods: metric: name: kafka_consumer_lag target: type: AverageValue averageValue: "100" ``` KEDA (Kubernetes Event-Driven Autoscaling): Scale on Kafka consumer lag, SQS queue depth, HTTP traffic, or any custom metric. Nginx as load balancer: Update upstream block to match replica count, or use dynamic service discovery (Consul, etcd) to auto-update upstream list.

Docker Scout: A cloud-based service (integrated into Docker Desktop and Docker Hub) that provides continuous vulnerability monitoring, policy compliance checking, and software bill of materials (SBOM) for container images. Key capabilities: 1. Vulnerability analysis: Scans OS packages and language dependencies against CVE databases (NVD, GHSA, GitHub Advisory). Updated continuously — alerts you when new CVEs affect images you've already pushed. 2. SBOM (Software Bill of Materials): Generates a complete inventory of all packages and libraries in the image. 3. Policy compliance: Define policies (e.g., "no CRITICAL CVEs", "base image must be less than 30 days old") and fail builds that violate them. 4. Base image recommendations: Suggests when your base image has a newer version with fewer vulnerabilities. Usage: ```bash # Analyze a local or remote image docker scout cves myapp:latest docker scout cves nginx:1.24 # even without pulling # Quick overview docker scout quickview myapp:latest # Compare two images docker scout compare myapp:new-version myapp:latest # Show SBOM docker scout sbom myapp:latest # Check policy compliance docker scout policy myapp:latest ``` CI/CD integration: ```yaml - uses: docker/scout-action@v1 with: command: cves image: myapp:${{ github.sha }} only-severities: critical,high exit-code: true # fail build on CVEs ``` Comparison with Trivy: • Trivy: Free, open-source, runs locally, no cloud dependency, broader scope (IaC, secrets, K8s) • Docker Scout: Cloud-based continuous monitoring, integrates with Docker Hub, policy engine, base image tracking • Use both: Trivy in CI for immediate feedback, Scout for ongoing production image monitoring

Stateful applications (databases, message brokers, caches) require persistent storage and careful lifecycle management in containers. Key principles: 1. Use named volumes: Container filesystem is ephemeral — use volumes for all persistent data 2. Backup strategy: Volumes survive container deletion but not host failure — implement backups 3. Single replica: Most stateful apps shouldn't have multiple write replicas without clustering 4. Upgrade carefully: Data format changes between versions require migration PostgreSQL: ```yaml services: db: image: postgres:16-alpine environment: POSTGRES_DB: myapp POSTGRES_USER: appuser POSTGRES_PASSWORD_FILE: /run/secrets/db_password # Docker secret volumes: - pgdata:/var/lib/postgresql/data - ./init.sql:/docker-entrypoint-initdb.d/init.sql # auto-execute on first start healthcheck: test: [CMD-SHELL, "pg_isready -U appuser"] restart: unless-stopped volumes: pgdata: ``` Data safety: ```bash # Backup docker exec db pg_dump -U appuser myapp | gzip > backup-$(date +%Y%m%d).sql.gz # Upgrade Postgres version safely: # 1. Stop app # 2. Backup # 3. Use pg_upgrade inside a migration container # 4. Switch to new version image # 5. Start and verify ``` Redis: ```yaml services: cache: image: redis:7-alpine command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD} volumes: [redisdata:/data] # /data contains AOF/RDB files ``` Kubernetes for stateful apps: Use StatefulSets (stable network identity, ordered scaling, per-pod PVCs) + StorageClass for dynamic provisioning. Consider managed services (RDS, ElastiCache) over self-managed containers for critical data.

AppArmor (Application Armor): A Linux kernel security module that enforces mandatory access control (MAC) policies. Confines programs to a limited set of allowed actions based on a profile — filesystem access, network operations, and capability use. Docker + AppArmor: • Docker applies a default AppArmor profile (docker-default) to all containers automatically on AppArmor-enabled systems (Ubuntu, Debian) • The profile restricts container capabilities beyond what seccomp and capability drops enforce • Provides defense-in-depth alongside seccomp and namespace isolation Default docker-default profile restrictions: • Denies writing to sensitive kernel files (tune2fs, hugetlb, etc.) • Denies ptrace outside the container • Restricts mount operations • Denies capability-escalation techniques Checking AppArmor status: ```bash # Check if AppArmor is enabled sudo aa-status # Verify a container's AppArmor profile docker inspect <container_id> | grep AppArmor # Should show: "AppArmorProfile": "docker-default" ``` Custom AppArmor profile: ```bash # Create profile cat > /etc/apparmor.d/docker-nginx <<EOF #include <tunables/global> profile docker-nginx flags=(attach_disconnected, mediate_deleted) { #include <abstractions/base> network inet tcp, network inet udp, /var/log/nginx/** w, /etc/nginx/** r, deny /etc/passwd r, # prevent reading sensitive files } EOF # Load profile sudo apparmor_parser -r -W /etc/apparmor.d/docker-nginx # Apply to container docker run --security-opt apparmor=docker-nginx nginx ``` Disable AppArmor (for debugging): ```bash docker run --security-opt apparmor=unconfined myimage ``` Kubernetes: AppArmor annotation on pod spec or securityContext.appArmorProfile (beta in K8s 1.30).

By default, Docker's json-file driver stores container logs indefinitely — a container with verbose logging can fill the host disk, causing system failures. Per-container log configuration: ```bash docker run \ --log-driver json-file \ --log-opt max-size=50m \ --log-opt max-file=5 \ myapp # Keeps at most 5 files of 50MB each = 250MB max per container ``` Docker Compose: ```yaml services: api: image: myapp logging: driver: json-file options: max-size: "50m" max-file: "5" compress: "true" # gzip rotated files ``` Global daemon default (daemon.json): ```json { "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } } ``` Applies to all containers that don't specify their own logging config. Checking log file location and size: ```bash # Find log file path for a container docker inspect --format='{{.LogPath}}' <container_id> # → /var/lib/docker/containers/<id>/<id>-json.log # Check disk usage of all container logs sudo du -sh /var/lib/docker/containers/*/*.log | sort -rh | head -20 # Docker system df shows total log size docker system df ``` External log drivers (no local files): ```yaml logging: driver: awslogs options: awslogs-region: us-east-1 awslogs-group: /docker/myapp awslogs-stream: api-container ``` Ships directly to CloudWatch — no local log files at all. Also available: fluentd, gcplogs, splunk, syslog. Emergency recovery: If /var/lib/docker is full, truncate (don't delete) large log files: ```bash truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log ```

Production Docker best practices — the essential checklist: Image best practices: • Multi-stage builds: Separate build and runtime environments • Minimal base images: distroless, alpine, or slim variants • Specific version tags: Never :latest in production (use git SHA or semver) • Pin base image digests: FROM nginx@sha256:abc... for reproducibility • Scan images: Trivy/Scout in CI, block CRITICAL CVEs • Sign images: Cosign for supply chain security Runtime security: • Run as non-root: USER appuser in Dockerfile • Read-only filesystem: --read-only + tmpfs for /tmp • Drop capabilities: --cap-drop ALL + add only needed • No privileged: Never --privileged in production • Resource limits: --memory and --cpus on every container • seccomp profiles: Use default or custom restrictive profiles Reliability: • Health checks: Define HEALTHCHECK in Dockerfile • Graceful shutdown: Exec form CMD, handle SIGTERM, use tini • Restart policies: restart: unless-stopped for long-running services • Dependency ordering: Compose depends_on with service_healthy condition Observability: • Log to stdout/stderr: 12-factor, no log files in containers • Log rotation: max-size + max-file or external log driver • Expose metrics: /metrics endpoint for Prometheus scraping • Structured logs: JSON with trace_id, service name CI/CD: • Immutable images: Build once, promote through environments • GitOps: Deployment config in git (ArgoCD/Flux) • Canary/blue-green: Zero-downtime deployments • Rollback: Deploy previous image tag (seconds, not minutes) Orchestration: • Kubernetes for production: Full autoscaling, self-healing, declarative config • Avoid Docker standalone for multi-host production: Use K8s or Swarm • Pod disruption budgets: Ensure minimum availability during rolling updates

Kaniko: A tool that builds container images from a Dockerfile inside a Kubernetes pod or container — without requiring a Docker daemon or privileged access. Developed by Google. Why Kaniko exists: In Kubernetes CI/CD (Jenkins on K8s, GitLab Runner on K8s), you need to build Docker images from pipeline pods. Options: 1. Docker-in-Docker (DinD): Requires --privileged — major security risk in multi-tenant clusters 2. Docker socket mount: Gives pod full host Docker access — security risk, host contention 3. Kaniko: Runs as unprivileged container, no daemon, no socket mount How Kaniko works: • Reads Dockerfile from build context (local, GCS, S3, git) • Executes each Dockerfile instruction within the container's user space • Handles layer extraction and building without kernel namespace features • Pushes final image to registry directly Kubernetes Pod (Kaniko build): ```yaml apiVersion: v1 kind: Pod spec: containers: - name: kaniko image: gcr.io/kaniko-project/executor:latest args: - "--dockerfile=Dockerfile" - "--context=git://github.com/org/repo.git#refs/heads/main" - "--destination=myregistry/myapp:1.0" - "--cache=true" - "--cache-repo=myregistry/myapp/cache" volumeMounts: - name: kaniko-secret mountPath: /kaniko/.docker volumes: - name: kaniko-secret secret: secretName: regcred items: - key: .dockerconfigjson path: config.json ``` Limitations: • Slower than native Docker (no BuildKit parallelism initially, though improving) • Some Dockerfile features not supported (e.g., certain RUN --mount types) • Cache warming requires a registry cache — no local layer cache Alternatives: Buildah (rootless), img (rootless), Jib (Java-specific, no Dockerfile needed).

Node.js-specific Docker optimization patterns: 1. Cache node_modules separately: ```dockerfile FROM node:20-alpine WORKDIR /app # Copy manifests first — cached until package.json/lock changes COPY package.json package-lock.json ./ RUN npm ci --omit=dev # install only production dependencies # Copy source (changes more frequently) COPY . . USER node EXPOSE 3000 CMD ["node", "server.js"] ``` 2. npm ci vs npm install: • npm ci: Faster, reproducible (uses lock file exactly), removes node_modules before installing • npm install: Updates lock file, slower, non-deterministic • Always use npm ci in Docker builds 3. --omit=dev (production deps only): Development dependencies (jest, eslint, typescript) don't belong in production images. 4. Build-time compilation (TypeScript): ```dockerfile FROM node:20-alpine AS builder WORKDIR /app COPY package*.json ./ RUN npm ci # include devDeps for build COPY . . RUN npm run build # compile TS → JS FROM node:20-alpine WORKDIR /app COPY package*.json ./ RUN npm ci --omit=dev # production deps only COPY --from=builder /app/dist ./dist CMD ["node", "dist/server.js"] ``` 5. Node.js signals and PID 1: ```dockerfile # Use exec form — node becomes PID 1, handles SIGTERM CMD ["node", "server.js"] ``` In Node.js, handle SIGTERM: ```javascript process.on("SIGTERM", async () => { await server.close(); await db.disconnect(); process.exit(0); }); ``` 6. Base image choice: • node:20-alpine: Small (~50MB), fast pulls, security-conscious • node:20-slim: Debian slim, better glibc compatibility for native modules • node:20: Full Debian — needed for complex native modules (sharp, canvas) 7. .dockerignore for Node.js: ``` node_modules .npm npm-debug.log dist .git *.md ```

Host network mode: The container shares the host's network stack directly. No network namespace isolation — the container uses the host's IP addresses, routing tables, and ports directly. ```bash docker run --network host nginx # nginx binds to host port 80 directly — no -p flag needed ``` Differences from bridge mode: • No network isolation: Container is on the host network, not in a separate namespace • No port mapping overhead: No DNAT iptables rules, no NAT for outbound • No container DNS: Container names not resolvable — must use host IPs or host DNS names • Port conflicts: Container can conflict with host services on same port Performance: Host network mode eliminates the veth pair, bridge, and iptables overhead — typically 10-20% better network throughput for latency-sensitive workloads. When to use host network: • Performance-critical networking: Message brokers (Kafka), high-frequency trading, low-latency services where network overhead matters • Network monitoring tools: Prometheus node_exporter, Wireshark, tcpdump — need to see host network traffic • Services that bind many ports: Reduces iptables rule complexity • Host discovery: Services that need to discover host network topology When NOT to use: • Security-sensitive workloads: No network isolation — a compromised container has full access to host network • Multi-tenant environments: Containers from different tenants could interfere • Most web services: Bridge network overhead is negligible for HTTP Linux only: Host network mode only works on Linux. On Docker Desktop (Mac/Windows), the host is the Linux VM inside Docker Desktop — not your Mac or Windows machine. docker run --network host on Mac → container is on the VM network, not your Mac network.

SBOM (Software Bill of Materials): A complete inventory of all software components, dependencies, and packages inside a container image. Like an ingredient list for software — tells you exactly what's in your image, which packages, at what versions, and under which licenses. Why SBOMs matter: • Security: When a new CVE is disclosed (e.g., Log4Shell), you can immediately query your SBOMs to identify which images contain the vulnerable library • License compliance: Identify GPL/LGPL dependencies that may have licensing implications • Regulatory compliance: NIST, Executive Order 14028 (US), EU Cyber Resilience Act require SBOMs for software • Audit: Know exactly what's running in production Generating SBOMs: 1. Syft (open source by Anchore): ```bash # Generate SBOM for an image syft myapp:latest -o spdx-json > sbom.spdx.json syft myapp:latest -o cyclonedx-json > sbom.cdx.json # Attach SBOM as image attestation synft attest --output-file sbom.spdx.json myapp:latest ``` 2. Docker Scout: ```bash docker scout sbom myapp:latest docker scout sbom --format spdx myapp:latest ``` 3. BuildKit (built-in): ```bash docker buildx build \ --sbom=true \ --output type=registry \ -t myapp:latest . # SBOM automatically attached as OCI artifact alongside image ``` 4. Trivy: ```bash trivy image --format cyclonedx myapp:latest > sbom.json ``` SBOM formats: • SPDX (Software Package Data Exchange): Linux Foundation standard • CycloneDX: OWASP standard, better tooling ecosystem Attesting with Cosign: ```bash cosign attest --predicate sbom.cdx.json --type cyclonedx myapp:latest cosign verify-attestation --type cyclonedx myapp:latest # verify later ```

Air-gapped environments: No internet access. Docker Hub, external registries, and package mirrors are unreachable. Requires pre-staging all required images and packages. Solution components: 1. Private registry mirror (Harbor or Registry v2): • Pull images from internet on a bastion host • Push to internal registry • Configure all Docker hosts to pull from internal registry daemon.json on air-gapped hosts: ```json { "registry-mirrors": ["https://internal-registry.company.com"], "insecure-registries": ["internal-registry.company.com"] } ``` 2. Saving and loading images offline: ```bash # On internet-connected host: docker pull nginx:1.25 docker pull postgres:16-alpine docker save nginx:1.25 postgres:16-alpine | gzip > images.tar.gz # Transfer images.tar.gz to air-gapped host (USB, secure file transfer) # On air-gapped host: docker load < images.tar.gz # Both images now available locally ``` 3. Building images in air-gapped environments: • All base images must be in internal registry • Package installation (apt, apk, pip, npm) must use internal mirrors • BuildKit cache export — pre-warm cache from external environment ```dockerfile # Point to internal mirrors RUN sed -i 's|http://deb.debian.org|http://internal-debian-mirror|g' /etc/apt/sources.list && \ apt-get update && apt-get install -y curl ``` 4. Helm charts and Kubernetes: ```bash # Save all required images for a Helm chart helm template myapp ./chart | grep image: | awk '{print $2}' | xargs -I{} docker pull {} ``` 5. Automation — crane tool: ```bash # Copy image between registries without local daemon crane copy nginx:1.25 internal-registry.company.com/nginx:1.25 crane catalog internal-registry.company.com # list all images ```

Multi-container patterns: Combining containers to achieve capabilities that a single container can't or shouldn't. These patterns appear in both Docker Compose and Kubernetes (as pod patterns). Sidecar pattern: Secondary container that extends or enhances the primary container, sharing the same network and optionally the same volumes. ```yaml services: app: image: myapp volumes: [logs:/app/logs] log-shipper: # sidecar: ships logs from volume to Elasticsearch image: fluentd volumes: [logs:/fluentd/log:ro] volumes: logs: ``` Other sidecar uses: Envoy proxy (service mesh), secret rotator, config watcher. Ambassador pattern: Proxy container that handles network communication on behalf of the main container. Main container talks to localhost; ambassador handles service discovery, retries, TLS. ```yaml services: app: image: myapp environment: DB_HOST: localhost:6432 # talk to ambassador db-proxy: # ambassador: connection pooling, TLS termination image: pgbouncer environment: DATABASES_HOST: actual-db-host ``` Adapter pattern: Transforms output of the main container to a standard format. E.g., convert legacy log format to structured JSON before shipping. Init container pattern: Runs setup tasks before the main container starts. Compose: depends_on with service_completed_successfully. Kubernetes: initContainers. ```yaml services: migrate: # init: runs once to completion image: myapp command: java -jar app.jar --run-migrations app: depends_on: migrate: condition: service_completed_successfully ``` Vs microservices: These are intra-application patterns (within one logical application). Microservices are separate applications with their own codebases, teams, and deployment pipelines. A microservice might use sidecar pattern internally.

When a container crashes and exits, standard docker exec won't work. Several techniques for post-mortem analysis. Step 1 — Get exit information: ```bash docker ps -a --filter name=myapp # Shows: STATUS = Exited (137) 5 minutes ago # Exit code 137 = SIGKILL (OOM or explicit kill) docker inspect myapp | jq '.[0].State' # Shows: ExitCode, Error, OOMKilled, FinishedAt ``` Step 2 — View logs from crashed container: ```bash docker logs myapp # logs from crashed container docker logs --tail 200 myapp # last 200 lines docker logs --timestamps myapp # with timestamps ``` Step 3 — Check OOM: ```bash docker inspect myapp | jq '.[0].State.OOMKilled' # true if OOM killed dmesg | grep -i oom # kernel OOM killer log journalctl -k | grep -i oom ``` Step 4 — Create a new container from same image for investigation: ```bash # Override entrypoint to get a shell instead of the crashing app docker run -it --entrypoint /bin/sh myapp # Check if required files/configs exist ls -la /app/ cat /app/config.yml env | grep DATABASE ``` Step 5 — Core dump analysis (Java): ```bash # If container crashed due to JVM error, look for hs_err_pid*.log docker cp crashed-container:/app/hs_err_pid1.log . cat hs_err_pid1.log | head -100 # Heap dump on OOM # Add to JAVA_OPTS: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heap.hprof docker cp crashed-container:/tmp/heap.hprof . # Analyze with Eclipse MAT or JProfiler ``` Step 6 — Container from committed state: ```bash # Commit crashed container filesystem for inspection docker commit crashed-container debug-image docker run -it debug-image /bin/sh # Examine filesystem state at crash time ```

Health checks determine if a container is healthy enough to receive traffic. Different aspects of health require different probe types. Dockerfile health check: ```dockerfile # HTTP health check (requires curl in image) HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=3 \ CMD curl -f http://localhost:8080/actuator/health || exit 1 # TCP connection check (no dependencies) HEALTHCHECK --interval=15s --timeout=5s --retries=3 \ CMD nc -z localhost 8080 || exit 1 # Redis health check HEALTHCHECK CMD redis-cli ping || exit 1 ``` Spring Boot health endpoint: ```yaml management: endpoint: health: show-details: always probes: enabled: true # enables /actuator/health/liveness and /actuator/health/readiness endpoints: web: exposure: include: health,info,metrics,prometheus ``` Kubernetes probes (more powerful than Docker health checks): ```yaml containers: - name: myapp livenessProbe: # restart pod if fails httpGet: path: /actuator/health/liveness port: 8080 initialDelaySeconds: 30 # wait for JVM to start periodSeconds: 10 failureThreshold: 3 timeoutSeconds: 5 readinessProbe: # remove from service if fails (no traffic) httpGet: path: /actuator/health/readiness port: 8080 initialDelaySeconds: 15 periodSeconds: 5 failureThreshold: 3 startupProbe: # one-time startup check (slow apps) httpGet: path: /actuator/health/liveness port: 8080 failureThreshold: 30 # allow up to 5 minutes startup periodSeconds: 10 ``` Custom readiness logic: • Liveness: Is the JVM alive? Basic responsiveness. • Readiness: Are all dependencies (DB, Kafka, cache) accessible? Only ready when app can serve traffic. • Startup: Allow slow startup without false liveness failures. Dockerize utility: Wait for dependencies before starting: ```dockerfile RUN wget -O dockerize https://github.com/jwilder/dockerize/releases/download/v0.7.0/dockerize-linux-amd64 CMD ["dockerize", "-wait", "tcp://db:5432", "-timeout", "60s", "java", "-jar", "app.jar"] ```

BuildKit cache mounts and secret mounts are advanced Dockerfile features enabled by the BuildKit backend. They solve two common build problems: slow dependency downloads and sensitive build-time credentials. Cache mounts (--mount=type=cache): Persist a directory across builds. Unlike regular RUN layers, cache mounts don't add to the image size and are shared across all builds. ```dockerfile # syntax=docker/dockerfile:1 FROM maven:3.9-eclipse-temurin-21 WORKDIR /app COPY pom.xml . # Maven .m2 repository persists between builds — no re-downloading RUN --mount=type=cache,target=/root/.m2 \ mvn dependency:go-offline COPY src/ src/ RUN --mount=type=cache,target=/root/.m2 \ mvn package -DskipTests ``` ```dockerfile # npm cache mount RUN --mount=type=cache,target=/root/.npm \ npm ci --cache /root/.npm # pip cache mount RUN --mount=type=cache,target=/root/.cache/pip \ pip install -r requirements.txt # apt cache mount RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ --mount=type=cache,target=/var/lib/apt,sharing=locked \ apt-get update && apt-get install -y curl ``` Secret mounts (--mount=type=secret): Pass sensitive values to RUN instructions without including them in any image layer. ```dockerfile # Requires the secret to be passed at build time RUN --mount=type=secret,id=github_token \ GITHUB_TOKEN=$(cat /run/secrets/github_token) \ ./download-private-deps.sh # Maven with private repo credentials RUN --mount=type=secret,id=maven_settings,target=/root/.m2/settings.xml \ mvn package ``` Build with secrets: ```bash docker buildx build \ --secret id=github_token,src=~/.github_token \ --secret id=maven_settings,src=./settings.xml \ -t myapp:latest . ``` The secret is available only during that RUN instruction. It doesn't appear in docker history or any image layer.

Docker is powerful for single-host deployment and local development. But it has clear limits that Kubernetes is designed to address. Docker limitations: 1. Single host: Docker (without Swarm) runs on one machine. If that machine fails, all containers fail. No built-in distribution across multiple hosts. 2. No autoscaling: Docker has no mechanism to automatically scale containers based on CPU, memory, or custom metrics. Manual scaling only. 3. Limited self-healing: Docker restart policies restart crashed containers, but can't reschedule them to healthy hosts when a host fails. 4. No service discovery at scale: Works within one host or Swarm cluster. No integration with cloud load balancers or DNS-based service discovery across data centers. 5. No rolling updates: Docker has no native rolling update mechanism (Swarm has basic one). No canary, blue-green, or traffic splitting built-in. 6. Storage complexity: Volume management across multiple hosts requires external volume drivers. No native PersistentVolumeClaim abstraction. 7. Config and secrets: No built-in hierarchical config management or integration with external secret stores at scale. When to move to Kubernetes: • Multi-host requirements: Need to run containers across more than one machine for capacity or HA • High availability: Need to survive node failures without downtime • Autoscaling: Traffic spikes require automatic horizontal scaling • Complex deployments: Multiple interdependent services with health-dependent rollouts • Team scale: Multiple teams deploying independent services — need namespace isolation and RBAC • Compliance: Need audit logs, network policies, pod security policies Stay with Docker Compose if: • Single-host workloads • Development and testing environments • Simple applications where K8s operational overhead isn't justified • Team doesn't have Kubernetes expertise yet Path: Docker Compose locally → Docker Swarm for simple multi-host → Kubernetes for production at scale.