Enterprise Architecture Kubernetes · DevSecOps

Enterprise CI/CD on Kubernetes:
Ephemeral Azure DevOps Agents, JIT Elastic Compute, and Zero Standing Privileges

How OSI Digital built an event-driven, least-privilege CI/CD platform on AKS with KEDA, Karpenter, rootless BuildKit, Azure Workload Identity and JFrog Artifactory.

November 2026 15 min read By OSI DevOps & Platform Engineering Team
Executive Summary

Ephemeral, Least-Privilege CI/CD on AKS

Teams that outgrow multi-tenant cloud runners (private networking, data sovereignty, or audits such as SOC 2 Type II, ISO 27001, PCI-DSS and FedRAMP) usually fall back to static, self-hosted build VMs, which carry three hidden costs:

  • Developer friction: jobs queue behind statically sized pools in peak merge windows (head-of-line blocking).
  • FinOps waste: capacity sized for Monday-morning peaks idles nights and weekends.
  • Audit liabilities: unrotated standing credentials and reusable tokens with production deployment rights, non-deterministic state drift, and unconfined Docker daemons that violate CIS Kubernetes benchmarks and widen the supply-chain attack surface.

OSI Digital replaced it with an event-driven, zero-trust platform on Azure Kubernetes Service (AKS):

  • KEDA (Kubernetes Event-driven Autoscaling) starts a build agent just in time for each job, per toolchain demand.
  • Karpenter provisions right-sized, Spot-first nodes, shields running builds from Karpenter-initiated disruption, and consolidates empty nodes after a cooldown.
  • Each agent runs one job (--once), unregisters and terminates: a clean, isolated workspace per job.
  • Short-lived Azure Workload Identity OIDC tokens replace static Personal Access Tokens (PATs).
  • Images build on decoupled rootless BuildKit daemons over mutual TLS; verified dependencies come through JFrog Artifactory.

Result: Zero Standing Privileges (ZSP) today, and a roadmap to verifiable supply-chain provenance (SLSA/SBOM).

1:1 Pod-to-Job
Isolated workspace per job on the seven ScaledJob toolchains
~30 s
Typical queue-to-agent-start wait, versus 2–5 min VM boots
9 Toolchains
Unified under a single declarative GitOps Helm library
0 Standing PATs
100% OIDC federated Azure Workload Identity authentication
What It Changes

Cost: Spot capacity up to 90% below on-demand list pricing, above a small always-warm floor. Speed (DORA lead time): pull-request bursts get pods within one polling interval plus node start time, up to 20 replicas per toolchain and a 128-vCPU NodePool limit; beyond that, jobs queue. Security: no perpetual tokens, stored passwords or Docker sockets. Governance: a new toolchain is a ~30-line values manifest and a pull request, not a VM image-baking pipeline.

FinOps baselining: savings depend on pipeline concurrency, job duration variance and legacy VM sizing; OSI Digital establishes an empirical FinOps baseline before migration to quantify net ROI.

“The most resilient and secure build agent is the one that ceases to exist the moment its job terminates.” Platform Design Principle

Legacy Fleet Anti-Patterns

The legacy estate ran pipelines on long-running, self-hosted VMs, with four anti-patterns common to mature engineering organizations:

01

Inefficient Capacity Provisioning & Peak Sizing

Static VM pools sized for peak Monday-morning load ran underutilized in business hours and idle overnight and at weekends.

02

Non-Deterministic Workspace Contamination

Persistent workers reused filesystems, caches and temp directories, so leftover artifacts caused intermittent, non-reproducible failures that cost significant triage time.

03

Unconfined Runtimes & Container Privilege Drift

Image builds used rootful Docker-in-Docker with privileged: true (a host-escape vector), and unpinned toolchains let unvetted package updates cause silent drift between releases.

04

Perpetual Standing Credentials & Secret Sprawl

Workers registered with organization-wide PATs stored in cleartext on disk, so any compromised job running untrusted pull-request code could exfiltrate access to repositories and registries.

Target Architecture & Zero-Trust Threat Model

The platform decouples build triggering from compute: it watches Azure DevOps queue depth, provisions isolated compute on demand, enforces least privilege during the job, and reclaims capacity when it ends.

Architecture: Azure DevOps queues a job, the KEDA controller scales agent pods in the Kubernetes cluster, Karpenter provisions Spot and On-Demand AKS nodes, and agent pods pull images and dependencies from JFrog Artifactory and mount caches from Azure Files Premium
Figure 1: Event-driven execution plane on AKS.

Event-Driven Build Execution Sequence

  1. Job Enqueued: a pipeline declares a toolchain demand (e.g., Java, .NET, Helm).
  2. Queue Evaluation: KEDA polls the Azure DevOps agent pool REST API every 30 seconds (sub-minute response within REST API rate limits) with an isolated, read-only Microsoft Entra managed identity.
  3. Just-In-Time Pod: KEDA creates a dedicated 1:1 Kubernetes Job. With node capacity available, a warm or new pod starts from cached node images, typically about 30 seconds or less from queue to agent start.
  4. Node Auto-Provisioning: if the cluster is saturated, Karpenter provisions a right-sized VM from an allowed SKU matrix (Spot preferred) while pre-pull DaemonSets warm the node.
  5. OIDC Token Exchange: the pod's projected, signed Kubernetes ServiceAccount token is exchanged via Azure Workload Identity for a short-lived Microsoft Entra token scoped only to Azure DevOps.
  6. Least-Privilege Execution: the container runs as unprivileged UID 1001 with all capabilities dropped; packages resolve through JFrog Artifactory and image builds go to rootless BuildKit over mTLS.
  7. Deregistration & Consolidation: the runner unregisters (--once), the pod and its volumes are destroyed, and Karpenter consolidates empty nodes after a 5-minute cooldown (outside the business-hours disruption freeze).
Zero-Trust Boundaries

1. Control Plane: KEDA talks only to the Azure DevOps queue, using a read-only Entra identity with no execution permissions in the pool.

2. Execution Plane: agent pods run as non-root UID 1001 with drop: ["ALL"] and default seccomp profiles, with no access to the host Docker daemon or the Kubernetes API (automountServiceAccountToken: false). The short-lived Entra registration token stays in process memory, never on disk, but same-UID steps can still read the agent's pool-scoped credential under /azp and the (still exchangeable) projected Workload Identity token file: run untrusted pull-request builds in a separate agent pool with a minimally privileged identity.

3. Container Compilation: rootless BuildKit accepts builds only over mutual TLS (mTLS); Kubernetes NetworkPolicies limit its egress to private container registries and the JFrog Artifactory virtual repositories that Dockerfile package installs resolve through.

4. Supply Chain: all third-party packages (Maven, npm, NuGet, PyPI) resolve only through authenticated JFrog Artifactory virtual repositories with automated vulnerability scanning.

Topology
Detailed Architecture & Trust Boundaries
▾
topology.txt ASCII Architecture
+---------------------------- Azure DevOps --------------------------------+
|  Pipeline Job Queued  (Demands: [Java])                                   |
+------------------+-------------------------------------^----------------+
                   | Queue Polled every 30s              | Agent Registers via
                   | (KEDA Identity: Pool Read-Only)     | Ephemeral Token
+------------------v------------------- AKS -------------+----------------+
|                                                                          |
|   KEDA  --ScaledJob-->  [ Agent Pod: --once ]  --mTLS-->  BuildKit       |
|                               |        |                 (Rootless,      |
|                               |        |                  On-Demand)     |
|                Pending Pod    |        | Dependencies       |            |
|   Karpenter <-----------------+        |                    | Push/Pull  |
|      |  Spot / On-Demand VM            |                    |            |
|      v                                 |                    |            |
|   [ Build-Agent NodePool ]             |                    |            |
|     + Pre-Pull DaemonSet               |                    |            |
|     + Warm Pause Pod (Priority -1)     |                    |            |
+----------------------------------------|--------------------|-----------+
                                         v                    v
                          JFrog Artifactory          Private Container Registry
                       (Maven/npm/NuGet/PyPI)          (Azure Private Endpoint)

Technical Implementation

1. Modular Helm Architecture: Shared Library & Parameterized Toolchains

Instead of one monolithic, multi-gigabyte image with every toolchain, the fleet is nine independent agent charts on one centralized Helm library chart, which enforces mandatory security contexts, Workload Identity wiring, volume configurations and KEDA trigger templates. Each chart belongs to one of three workload classes (standard, light, heavy) or the stateful singleton, which standardizes its scheduling, capability routing and compute allocation. Values compose in three layers: toolchain values, workload-class defaults and an environment overlay:

Workload Class Toolchains & Runtimes Routing Demands Kubernetes Primitive Compute & Node Sizing
Node sizes; per-pod requests are set per chart (e.g. 1 vCPU / 1.5 GiB for jdk17)
standard
General Compilers
Java (JDK 17/21, Maven), .NET (SDK 10 LTS), Node.js (v22 LTS) Java, DotNet, Node Ephemeral ScaledJob
1:1 pod lifecycle (--once)
Azure Spot First (D/E/F series nodes, 4–8 vCPU / 8–16 GiB; on-demand fallback)
light
Automation & IaC
Terraform / Azure CLI, Helm / kubectl, Python 3.11+, SQL DACPAC Terraform, Helm, Python, SQL Ephemeral ScaledJob
Rapid sub-minute burst
Azure Spot First (High bin-packing density on 4 vCPU / 8–16 GiB nodes)
heavy
E2E Browser Suites
Selenium, Headless Chromium & Firefox test suites Selenium Warm ScaledObject
Persistent listener pool (30m cooldown)
Dedicated On-Demand (8–16 vCPU / 32+ GiB nodes; zero Spot eviction risk)
stateful
Singleton Scanner
OWASP Dependency-Check (NVD vulnerability database) OWASP Dedicated StatefulSet
Singleton runner, daily sidecar sync
On-Demand + Managed Disk (Persistent retained disk for multi-GB DB)
YAML
Declarative Values Composition Stack
▾
values-stack.yaml YAML
# 1. charts/agents/jdk17/values.yaml — toolchain-specific specification
workloadClass: standard
agent:
  name: jdk17
image:
  repository: <acr-registry>.azurecr.io/build-agents-jdk17
  tag: v1.2.0@sha256:<digest>
capabilities:           # Primary capability matches Azure DevOps pipeline demands
  - Java
  - JDK17
  - Maven
  - Docker                # Docker/BuildKit = buildctl client to the remote BuildKit daemon
  - BuildKit
resources:
  requests: { cpu: 1000m, memory: 1500Mi }  # Burst capacity without CPU throttling limits

# 2. charts/agents/_classes/standard.yaml — workload scheduling & scaling class
keda:
  scalingMode: ScaledJob   # Strict 1:1 job-to-pod lifecycle
  minReplicaCount: 1       # Pre-warmed listener registered to eliminate pod bootstrap delay
  pollingInterval: 30      # 30s evaluation window prevents downstream REST throttling
scheduling:
  protectFromDisruption: true   # karpenter.sh/do-not-disrupt on all running build pods
  nodeSelector: { workload: build-agent }

# 3. envs/agents/jdk17.yaml — environment-specific authentication bindings
workloadIdentity:
  enabled: true
  clientId: "<managed-identity-client-id>"
  tenantId: "<entra-tenant-id>"
buildkit:
  tls: { enabled: true }

2. Ephemeral Agents with KEDA ScaledJobs

Seven of the nine toolchains run as KEDA ScaledJobs: a matching queued job starts a Kubernetes Job whose pod runs the Azure Pipelines agent with --once, deregisters and terminates; Kubernetes then destroys the pod and its ephemeral storage. backoffLimit: 0 (no automatic pod retries on build failures) and a hard 2-hour activeDeadlineSeconds stop failed or hung builds from leaving runaway orphan processes.

Architectural Exceptions: two workloads deliberately avoid ScaledJobs:

YAML
KEDA ScaledJob & TriggerAuthentication CRDs
▾
keda-scaledjob.yaml YAML
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: build-agent-jdk17
  namespace: build-agents
spec:
  pollingInterval: 30
  minReplicaCount: 1                 # Pre-warmed listener; immediately replenished on execution
  maxReplicaCount: 20                # Per-toolchain blast radius ceiling
  scalingStrategy:
    strategy: accurate               # Direct queue depth matching
  successfulJobsHistoryLimit: 5
  failedJobsHistoryLimit: 20         # Retain failed job pods for post-mortem forensics
  jobTargetRef:
    parallelism: 1
    completions: 1
    backoffLimit: 0                  # Prevent automatic pod retries on build failures
    activeDeadlineSeconds: 7200      # Hard 2-hour execution boundary
    ttlSecondsAfterFinished: 600     # Automated pod and volume cleanup after 10 minutes
    template:
      metadata:
        annotations:
          karpenter.sh/do-not-disrupt: "true"
        labels:
          azure.workload.identity/use: "true"
      spec:
        restartPolicy: Never
        serviceAccountName: build-agent-jdk17
        automountServiceAccountToken: false   # No Kubernetes API token; Workload Identity projects its own
        nodeSelector: { workload: build-agent }
        tolerations:                     # Required: the NodePool taints nodes workload=build-agent:NoSchedule
          - { key: workload, operator: Equal, value: build-agent, effect: NoSchedule }
        securityContext:
          runAsNonRoot: true
          runAsUser: 1001
          seccompProfile: { type: RuntimeDefault }
        containers:
          - name: agent
            image: <acr-registry>.azurecr.io/build-agents-jdk17@sha256:<digest>
            env:
              - { name: AZP_URL,  value: "https://dev.azure.com/<org>" }
              - { name: AZP_POOL, value: "<pool-name>" }
            resources:
              requests: { cpu: 1000m, memory: 1500Mi }   # Karpenter sizes nodes from requests
            securityContext:
              allowPrivilegeEscalation: false
              capabilities: { drop: ["ALL"] }
              # readOnlyRootFilesystem: true   # rolling out; needs emptyDir mounts for /azp and /tmp
            args: ["--once"]                 # Execute single job, unregister, and terminate
  triggers:
    - type: azure-pipelines
      metadata:
        poolID: "<pool-id>"
        organizationURLFromEnv: AZP_URL
        targetPipelinesQueueLength: "1"
        activationTargetPipelinesQueueLength: "0"
        demands: "Java"                   # Primary capability demand match
        requireAllDemands: "true"
      authenticationRef:
        name: keda-trigger-auth
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth
  namespace: build-agents
spec:
  podIdentity:
    provider: azure-workload            # Read-only OIDC queue interrogation; zero static PAT
    identityId: "<keda-identity-client-id>"

3. Single-Demand Job Routing

KEDA matches each pipeline's declared demands against the scaler's demand list. Multi-capability matching causes Cartesian scaling bugs when business units share a queue (see Incident 01), so each toolchain scaler matches exactly one routing capability (e.g., Java, DotNet, Helm). Utilities baked into every runner image (e.g., Docker, BuildKit, git) are advertised as Azure DevOps capabilities but never used as KEDA scaling criteria.

4. Unified Karpenter NodePool with Spot Prioritization

One single heterogeneous Karpenter NodePool, rather than fragmented node groups, spans D, E and F-series Azure VM families (4 to 32 vCPUs). Karpenter picks the most cost-effective SKU for pending agent pods' resource requests, prioritizing Azure Spot capacity.

Disruption Handling: karpenter.sh/do-not-disrupt: "true" on every ephemeral build pod blocks Karpenter's own voluntary disruption (consolidation, drift and expiry) while a job runs, but not Azure Spot evictions: when Azure reclaims a Spot VM the pod is lost and, with backoffLimit: 0, the run fails and must be retried. Spot therefore suits short, retry-tolerant jobs (use pipeline-level retry on infrastructure failure); long-running or preemption-sensitive workloads request On-Demand instances in the same shared pool via a nodeSelector. Consolidation is WhenEmpty with a 5-minute stabilization window, and a disruption budget blocks voluntary node churn during business hours (which also delays reclaiming empty nodes in that window). Budget schedules are evaluated in UTC, so set the cron to your engineering hours:

YAML
Karpenter NodePool & AKSNodeClass Specification
▾
karpenter-nodepool.yaml YAML
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: build-agents
spec:
  template:
    metadata:
      labels:
        workload: build-agent
    spec:
      nodeClassRef:
        group: karpenter.azure.com
        kind: AKSNodeClass
        name: build-agents
      taints:
        - { key: workload, value: build-agent, effect: NoSchedule }
      requirements:
        - { key: kubernetes.io/arch,            operator: In, values: ["amd64"] }
        - { key: karpenter.sh/capacity-type,     operator: In, values: ["spot", "on-demand"] }
        - { key: karpenter.azure.com/sku-family, operator: In, values: ["D", "E", "F"] }
        - { key: karpenter.azure.com/sku-cpu,    operator: In, values: ["4", "8", "16", "32"] }
      expireAfter: 168h                # Weekly rolling node refresh
      terminationGracePeriod: 24h      # Safeguard active builds during node drains
  limits:
    cpu: "128"                         # Cluster blast-radius cap preventing runaway autoscaling
    memory: 512Gi
  disruption:
    consolidationPolicy: WhenEmpty     # Reclaim nodes strictly when all pods terminate
    consolidateAfter: 5m
    budgets:
      # Schedules are evaluated in UTC: set the cron to your engineering hours.
      # nodes: "0" blocks ALL voluntary disruption in the window, including reclaiming empty nodes.
      - { nodes: "0", schedule: "0 8 * * MON-FRI", duration: 8h }
      - { nodes: "20%" }
---
apiVersion: karpenter.azure.com/v1beta1
kind: AKSNodeClass
metadata:
  name: build-agents
spec:
  vnetSubnetID: <subnet-resource-id>
  osDiskSizeGB: 512
  imageFamily: Ubuntu2204

5. Multi-Tier Cold-Start Optimization

Ephemeral agents need start-up that rivals warm runners. Unoptimized, a Kubernetes pod takes 2 to 5 minutes (node provisioning, image layer pulls, toolchain downloads); four tiers keep the typical queue-to-agent-start wait at about 30 seconds or less. The balloon pod hides cold node provisioning only while its pre-warmed node has spare capacity:

Cold-Start Overhead DriverUnoptimized LatencyArchitectural Remediation
Node Provisioning & VM Boot60–120 sNegative-priority balloon pause pods hold a pre-warmed node; a real build pre-empts the pause pod immediately, triggering background node scale-out.
Container Image Pulling30–90 sA system-node-critical pre-pull DaemonSet loads all agent images into each new node's local containerd cache immediately.
Agent Binary Installation10–20 sThe official Azure DevOps agent binary is baked into the container image; runtime external downloads are disabled.
Dependency Resolution & Restore15–60 sJFrog Artifactory virtual repositories (a nearby private proxy cache) plus read-only, pre-seeded package caches on Azure Files; anything writable stays in the per-pod emptyDir.
Storage Architecture: Cache Trust Boundaries vs. Workspace Isolation

npm's cache (~/.npm) is content-addressed and integrity-checked, so sharing it is low risk. Maven (~/.m2/repository) and NuGet (~/.nuget/packages) caches are keyed by name and version and can hold mutable SNAPSHOTs and resolver metadata: a writable shared cache would let an untrusted pull-request build plant an artifact a later release build consumes, and concurrent writers over Azure Files risk corruption. So only read-only, pre-seeded caches are shared; writable caches are per-pod and discarded, and Artifactory provides proximity. Workspaces, build trees, temp directories and credential volumes are ephemeral emptyDir volumes destroyed with the pod.

6. Rootless Container Builds with Mutual-TLS BuildKit

Docker-in-Docker (DinD) needs privileged: true and host Docker socket mounts, violating security baselines, so all pipelines use a centralized rootless BuildKit daemon via the lightweight buildctl CLI:

7. Short-Lived Credentials with Azure Workload Identity (OIDC)

Every build agent service account is federated to an Azure Managed Identity via Azure Workload Identity; that identity is a Basic-access Azure DevOps user, and no Personal Access Token (PAT) is active in the organization. At pod initialization, the Kubernetes admission webhook projects an OIDC token to /var/run/secrets/azure/tokens/azure-identity-token.

A lightweight entrypoint exchanges it with Microsoft Entra ID (an OAuth 2.0 client-assertion JWT exchange using only curl and jq, no Azure CLI) for a short-lived token scoped to Azure DevOps (Resource ID: 499b84ac-1321-427f-aa17-267ca6975798). The token registers the agent once and is then unset; VSO_AGENT_IGNORE also keeps credential variables out of step environments. Deregistration re-acquires a fresh token, because Entra access tokens last roughly 60 to 90 minutes and a job can run for 2 hours. Steps running as the same UID can still read the projected token file (see the Execution Plane boundary), so protection comes from short lifetime and narrow scope, not from hiding the file:

Bash
Agent Bootstrap & Workload Identity Token Exchange
▾
start.sh Bash
#!/bin/bash
set -euo pipefail
AGENT_DIR=/azp

# 1. Workload Identity OIDC exchange (re-runnable: also used at teardown)
get_token() {
  curl -sf --retry 3 -X POST \
    "${AZURE_AUTHORITY_HOST%/}/${AZURE_TENANT_ID}/oauth2/v2.0/token" \
    -d "grant_type=client_credentials" \
    -d "client_id=${AZURE_CLIENT_ID}" \
    -d "client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer" \
    --data-urlencode "client_assertion=$(cat "${AZURE_FEDERATED_TOKEN_FILE}")" \
    -d "scope=499b84ac-1321-427f-aa17-267ca6975798/.default" \
    | jq -r '.access_token // empty'
}

# Fail closed: never fall back to an ambient PAT
unset AZP_TOKEN
[ -f "${AZURE_FEDERATED_TOKEN_FILE:-}" ] || { echo "[FATAL] No federated token file" >&2; exit 1; }
TOKEN="$(get_token || true)"
[ -n "${TOKEN}" ] || { echo "[FATAL] Workload Identity token exchange failed" >&2; exit 1; }

# 2. Register, then drop the token from the process (the Entra token is never written to disk;
#    the agent itself stores its pool-scoped registration credential under /azp)
# The agent's "PAT" auth scheme accepts the Entra access token; no PAT exists.
"${AGENT_DIR}/config.sh" --unattended --replace --agent "$(hostname)" \
  --url "${AZP_URL}" --pool "${AZP_POOL}" --auth PAT --token "${TOKEN}"
unset TOKEN
export VSO_AGENT_IGNORE="AZP_TOKEN,AZP_TOKEN_FILE"

# 3. Deregister with a FRESH token: the registration token may expire during a 2h job
cleanup() {
  local t
  t="$(get_token || true)"
  if [ -n "${t}" ]; then
    "${AGENT_DIR}/config.sh" remove --unattended --auth PAT --token "${t}" \
      || echo "[WARN] deregistration failed; a stale agent may remain" >&2
  else
    echo "[WARN] could not obtain a token to deregister" >&2
  fi
}
trap cleanup EXIT
trap 'exit 143' TERM

"${AGENT_DIR}/run.sh" "$@" &     # pass --once
wait $!

8. Hardened Images & Digest Pinning

Every base-image layer is sha256-pinned and all toolchain binaries are version-locked. Agents run as unprivileged UID 1001, drop all Linux kernel capabilities, and are moving to readOnlyRootFilesystem: true (rolling out; see the roadmap). Vertical Pod Autoscaler (VPA) runs only in Recommendation mode, profiling compiler resource use without disrupting active builds:

Dockerfile
Hardened Multi-Stage Dockerfile
▾
Dockerfile.jdk17 Dockerfile
# Stage 1: Download the agent binary pinned to an exact release
FROM debian:bookworm-slim@sha256:<digest> AS agent-builder
ARG AGENT_VERSION=4.273.0
RUN apt-get update \
 && apt-get install -y --no-install-recommends ca-certificates curl \
 && mkdir -p /azp \
 && curl -fsSL "https://download.agent.dev.azure.com/agent/${AGENT_VERSION}/vsts-agent-linux-x64-${AGENT_VERSION}.tar.gz" \
    | tar -xz -C /azp

# Stage 2: Hardened runtime container (pin apt package versions in production)
FROM ubuntu:24.04@sha256:<digest> AS runtime
RUN apt-get update \
 && apt-get install -y --no-install-recommends ca-certificates curl jq git openjdk-17-jdk-headless maven \
 && rm -rf /var/lib/apt/lists/*
# buildctl client only (no Docker daemon): builds run on the remote rootless BuildKit service
COPY --from=moby/buildkit:rootless@sha256:<digest> /usr/bin/buildctl /usr/local/bin/buildctl
COPY --from=agent-builder /azp /azp
RUN /azp/bin/installdependencies.sh \
 && groupadd -g 1001 agent && useradd -m -u 1001 -g agent agent \
 && chown -R agent:agent /azp
COPY --chown=agent:agent --chmod=0755 start.sh /azp/start.sh
USER 1001
ENV AGENT_ALLOW_RUNASROOT=0
ENTRYPOINT ["/azp/start.sh"]

9. Admission Governance with Kyverno

Kyverno validates every resource admitted to the build namespace against zero-trust baselines, preventing configuration drift; two policies are enforced and two run in Audit mode ahead of enforcement:

Kyverno Admission PolicyEnforced Operational BaselineAdmission Mode
Disallow Kernel CapabilitiesRejects pods that add capabilities such as NET_RAW, SYS_ADMIN or SYS_PTRACE.Enforced
Mandate Image Digest PinningBlocks mutable tags such as :latest; requires explicit sha256 digests.Audit
Require FinOps Workload LabelingRejects unlabelled workloads to keep FinOps chargeback accounting precise.Enforced
Mandate Pod Disruption BudgetsRequires PDB declarations for all multi-replica shared infrastructure components.Audit

10. GitOps Delivery with Argo CD ApplicationSets

An Argo CD ApplicationSet continuously reconciles all nine toolchain charts from Git (GitOps) with automated prune and self-heal drift remediation. Sync-waves create the BuildKit mTLS certificates and KEDA TriggerAuthentication before any agent scaler starts, and CI contract tests run helm lint, helm-unittest and Checkov static analysis on all generated manifests before merge.

YAML
Argo CD ApplicationSet Specification
▾
applicationset-agents.yaml YAML
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: build-agents
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - { agent: jdk17,     class: standard }
          - { agent: dotnet10,  class: standard }
          - { agent: node22,    class: standard }
          - { agent: python,    class: light }
          - { agent: helm,      class: light }
          - { agent: terraform, class: light }
          - { agent: sql,       class: light }
          - { agent: selenium,  class: heavy }
          - { agent: owasp,     class: stateful }
  template:
    metadata:
      name: 'agent-{{ .agent }}'
      annotations:
        argocd.argoproj.io/sync-wave: "1"
    spec:
      project: build-agents
      source:
        repoURL: <git-repo>
        path: 'charts/agents/{{ .agent }}'
      destination:
        server: https://kubernetes.default.svc
        namespace: build-agents
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [ServerSideApply=true]

Production Post-Mortems

Engineering Post-Mortem Note

Found during scale validation, each incident came from valid components interacting under edge conditions, not overt misconfiguration, and lists the automated control added to prevent recurrence.

Incident 01: Scaler Over-Matching & Starvation
1

Symptom

Queued jobs stalled in pending status while KEDA reported zero eligible work. After switching to disjunctive demand matching, one scaler spawned 28 pods within 120 seconds for jobs it lacked the toolchain to run.

2

Root Cause Analysis (RCA)

The scaler demand list mirrored the image’s complete capability set. Conjunction (requireAllDemands: true) failed because pipelines requested only their primary toolchain; disjunction (requireAllDemands: false) scaled on generic shared tags such as Docker, intercepting jobs across the multi-tenant pool.

3

Remediation

Each scaler now evaluates one atomic routing capability. A Helm templating bug that silently converted explicit booleans was fixed and locked down with automated unit contract tests.

Incident 02: Silent Fallback to a Legacy PAT
1

Symptom

Canary runners on Workload Identity registered and passed every pipeline suite, yet Entra ID audit logs showed zero OIDC token exchanges.

2

Root Cause Analysis (RCA)

The shared baseline values manifest still injected a legacy PAT environment variable, which the bootstrap script preferred over federated exchange, so runners silently authenticated with the deprecated PAT.

3

Remediation

The Helm library now prunes PAT environment variables whenever Workload Identity is active, and CI regression tests fail the build if static credentials appear in rendered manifests.

Incident 03: Missing Subnet Route for Auto-Provisioned Nodes
1

Symptom

Karpenter created Azure VMs for pending pods, but the new nodes never registered with the Kubernetes API server, leaving pods Pending indefinitely.

2

Root Cause Analysis (RCA)

The AKSNodeClass subnet lacked the User-Defined Route (UDR) table sending egress through the enterprise Next-Generation Firewall, so kubelets booted but could not reach the AKS API server or registry endpoints.

3

Remediation

Subnet network security group and route-table bindings are now checked in automated pre-flight Terraform validation, ensuring routing parity across all auto-provisioned subnets.

Incident 04: Fleet-Wide Scaler Outage from Out-of-Band State
1

Symptom

All eight KEDA scalers (seven ScaledJobs and the Selenium ScaledObject) went NotReady simultaneously, halting cluster autoscaling.

2

Root Cause Analysis (RCA)

Agent charts referenced an imperatively created TriggerAuthentication whose name diverged from the GitOps-managed specification; routine cleanup of untracked cluster resources deleted it, breaking every dependent scaler.

3

Remediation

All TriggerAuthentication objects are now declared in the base platform release and ordered by Argo CD sync-waves, and manual cluster modifications are automatically reconciled.

Platform Tenets & Guardrails

Reliability & Velocity

  • Atomic Demand Routing: one primary routing demand per KEDA scaler, preventing multi-tenant cross-scaling.
  • 30-Second Polling: sub-minute queue response within Azure DevOps REST API rate quotas.
  • Cold-Start Defenses: balloon pods, pre-pull DaemonSets and baked binaries keep queue-to-start at about 30 seconds or less.
  • Bounded Failure Execution: zero retries (backoffLimit: 0) and a hard 2-hour deadline prevent runaway orphan processes.

Security & Supply Chain

  • Zero Standing Privileges: ephemeral OIDC token exchange only; static PATs eliminated fleet-wide.
  • Digest Pinning: sha256 digests for all base images and stages; the :latest admission check is moving from Audit to Enforce.
  • Rootless Compilation: container builds isolated in rootless BuildKit daemons over mutual TLS.
  • Fail-Closed Admission: Kyverno rejects capability escalation and unlabelled workloads; digest-pinning and PDB policies stay in Audit until existing violations are cleared.

FinOps & Governance

  • Spot Bin-Packing: maximum Spot adoption across VM families, with graceful on-demand fallback.
  • Disruption Budgets: WhenEmpty consolidation and zero voluntary disruption in business hours (schedules in UTC).
  • Blast-Radius Ceilings: per-toolchain replica caps and node-pool CPU quotas contain rogue pipeline loops.
  • Cache Trust Boundaries: share only read-only, pre-seeded caches over Azure Files; writable caches and workspaces stay per-pod.

Roadmap: Supply-Chain Hardening & Telemetry

The ephemeral execution plane is in production; next come supply-chain verification and telemetry:

Active Deployment

Fleet-Wide Read-Only Root Filesystem

Moving all agent toolchains to enforced readOnlyRootFilesystem: true.

Phase 2 Roadmap

SLSA Level 3 Compliance & Cryptographic Attestation

Signing all internally built runner images and container artifacts with Cosign/Sigstore, and generating Software Bills of Materials (SBOMs) with in-toto attestations verified at admission by Kyverno.

Phase 3 Roadmap

End-to-End DORA & Platform SLO Observability

Prometheus and OpenTelemetry metrics in executive Grafana dashboards: Queue-to-Execution Lead Time (P50/P95/P99), Cold-Start Initialization Latency, Spot Preemption Frequency, and FinOps Unit Cost Per Pipeline Build.

Conclusion

Self-hosted build infrastructure is among the most privileged yet least-examined parts of an enterprise technology stack. Making it ephemeral and event-driven turns a brittle collection of static machines into a resilient, cloud-native utility, improving economics and security posture at the same time.

The surest way to remove the security risk and capital waste of long-lived build infrastructure is to minimize it and keep whatever remains short-lived and tightly scoped. If your pipelines still run on static build VMs, OSI Digital's Platform Engineering practice can guide a phased, validated, low-risk modernization.

Frequently Asked Questions

How does KEDA with Karpenter compare to GitHub Actions Runner Controller (ARC) on Kubernetes?
ARC only orchestrates GitHub Actions runners. For Azure DevOps, KEDA ScaledJobs driven by queue depth plus Karpenter give the equivalent ephemeral model, with 1:1 job isolation and Spot-first elasticity. The pattern (queue-driven ephemeral pods, rootless builds, federated identity) is portable, but this implementation is Azure-specific (AKS Karpenter AKSNodeClass, Azure Workload Identity, Azure DevOps).
How does Karpenter manage Spot instance preemption risks for mission-critical builds?
Build pods carry karpenter.sh/do-not-disrupt: "true", which blocks Karpenter's voluntary disruption but not Azure Spot evictions, so short jobs rely on pipeline-level retry. Interruption-sensitive workloads (browser test suites, the BuildKit daemon) select on-demand capacity via nodeSelector in the same NodePool.
How does this architecture satisfy SOC 2, ISO 27001, and CIS Kubernetes compliance standards?
Short-lived Workload Identity OIDC tokens replace PATs (Zero Standing Privileges), and pods run non-root with drop: ["ALL"]. Rootless BuildKit over mTLS replaces privileged Docker-in-Docker, aligning with CIS Kubernetes benchmark controls apart from one documented seccomp exception.
What is the operational maintenance overhead of managing nine distinct toolchain images?
All nine toolchains inherit one Helm library chart for security policy, Workload Identity, volumes and KEDA triggers. Each toolchain adds only ~30 lines of values, and CI builds and scans base images automatically.
Can this platform operate in strictly air-gapped or private-endpoint-only VPC/VNet topologies?
Private-endpoint-only: largely yes; strictly air-gapped: no. Registry, Artifactory and BuildKit traffic can stay on Azure Private Endpoints, but agents must reach Azure DevOps (dev.azure.com) and Microsoft Entra ID, and the OWASP sync needs the NVD or an internal mirror; these paths can be allow-listed through a firewall or proxy.
What does an implementation engagement look like with OSI Digital?
We assess your pipeline inventory, toolchain matrix and regulatory requirements, then roll out in phased, reversible increments validated on canary runners before fleet cutover, with test suites, runbooks and team enablement included. Contact our DevOps practice to start an environment review.

Modernize Your CI/CD Execution Infrastructure

Move your Azure DevOps pipelines to an on-demand, least-privilege, cost-optimized Kubernetes platform with OSI Digital's platform engineering practice.

Schedule an Architecture Strategy Session →