Zero-Downtime, Progressive Secret & Certificate Rotations for Enterprise Kubernetes
Dynamic Secret Operator (DSO) is a Kubernetes operator engineered to eliminate CrashLoopBackOff outages during credential rotations across multi-cloud environments.
Traditional secret management tools mutate secrets in-place, instantly crashing downstream pods if a rotated database credential, API key, or TLS certificate is malformed, not yet active, or fails handshakes. DSO solves this by adopting ADR-002: Immutable Revisions.
Featuring an extensible, provider-agnostic source abstraction layer, DSO supports:
- Event-Driven Multi-Cloud Ingestion supporting AWS, GCP, and Azure via cloud-native message queues (Amazon SQS, Google Cloud Pub/Sub, Azure Service Bus) & Zero-Trust Federated Workload Identity.
- Universal Multi-Cloud Synergy with External Secrets Operator (ESO) for AWS Secrets Manager, Google Cloud Secret Manager, HashiCorp Vault, and Akeyless.
- eBPF Canary Isolation with CiliumNetworkPolicy and Hubble packet telemetry.
- Supply Chain Security with SLSA Level 3 build provenance, keyless Cosign OIDC signing, and SPDX SBOMs.
DSO shifts secret rotation from a risky "push and pray" operation to a safe, event-driven Progressive Delivery pipeline.
flowchart TD
subgraph MultiCloud ["☁️ Multi-Cloud Secret Backends & Ingestion Architectures"]
direction TB
subgraph ModeESO ["ESO Mode: Universal Multi-Cloud Ingestion (ESO-Native)"]
VAULT_ALL["AWS Secrets Manager / GCP Secret Manager / Vault / Key Vault"]
ESO["External Secrets Operator<br/>(SecretStore + ExternalSecret)"]
SYNC_SEC["Intermediate Secret<br/>(dso.quantumsys.dev/managed: watch)"]
VAULT_ALL -->|"Sync (Drift / Polling / Webhook)"| ESO
ESO -->|"Writes Synced Secret"| SYNC_SEC
end
subgraph ModeEventDriven ["Event-Driven Mode: Universal Direct Ingestion (Multi-Cloud Push)"]
direction LR
subgraph CloudVaults ["Cloud Vaults & Event Routers"]
AKV["Azure Key Vault<br/>+ Event Grid"]
AWS_SM["AWS Secrets Manager<br/>+ EventBridge / SNS"]
GCP_SM["GCP Secret Manager<br/>+ Cloud Pub/Sub"]
VAULT_EVT["HashiCorp Vault<br/>+ Event Streams"]
end
subgraph CloudQueues ["Reliable Message Queues & Streaming"]
ASB["Azure Service Bus Queue<br/>(Peek-Lock Delivery)"]
SQS["Amazon SQS Queue<br/>(Ack/Nack Delivery)"]
PUBSUB["GCP Pub/Sub<br/>(Streaming Subscription)"]
KAFKA["Kafka / NATS Broker<br/>(Event Streaming)"]
end
AKV -->|"SecretNewVersionCreated"| ASB
AWS_SM -->|"Rotation Event"| SQS
GCP_SM -->|"Secret Version Add"| PUBSUB
VAULT_EVT -->|"Audit / Event Stream"| KAFKA
end
end
subgraph K8sCluster ["☸️ Kubernetes Cluster Architecture"]
subgraph DSO_System ["dso-system Namespace"]
DSO["⚙️ Dynamic Secret Operator<br/>(Pluggable Event Handlers & Providers)"]
OTEL["📊 OpenTelemetry & Prometheus<br/>(:8080/metrics)"]
end
subgraph AppNamespace ["Target Application Namespace"]
DSP["📄 DynamicSecretPolicy<br/>(Declarative Strategy & Probes)"]
REV_SEC["🔒 Immutable SecretRevision<br/>(app-rev-a1b2c3d4)"]
CANARY["🐤 Ephemeral Canary Pod<br/>(+ NetPol / Cilium eBPF Sandbox)"]
PROBES["🩺 Synthetic Validation Probes<br/>(HTTP / TLS / PG / MySQL / Job)"]
PROD["🚀 Production Workload<br/>(Deployment / StatefulSet / Rollout)"]
end
end
%% Ingestion Flows
SYNC_SEC -.->|"spec.source.k8sSecret (Watch / Informer)"| DSO
CloudQueues -.->|"spec.source.* (Zero-Polling Push Stream)"| DSO
DSO -.->|"Fetch Payload via Cloud Identity (Workload Identity / IRSA)"| MultiCloud
%% Progressive Delivery State Machine
DSP -->|"1. Reconcile Policy"| DSO
DSO -->|"2. Materialize Revision"| REV_SEC
DSO -->|"3. Provision Isolated Sandbox"| CANARY
CANARY -->|"Mounts"| REV_SEC
DSO -->|"4. Execute Synthetic Probes"| PROBES
PROBES -->|"Validate Real Traffic"| CANARY
DSO -->|"5. Zero-Downtime Rollover & GitOps Patch"| PROD
DSO provides two distinct, enterprise-grade ingestion models designed to fit any cloud topology:
- Concept: Operates alongside the CNCF standard External Secrets Operator (ESO) to synchronize secrets from 30+ external secret stores (AWS Secrets Manager, Google Secret Manager, HashiCorp Vault, Azure Key Vault, Akeyless, CyberArk) into intermediate Kubernetes Secrets.
- How it Works: When an upstream secret updates, ESO refreshes the Kubernetes Secret carrying the label
dso.quantumsys.dev/managed: "watch". DSO observes the hash drift through standard controller-runtime informers and initiates the progressive canary rollout. - Zero Cloud IAM Overhead: DSO requires zero cloud IAM permissions in this mode, running purely on native Kubernetes RBAC.
- Concept: Sub-second, reactive push notifications triggered directly by cloud event brokers and queue streams. Rather than relying on periodic polling intervals, upstream secret rotations immediately push notification events to DSO.
- Multi-Cloud Architecture: Event-driven ingestion is an architectural pattern supported across all major cloud providers and hybrid infrastructures:
- 🟦 Microsoft Azure:
Azure Key Vault→Azure Event Grid→Azure Service Bus Queue(with Peek-Lock delivery and Azure Workload Identity). - 🟧 Amazon Web Services (AWS):
AWS Secrets Manager→Amazon EventBridge / SNS→Amazon SQS Queue(with message visibility timeouts and AWS IAM Roles for Service Accounts - IRSA / EKS Pod Identity). - 🟥 Google Cloud Platform (GCP):
Google Cloud Secret Manager→Cloud Pub/Sub Topic & Subscription(with streaming pull and GCP Workload Identity Federation). - 🟪 HashiCorp Vault & Hybrid/On-Prem:
Vault Event Streams / Audit Webhooks→Apache Kafka / NATS / Event Broker(with mutual TLS / SPIFFE identities).
- 🟦 Microsoft Azure:
- Sub-Second Latency & Zero Polling: Eliminates API rate-limit throttling and polling delays, ensuring rotation events trigger canary validation within milliseconds of upstream modification.
Regardless of whether secret ingestion is driven by ESO Mode (watching intermediate secrets) or Event-Driven Mode (reactive push streams), DSO never promotes a rotated secret directly to production workloads.
Instead, the operator intercepts the event and executes a zero-trust Progressive Delivery & Canary Validation Pipeline:
stateDiagram-v2
direction LR
[*] --> Ingestion: Upstream Secret Rotation (ESO Sync / Cloud Queue Push)
Ingestion --> RevisionPrepared: Materialize Immutable SecretRevision
RevisionPrepared --> CanaryProvisioning: Deploy Ephemeral Canary + Network Sandbox
CanaryProvisioning --> Validating: Run spec.validationProbes
state Validating {
[*] --> CheckProbes: HTTP / TLS / PostgreSQL / MySQL / Job
CheckProbes --> Passed: All Probes Succeed ✅
CheckProbes --> Failed: Any Probe Fails or Times Out ❌
}
Passed --> Promoting: Zero-Downtime Rolling Update & Argo CD Auto-Patch
Promoting --> Completed: Production Verified & Teardown Canary
Failed --> RolledBack: Trip Circuit Breaker & Preserve Production
Completed --> [*]
RolledBack --> [*]
When an upstream rotation occurs, DSO transitions through distinct lifecycle phases before production workloads are touched:
RevisionPrepared: Computes the SHA-256 checksum of the new payload and creates an immutable, cryptographically named Kubernetes Secret (<workload>-rev-<sha256>).CanaryProvisioning: Provisions an isolated canary deployment (<workload>-canary) and applies an ephemeralNetworkPolicy(or Cilium eBPF egress sandbox). Only this canary pod mounts the candidateSecretRevision.Validating: This is wherevalidationProbesenter the lifecycle. The operator runs all synthetic probes configured inspec.validationProbesagainst the candidate revision and canary sandbox.
DSO provides a modular, pluggable probe execution engine with strict anti-leakage sanitization:
HTTP/HTTPS: Issues HTTP requests against canary endpoints, verifying status codes (200-399) withinqueryTimeout.TLS: Executes a mutual or one-way TLS handshake, verifying certificate validity dates (not expired, already active) and asserting that the leaf certificate's SHA-256 thumbprint matches the new secret.PostgreSQL&MySQL: Opens a real database connection using the rotated username/password and executes a live verification query (SELECT 1). All DSNs and credentials are automatically redacted from logs and OpenTelemetry traces.Job(Bring Your Own Container): Ephemerally schedules an arbitrary Kubernetes Batch Job (batch/v1.Job) in the target namespace (e.g.,redis-cli, custom shell scripts, compliance validators). DSO injects the candidate secret name asDSO_REVISION_SECRET_NAME, monitors completion, and parses container exit codes.- Automatic Canary Redirection: For probes targeting the workload, DSO automatically discovers the ready canary pod IP address and redirects the probe's network traffic to the isolated canary instance.
When all configured validation probes pass successfully:
- Workload Promotion (
Promoting/RolloutProgressing):- DSO safely patches the production workload (
Deployment,StatefulSet,DaemonSet, orArgo Rollout) to mount the validatedSecretRevision. - The workload rolls out with zero downtime using native Kubernetes rolling upgrade strategies.
- DSO safely patches the production workload (
- GitOps Harmony (Argo CD Auto-Patch):
- DSO dynamically patches the target Argo CD
Application'sspec.ignoreDifferenceswith JSON Pointers to the modified revision annotation and secret volumes, preventing Argo CD self-heal revert loops.
- DSO dynamically patches the target Argo CD
- Sandbox Teardown & Cleanup:
- The ephemeral canary deployment and network sandbox policies are automatically deleted.
- The policy status transitions to
PromotionCompleted: True. - Consecutive failure counters are reset to zero (
consecutiveFailures: 0).
- Queue Event ACK (Event-Driven Mode):
- The message lock on the event broker (e.g. Azure Service Bus Peek-Lock, AWS SQS visibility timeout, GCP Pub/Sub ack) is completed and acknowledged, finalizing the event stream.
If any probe fails, times out, or the canary pod enters CrashLoopBackOff:
- Absolute Production Isolation (Zero Outages):
- Production pods are NEVER touched and NEVER patched.
- Production workloads remain running normally with their existing, functional secret revision without disruption.
- Masked Diagnostics & Observability:
- DSO captures the exact failure cause (e.g.,
password authentication failed,certificate expired,connection timed out) and redacts all secrets before recording the error intostatus.conditions. - Emits failure metrics to Prometheus (
dso_probe_failures_total) and exports OpenTelemetry tracing spans.
- DSO captures the exact failure cause (e.g.,
- Transient Retries & Circuit Breaker Protection:
- DSO increments the failure counter (
consecutiveFailures++) and retries with exponential backoff (in Event-Driven Mode, the message is released/NACKed to allow retry). - If failures reach
spec.rollbackConfig.circuitBreakerThreshold(default: 3):- DSO trips the circuit breaker and halts further rollout attempts to prevent authentication lockouts or denial of service against upstream backends.
- The policy transitions to
RolledBack: Truewith reasonValidationProbeFailed. - The failed canary deployment and network policies are deleted to reclaim resources.
- DSO increments the failure counter (
- Self-Healing on Upstream Correction:
- As soon as an administrator or CI/CD pipeline pushes a valid secret to the upstream vault (or ESO syncs a corrected secret), DSO detects the new SHA-256 hash drift, resets the circuit breaker, and automatically starts a new canary validation cycle.
| Feature | Description |
|---|---|
| Pluggable Provider Architecture | Extensible source backend abstraction (source.Provider) supporting native Event-Driven Multi-Cloud Push (Azure Key Vault, AWS Secrets Manager, GCP Secret Manager, Vault) and Universal ESO-Native intermediate watch. |
| Zero-Trust Passwordless Auth | Integrates natively with cloud federated identities: Azure Workload Identity, AWS IAM Roles for Service Accounts (IRSA) / EKS Pod Identity, and GCP Workload Identity. No static credentials, client secrets, or long-lived tokens. |
| Immutable SecretRevisions | Materializes cryptographically hashed, immutable Kubernetes Secrets (<workload>-rev-<sha256>), preventing in-place race conditions. |
| Progressive Canary Validation | Spins up isolated canary workloads with strict NetworkPolicy or eBPF CiliumNetworkPolicy ingress rules and executes synthetic validation probes before touching production. |
| eBPF & Hubble Observability | Optional native generation of cilium.io/v2.CiliumNetworkPolicy for granular L3/L4/L7 egress sandboxing and real-time Hubble packet telemetry. |
| Comprehensive Probe Engine | Built-in probes for HTTP, TLS (certificate expiration and thumbprint matching), PostgreSQL, and MySQL (SELECT 1). |
| Extensible Job-Based Probes | "Bring Your Own Container" (type: Job) lets users supply a standard batch/v1.JobTemplateSpec (e.g., redis:alpine, kafka-consumer, custom scripts). The operator creates the Job ephemerally in the target namespace, automatically injects DSO_REVISION_SECRET_NAME into container environments, monitors completion, captures failure logs into CRD Conditions, and auto-cleans up — with zero driver bloat in the operator binary. |
| Anti-Leakage Error Sanitization | Intercepts all database and transport errors, stripping passwords, tokens, and raw DSNs before emitting logs or OpenTelemetry spans. |
| Scoped Secret Ingestion | Restricts controller-runtime caches to operator-managed secrets (dso.quantumsys.dev/managed), isolating cluster secrets and minimizing memory exposure. |
| Circuit Breaker & Backoff | Exponential backoff and threshold-based circuit breaker halts retry storms and preserves intact production workloads on bad credential updates. |
| Supply Chain Security & SLSA L3 | Built on zero-CVE Chainguard Static Distroless, cryptographically signed keylessly via Sigstore / Cosign OIDC, with attached SPDX SBOMs and verifiable SLSA Level 3 Build Provenance. |
- 🛡️ Immutable Revisions (ADR-002): Eliminates in-place mutation drift. Rotations generate unique, immutable SecretRevisions (
<workload>-rev-<sha256>). Production pods are entirely shielded from bad credentials until the new revision passes all canary tests. - 🩺 Comprehensive Validation Probes: Ship with confidence using built-in synthetic probes for PostgreSQL, MySQL (executing
SELECT 1), HTTP/S, and TLS (validating certificate expiration and SHA-256 thumbprint matching). - 📜 Native
kubernetes.io/tlsAuto-Parsing: No more manual scripting. DSO automatically intercepts Key Vault Certificate payloads (PEM/PKCS#12), splits them intotls.crtandtls.key, and creates nativekubernetes.io/tlsSecrets ready for immediate consumption by Nginx Ingress, Istio, or Gateway API. - 🐙 GitOps & Argo CD Harmony: In-cluster mutations typically cause infinite reconciliation loops with Argo CD's Self-Heal. DSO automatically calculates and injects safe
ignoreDifferencesJSON Pointers into parent Argo CDApplicationresources, utilizingRetryOnConflictto avoid 409 Conflict storms during concurrent updates. - ♻️ Enterprise Resiliency & Etcd GC:
- Circuit Breakers: Tracks consecutive failures and halts reconciliation to prevent cascading cluster damage. Supports automatic drift-recovery the moment an upstream admin fixes the secret in Key Vault.
- Sliding Window GC: Automatically garbage-collects orphaned SecretRevisions in
etcd, keeping only theCurrentandDesiredrevisions to prevent API server bloat. - Backpressure Handling: Leverages reliable queue delivery (e.g. Azure Service Bus Peek-Lock, AWS SQS visibility timeouts, GCP Pub/Sub nacks) with explicit timeout context NACKs to ensure rotation events are safely preserved during cluster CPU/Queue saturation.
- 🚥 Native Rollout Compatibility: Works natively with standard Kubernetes
Deployment,StatefulSet, andDaemonSetresources, as well as native support for Argo Rollouts (Blue/Green) for advanced traffic shifting.
The DynamicSecretPolicy CRD is your declarative interface for secret ingestion, isolated canary sandboxing, synthetic validation probes, and automated rollbacks.
DSO features a modular architecture where the secret source backend (spec.source) and the validation probe type (spec.validationProbes) are completely pluggable and decoupled:
| Field Block | Purpose | Supported Options |
|---|---|---|
spec.source |
Pluggable secret backend ingestion | K8sSecret (ESO Universal Multi-Cloud), AzureKeyVault, AWSSecretsManager, GCPSecretManager, Vault |
spec.workloadSelector |
Target workload resource to protect and roll over | Deployment, StatefulSet, DaemonSet, Rollout (Argo Rollouts Blue/Green) |
spec.validationProbes |
Synthetic zero-trust canary validation checks | HTTP / HTTPS, TLS (handshake & thumbprint), PostgreSQL, MySQL (SELECT 1), Job (Bring Your Own Container) |
spec.rollbackConfig |
Resiliency, backoff, and circuit breaker protection | autoRollback: true, circuitBreakerThreshold: 3 |
For complete, copy-paste ready DynamicSecretPolicy manifests featuring diverse probe types (HTTP, TLS, PostgreSQL, MySQL, Job) tailored to each cloud ecosystem, consult our dedicated provider documentation:
- 🟢 Universal Multi-Cloud via ESO Guide – Production Ready (Patterns for HTTP Health, Redis Job, PostgreSQL/MySQL, and Ingress TLS probes)
- 🟢 Microsoft Azure Key Vault Guide – Production Ready (Patterns for Relational Database, Ingress TLS Handshake, Microservice HTTP, and Redis Job probes)
- 🟡 Amazon Web Services (AWS) Guide – In Development (Roadmap v0.3) (Patterns for Aurora MySQL, Microservice HTTP, Ingress TLS, and ElastiCache Redis Job probes)
- 🟡 Google Cloud Platform (GCP) Guide – In Development (Roadmap v0.3) (Patterns for Memorystore Redis Job, Microservice HTTP, Cloud SQL Database, and Ingress TLS probes)
- 📖 Pluggable Providers Overview – Architectural model of the provider registry and ingestion engine
- 📖 Comprehensive CRD API Reference – Full schema field definitions, status conditions, and Kyverno policy rules
DSO's core progressive delivery engine—the state machine, immutable SecretRevisions, isolated canary sandboxing, and synthetic validation probes—is entirely cloud-agnostic.
- ✅ Phase 1 (Production Ready - v0.2.x):
- ESO Mode (Universal Multi-Cloud): Seamless progressive delivery for 30+ secret backends via External Secrets Operator across EKS, GKE, AKS, and bare-metal Kubernetes.
- Event-Driven Mode (Azure): Native sub-second push ingestion via Azure Key Vault, Event Grid, and Service Bus Peek-Lock queues with Azure Workload Identity.
- 🚀 Phase 2 (Roadmap - v0.3.x):
- Event-Driven Mode (AWS): Native push adapter for AWS Secrets Manager via Amazon EventBridge / SNS & Amazon SQS using AWS IAM Roles for Service Accounts (IRSA) and EKS Pod Identity.
- Event-Driven Mode (Google Cloud): Native push adapter for Google Cloud Secret Manager via Cloud Pub/Sub using GCP Workload Identity Federation.
- Event-Driven Mode (HashiCorp Vault & On-Prem): Native push adapter for Vault Event Streams and Audit Webhooks via Kafka / NATS / CloudEvents.
Explore our comprehensive reference architecture examples for testing:
- Azure Multi-Secret Rotation: Multi-secret workload auto-rotation consuming PostgreSQL, Redis, and Payment API keys with dedicated validation probes.
- Azure Fullstack DB Rotation: Live AKS cluster integration with Azure Key Vault, Service Bus, and Workload Identity.
- Azure Job-Based Redis Probe: Ephemeral Batch Job probe running custom CLI validation scripts against rotated Redis cache tokens.
- Azure Argo Rollouts Blue/Green: Live AKS Blue/Green promotion triggered by Azure Key Vault rotations.
- Azure TLS Certificate Rotation: Live AKS TLS Gateway with Azure Key Vault SSL certificate auto-parsing.
- Azure Nginx Color Canary: Live AKS Nginx background color rotation with Azure Key Vault.
- ESO Argo Rollouts Blue/Green: Decoupled Blue/Green rollout triggered by synced Kubernetes secrets.
- ESO TLS Certificate Rotation: Decoupled TLS ingress certificate rotation with synthetic TLS handshake verification.
- ESO Nginx Color Canary: Canary rollout with synthetic Job hex color verification and Argo CD drift protection.
- ESO Fullstack DB Rotation: Zero-downtime PostgreSQL credential rollover via synced secrets.
- ESO Job-Based Redis Probe: Ephemeral validation Job running
redis-cli PINGon synced secret update. - ESO Multi-Secret Rotation: Multi-secret microservice with independent validation probes per volume.
For comprehensive details on enterprise integration, architecture, and operation:
- Getting Started Guide (5-Minute Quickstart)
- Operating Modes Guide (ESO Mode vs Event-Driven Mode)
- API Reference
- Configuration & Enterprise Tuning
- Troubleshooting & Runbooks
- GitOps: Argo CD Self-Heal Integration
- Security & Threat Model
- Pluggable Providers Overview
Architecture Decision Records (ADRs):
- ADR-001: Azure Service Bus Peek-Lock vs Webhooks
- ADR-002: Immutable Revisions vs Mutable Updates
- ADR-003: Decoupling Secret Ingestion & ESO Standard
We actively welcome community contributions, PRs, and provider plugin development to help make DSO the universal standard for progressive secret delivery across all major cloud providers.
For runtime flags and enterprise concurrency tuning, see the Configuration Guide.
For operational runbooks and DLQ management, refer to the Troubleshooting Guide.
To report vulnerabilities, please read our Security Policy.