Note: This project is a work in progress. The API is
v1alpha1and may change between versions. If you run into issues, please open a GitHub issue.
A Kubernetes operator for managing ML models deployed on Baseten.
A BasetenModel Kubernetes custom resource declares the desired state of a Baseten model in a target environment. The baseten-operator uses the Baseten Management API to (1) create and promote a model deployment to the target environment, (2) reconcile autoscaling and promotion settings for the environment, and (3) scale in and clean up orphaned model deployments not associated with any environment (preventing cost overruns from GPU resource leakage).
Use Kubernetes-native continuous delivery tooling (e.g. Argo CD, Flux). No need to build dedicated tooling to manage models on Baseten.
Easily manage deployments across:
- environments, regions, and tenants. A single model may need to run across multiple environments (dev, staging, prod), regions (US, EU), and tenants (for subsets of customers). Each deployment is another Kubernetes custom resource in the same continuous delivery flow.
- in-house and managed inference infra providers. ML teams often deploy the same model across in-house and managed infra providers. With the operator, Baseten deployments follow the same Kubernetes-based continuous delivery pattern as in-house infra running on Kubernetes.
- Prevent cost overruns from leaked GPUs. Orphaned deployments not associated with any environment are scaled to zero and deleted on a schedule, with configurable safeguards.
- Self-heal on transient failures. Failed deployments (
FAILED,DEPLOY_FAILED,BUILD_FAILED) are retried with exponential backoff for a fixed time window. When a deployment attached to an environment is markedINACTIVEby Baseten's TTL, the operator reactivates it. - Prevent drift from click-ops. Changes made directly in the Baseten UI are detected and reverted to match the
BasetenModelcustom resource. The custom resource remains the single source of truth. - Operator and Baseten UI work together. Pause reconciliation per-resource to manage a deployment directly in the Baseten UI for incident response or iterative tuning, then codify the result back into the
BasetenModelcustom resource. - Multi-region failover support. For multi-region Argo setups, run one region with
spec.mode: Reconcileand the rest withspec.mode: Observe. Observers report state and drift without mutating Baseten. Fail over by deactivating the primary Argo app and flipping a secondary's mode toReconcile. - Safely delete models.
deletionPolicy: DeleteWithGuardrailsenables declarative model teardown with availability protections: the operator refuses to delete while any environment associated with the model has non-zero replicas, names the offending envs in status, and self-heals once they drain. The defaultRetainpreserves the upstream model when CRs are removed, so a straykubectl deletecannot take down a model serving production traffic. (Deleteis also available for unconditional teardown.) - Separate ML platform ownership from model ownership.
- Model owners build, test, and tune models using the Baseten CLI or UI, then codify the working configuration as a
BasetenModelcustom resource. From that point theBasetenModelcustom resource is version-controlled and auditable. - The ML platform team can own the operator and set common custom resource defaults for autoscaling, promotion, and cleanup.
- Model owner access to production can be scoped down while the platform team retains UI access for incident response.
- Model owners build, test, and tune models using the Baseten CLI or UI, then codify the working configuration as a
Complete BasetenModel CR with all supported fields
apiVersion: models.baseten.com/v1alpha1
kind: BasetenModel
metadata:
name: my-model-production
spec:
modelName: "my-llm-model" # required — model name in Baseten
mode: Reconcile # optional — Reconcile (default), Observe, or Pause
deletionPolicy: Retain # optional — Retain (default), DeleteWithGuardrails, or Delete
# Option A: Promote a CI/CD-created deployment
# sourceDeploymentName: "img-1.0-wgt-1.0-p-1.3" # created by CI/CD via truss push
# Option B: Operator creates the deployment via truss push
trussConfig:
pythonVersion: "py312" # optional (e.g., py311, py312)
resources: # required
accelerator: "H100:1" # required (e.g., H100:2, A100:4, L4)
# useGpu: true # optional
baseImage: # required
image: "us-docker.pkg.dev/my-project/my-repo/vllm:0.16.0"
dockerAuth: # optional — only if private registry
authMethod: "GCP_SERVICE_ACCOUNT_JSON"
secretName: "docker-registry-secret"
registry: "us-docker.pkg.dev"
dockerServer: # optional
# noBuild: true # skip image build if base image is ready
startCommand: "sh -c 'bash /app/data/setup.sh'"
readinessEndpoint: "/health"
livenessEndpoint: "/health"
predictEndpoint: "/v1/completions"
serverPort: 8000 # 8080 is reserved by Baseten
runtime: # optional
predictConcurrency: 256
modelMetadata: # optional
tags: ["openai-compatible"]
secrets: # optional — keys only, values stored in Baseten
docker-registry-secret: ""
environmentVariables: # optional
DD_SITE: "us5.datadoghq.com"
setupScript: # optional
configMapRef: # large scripts → ConfigMap
name: my-model-setup
key: setup.sh
# inline: | # or inline for small scripts
# #!/bin/bash
# echo "done"
environment:
name: production # required — lowercase alphanumeric only
autoscaling: # optional — all fields optional, drift-reconciled
minReplicas: 2
maxReplicas: 20
concurrencyTarget: 20
autoscalingWindow: 600 # seconds
scaleDownDelay: 120 # seconds
targetUtilizationPercentage: 70
promotionSettings: # optional — all fields optional, drift-reconciled
# Promote-time flags (passed to POST /promote, not stored on environment)
scaleDownPreviousDeployment: true # default: true
preserveInstanceType: true # default: true
# Environment-level settings (reconciled — drift detected and fixed)
redeployOnPromotion: false # default: false
rollingDeploy: false # default: false
promotionCleanupStrategy: "SCALE_TO_ZERO" # KEEP, SCALE_TO_ZERO, DEACTIVATE
rampUpWhilePromoting: true # enables canary traffic shifting
rampUpDurationSeconds: 600 # 10 stages over this window
rollingDeployConfig:
strategy: "REPLICA"
maxSurgePercent: 25 # default: 10 (0-100)
maxUnavailablePercent: 25 # 0-100
stabilizationTimeSeconds: 300 # seconds
# Optional: clean up old deployments that accumulate after promotions
# orphanDeploymentCleanup:
# scaleToZero: true # Set min_replica=0 on orphans
# delete: true # Delete old inactive orphans
# deleteAfterDays: 30
# minToKeep: 10
# intervalMinutes: 10080 # WeeklyPromote deployments to any environment by updating a single field. The operator handles the full lifecycle: validate source deployment, configure environment, promote, poll until active.
spec:
modelName: "my-llm-model"
sourceDeploymentName: img-1.0-wgt-1.0-p-1.3 # from CI/CD
environment:
name: production
autoscaling:
minReplicas: 2
maxReplicas: 20Define your deployment inline and the operator handles the truss push and promotion. Deployment names are deterministic from config content, so the same config across environments results in only one push.
spec:
modelName: "my-llm-model"
trussConfig:
resources:
accelerator: "H100:1"
baseImage:
image: "us-docker.pkg.dev/my-project/my-repo/vllm:0.16.0"
dockerServer:
predictEndpoint: "/v1/completions"
serverPort: 8000
setupScript:
configMapRef:
name: my-model-setup
key: setup.sh
environment:
name: productionIf someone changes autoscaling or promotion settings in the Baseten UI, the operator detects the drift on the next reconcile and corrects it back to the desired state defined in the CR. Both autoscaling and promotion settings are reconciled in a single API call.
When a deployment fails (FAILED, DEPLOY_FAILED, BUILD_FAILED), the operator automatically retries via the Baseten API. No manual intervention needed.
| Parameter | Value |
|---|---|
| Backoff | Exponential: 2m base, doubling, 30m cap |
| Jitter | 0-50% random |
| Deadline | 2 hours from first failure |
| Concurrency | Safe across multiple CRs on the same model |
After 2 hours the operator stops retrying and emits a DeploymentRetryExhausted warning. BUILD_STOPPED is never retried (intentional user action).
Old deployments accumulate after promotions, leaking GPU capacity. The operator can automatically scale them to zero and delete stale ones:
orphanDeploymentCleanup:
scaleToZero: true # release GPUs on orphans
delete: true # delete old inactive deployments
deleteAfterDays: 30 # keep 30-day rollback window
minToKeep: 10 # always preserve 10 newest
intervalMinutes: 10080 # run weeklyReady + Progressing conditions provide tri-state health out of the box:
| Ready | Progressing | Argo CD Status |
|---|---|---|
| True | * | Healthy (green) |
| False | True | Progressing (yellow) |
| False | False | Degraded (red) |
Argo CD health check config
Add to your argocd-cm ConfigMap:
resource.customizations.health.models.baseten.com_BasetenModel: |
hs = {}
if obj.status ~= nil and obj.status.conditions ~= nil then
local ready = nil
local progressing = nil
for i, condition in ipairs(obj.status.conditions) do
if condition.type == "Ready" then ready = condition
elseif condition.type == "Progressing" then progressing = condition end
end
if ready ~= nil then
if ready.status == "True" then
hs.status = "Healthy"; hs.message = ready.message or ""; return hs
end
if progressing ~= nil and progressing.status == "True" then
hs.status = "Progressing"; hs.message = progressing.message or ""; return hs
end
hs.status = "Degraded"; hs.message = ready.message or ""; return hs
end
end
hs.status = "Progressing"; hs.message = "Waiting for reconciliation"; return hsSee Argo CD Resource Health docs for more on custom health checks.
spec.mode controls the operator's behavior per CR:
Reconcile(default) — normal active reconciliation. Reads desired state, reconciles drift, promotes deployments, retries failures.Observe— read-only. The operator reads Baseten state and refreshes status (including detected drift) but never mutates: no creates, promotions, updates, deletes, or truss pushes. Useful in a multi-region setup so secondary regions retain visibility without dual-writing alongside the primary. Failover by flipping a region toReconcilevia GitOps.Pause— no API calls, no requeue, last known status preserved. Useful during incidents or when configuring the model manually in the Baseten UI.
spec:
mode: Observe # this region only watches; the primary region runs as Reconcilespec.deletionPolicy controls what happens to the upstream Baseten model when the CR is deleted. Only honored in Reconcile mode; Pause and Observe always retain.
DeleteWithGuardrails— recommended when you want declarative deletion without availability risk. The operator callsListEnvironmentsfirst and refuses to delete while any environment hasmin_replica > 0or active replicas. Blocks withDELETE_BLOCKEDstatus naming the offending envs (e.g.,prod (min=2, active=4)), emits aModelDeleteBlockedevent, and requeues every 30s. Once every env drains, the next reconcile proceeds withDeleteModel. Mitigates the case where aDeletecascade would nuke envs not tracked by this CR's spec, including ones that may still be serving production traffic.Retain(default) — CR is removed, model is preserved in Baseten. Safe choice for click-ops models or anything you might want back.Delete— CR deletion cascades toDeleteModel, removing the model and all its environments and deployments. Irrevocable, no checks.
spec:
deletionPolicy: DeleteWithGuardrailsConfigure rolling deployments and canary traffic ramp-up for zero-downtime promotions:
environment:
promotionSettings:
rollingDeploy: true
rampUpWhilePromoting: true
rampUpDurationSeconds: 600
promotionCleanupStrategy: "SCALE_TO_ZERO"The status.message field shows what the operator is doing. These are the messages you'll see on the resource (and in Argo CD):
# Steady state
active: depl-vllm-0.17.0-2cb9685f.1775156532 (3 replicas, min:1 max:5) in production environment
active: depl-vllm-0.17.0-2cb9685f.1775156532 (0 replicas, min:0 max:5, SCALED_TO_ZERO) in staging environment
# Promotion in progress
active: depl-vllm-0.16.0-a3f7c2b1 (3 replicas) | promoting: depl-vllm-0.17.0-2cb9685f (BUILDING, 0 replicas)
active: depl-vllm-0.16.0-a3f7c2b1 (3 replicas) | promoting depl-vllm-0.17.0-2cb9685f to production
active: depl-vllm-0.16.0-a3f7c2b1 (3 replicas) | promoted depl-vllm-0.17.0-2cb9685f to production (DEPLOYING)
# Pause mode
reconciliation paused | last status: active: depl-vllm-0.17.0-2cb9685f (3 replicas, min:1 max:5) in production environment
# Observe mode (read-only secondary; steady-state messages match Reconcile mode)
active: depl-vllm-0.17.0-2cb9685f (5 replicas, min:5 max:5) in production environment | drift: minReplicas 2→5; awaiting reconciliation
no active deployment | spec source: depl-vllm-0.17.0-2cb9685f (not yet promoted; awaiting reconciliation)
model "my-llm-model" not found in Baseten; observe mode does not create
environment "production" does not exist; observe mode does not create
All status messages
Steady state:
active: {name} ({replicas} replicas, min:{min} max:{max}) in {env} environment
active: {name} ({replicas} replicas, min:{min} max:{max}, SCALED_TO_ZERO) in {env} environment
Promotion lifecycle:
promoting {source} to {env}
promoted {source} to {env} ({status})
promotion of '{source}' to {env} failed: {error}
active: {current} ({replicas} replicas) | promoting: {candidate} ({status}, {replicas} replicas)
active: {current} ({replicas} replicas) | candidate {candidate} failed: {status}
Waiting to promote:
waiting to promote: source deployment {name} is BUILDING
waiting to promote: source deployment {name} is ACTIVATING
waiting to promote: source deployment {name} is {status}
waiting to promote: source deployment {name} has status {status}
unable to promote: source deployment {name} not found
unable to promote: source deployment {name} is {status} (retries exhausted after 2h0m0s)
unable to promote: failed to activate source deployment {name}: {error}
Truss push (trussConfig workflow):
truss push started for {name}
truss push in progress for {name}
truss push failed for {name}: {error}
truss push failed: {error}
unable to check deployment {name}: {error}
waiting for push capacity for {name}
Retries:
retrying deployment {name} (was {status}, attempt {n})
Environment management:
Created environment {env}
Failed to create environment '{env}': {error}
Failed to get environment '{env}': {error}
failed to update settings for {env}: {error}
Errors:
Failed to lookup model '{name}': {error}
Pause mode:
reconciliation paused | last status: {previous message}
Observe mode:
{steady-state message — same shape as Reconcile mode}
{steady-state message} | drift: {field} {spec_value}→{actual_value}; awaiting reconciliation
{state} | spec source: {expected_name} (not yet promoted; awaiting reconciliation)
model "{name}" not found in Baseten; observe mode does not create
environment "{name}" does not exist; observe mode does not create
The status.deploymentStatus field contains the machine-readable status (ACTIVE, PROMOTING, BUILDING, FAILED, etc.) used by the Argo CD health check.
Every operation emits standard Kubernetes events, visible via kubectl describe:
Normal Events:
| Reason | Description |
|---|---|
EnvironmentCreated |
Environment created with autoscaling settings |
AutoscalingUpdated |
Autoscaling settings corrected due to drift |
PromotionSettingsUpdated |
Promotion settings corrected due to drift |
DeploymentPromoted |
Deployment promoted to environment |
DeploymentActive |
Deployment is now active after promotion |
TrussPushStarted |
Truss push launched to create deployment |
TrussPushCompleted |
Deployment created successfully via truss push |
ReconciliationPaused |
Reconciliation paused via spec.mode: Pause |
OrphanDeploymentsScaledIn |
Orphan deployments scaled to zero |
OrphanDeploymentsDeleted |
Stale orphan deployments deleted |
DeploymentRetried |
Failed deployment retried via Baseten API |
ModelDeleted |
Upstream model deleted from Baseten on CR teardown (deletionPolicy: Delete or DeleteWithGuardrails) |
Warning Events:
| Reason | Description |
|---|---|
ModelNotFound |
Model lookup failed or not found in Baseten |
EnvironmentCreateFailed |
Environment creation failed |
AutoscalingUpdateFailed |
Autoscaling settings update failed |
PromotionSettingsUpdateFailed |
Promotion settings update failed |
SourceDeploymentNotFound |
Source deployment not found in Baseten |
SourceDeploymentFailed |
Source deployment is in terminal failure state |
PromotionFailed |
Promotion API call failed or candidate failed |
PromotionBlocked |
Existing promotion in progress (double-promote guard) |
SetupScriptNotFound |
ConfigMap for setup script not found |
TrussPushFailed |
Truss push preparation failed or stale push timed out |
DeploymentRetryFailed |
Retry API call failed or was declined |
DeploymentRetryExhausted |
Failing for over 2h, operator stopped retrying |
ModelDeleteBlocked |
DeleteWithGuardrails refused to delete: an env still has min_replica > 0 or active replicas |
ModelDeleteFailed |
DeleteModel API call failed during CR teardown |
Install via the Helm chart published to GHCR:
helm install baseten-operator oci://ghcr.io/abridgeai/charts/baseten-operator \
--namespace baseten-operator-system --create-namespace \
--version 0.3.1
kubectl create secret generic baseten-operator-api-key \
--namespace baseten-operator-system \
--from-literal=api-key=YOUR_BASETEN_API_KEYSee charts/baseten-operator/README.md for the full values reference and GitOps (Argo CD / Config Sync) examples.
make build # build binary
make test # unit tests
make test-e2e # e2e tests (Kind cluster)
make lint # linter
make manifests generate # regenerate CRDs after type changeskind create cluster --name baseten-dev
make docker-build IMG=baseten-operator:dev
kind load docker-image baseten-operator:dev --name baseten-dev
make install && make deploy IMG=baseten-operator:devSee test/kind/README.md for a full walkthrough.
Contributions are welcome! Please see CONTRIBUTING.md for guidelines.
Copyright 2025 Abridge AI, Inc. Licensed under the Apache License, Version 2.0.