v1.9.4

This is Training Operator v1.9.4 release.

New Features

• [release-1.9] Bump v1 Kubernetes dependencies to 1.36 (#3924 by @robert-bell)

Bug Fixes

• fix: respect cleanPodPolicy when job exceeds backoffLimit (#3420 by @AviadHayumi)
• fix(sdk): always add queue label regardless of queue existence (#3221 by @puwun)
• fix(docker): support multi-arch build for kubectl-delivery image (#3078 by @jsl9208)

v2.3.0

This is Kubeflow Trainer v2.3.0 release.

helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer \
    --namespace kubeflow-system \
    --create-namespace \
    --version 2.3.0 \
    --set runtimes.defaultEnabled=true

Breaking Changes

• feat(runtimes): BREAKING CHANGE: Remove Runtime Finalizers (#3716 by @amirialy)
• feat(manifests): BREAKING CHANGE: Move CRD to Helm chart template directory (#3655 by @andreyvelich)
  • IMPORTANT: if you are upgrading from Kubeflow Trainer v2.0, v2.1 or v2.2, you must first upgrade to v2.3 before upgrading to any later version. See this migration guide for the details.

🚀 Features

• feat: KEP-3562: Introduce OptimizationJob CRD (#3565 by @aniket2405)
• feat(runtimes): make MPI launcher depend on worker readiness (#3748 by @andreyvelich)
• feat: expose Image and Command on PodSet Container (#3674 by @reckless-sherixx)
• feat: move enableHTTP2 setting to Configuration object (#3339 by @krishdef7)
• feat: implement runtime snapshot mechanism (KEP-2599) (#3580 by @robert-bell)
• feat(initializer): validate initializer Pod uses correct volume name (#3714 by @HarshPopat23)
• feat(docs): add theme toggle to top navbar (#3682 by @adityabavadekar)
• feat(runtimes): Add label to disable webhook validation for Runtimes (#3683 by @andreyvelich)
• feat: add Flux integration and e2e tests (#3561 by @amirialy)
• feat(examples): add interactive JAX distributed TPU smoke test notebook (#3531 by @richabanker)
• feat(ci): add govulncheck CVE detection workflow for PR checks (#3557 by @ChughShilpa)
• feat(docs): add dedicated documentation website (#3308 by @Sridhar1030)
• feat: inject PET_* envs into init containers via envInjection config (#3516 by @panpan0000)
• feat(ci): automate release process (#3536 by @Krishna-kg732)
• feat(api): Add terminationGracePeriodSeconds to PodSpecPatch in TrainJob (#3324 by @krishdef7)
• feat(ci): add Python dependency scanning to OSV-Scanner workflow (#3530 by @Fiona-Waters)
• feat(ci): add nightly OSV-Scanner vulnerability scan workflow (#3518 by @Fiona-Waters)
• feat(docs): KEP-2599: Decouple runtime lifecycle from TrainJobs to simplify updating runtimes (#3428 by @robert-bell)
• feat: KEP for inject PET envs into init-container (#3417 by @panpan0000)
• feat: storageuri targetjob validation fix (#3510 by @pulkit-999)
• feat(docs): add AI policy reference to contributing guide (#3493 by @andreyvelich)
• feat(runtimes): add validation for reserved MPI environment variables (#3491 by @adiprathapa)
• feat: update k8s to 0.36 (#3470 by @kannon92)
• feat: merge gpu-cluster setup into single e2e script via flag (#3281 by @aniket2405)
• feat(examples): Add kubectl-friendly YAML examples for TrainJob and TrainingRuntime (#2925 by @NarayanaSabari)
• feat(docs): Kubeflow Trainer ROADMAP 2026 (#3242 by @andreyvelich)
• feat: add e2e tests to Helm CI workflow (#3253 by @Goku2099)
• feat(docs): Add Kubeflow Trainer v2.2 release announcement to README (#3435 by @andreyvelich)
• feat: add Megatron-Core GPT Tensor Parallelism example notebook (#3201 by @XploY04)
• feat: customize R generation to include GPUs (#3325 by @vsoch)
• feat(docs): Changelog for Kubeflow Trainer v2.2.0 (#3369 by @andreyvelich)
• feat(api): Set RuntimePatch.Time field automatically during admission (#3319 by @astefanutti)

🐛 Bug Fixes

• [release-2.3] fix(operator): improve status server bearer token parsing and add helper tests (#3831 by @google-oss-robot)
• [release-2.3] fix(ci): Update Kubeflow SDK installation for E2Es (#3827 by @google-oss-robot)
• [release-2.3] fix(ci): Remove concurrency from Helm publish action (#3793 by @google-oss-robot)
• fix(ci): Publish Helm Charts in release workflow (#3789 by @google-oss-robot)
• [release-2.3] fix(ci): Publish images during the GitHub release workflow (#3786 by @google-oss-robot)
• fix(runtimes): propagate PodSets.Count into TemplateSpec Parallelism/Completions (#3651 by @h0pers)
• fix(runtimes): merge resourcesPerNode with Runtime container resources (#3602 by @Sridhar1030)
• fix: correct typos in AGENTS.md and deepspeed example notebook (#3765 by @ZhongJinHacker)
• fix(api): increase the length limit on train job commands and args (#3689 by @robert-bell)
• fix(docs): update Trainer documentation links (#3711 by @h0pers)
• fix(operator): honor managedBy field in TrainJob reconciler (#3681 by @kaisoz)
• fix(docs): add favicon, remove year from copyright (#3679 by @Sridhar1030)
• fix(docs): restyle top nav and moved ads to footer (#3656 by @Sridhar1030)
• fix(runtimes): set defaultMode on MPI SSH Secret volume (#3649 by @h0pers)
• fix: do not declare webhook cert data in Helm chart Secret (#3627 by @andreyvelich)
• fix(docs): improve top nav light mode contrast and add hero logo (#3625 by @Sridhar1030)
• fix(ci): Fix the release workflow updates (#3623 by @andreyvelich)
• fix(ci): Update Python API models version in make release (#3603 by @andreyvelich)
• fix(cache): use status code (not identity) to detect Service AlreadyExists (#3507 by @1fanwang)
• fix(ci): Guard git-cliff against empty user names (#3597 by @andreyvelich)
• fix(ci): Fix git-cliff for patch release Changelog (#3591 by @andreyvelich)
• fix(operator): guard against nil trainer PodSet in JAX EnforceMLPolicy (#3563 by @immanuwell)
• fix: apply clientConnection QPS/burst to the manager client (#3432 by @abhijeet-dhumal)
• fix: use the CI image in helm e2e tests (#3588 by @robert-bell)
• fix: migrate to newer name for cncf runners (#3556 by @jaiakash)
• fix(operator): support multi-replica endpoint generation in IdentifyP… (#3539 by @fedebongio)
• fix(examples): use namespaced SQuAD dataset (#3560 by @XploY04)
• fix(runtimes): add validation for LoRA multi-node and immutable trainer args (#3302 by @krishdef7)
• fix(cache): validate cache_index schema collisions in worker dat… (#3216 by @Hitanshi7556)
• fix: remove unnecessary setcap CAP_NET_BIND_SERVICE from MPI runtime docker file (#3286 by @kapil27)
• fix(api): allow atomic RuntimePatches update on unsuspend (#3469 by @NarayanaSabari)
• fix(manifests): Use released version for JobSet and LWS images (#3452 by @andreyvelich)
• fix(examples): unblock Megatron TP notebook on GPU E2E (#3434 by @XploY04)
• fix(ci): Bump Python to 3.12 in E2Es (#3330 by @andreyvelich)
• fix: use global rank for dataset download guard in container backend notebook (#3418 by @Fiona-Waters)
• fix(initializer): add missing glob wildcard to...

v2.3.0-rc.3

Kubeflow Trainer Official Release v2.3.0-rc.3

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

v2.3.0-rc.2

Kubeflow Trainer Official Release v2.3.0-rc.2

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

v2.3.0-rc.1

Kubeflow Trainer Official Release v2.3.0-rc.1

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

v2.3.0-rc.0

Kubeflow Trainer Official Release v2.3.0-rc.0

Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

v2.2.1

This is the Kubeflow Trainer v2.2.1 release 🚀

🚀 Features

• feat(ci): automate release process (#3536) (#3590 by @Krishna-kg732)
• feat: update k8s to 0.36 for release-2.2 (#3514 by @kannon92)

🐛 Bug Fixes

• [release-2.2] fix(ci): Update Python API models version in make release (#3622 by @andreyvelich)
• [release-2.2] fix(ci): Guard git-cliff against empty user names (#3598 by @andreyvelich)
• [release-2.2] fix(ci): Fix git-cliff for patch release Changelog (#3591) (#3593 by @andreyvelich)
• [release-2.2] fix(examples): use namespaced SQuAD dataset (#3560) (#3592 by @XploY04)
• [release-2.2] fix(api): allow atomic RuntimePatches update on unsuspend (#3489 by @NarayanaSabari)
• [release-2.2] fix(manifests): Use released version for JobSet and LWS images (#3453 by @andreyvelich)

⚙️ Miscellaneous Tasks

• chore(ci): Remove Trivy Action (#3389) (#3391 by @andreyvelich)

v2.2.0

This is the Kubeflow Trainer v2.2.0 release 🚀

You can now deploy Trainer control plane and runtimes with a single Helm install:

helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer \
    --namespace kubeflow-system \
    --create-namespace \
    --version 2.2.0 \
    --set runtimes.defaultEnabled=true

Install Kubeflow Python SDK:

pip install kubeflow

For more information, please see the Kubeflow Trainer docs.

Breaking Changes

• feat(api): BREAKING CHANGE: Replace PodTemplateOverrides with RuntimePatches API (#3309 by @astefanutti)
• feat(api): BREAKING CHANGE: Remove numProcPerNode from Torch API (#3239 by @andreyvelich)
• feat(api): BREAKING CHANGE: Remove ElasticPolicy API (#3235 by @andreyvelich)
• feat(api): Fix immutability of the TrainJob APIs (#3157 by @andreyvelich)

New Features

XGBoost & JAX Runtimes

• feat(runtimes): Add XGBoost runtime(KEP-2598) (#3200 by @Krishna-kg732)
• feat(docs): KEP-2598 XGBoost Runtime for Trainer V2 (#3118 by @Krishna-kg732)
• feat(runtimes): Add JAX training runtime (#3151 by @kaisoz)
• feat(cache): KEP-2655: Adding default runtime with cache and example (#2923 by @akshaychitneni)

Flux Runtime for MPI and HPC Workloads

• feat: support for flux framework as hpc manager (#3188 by @vsoch)
• feat: KEP 2841 Flux Policy to support Flux Framework (#2909 by @vsoch)

TrainJob Lifecycle

• feat(api): Set RuntimePatch.Time field automatically during admission (#3363 by @astefanutti)
• feat: add support for tracking TrainJob progress and training metrics (#3227 by @robert-bell)
• feat(docs): KEP-2779: Track TrainJob progress and expose training metrics (#2905 by @robert-bell)
• feat: add activeDeadlineSeconds (#3258 by @XploY04)
• feat(docs): proposal for adding TTLSecondsAfterFinished and ActiveDeadlineSeconds fields to TrainJob CRD (#3068 by @XploY04)
• chore: upstream istio support - superseding 3189 (#3259 by @sameerdattav)
• feat(runtimes): Use JobSet VolumeClaimPolicies APIs for LLM Runtimes (#3150 by @andreyvelich)
• feat(cache): KEP-2655 - Supporting readiness probes on cache nodes (#2904 by @akshaychitneni)
• feat(initializer): add s3 model and dataset initializers (#2728 by @rudeigerc)
• feat(api): Add securityContext support to PodTemplateSpecOverride in TrainJob (#3066 by @Sanskarzz)

Bug Fixes

• fix(initializer): add missing glob wildcard to .pt and .pth ignore p… (#3364 by @ghazariann)
• fix(ci): re-enable XGBoost E2E test (#3348 by @Krishna-kg732)
• fix(examples): Verify TrainJob Completion (#3344 by @andreyvelich)
• fix(docs): Update steps in release document (#3342 by @andreyvelich)
• fix: back to a10-1 for gpu e2e and time slicing (#3340 by @jaiakash)
• fix(ci): Generate valid release version for Python package (#3334 by @andreyvelich)
• fix(test): Fix Data Cache runtime in Helm Charts (#3241 by @andreyvelich)
• fix: failing e2e and gpu e2e tests (#3234 by @jaiakash)
• fix: align torch-distributed-with-cache runtime logic with unit tests (#3226 by @Goku2099)
• fix(ci): correct duplicate step name in test-go.yaml (#3202 by @puwun)
• fix: align torchao with torch 2.9.1 to fix GPU e2e failure (#3203 by @Goku2099)
• fix: Defer kubernetes imports to method level for use with local mode (#3167 by @Fiona-Waters)
• fix: service account test filename (#3153 by @aniketpati1121)
• fix(manifests): Remove jobset and lws patches from kustomize deployment (#3141 by @yosri-brh)
• fix: enable read-only root filesystem for trainer manager (#3119 by @Goku2099)
• fix: resourcePerNode override not applied with Volcano scheduler (#2982 by @sksingh2005)
• fix(operator): Prevent JobSet recreation when its TTL has expired (#3013 by @astefanutti)
• fix(operator): Use Patch to update TrainJob status (#3009 by @astefanutti)
• fix(manifests): Fix RBAC for ClusterTrainingRuntime Access (#3022 by @andreyvelich)
• fix(manifests): fix Prometheus metrics port mismatch (#3056 by @ChughShilpa)
• fix(manifests): Fix boolean values defaulting in Helm charts (#2913 by @astefanutti)
• fix(manifests): Fix Helm charts image name (#2915 by @andreyvelich)
• fix(manifests): Remove the default tag from the controller image (#2916 by @andreyvelich)
• fix: add appVersion field to Helm chart for Kubeflow Trainer (#3044 by @milinddethe15)
• fix(runtimes): Update pip version in the MLX runtime (#2908 by @andreyvelich)
• fix(examples): Fix SSL certificate error for local MNIST example (#2971 by @astefanutti)
• fix(ci): Fix kube-api-linter install (#3023 by @astefanutti)
• fix(ci): Fix new contributors GH actions workflow lint errors (#3024 by @astefanutti)
• fix(ci): Fix the Kubeflow SDK installation with Docker (#2926 by @andreyvelich)

Misc

• chore(runtimes): persist runtimes map and expose Runtimes function (#3367 by @kaisoz)
• chore: Remove deprecated Python models (#3318 by @andreyvelich)
• fix: rename JAX and Torch runtime plugin tests to descriptive names (#3283 by @Amir380-A)
• fix(examples): add parameters to Fashion MNIST training function (#3301 by @krishdef7)
• chore(ci): Ignore Coveralls Errors (#3260 by @andreyvelich)
• fix: Enforce single ML policy constraint with CEL validation for Torch, MPI, and JAX (#3225 by @Krishna-kg732)
• feat: Helm test workflow (#3228 by @Goku2099)
• feat: add production-ready MNIST example for PyTorch (#3063 by @Snehadas2005)
• feat(runtimes): add support for ClusterTrainingRuntimes in Helm chart (#3124 by @khushiiagrawal)
• feat(cache): add Helm chart configuration for data_cache (#3080 by @khushiiagrawal)
• feat(examples): add torch.compile to PyTorch local examples (#3076 by @Ishtiyaque-Alam)
• feat(manifests): Publish Trainer Helm Charts (#2906 by @adity1raut)
• fix(test): Ignore version increment in Helm Chart lint (#3240 by @andreyvelich)
• feat: Code Quality Checks workflow (#3224 by @Goku2099)
• chore: Add comprehensive unit tests for Config API (#2893 by @kapil27)
• chore(operator): Use SSA throughout runtime framework (#2877 by @astefanutti)
• chore(operator): Remove Unstructured objects caching (#3010 by @astefanutti)
• chore: Expose trainer API version via public ConfigMap (#3083 by @sameerdattav)
• chore: changed latest to dev in trainer manifests (#3146 by @sameerdattav)
• chore: use named ports for manager deployment and service (#3100 by @Goku2099)
• chore: fix make helm-lint (#3103 by @robert-bell)
• feat: add scaffolding for ...

v2.2.0-rc.1

This is Kubeflow Trainer v2.2.0-rc.1 pre-release:

helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer \
    --namespace kubeflow-system \
    --create-namespace \
    --version 2.2.0-rc.1 \
    --set runtimes.defaultEnabled=true

v2.2.0-rc.0

This is Kubeflow Trainer v2.2.0-rc.0 pre-release:

helm install kubeflow-trainer oci://ghcr.io/kubeflow/charts/kubeflow-trainer \
    --namespace kubeflow-system \
    --create-namespace \
    --version 2.2.0-rc.0 \
    --set runtimes.defaultEnabled=true