feat(csi)!: adopt CSI spec v1.13.0 and migrate to the VolumeHealth API - #364
Closed
sticky-gecko[bot] wants to merge 8 commits into
Closed
feat(csi)!: adopt CSI spec v1.13.0 and migrate to the VolumeHealth API#364sticky-gecko[bot] wants to merge 8 commits into
sticky-gecko[bot] wants to merge 8 commits into
Conversation
…v1.12.0 → v1.13.0)
onedr0p
enabled auto-merge
July 29, 2026 09:20
Contributor
Author
Edited/Blocked NotificationRenovate will not automatically rebase this PR, because it does not recognize the last commit author and assumes somebody else may have edited the PR. You can manually request rebase by checking the rebase/retry box above. |
CSI spec v1.13 removes the alpha VolumeCondition message, both VOLUME_CONDITION service capabilities and the volume_condition fields on NodeGetVolumeStats, ControllerGetVolume and ListVolumes, replacing them with VolumeHealth and dedicated RPCs. Answer ControllerGetVolumeHealth, ControllerListVolumeHealth and NodeGetVolumeHealth, reporting split-brain as DATA_LOSS, a failed backing disk or a lagging replica as DEGRADED, and a volume whose device never materialized as INACCESSIBLE. The new message carries a list, so every condition that holds is reported instead of only the highest-ranked one. csi-test has no release that compiles against the new spec, so pin the master pseudo-version until one is cut. BREAKING CHANGE: the external-health-monitor sidecar and its sidecars.healthMonitor values are removed from the chart. external-health-monitor v0.18.0 exits non-zero against a driver that no longer advertises VOLUME_CONDITION, so keeping it would crash-loop the controller pod, and its support for the health RPCs is still unmerged upstream. Volume health remains available through the miroir_volume_* metrics and the CSI health RPCs. Signed-off-by: Devin Buhl <devin@buhl.casa>
onedr0p
changed the title
Follow-ups on the VolumeHealth migration: - ControllerGetVolume dropped the whole Status message when its only populated member went away. v1.13 reserved VolumeCondition inside VolumeStatus, not VolumeStatus itself, which is still REQUIRED. - Gate the phase and split-brain mappings on status.activated/formatted, the same latch split-brain auto-recovery keys on. A never-activated birth split reports DEGRADED/SplitBrainRecovering instead of the DATA_LOSS that would send a CO restoring a snapshot of an empty volume; Creating on a volume that has already served is no longer reported as healthy; Failed with a surviving leg is DEGRADED, not INACCESSIBLE, since a peer keeps serving the pod. - Sort the entries by severity so "most-urgent first" holds regardless of the order the mappers ran in. - NodeGetVolumeHealth answers for the node that asks: only that node's replica status feeds the per-replica signals, plus the live kernel view for the quorum loss the CRD never carries. It was a byte- identical copy of the controller RPC, reporting peers' faults as the local node's while missing the leg that cannot be staged there. - The health listing's index space shrinks as volumes recover, so a positional token routinely falls off the end between two pages of one listing. Return an empty page there instead of aborting; ListVolumes keeps aborting, its set only moves on create/delete. - Guard the removed sidecars.healthMonitor values with a fail, matching the chart's other removed-key guards, and document 0.12.0 in the upgrade notes. - Correct the doc comments and monitoring.md prose that claimed PVC events and kubelet volume-health conditions are live today. - Extract sortedVolumes and Node.getVolume, wire requireVolume into the remaining identical lookups, and assert gRPC codes rather than err != nil in the health tests. Signed-off-by: Devin Buhl <devin@buhl.casa>
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 29, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 29, 2026
staticcheck only knows t.Fatalf terminates when the testing package facts are part of the run; with a restored golangci-lint cache they are not always there, and SA5011 fires on the deref that follows. Moving it into an else branch makes the guard structural. Signed-off-by: Devin Buhl <devin@buhl.casa>
Contributor
sticky-gecko
Bot
deleted the
renovate/github.com-container-storage-interface-spec-1.x
branch
July 30, 2026 11:13
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates
github.com/container-storage-interface/specfromv1.12.0tov1.13.0, and migrates the driver onto the API that replaces the one v1.13.0 deleted.Why this is more than a version bump
The v1.13.0 release notes say "Breaking changes/Deprecations: None", but the release removed the alpha
VolumeConditionAPI outright:VolumeConditionmessage,ControllerServiceCapability.RPC.VOLUME_CONDITION(11) andNodeServiceCapability.RPC.VOLUME_CONDITION(4),volume_conditionfield onNodeGetVolumeStatsResponse,ControllerGetVolumeResponse.VolumeStatusandListVolumesResponse.VolumeStatus.internal/csiused all of it, so the bare bump failed to compile (Go Test / Go Lint / Build / every E2E leg).What replaces it
CSI v1.13.0 adds an alpha
VolumeHealthAPI — aVolumeHealthErrorTypeenum (DEGRADED,INACCESSIBLE,DATA_LOSS) plus dedicated RPCs. The driver now implements all three and advertises the matching capabilities (GET_VOLUME_HEALTH,LIST_VOLUME_HEALTHon the controller;GET_VOLUME_HEALTHon the node):DATA_LOSSSplitBrainDEGRADEDBackingDiskFailedINACCESSIBLEProvisioningFailedDEGRADEDReplicaOutOfSyncVolumeHealthcarries a list, so every condition that holds is now reported (ordered most-urgent first) instead of only the highest-ranked one — a split-brain volume reports the failed disk and the degraded replication alongside it.ControllerListVolumeHealthreturns only the abnormal set, paginated with the same positional tokensListVolumesuses.csi-test
csi-test/v5 v5.5.0(latest tag) still references the removed constants and does not compile against the new spec. Fixed on master by csi-test#614 but unreleased, so this pins the master pseudo-versionv5.5.1-0.20260728125401-ae8a63a708ab. Swap it for the tag once one is cut. The sanity suite passes against it.BREAKING: the health-monitor sidecar is removed
external-health-monitorv0.18.0 is still built against spec v1.12.0 and callsklog.FlushAndExit(…, 1)at startup when the driver does not advertiseVOLUME_CONDITION. Left in place it would put the controller pod intoCrashLoopBackOfffor anyone runningsidecars.healthMonitor.enabled: true, so the sidecar and the wholesidecars.healthMonitorvalues block are gone from the chart. Its support for the new RPCs is in external-health-monitor#376, still unmerged.Upgrade action: if you set
sidecars.healthMonitor.*, drop it from your values. The generated schema does not setadditionalProperties, so a stale key is ignored rather than rejected — the sidecar simply stops being rendered, no failed upgrade. Volume health is unchanged on themiroir_volume_*metrics and the shipped alerts/dashboard, which is where the chart already pointed operators.What this costs on Kubernetes 1.36
Nothing on a stock cluster. In
release-1.36,CSIVolumeHealthis{Version: 1.21, Default: false, PreRelease: Alpha}(pkg/features/kube_features.go), and kubelet's entire volume-condition path sits insideif utilfeature.DefaultFeatureGate.Enabled(features.CSIVolumeHealth)(pkg/volume/csi/csi_client.go). With the gate off — the default since 1.21 — kubelet never queries the capability and never readsvolume_condition, sokubelet_volume_stats_health_status_abnormalwas never being emitted for miroir volumes in the first place.On a cluster that explicitly enables the gate there are two effects: that metric stops appearing, and raw-block volumes lose a side effect. miroir returns an empty
NodeGetVolumeStatsResponsefor a raw-block path (statfs there would report the host filesystem, not the volume), and kubelet errorsfailed to get usage from response. usage is nilwhen the driver offers neither usage nor the volume-condition capability. The capability used to suppress that; now it doesn't. It is a stats-collection error only — mounting and pod health are unaffected — and it is already the behaviour on every default-gate cluster.kubelet's replacement (
pkg/kubelet/volumemanager/volumehealth, callingNodeGetVolumeHealthand folding results intoPodStatus.VolumeHealthandCSINode.Status.StorageHealth) is on Kubernetes master, so it lands in a release after v1.36. The driver is ready ahead of it; until then themiroir_volume_*gauges remain the surface to alert on.Verified locally
go build ./...,go vet ./..., the full unit suite,mise run test-sanity,golangci-lint run,helm lint,helm-unittest(119 tests),helm-docs-checkand the docs build all pass.Upstream release notes: container-storage-interface/spec v1.13.0
Compare Source
Breaking changes/Deprecations:
Additions:
Clarifications/Corrections/Fixes:
Dependencies:
Full Changelog: container-storage-interface/spec@v1.12.0...v1.13.0
Configuration
📅 Schedule: (UTC)
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. Do not rebase — the migration commit on this branch would be lost.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate and extended by hand with the API migration.