Snapshots of PVCs fails on the tenant cluster PVCs

[skakram]

While attempting to create a Snapshot of a PVC provisioned in the Tenant cluster it immediately Errors out

VolumeSnapshot Class (Removed managedfields for brevity)

apiVersion: snapshot.storage.k8s.io/v1
deletionPolicy: Delete
driver: csi.kubevirt.io
kind: VolumeSnapshotClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |      {"apiVersion":"snapshot.storage.k8s.io/v1","deletionPolicy":"Delete","driver":"csi.kubevirt.io","kind":"VolumeSnapshotClass","metadata":{"annotations":{},"labels":{"app":"kubevirt-csi-driver"},"name":"kubevirt-csi-snapclass"}}
  creationTimestamp: '2026-02-09T12:12:41Z'
  generation: 1
  labels:
    app: kubevirt-csi-driver
  name: kubevirt-csi-snapclass

I have the Snapshot class mapping in the driver-config config map

data:
  infraClusterLabels: csi-driver/cluster=tenant
  infraClusterNamespace: ocp-nested
  infraStorageClassEnforcement: |
    allowAll: false
    allowDefault: true
    allowList: [sc-gold-dc1, sc-gold-dc2]
    storageSnapshotMapping: 
      - StorageClasses:
        - sc-gold-dc1
        VolumeSnapshotClasses:
        - snapshotclass-gold-dc1
      - StorageClasses:
        - sc-gold-dc2
        VolumeSnapshotClasses:
        - snapshotclass-gold-dc2

Is there something missing inorder to have snapshots configured. I have the corresponding snapshotclass on the infra cluster snapshotclass-gold-dc1 and snapshotclass-gold-dc2 which work fine

Error in the csi-snapshotter pod logs

E0212 14:10:55.062550 1 snapshot_controller.go:124] checkandUpdateContentStatus [snapcontent-861f45e1-bf89-4964-9162-5752c5cf5df6]: error occurred failed to add VolumeSnapshotBeingCreated annotation on the content snapcontent-861f45e1-bf89-4964-9162-5752c5cf5df6: "snapshot controller failed to update snapcontent-861f45e1-bf89-4964-9162-5752c5cf5df6 on API server: VolumeSnapshotContent.snapshot.storage.k8s.io \"snapcontent-861f45e1-bf89-4964-9162-5752c5cf5df6\" is invalid: spec: Invalid value: \"object\": sourceVolumeMode is required once set"
E0212 14:10:55.062589 1 snapshot_controller_base.go:265] could not sync content "snapcontent-861f45e1-bf89-4964-9162-5752c5cf5df6": failed to add VolumeSnapshotBeingCreated annotation on the content snapcontent-861f45e1-bf89-4964-9162-5752c5cf5df6: "snapshot controller failed to update snapcontent-861f45e1-bf89-4964-9162-5752c5cf5df6 on API server: VolumeSnapshotContent.snapshot.storage.k8s.io \"snapcontent-861f45e1-bf89-4964-9162-5752c5cf5df6\" is invalid: spec: Invalid value: \"object\": sourceVolumeMode is required once set"

Environment :

Tenant Cluster : OpenShift version 4.20
Infra cluster : OpenShift Version 4.18

[Mehben]

I ran into the same issue and tracked down the root cause. There are three things needed for snapshots to work in a split (non-HyperShift) deployment:

1. The infraStorageClassEnforcement key must exist in the driver-config ConfigMap

The controller needs to know which infra VolumeSnapshotClass to use when creating snapshots on the infra cluster. Without this, the driver either defaults to AllowDefault: true (which creates the infra snapshot without a class) or fails silently.

Add the infraStorageClassEnforcement key to your existing driver-config ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: driver-config
  namespace: <infra-namespace>
data:
  infraClusterLabels: csi-driver/cluster=tenant
  infraClusterNamespace: <infra-namespace>
  infraStorageClassEnforcement: |
    allowAll: true
    allowDefault: false
    storageSnapshotMapping:
      - storageClasses:
          - sc-gold-dc1
        volumeSnapshotClasses:
          - snapshotclass-gold-dc1
      - storageClasses:
          - sc-gold-dc2
        volumeSnapshotClasses:
          - snapshotclass-gold-dc2

The key fields:

• allowAll: true : allows all storage classes for volume provisioning
• allowDefault: false : forces the driver to use the explicit mapping instead of relying on a default snapshot class on the infra cluster
• storageSnapshotMapping : maps each infra StorageClass to its corresponding infra VolumeSnapshotClass(es)

2. The INFRA_STORAGE_CLASS_ENFORCEMENT env var must be set on the csi-driver container

The controller Deployment needs an environment variable that reads the ConfigMap key. The upstream deploy/controller-infra/base/deploy.yaml has this as optional: true, so it's easy to miss.

Make sure your csi-driver container in the controller Deployment includes:

env:
  - name: INFRA_STORAGE_CLASS_ENFORCEMENT
    valueFrom:
      configMapKeyRef:
        name: driver-config
        key: infraStorageClassEnforcement
        optional: true

Also verify that the driver-config ConfigMap is mounted as a volume:

volumeMounts:
  - name: driver-config
    mountPath: /var/run/kubevirt-csi-driver

3. The tenant VolumeSnapshotClass needs the infraSnapshotClassName parameter

The driver uses this parameter to match the tenant snapshot class to the corresponding infra snapshot class. Without it, the mapping function in buildStorageClassSnapshotClassMapping (client.go) can't resolve the infra↔tenant relationship.

Your tenant VolumeSnapshotClass should look like:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: kubevirt-csi-snapclass
driver: csi.kubevirt.io
deletionPolicy: Delete
parameters:
  infraSnapshotClassName: snapshotclass-gold-dc1

The infraSnapshotClassName value must match one of the entries in volumeSnapshotClasses from the ConfigMap mapping.

4. Version skew between csi-snapshotter and snapshot CRDs

Your error about sourceVolumeMode is required once set is a separate issue caused by a version mismatch. The VolumeSnapshotContentSpec CRD has a CEL validation rule:

XValidation: rule="!has(oldSelf.sourceVolumeMode) || has(self.sourceVolumeMode)"

This means once sourceVolumeMode is set on a VolumeSnapshotContent, it cannot be removed. If the csi-snapshotter sidecar (or snapshot-controller) uses client types from an older version that doesn't include the sourceVolumeMode field, any update it makes will silently drop the field, and the API server rejects it.

You can check for a mismatch with:

# csi-snapshotter sidecar image version (infra side)
kubectl get pod -n <infra-namespace> <csi-controller-pod> -o jsonpath='{.spec.containers[?(@.name=="csi-snapshotter")].image}' 

# snapshot-controller version (tenant side) 
kubectl get pod -n kube-system <snapshot-controller-pod> -o jsonpath='{.spec.containers[0].image}'

# CRD version on the tenant cluster 
kubectl get crd volumesnapshotcontents.snapshot.storage.k8s.io -o jsonpath='{.metadata.annotations.controller-gen\.kubebuilder\.io/version}'

Make sure the csi-snapshotter sidecar, the snapshot-controller, and the snapshot CRDs are all from compatible versions of kubernetes-csi/external-snapshotter. In your case (OpenShift 4.18 infra / 4.20 tenant), the OCP-bundled versions likely diverge.

What happens without these

Without the mapping, getSnapshotClassNameFromVolumeClaimName() in client.go returns an empty string. The infra VolumeSnapshot gets created without a VolumeSnapshotClassName. The infra snapshot-controller then fails with:

Failed to set default snapshot class with error cannot find default snapshot class

...and the snapshot stays readyToUse: false forever.

After applying

Restart the controller pod after updating the ConfigMap so the env var is re-read. Delete any stuck snapshots on both the tenant and infra side, then recreate on the tenant. The csi-driver should now create the infra snapshot with the correct VolumeSnapshotClass, and readyToUse should become true within seconds.

# k get volumesnapshot
NAME                 READYTOUSE   SOURCEPVC           SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS            SNAPSHOTCONTENT                                    CREATIONTIME   AGE
test-kubevirt-snap   true         test-kubevirt-csi                           10Gi          kubevirt-csi-snapclass   snapcontent-f217578d-f3fc-46ec-9b27-de543f207e8a   28m            28m

Environment :

Tenant Cluster : RKE2 1.33.5
Infra cluster : OpenShift Version 4.17

I may submit a little doc about that, if it's okay for maintainers.

[kubevirt-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

/lifecycle stale