Regression in hpe-csi-driver helm 3.0.2: `DeleteVolume` fails with 409 "resource in use" on Primera/Alletra B10000 due to race condition / premature success in `ControllerUnpublishVolume`

[aki-kamada]

Environment:

• HPE CSI Driver version: Upgraded from 2.5.2 to 3.0.2 (Deployed via Helm)
• Storage Arrays: HPE Primera and HPE Alletra B10000 (Alletra Storage MP)
• Storage Protocol: iSCSI
• Orchestrator: Kubernetes 1.32.9
• Context: Real production services

Bug Summary & Root Cause Analysis:
We are actively using the HPE CSI driver in our real production services with HPE Primera and Alletra B10000 arrays via iSCSI. After upgrading to version 3.0.2, deleting PVCs frequently leaves the PVs in a terminating/released state forever. We have successfully reproduced the issue and isolated the root cause to a regression/race condition in how the 3.0.2 driver handles VLUN removal over iSCSI.

The root cause: ControllerUnpublishVolume is returning SUCCESS to Kubernetes before the physical storage array has actually finished removing the VLUN mapping.
Because K8s receives a premature success response, it immediately calls DeleteVolume. The DeleteVolume call correctly fails (returning 409 Conflict from the array, mapped to HTTP 500 by the CSP) because the array protects the VV since the VLUN is technically still mapped. This causes an infinite retry loop in K8s.

This is an intermittent race condition that becomes highly reproducible when bulk deleting Pods/PVCs simultaneously.

Steps to Reproduce (Real Execution Trace):
Executing multiple deletions simultaneously triggers the race condition.

# 1. Bulk delete multiple deployments and PVCs simultaneously
$ kubectl delete -f deployment-pvc2000-standard01.yaml & kubectl delete -f deployment-pvc2000-standard02.yaml & kubectl delete -f deployment-pvc2000-standard03.yaml & kubectl delete -f first-test-pvc2000-velero-standard01.yaml & kubectl delete -f first-test-pvc2000-velero-standard02.yaml & kubectl delete -f first-test-pvc2000-velero-standard03.yaml
[1] 46887
[2] 46888
[3] 46889
...
[1]   Done                    kubectl delete -f deployment-pvc2000-standard01.yaml
[2]   Done                    kubectl delete -f deployment-pvc2000-standard02.yaml
[3]   Done                    kubectl delete -f deployment-pvc2000-standard03.yaml

# 2. Most PVs are successfully deleted:
$ kubectl get pv pvc-17d2e393-64a4-4705-9996-46523049244b
Error from server (NotFound): persistentvolumes "pvc-17d2e393-64a4-4705-9996-46523049244b" not found

$ kubectl get pv pvc-910a9d7f-88a0-442e-a516-8db67947e35c
Error from server (NotFound): persistentvolumes "pvc-910a9d7f-88a0-442e-a516-8db67947e35c" not found

# 3. However, some PVs get completely stuck due to the 409 Conflict:
$ kubectl get pv pvc-808d2c4b-f292-4aab-a1c9-5f55c362b5ec
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                       STORAGECLASS     REASON   AGE
pvc-808d2c4b-f292-4aab-a1c9-5f55c362b5ec   3Gi        RWO            Delete           Released   kamada/my-first-pvc2000-velero-standard03   hpe-standard03            3m20s

Evidence 1: Timing comparison from CSI Controller Logs (The Smoking Gun)
We compared the logs of a successful deletion (pvc-17d2e...) vs the failed deletion (pvc-808d2c...). A physical Primera/Alletra array takes about ~2 seconds to unmap a VLUN.

[Failed Run - Bug occurring for pvc-808d2c...]
The driver returns success for Unpublish in just 0.48 seconds, which is physically impossible for the array to actually complete the unmap. K8s immediately calls DeleteVolume and gets rejected.

# 1. Unpublish is called
I0227 02:46:15.186976 1 connection.go:264] "GRPC call" driver="csi.hpe.com" method="/csi.v1.Controller/ControllerUnpublishVolume" request="{\"node_id\":\"...\",\"volume_id\":\"pvc-808d2c4b-f292-4aab-a1c9-5f55c362b5ec\"}"

# 2. Only 0.48 seconds later, DeleteVolume is called (Driver returned success prematurely!)
I0227 02:46:15.666214 1 connection.go:264] "GRPC call" method="/csi.v1.Controller/DeleteVolume" request="{\"volume_id\":\"pvc-808d2c4b-f292-4aab-a1c9-5f55c362b5ec\"}"

# 3. DeleteVolume immediately fails because the VLUN is still there on the array
time="2026-02-27T02:46:15Z" level=error msg="Error deleting the volume pvc-808d2c4b-f292-4aab-a1c9-5f55c362b5ec, err: rpc error: code = Internal desc = Error while deleting volume ... err: Request failed with status code 500

[Successful Run - Normal behavior for pvc-17d2e...]
In a successful run, the gap between Unpublish and DeleteVolume is ~2.08 seconds, meaning it properly waited for the array.

# 1. Unpublish is called
I0227 02:46:17.638852 1 connection.go:264] "GRPC call" driver="csi.hpe.com" method="/csi.v1.Controller/ControllerUnpublishVolume" request="{\"volume_id\":\"pvc-17d2e393-64a4-4705-9996-46523049244b\"}"

# 2. ~2.08 seconds later, DeleteVolume is called
I0227 02:46:19.718236 1 connection.go:264] "GRPC call" method="/csi.v1.Controller/DeleteVolume" request="{\"volume_id\":\"pvc-17d2e393-64a4-4705-9996-46523049244b\"}"

# 3. Volume deleted successfully
I0227 02:46:20.041216 1 controller.go:1574] "Volume deleted" PV="pvc-17d2e393-64a4-4705-9996-46523049244b"

Evidence 2: Storage Array verifies the VLUN was NEVER removed
Checking the array directly via CLI (showvlun) confirms the VLUN remains indefinitely for the stuck volume:

fuga-int-c00-g02-hxc0002 cli% showvlun -vv pvc-808d2c4b-f292-4aab-a1c9-5f55c362b5ec
Lun VVName                          HostName          -Host_WWN/iSCSI_Name-  Port        Type
  0 pvc-808d2c4b-f292-4aab-a1c9-... iqn-hoge-w4523-dev ----------------      0:4:1 matched set

Attached Logs:
Please find the attached 20260227-hpe-csi-driver-3.0.2-errorlog.zip which contains the full hpe-csi-controller and CSP logs capturing this behavior.

Expected Behavior:
Just like in version 2.5.2, ControllerUnpublishVolume MUST synchronously wait for the array to actually complete the VLUN removal operation, and ONLY return success to K8s when it is fully unmapped. The current async/non-blocking behavior (or swallowed error) in 3.0.2 breaks the deletion sequence completely. This is critically impacting our production environments.

---

Thanks,
Aki

20260227-hpe-csi-driver-3.0.2-errorlog.zip

[datamattsson]

Thanks for reporting this. I've notified engineering about this. Stay tuned.

[diganta-001]

We conducted a test with 100 Pods and 100 PVCs, deleting them asynchronously, but we were not able to observe any error messages. To help us investigate further, could you please run the log collector script and share the complete logs, specifically the hpe-csi-controller.log file ?

[aki-kamada]

Hi, diganta-001-san.

Please give me some time. I will rerun the reproduction test, collect the logs using this script, and provide them to you. I will definitely get back to you later.
https://github.com/hpe-storage/csi-driver/blob/master/hpe-logcollector.sh

Thanks,
Aki

[aki-kamada]

Hi, diganta-001-san.

I have conducted the reproduction test again, executed hpe-logcollector.sh, and attached the collected logs for your review.

Attached Logs:

• Local execution environment log: client-side-hpe-csi-log.zip
• Worker node /var/log log: worker-node-hpe-csi-log.zip

Note regarding the attached logs:
Please understand that I had to make the following adjustments before uploading them to this public GitHub issue:

• Certain internal names (such as specific service names) have been masked with xxxx. However, the namespace kamada used for the reproduction is generally left unmasked.
• Because the worker node logs are extremely large, I am only submitting the logs generated today (the date of reproduction) specifically from the node w4519 where the issue was reproduced.

The reproduction was performed using the exact same steps as before.

Reproduction Steps

# Deleting 3 deployments and 3 PVCs simultaneously
$ kubectl delete -f deployment-pvc2000-standard01.yaml & kubectl delete -f deployment-pvc2000-standard02.yaml & kubectl delete -f deployment-pvc2000-standard03.yaml & kubectl delete -f first-test-pvc2000-velero-standard01.yaml & kubectl delete -f first-test-pvc2000-velero-standard02.yaml & kubectl delete -f first-test-pvc2000-velero-standard03.yaml

Observation (Kubernetes):
I confirmed that only the PVs for first-test-pvc2000-velero-standard02.yaml and first-test-pvc2000-velero-standard03.yaml are left behind (orphaned) in the Released state.

$ kubectl sudo get pv | grep "Released"| grep "kamada"
pvc-a964fa85-9406-447c-b6ed-9ed7e1849893   3Gi         RWO            Delete           Released      kamada/my-first-pvc2000-velero-standard02                                      hpe-standard02                    <unset>                          5m13s
pvc-cd80be69-8449-4731-918c-4acbd75596e5   3Gi         RWO            Delete           Released      kamada/my-first-pvc2000-velero-standard03                                      hpe-standard03                    <unset>                          5m10s

Evidence from Primera Storage

1. hpe-standard02
We can see that the VV with ID 1648 (pvc-a964fa85...) was not deleted and remains on the storage side.

hxb0003-cage0 cli% showvv
                                                                                               -Rsvd(MiB)-- -(MiB)-
  Id Name                             Prov Compr Dedup Type CopyOf BsId Rd -Detailed_State-   Snp    Usr   VSize
   2 .mgmtdata                        full NA    NA    base ---       2 RW normal               0 524288  524288
   1 .srdata                          full NA    NA    base ---       1 RW normal               0  81920   81920
   0 admin                            full NA    NA    base ---       0 RW normal               0  10240   10240
1510 pvc-2be97f7e-59f5-4c52-9915-121 tpvv No    No    base ---    1510 RW normal            1024   1024   30720
1513 pvc-4c31dde3-f211-4b16-9550-a21 tpvv No    No    base ---    1513 RW normal            1024  99328  716800
1511 pvc-6b1bffe3-8381-48b0-a09f-97a tpvv No    No    base ---    1511 RW normal            1024   3072   30720
1638 pvc-14e6db23-f074-4c57-8d2e-0f1 tpvv No    No    base ---    1638 RW normal            1024   1024    1024
1516 pvc-178bbc90-819a-4dd0-abb2-328 tpvv No    No    base ---    1516 RW normal            1024   1024    1024
1605 pvc-553aeaee-c90a-4461-a7f6-aaf tpvv No    No    base ---    1605 RW normal            1024   1024    1024
1639 pvc-709ba4d3-f466-4f84-974d-773 tpvv No    No    base ---    1639 RW normal            1024   1024    1024
 111 pvc-72778dee-567c-4702-b27e-4d6 tpvv No    No    base ---     111 RW normal            1024   1024    3072
1514 pvc-78771baa-6ebb-4d9a-a09a-c83 tpvv No    No    base ---    1514 RW normal            1024   1024    5120
1517 pvc-478817e2-80cb-4934-8577-5a8 tpvv No    No    base ---    1517 RW normal            1024   5120    5120
1648 pvc-a964fa85-9406-447c-b6ed-9ed tpvv No    No    base ---    1648 RW normal            1024   1024    3072
1508 pvc-ad7d9207-ca87-4258-a0d6-04e tpvv No    No    base ---    1508 RW normal            1024  54272  512000
1505 pvc-aea5e5ee-8870-4d3a-a62d-2a1 tpvv No    No    base ---    1505 RW normal            1024  54272  512000
1632 pvc-b18b1c2b-6d96-4492-929b-5dd tpvv No    No    base ---    1632 RW normal            1024   1024    1024
1509 pvc-bf1099f4-e4e5-4b14-b00f-266 tpvv No    No    base ---    1509 RW normal            1024   1024   30720
1507 pvc-ca772b29-9aa8-4cb8-aedf-3ec tpvv No    No    base ---    1507 RW normal            1024  54272  512000
1506 pvc-d3c0b489-fb26-48f5-a0c4-15b tpvv No    No    base ---    1506 RW normal            1024  54272  512000
1515 pvc-d3240018-218b-4262-9641-245 tpvv No    No    base ---    1515 RW normal            1024   1024    1024
1647 pvc-ee472967-2bb7-4c12-a874-4fa tpvv No    No    base ---    1647 RW normal            1024   1024    1024
---------------------------------------------------------------------------------------------------------------
  22 total                                                                                 19456 953344 3496960

2. hpe-standard03
Similarly, we can see that the VV with ID 56 (pvc-cd80be69...) also remains left behind.

hxb0006 cli% showvv
                                                                                             -Rsvd(MiB)-- -(MiB)-
Id Name                             Prov Compr Dedup Type CopyOf BsId Rd -Detailed_State-   Snp    Usr   VSize
 2 .mgmtdata                        full NA    NA    base ---       2 RW normal               0 524288  524288
 1 .srdata                          full NA    NA    base ---       1 RW normal               0  81920   81920
 0 admin                            full NA    NA    base ---       0 RW normal               0  10240   10240
54 pvc-0c01579f-af25-4c6f-910d-587 tpvv No    No    base ---      54 RW normal            1024   1024    1024
39 pvc-9a841432-5f13-4c74-9f72-d11 tpvv No    No    base ---      39 RW normal            1024   1024    5120
43 pvc-50c1d08c-e019-4dad-99c5-fd6 tpvv No    No    base ---      43 RW normal            1024   1024    1024
37 pvc-67eca9e2-9373-4a59-99aa-92e tpvv No    No    base ---      37 RW normal            1024   1024    1024
32 pvc-770e5ce9-193b-4cdd-8b15-934 tpvv No    No    base ---      32 RW normal            1024   1024   20480
33 pvc-776f0b6c-5179-471c-a79c-856 tpvv No    No    base ---      33 RW normal            1024   1024   30720
45 pvc-811f0b1f-c818-4d83-9626-013 tpvv No    No    base ---      45 RW normal            1024   1024    1024
38 pvc-48739f0f-4eb3-4fcb-802b-d19 tpvv No    No    base ---      38 RW normal            1024   1024    1024
48 pvc-ba290b60-ac7a-4f13-80a4-2a0 tpvv No    No    base ---      48 RW normal            1024   1024    1024
19 pvc-bec2b33a-6fbf-457a-92bf-dc5 tpvv No    No    base ---      19 RW normal            1024   1024    5120
56 pvc-cd80be69-8449-4731-918c-4ac tpvv No    No    base ---      56 RW normal            1024   1024    3072
55 pvc-ddb44e09-5aa0-4103-bc0d-958 tpvv No    No    base ---      55 RW normal            1024   1024    1024
24 pvc-e4f6ceb3-621e-403a-91f1-202 tpvv No    No    base ---      24 RW normal            1024   1024    5120
11 pvc-e24ec4c9-b3fd-43a6-8686-164 tpvv No    No    base ---      11 RW normal            1024   1024    5120
-------------------------------------------------------------------------------------------------------------
17 total                                                                                 14336 630784  698368

Thanks,
Aki

client-side-hpe-csi-log.zip

worker-node-hpe-csi-log.zip