PowerFlex/ScaleIO MDM and host SDC connection enhancements #11047

sureshanaparti · 2025-06-17T07:00:03Z

Description

This PR enhances the PowerFlex/ScaleIO MDM and host SDC connections, includes the following changes (and some code improvements).

Introduced timeout configuration 'powerflex.mdm.change.apply.timeout.ms' (Default value: 1000 ms) at zone scope, to wait after MDM changes made on Host with ScaleIO SDC. Also, Changes to apply timeout after making MDM changes for ScaleIO in prepare and unprepare logic.
Introduced configuration flag 'powerflex.block.sdc.unprepare' (Default is false) at zone scope, to enable/disable blocking unprepare ScaleIO SDC connection when SDC client restart required and there are volumes attached to the Host. Added validation to fail Host disconnect from Storage Pool if there are Volumes attached and SDC client MDM removal requires scini service to be restarted.
Introduced configuration flag 'powerflex.mdm.validate.on.connect' (Default is false) at zone scope, to enable/disable validation of MDM addresses on Host from configuration file and from memory (using CLI) matches or not.
Added detection of MDM removal support via CLI. If MDM removal support via CLI supported then use CLI, Otherwise fall back to edit drv_cfg.txt and restart scini as earlier. Tested with /opt/emc/scaleio/sdc/bin/drv_cfg --version: DellEMC PowerFlex Version: R3_6.4000.124, with cmd: /opt/emc/scaleio/sdc/bin/drv_cfg --remove_mdm.
Added wait time after SDC service start/restart/stop, and retries to fetch SDC id/guid.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested the new settings, and PowerFlex SDC connections (MDM add/remove) with VM & Volume operations.

How did you try to break this feature and the system with this change?

…/unprepare, validate Storage Pool can be created in Agent. - Implemented validation to fail Host disconnect from Storage Pool if there are Volumes attached and SDC client MDM removal requires scini service to be restarted - Implemented Storage Pool validation by checking whether MDM addresses from configuration file and from memory (using CLI) matches, otherwise file ModifyStoragePool command. - Introduced configuration key to apply timeout after making MDM changes for ScaleIO: powerflex.mdm.change.apply.timeout.ms (default 1000ms) - Implemented logic to apply timeout after making MDM changes for ScaleIO in prepare and unprepare logic - Added detection of MDM removal support via CLI - If MDM removal support via CLI supported then use CLI, fall back to edit drv_cfg.txt and restart scini instead

… pool when SDC client restart required and there are volumes attached to the Host

…tack/storage/datastore/manager/ScaleIOSDCManager.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

…tack/storage/datastore/util/ScaleIOUtil.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

… validation

…y_guid, --rescan in drv_cfg cmd as it is not supported. --file parameter is supported with --add_mdm, --mod_mdm_ip, --remove_mdm, --set_guid, --set_mdm_password, --reset_mdm_password

sureshanaparti · 2025-06-17T07:05:11Z

@blueorangutan package

blueorangutan · 2025-06-17T07:06:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-06-17T07:12:24Z

Codecov Report

Attention: Patch coverage is 25.93857% with 217 lines in your changes missing coverage. Please review.

Project coverage is 16.57%. Comparing base (41b4f0a) to head (38db39e).
Report is 39 commits behind head on main.

Files with missing lines	Patch %	Lines
...cloudstack/storage/datastore/util/ScaleIOUtil.java	27.39%	99 Missing and 7 partials ⚠️
.../hypervisor/kvm/storage/ScaleIOStorageAdaptor.java	21.79%	54 Missing and 7 partials ⚠️
...re/lifecycle/ScaleIOPrimaryDataStoreLifeCycle.java	14.81%	21 Missing and 2 partials ⚠️
...torage/datastore/provider/ScaleIOHostListener.java	0.00%	12 Missing ⚠️
...s/src/main/java/com/cloud/utils/script/Script.java	0.00%	9 Missing ⚠️
...orage/datastore/manager/ScaleIOSDCManagerImpl.java	0.00%	4 Missing ⚠️
...rapper/LibvirtModifyStoragePoolCommandWrapper.java	77.77%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #11047      +/-   ##
============================================
- Coverage     16.57%   16.57%   -0.01%     
- Complexity    13870    13967      +97     
============================================
  Files          5719     5743      +24     
  Lines        507200   510717    +3517     
  Branches      61574    62097     +523     
============================================
+ Hits          84093    84667     +574     
- Misses       413688   416580    +2892     
- Partials       9419     9470      +51

Flag	Coverage Δ
uitests	`3.90% <ø> (-0.07%)`	⬇️
unittests	`17.47% <25.93%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull Request Overview

This PR enhances the handling of ScaleIO MDM and host SDC connections by introducing new configuration keys for timeouts, validation, and blocking behavior along with updating the underlying command execution and logging mechanisms.

Introduced new configuration keys, including MdmsChangeApplyTimeout, ValidateMdmsOnConnect, and BlockSdcUnprepareIfRestartNeededAndVolumesAreAttached.
Updated MDM add/remove logic to use varargs and improved command execution via Script.executeCommand.
Adjusted test cases and adapter logic to account for the new ScaleIO configurations.

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
utils/src/main/java/com/cloud/utils/script/Script.java	Added a new executeCommand(String) method for command execution including stdout/stderr handling.
plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/util/ScaleIOUtil.java	Refactored MDM add/remove methods and updated command templates, patterns, and file read operations.
plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/manager/ScaleIOSDCManagerImpl.java	Updated configuration details sent to hosts by including new timeout settings.
plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/manager/ScaleIOSDCManager.java	Introduced new config key definitions and updated the getConfigKeys() return values.
plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/ScaleIOPrimaryDataStoreLifeCycle.java	Injected the new configuration details during maintain and cancellation procedures.
plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/client/ScaleIOGatewayClient.java	Added a short documentation comment for STORAGE_POOL_MDMS.
plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/storage/ScaleIOStorageAdaptorTest.java	Updated test mocks to reflect changes in command executions for MDM removal.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/ScaleIOStorageAdaptor.java	Introduced validation of MDM state and timeout application after MDM changes.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtModifyStoragePoolCommandWrapper.java	Wrapped storage pool creation in a try/catch block to better handle CloudRuntimeException.

Comments suppressed due to low confidence (1)

plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/util/ScaleIOUtil.java:315

[nitpick] The ordering of stdout and stderr here is reversed compared to other usages of Script.executeCommand; consider using a consistent ordering (stdout as first, stderr as second) to avoid confusion.

String stdErr = result.first(); String stdOut = result.second();

utils/src/main/java/com/cloud/utils/script/Script.java

blueorangutan · 2025-06-17T08:01:11Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 13811

sureshanaparti · 2025-06-17T08:13:05Z

@blueorangutan package

blueorangutan · 2025-06-17T08:14:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-17T09:38:10Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13812

harikrishna-patnala

code LGTM

sureshanaparti · 2025-06-19T06:12:37Z

@blueorangutan test

blueorangutan · 2025-06-19T06:14:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-06-19T06:20:24Z

[SF] Trillian Build Failed (tid-13562)

harikrishna-patnala · 2025-06-23T06:21:07Z

@blueorangutan test

blueorangutan · 2025-06-23T06:22:03Z

@harikrishna-patnala a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-06-24T02:36:46Z

[SF] Trillian test result (tid-13586)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 69826 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11047-t13586-kvm-ol8.zip
Smoke tests completed. 136 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestAccounts>:setup	`Error`	0.00	test_accounts.py
ContextSuite context=TestAddVmToSubDomain>:setup	`Error`	0.00	test_accounts.py
test_DeleteDomain	`Error`	15.92	test_accounts.py
test_forceDeleteDomain	`Failure`	15.66	test_accounts.py
ContextSuite context=TestRemoveUserFromAccount>:setup	`Error`	16.38	test_accounts.py
ContextSuite context=TestTemplateHierarchy>:setup	`Error`	1537.55	test_accounts.py
ContextSuite context=TestDeployVmWithAffinityGroup>:setup	`Error`	0.00	test_affinity_groups_projects.py
ContextSuite context=TestAnnotations>:setup	`Error`	0.00	test_annotations.py
ContextSuite context=TestAsyncJob>:setup	`Error`	0.00	test_async_job.py
ContextSuite context=TestClusterDRS>:setup	`Error`	0.00	test_cluster_drs.py

…h SDC id/guid and some code improvements

...leio/src/main/java/org/apache/cloudstack/storage/datastore/provider/ScaleIOHostListener.java

sureshanaparti · 2025-06-25T06:41:19Z

@blueorangutan package

blueorangutan · 2025-06-25T06:42:07Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-25T08:04:14Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13907

rohityadavcloud · 2025-06-26T07:52:12Z

@sureshanaparti as these are general bugfix/improvements for powerflex storage, should these be also applicable for 4.20 branch (esp these changes are scaleio/powerflex storage plugin related with no DB changes)?

mprokopchuk and others added 10 commits June 17, 2025 11:25

Add feature flag to enable/disable blocking unprepare ScaleIO storage…

3cf8eb1

… pool when SDC client restart required and there are volumes attached to the Host

Update plugins/storage/volume/scaleio/src/main/java/org/apache/clouds…

31518c2

…tack/storage/datastore/manager/ScaleIOSDCManager.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

Update plugins/storage/volume/scaleio/src/main/java/org/apache/clouds…

aa982eb

…tack/storage/datastore/manager/ScaleIOSDCManager.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

Update plugins/storage/volume/scaleio/src/main/java/org/apache/clouds…

c4251de

…tack/storage/datastore/util/ScaleIOUtil.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

- Addressed code review comments: Added checking mdm removal response…

2a78099

… validation

Updated scope to Zone for the new powerflex settings

fc9404e

Get powerflex storage config settings from zone, and some fixes

1d504ef

Removed --file parameter used with --query_mdms, --query_vols, --quer…

1bc40eb

…y_guid, --rescan in drv_cfg cmd as it is not supported. --file parameter is supported with --add_mdm, --mod_mdm_ip, --remove_mdm, --set_guid, --set_mdm_password, --reset_mdm_password

rebase fixes

2be6c48

boring-cyborg bot added component:kvm component:primary-storage component:storage labels Jun 17, 2025

typo fix

661082d

DaanHoogland added this to the 4.21.0 milestone Jun 17, 2025

sureshanaparti requested review from harikrishna-patnala and Copilot June 17, 2025 07:55

Copilot AI reviewed Jun 17, 2025

View reviewed changes

utils/src/main/java/com/cloud/utils/script/Script.java Outdated Show resolved Hide resolved

review fix

0ae2156

harikrishna-patnala approved these changes Jun 19, 2025

View reviewed changes

Added wait time after SDC service start/restart/stop, retries to fetc…

da47703

…h SDC id/guid and some code improvements

sureshanaparti commented Jun 25, 2025

View reviewed changes

...leio/src/main/java/org/apache/cloudstack/storage/datastore/provider/ScaleIOHostListener.java Outdated Show resolved Hide resolved

sureshanaparti commented Jun 25, 2025

View reviewed changes

...leio/src/main/java/org/apache/cloudstack/storage/datastore/provider/ScaleIOHostListener.java Outdated Show resolved Hide resolved

Apply suggestions from code review

38db39e

PowerFlex/ScaleIO MDM and host SDC connection enhancements #11047

Are you sure you want to change the base?

PowerFlex/ScaleIO MDM and host SDC connection enhancements #11047

Uh oh!

Conversation

sureshanaparti commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

codecov bot commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

sureshanaparti commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented Jun 19, 2025

Uh oh!

blueorangutan commented Jun 19, 2025

Uh oh!

blueorangutan commented Jun 19, 2025

Uh oh!

harikrishna-patnala commented Jun 23, 2025

Uh oh!

blueorangutan commented Jun 23, 2025

Uh oh!

blueorangutan commented Jun 24, 2025

Uh oh!

Uh oh!

Uh oh!

sureshanaparti commented Jun 25, 2025

Uh oh!

blueorangutan commented Jun 25, 2025

Uh oh!

blueorangutan commented Jun 25, 2025

Uh oh!

rohityadavcloud commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

sureshanaparti commented Jun 17, 2025 •

edited

Loading

codecov bot commented Jun 17, 2025 •

edited

Loading

rohityadavcloud commented Jun 26, 2025 •

edited

Loading