Introducing Storage Access Groups for better management for host and storage connections #10381

harikrishna-patnala · 2025-02-13T04:46:23Z

Description

Documentation PR apache/cloudstack-documentation#503

In CloudStack, when a primary storage is added at the Zone or Cluster scope, it is by default connected to all hosts within that scope. This default behavior can be refined using storage access groups, which allow operators to control and limit which hosts can access specific storage pools.

Storage access groups can be assigned to hosts, clusters, pods, zones, and primary storage pools. When a storage access group is set on a cluster/pod/zone, all hosts within that scope inherit the group. Connectivity between a host and a storage pool is then governed by whether they share the same storage access group.

A storage pool with a storage access group will connect only to hosts that have the same storage access group. A storage pool without a storage access group will connect to all hosts, including those with or without a storage access group.

Example:
Consider a CloudStack environment with 10 clusters, each with 5 hosts, totaling 50 hosts. When a zone-wide primary storage is added, it will by default connect to all 50 hosts. If the operator wants the storage to connect only to selected hosts in Cluster 1 and Cluster 2, they can assign a storage access group to:

```
The primary storage
```

The hosts in Cluster 1 and Cluster 2 or directly assign the storage access group to Cluster 1 and Cluster 2, so all their hosts inherit it.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

codecov · 2025-02-13T05:07:07Z

Codecov Report

Attention: Patch coverage is 25.54517% with 1434 lines in your changes missing coverage. Please review.

Project coverage is 16.52%. Comparing base (d5ba23c) to head (a90227d).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...n/java/com/cloud/resource/ResourceManagerImpl.java	55.79%	206 Missing and 42 partials ⚠️
.../storage/datastore/db/PrimaryDataStoreDaoImpl.java	5.34%	124 Missing ⚠️
...ain/java/com/cloud/storage/StorageManagerImpl.java	49.59%	90 Missing and 34 partials ⚠️
...ain/java/com/cloud/api/query/QueryManagerImpl.java	0.00%	123 Missing ⚠️
.../src/main/java/com/cloud/host/dao/HostDaoImpl.java	0.00%	61 Missing ⚠️
...orage/dao/StoragePoolAndAccessGroupMapDaoImpl.java	0.00%	51 Missing ⚠️
...mmand/admin/storage/ConfigureStorageAccessCmd.java	0.00%	44 Missing ⚠️
...stack/api/response/StorageAccessGroupResponse.java	0.00%	43 Missing ⚠️
...torage/motion/StorageSystemDataMotionStrategy.java	0.00%	40 Missing ⚠️
...om/cloud/api/query/dao/StoragePoolJoinDaoImpl.java	0.00%	29 Missing ⚠️
... and 59 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #10381      +/-   ##
============================================
+ Coverage     16.49%   16.52%   +0.02%     
- Complexity    13705    13790      +85     
============================================
  Files          5708     5716       +8     
  Lines        504083   506016    +1933     
  Branches      61101    61380     +279     
============================================
+ Hits          83139    83604     +465     
- Misses       411665   413052    +1387     
- Partials       9279     9360      +81

Flag	Coverage Δ
uitests	`3.97% <ø> (-0.03%)`	⬇️
unittests	`17.39% <25.54%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

harikrishna-patnala · 2025-02-13T06:38:31Z

@blueorangutan package

blueorangutan · 2025-02-13T06:40:04Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-02-13T07:55:10Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 12435

harikrishna-patnala · 2025-02-14T04:49:52Z

@blueorangutan package

blueorangutan · 2025-02-14T04:52:03Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-02-14T06:08:53Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12456

rohityadavcloud · 2025-02-14T06:37:07Z

@blueorangutan test

blueorangutan · 2025-02-14T06:38:03Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

harikrishna-patnala · 2025-02-14T08:17:01Z

@blueorangutan package

blueorangutan · 2025-02-14T08:18:04Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-02-14T09:40:56Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12459

harikrishna-patnala · 2025-02-14T10:56:15Z

@blueorangutan test

blueorangutan · 2025-02-14T10:58:03Z

@harikrishna-patnala a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-02-15T02:17:29Z

[SF] Trillian test result (tid-12407)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 52229 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10381-t12407-kvm-ol8.zip
Smoke tests completed. 139 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_11_isolated_network_with_dynamic_routed_mode	`Error`	2.27	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	1.37	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	1.37	test_ipv4_routing.py
test_06_purge_expunged_vm_background_task	`Failure`	389.95	test_purge_expunged_vms.py

rohityadavcloud · 2025-03-25T11:21:01Z

@blueorangutan package

blueorangutan · 2025-03-25T11:22:04Z

@rohityadavcloud a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-03-25T12:30:34Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 12884

rohityadavcloud · 2025-03-27T08:08:52Z

@blueorangutan test

blueorangutan · 2025-03-27T08:10:04Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

github-actions · 2025-03-27T17:36:07Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

blueorangutan · 2025-03-27T23:49:28Z

[SF] Trillian test result (tid-12828)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 53608 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10381-t12828-kvm-ol8.zip
Smoke tests completed. 141 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

harikrishna-patnala · 2025-05-16T09:47:17Z

@blueorangutan package

blueorangutan · 2025-05-16T09:48:04Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-16T11:00:48Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13411

github-actions · 2025-05-16T16:35:36Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

… connections

… list APIs

harikrishna-patnala · 2025-05-19T04:26:38Z

@blueorangutan package

blueorangutan · 2025-05-19T04:28:03Z

@harikrishna-patnala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-19T05:29:12Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13437

rohityadavcloud · 2025-05-20T06:18:01Z

@blueorangutan test

blueorangutan · 2025-05-20T06:20:03Z

@rohityadavcloud a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-05-21T00:11:03Z

[SF] Trillian test result (tid-13356)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 61322 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10381-t13356-kvm-ol8.zip
Smoke tests completed. 140 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_deployVMInSharedNetwork	`Failure`	71.36	test_network.py
ContextSuite context=TestSharedNetworkWithConfigDrive>:teardown	`Error`	72.47	test_network.py

boring-cyborg bot added the component:api label Feb 13, 2025

harikrishna-patnala added this to the 4.21.0 milestone Feb 13, 2025

apache deleted a comment from blueorangutan Feb 13, 2025

harikrishna-patnala requested a review from kiranchavala February 14, 2025 05:11

rohityadavcloud requested review from shwstppr, sureshanaparti and weizhouapache February 14, 2025 06:36

github-actions bot added the status:has-conflicts label Mar 27, 2025

github-actions bot removed the status:has-conflicts label May 16, 2025

github-actions bot added the status:has-conflicts label May 16, 2025

harikrishna-patnala added 14 commits May 19, 2025 09:37

Introducing Storage Access Groups to define the host and storage pool…

d230582

… connections

Added ListStorageAccessGroups API and storageaccessgroup parameter in…

c93fa0e

… list APIs

UI fix

39244ac

view fix

3c56d34

Added more unit tests

507f6fc

Fix duplicate storage access groups

fdd95fe

build fixes

acf8a33

Lint fix

7d8cede

Handle storage pool disconnect in a host

12fb59c

restrict storage access groups for local storage

dc4b173

duplicate imports

d658999

import issue

c48b755

Version changes

e6ba3be

Fix log message

a90227d

harikrishna-patnala force-pushed the StorageAccessGroups branch from d4e4746 to a90227d Compare May 19, 2025 04:26

github-actions bot removed the status:has-conflicts label May 19, 2025

harikrishna-patnala merged commit b17808b into apache:main May 19, 2025
23 of 26 checks passed

harikrishna-patnala deleted the StorageAccessGroups branch May 19, 2025 06:03

Introducing Storage Access Groups for better management for host and storage connections #10381

Introducing Storage Access Groups for better management for host and storage connections #10381

Uh oh!

Conversation

harikrishna-patnala commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

codecov bot commented Feb 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

harikrishna-patnala commented Feb 13, 2025

Uh oh!

blueorangutan commented Feb 13, 2025

Uh oh!

blueorangutan commented Feb 13, 2025

Uh oh!

harikrishna-patnala commented Feb 14, 2025

Uh oh!

blueorangutan commented Feb 14, 2025

Uh oh!

blueorangutan commented Feb 14, 2025

Uh oh!

rohityadavcloud commented Feb 14, 2025

Uh oh!

blueorangutan commented Feb 14, 2025

Uh oh!

harikrishna-patnala commented Feb 14, 2025

Uh oh!

blueorangutan commented Feb 14, 2025

Uh oh!

blueorangutan commented Feb 14, 2025

Uh oh!

harikrishna-patnala commented Feb 14, 2025

Uh oh!

blueorangutan commented Feb 14, 2025

Uh oh!

blueorangutan commented Feb 15, 2025

Uh oh!

rohityadavcloud commented Mar 25, 2025

Uh oh!

blueorangutan commented Mar 25, 2025

Uh oh!

blueorangutan commented Mar 25, 2025

Uh oh!

rohityadavcloud commented Mar 27, 2025

Uh oh!

blueorangutan commented Mar 27, 2025

Uh oh!

github-actions bot commented Mar 27, 2025

Uh oh!

blueorangutan commented Mar 27, 2025

Uh oh!

harikrishna-patnala commented May 16, 2025

Uh oh!

blueorangutan commented May 16, 2025

Uh oh!

blueorangutan commented May 16, 2025

Uh oh!

github-actions bot commented May 16, 2025

Uh oh!

harikrishna-patnala commented May 19, 2025

Uh oh!

blueorangutan commented May 19, 2025

Uh oh!

blueorangutan commented May 19, 2025

Uh oh!

Uh oh!

rohityadavcloud commented May 20, 2025

Uh oh!

blueorangutan commented May 20, 2025

Uh oh!

harikrishna-patnala commented Feb 13, 2025 •

edited

Loading

codecov bot commented Feb 13, 2025 •

edited

Loading