[Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM #10770

nvazquez · 2025-04-24T09:55:06Z

Description

This PR improves the listing of Vmware VMs for KVM migration tool to reduce the timeout observed while listing VMs on a datacenter. The improvement is performed in two stages:

First stage: Datacenter VMs listing -> use thin listing of VMs instead of obtaining all its disks and NICs (Invoke the listVmwareDcVms API setting the datacentername parameter only)
Second stage: When selecting a VM to import -> perform the full information retrieval of the selected VM including its disks and NICs information (Invoke the listVmwareDcVms API setting the datacentername, hostname and virtualmachinename parameter)

It was originally added by PR: #9925 but reverted as it caused unintended regression affecting the VM console access (#10634).

Fixes: #9782

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested against Vmware vCenter datacenter with 200+ VMs, listing times reduced significantly from 30 seconds to <1 second for the entire datacenter VMs

Same command time difference against the same vCenter: (time cmk list vmwaredcvms vcenter=XXXX datacentername=XXXX username=XXXX password=XXXX page=1 pagesize=10 filter=name)

Before:

real	0m35.755s
user	0m0.087s
sys	0m0.016s

After:

real	0m0.468s
user	0m0.076s
sys	0m0.013s

nvazquez · 2025-04-24T09:56:30Z

@blueorangutan package

blueorangutan · 2025-04-24T09:58:04Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-04-24T10:03:28Z

Codecov Report

Attention: Patch coverage is 0.86207% with 115 lines in your changes missing coverage. Please review.

Project coverage is 15.16%. Comparing base (55c8138) to head (ab29fd8).
Report is 11 commits behind head on 4.19.

Files with missing lines	Patch %	Lines
...in/java/com/cloud/hypervisor/vmware/mo/BaseMO.java	1.88%	52 Missing ⚠️
...d/hypervisor/vmware/manager/VmwareManagerImpl.java	0.00%	51 Missing ⚠️
...ack/api/command/admin/zone/ListVmwareDcVmsCmd.java	0.00%	9 Missing ⚠️
...a/com/cloud/hypervisor/vmware/mo/DatacenterMO.java	0.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               4.19   #10770    +/-   ##
==========================================
  Coverage     15.16%   15.16%            
- Complexity    11332    11335     +3     
==========================================
  Files          5412     5414     +2     
  Lines        475033   475197   +164     
  Branches      57963    57990    +27     
==========================================
+ Hits          72048    72073    +25     
- Misses       394930   395063   +133     
- Partials       8055     8061     +6

Flag	Coverage Δ
uitests	`4.28% <ø> (-0.02%)`	⬇️
unittests	`15.89% <0.86%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

DaanHoogland

clgtm

blueorangutan · 2025-04-24T11:19:05Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13166

nvazquez · 2025-04-24T11:48:03Z

@blueorangutan test ol8 vmware-70u3

blueorangutan · 2025-04-24T11:50:03Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

.../vmware/src/main/java/org/apache/cloudstack/api/command/admin/zone/ListVmwareDcHostsCmd.java

blueorangutan · 2025-04-25T10:37:55Z

[SF] Trillian test result (tid-13127)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 79615 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10770-t13127-vmware-70u3.zip
Smoke tests completed. 127 look OK, 6 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_role_account_acls	`Failure`	3.97	test_dynamicroles.py
test_role_account_acls_multiple_mgmt_servers	`Failure`	3.33	test_dynamicroles.py
test_01_events_resource	`Error`	355.18	test_events_resource.py
test_04_deploy_vm_for_other_user_and_test_vm_operations	`Error`	122.44	test_network_permissions.py
test_01_restore_vm	`Error`	3607.45	test_restore_vm.py
test_02_restore_vm_allocated_root	`Error`	10.30	test_restore_vm.py
test_12_destroy_cpvm	`Error`	6.36	test_ssvm.py
test_01_deploy_vm_on_specific_host	`Error`	19.06	test_vm_deployment_planner.py
test_02_deploy_vm_on_specific_cluster	`Error`	3607.06	test_vm_deployment_planner.py
test_03_deploy_vm_on_specific_pod	`Error`	2.57	test_vm_deployment_planner.py
test_04_deploy_vm_on_host_override_pod_and_cluster	`Error`	4.56	test_vm_deployment_planner.py
test_05_deploy_vm_on_cluster_override_pod	`Error`	15.80	test_vm_deployment_planner.py

…/api/command/admin/zone/ListVmwareDcHostsCmd.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

DaanHoogland · 2025-04-25T13:21:15Z

@blueorangutan package

blueorangutan · 2025-04-25T13:22:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-04-25T14:40:45Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13184

nvazquez · 2025-04-25T14:48:48Z

@blueorangutan test ol8 vmware-70u3

blueorangutan · 2025-04-25T14:50:04Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

blueorangutan · 2025-04-26T05:54:52Z

[SF] Trillian test result (tid-13159)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 51831 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10770-t13159-vmware-70u3.zip
Smoke tests completed. 130 look OK, 3 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_role_account_acls	`Failure`	3.55	test_dynamicroles.py
test_role_account_acls_multiple_mgmt_servers	`Failure`	3.54	test_dynamicroles.py
test_01_events_resource	`Error`	320.53	test_events_resource.py
test_01_events_resource	`Error`	320.54	test_events_resource.py
test_04_deploy_vm_for_other_user_and_test_vm_operations	`Error`	117.94	test_network_permissions.py

.../vmware/src/main/java/org/apache/cloudstack/api/command/admin/zone/ListVmwareDcHostsCmd.java

rohityadavcloud

Left one remark, otherwise LGTM - this needs testing

nvazquez · 2025-05-05T21:04:37Z

Hi @sureshanaparti @DaanHoogland @rohityadavcloud I could improve the listing reducing the time significantly by excluding the disks and NICs information on the initial listing and then obtaining them when the user selects the VM (I have updated the PR description to explain the new behavior). Can you please re-review?

nvazquez · 2025-05-05T21:04:44Z

@blueorangutan package

blueorangutan · 2025-05-05T21:06:04Z

@nvazquez a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-05-05T22:02:48Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13265

nvazquez · 2025-05-05T22:23:50Z

@blueorangutan test ol8 vmware-70u3

blueorangutan · 2025-05-05T22:26:03Z

@nvazquez a [SL] Trillian-Jenkins test job (ol8 mgmt + vmware-70u3) has been kicked to run smoke tests

DaanHoogland

clgg,

DaanHoogland · 2025-05-06T06:30:19Z

vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/BaseMO.java

+                powerState == VirtualMachinePowerState.POWERED_OFF ? UnmanagedInstanceTO.PowerState.PowerOff : UnmanagedInstanceTO.PowerState.PowerUnknown;
+    }
+
+    protected List<UnmanagedInstanceTO> convertVmsObjectContentsToUnmanagedInstances(List<ObjectContent> ocs, String keyword) throws Exception {


complexity of this method is a bit high, would be nice to modularise it a bit.

DaanHoogland · 2025-05-06T11:25:00Z

as this functionality was implemented in 4.19.2 (but with side effects) and thus would be a regression if not merged, I marked this as BLOCKER.

blueorangutan · 2025-05-06T12:55:44Z

[SF] Trillian test result (tid-13205)
Environment: vmware-70u3 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 49843 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr10770-t13205-vmware-70u3.zip
Smoke tests completed. 133 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

nvazquez added 3 commits April 24, 2025 06:49

Improve Vmware VMs retrieval

b770d27

Fix imports

8cac953

Fixes and improvements

16eab4d

nvazquez added type:bug component:vmware component:UI labels Apr 24, 2025

nvazquez added this to the 4.19.3 milestone Apr 24, 2025

nvazquez requested review from rohityadavcloud, kiranchavala, DaanHoogland, harikrishna-patnala, Pearl1594, sureshanaparti, vladimirpetrov, borisstoyanov and weizhouapache April 24, 2025 09:55

nvazquez linked an issue Apr 24, 2025 that may be closed by this pull request

[UI] Timeout listing Vmware Datacenter VMs for migration to KVM #9782

Open

nvazquez mentioned this pull request Apr 24, 2025

[UI] Timeout listing Vmware Datacenter VMs for migration to KVM #9782

Open

DaanHoogland approved these changes Apr 24, 2025

View reviewed changes

DaanHoogland added the status:needs-testing label Apr 24, 2025

Pearl1594 added this to ACS 4.20.1 Apr 24, 2025

Pearl1594 moved this to In Progress in ACS 4.20.1 Apr 24, 2025

sureshanaparti reviewed Apr 25, 2025

View reviewed changes

.../vmware/src/main/java/org/apache/cloudstack/api/command/admin/zone/ListVmwareDcHostsCmd.java Outdated Show resolved Hide resolved

Update plugins/hypervisors/vmware/src/main/java/org/apache/cloudstack…

c4a2224

…/api/command/admin/zone/ListVmwareDcHostsCmd.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

rohityadavcloud reviewed Apr 29, 2025

View reviewed changes

.../vmware/src/main/java/org/apache/cloudstack/api/command/admin/zone/ListVmwareDcHostsCmd.java Outdated Show resolved Hide resolved

rohityadavcloud approved these changes Apr 29, 2025

View reviewed changes

rohityadavcloud assigned vladimirpetrov and borisstoyanov May 5, 2025

Refactor to improve listing

a7cfdb6

nvazquez changed the title ~~[Vmware][UI] Fix Timeout listing Vmware Datacenter VMs for migration to KVM~~ [Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM May 5, 2025

nvazquez requested review from sureshanaparti, DaanHoogland and rohityadavcloud May 5, 2025 21:03

Remove commented line

ab29fd8

DaanHoogland reviewed May 6, 2025

View reviewed changes

DaanHoogland added Severity:Critical Critical bug Severity:BLOCKER and removed Severity:Critical Critical bug labels May 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM #10770

[Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM #10770

nvazquez commented Apr 24, 2025 •

edited

Loading

nvazquez commented Apr 24, 2025

blueorangutan commented Apr 24, 2025

codecov bot commented Apr 24, 2025 •

edited

Loading

DaanHoogland left a comment

blueorangutan commented Apr 24, 2025

nvazquez commented Apr 24, 2025

blueorangutan commented Apr 24, 2025

blueorangutan commented Apr 25, 2025

DaanHoogland commented Apr 25, 2025

blueorangutan commented Apr 25, 2025

blueorangutan commented Apr 25, 2025

nvazquez commented Apr 25, 2025

blueorangutan commented Apr 25, 2025

blueorangutan commented Apr 26, 2025

rohityadavcloud left a comment

nvazquez commented May 5, 2025

nvazquez commented May 5, 2025

blueorangutan commented May 5, 2025

blueorangutan commented May 5, 2025

nvazquez commented May 5, 2025

blueorangutan commented May 5, 2025

DaanHoogland left a comment

DaanHoogland May 6, 2025

DaanHoogland commented May 6, 2025

blueorangutan commented May 6, 2025

[Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM #10770

Are you sure you want to change the base?

[Vmware] Improve listing of Vmware Datacenter VMs for migration to KVM #10770

Conversation

nvazquez commented Apr 24, 2025 • edited Loading

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

nvazquez commented Apr 24, 2025

blueorangutan commented Apr 24, 2025

codecov bot commented Apr 24, 2025 • edited Loading

Codecov Report

DaanHoogland left a comment

Choose a reason for hiding this comment

blueorangutan commented Apr 24, 2025

nvazquez commented Apr 24, 2025

blueorangutan commented Apr 24, 2025

blueorangutan commented Apr 25, 2025

DaanHoogland commented Apr 25, 2025

blueorangutan commented Apr 25, 2025

blueorangutan commented Apr 25, 2025

nvazquez commented Apr 25, 2025

blueorangutan commented Apr 25, 2025

blueorangutan commented Apr 26, 2025

rohityadavcloud left a comment

Choose a reason for hiding this comment

nvazquez commented May 5, 2025

nvazquez commented May 5, 2025

blueorangutan commented May 5, 2025

blueorangutan commented May 5, 2025

nvazquez commented May 5, 2025

blueorangutan commented May 5, 2025

DaanHoogland left a comment

Choose a reason for hiding this comment

DaanHoogland May 6, 2025

Choose a reason for hiding this comment

DaanHoogland commented May 6, 2025

blueorangutan commented May 6, 2025

nvazquez commented Apr 24, 2025 •

edited

Loading

codecov bot commented Apr 24, 2025 •

edited

Loading