[Test] L2 & L3 Test Case Stratification Design for Omni Model #1272

yenuo26 · 2026-02-09T02:57:51Z

PLEASE FILL IN THE PR DESCRIPTION HERE ENSURING ALL CHECKLIST ITEMS (AT THE BOTTOM) HAVE BEEN CONSIDERED.

Purpose

L2 & L3 Test Case Stratification Design for Omni Model: refer to #1218
Related documentation can be found：#1167

The main changes are as follows:
1.Added test-merge.yaml to manage merge-level test suites in the future.
2.Standardized the existing Omni3 online use cases and integrated both L2 and L3 level execution logic into a single script, differentiated during execution via the --run-level parameter.
3.Removed the default configuration test scenarios from the original online use cases, retaining only the async_chunk scenario. Default configurations will be covered by offline use cases.
4.Removed the test_build_and_log_summary test case (a new UT case covering this logic will be submitted in #891) and migrated test_async_omni.py to the tests/engine directory.

Test Plan

1.run offline case
/workspace/.venv/bin/python -m pytest -sv tests/e2e/offline_inference/test_qwen3_omni.py --html=report.html --self-contained-html
2.run online case
L2
/workspace/.venv/bin/python -m pytest -sv tests/e2e/online_serving/test_qwen3_omni.py -m core_model --run-level="core_model" --html=report.html --self-contained-html
/workspace/.venv/bin/python -m pytest -sv tests/e2e/online_serving/test_qwen3_omni.py --html=report.html --self-contained-html
L3
/workspace/.venv/bin/python -m pytest -sv tests/e2e/online_serving/test_qwen3_omni.py -m advanced_model --run-level="advanced_model" --html=report.html --self-contained-html
3.run abort case
/workspace/vllm-omni# /workspace/.venv/bin/python -m pytest -sv tests/engine/test_abort.py --html=report.html --self-contained-html

Test Result

1.offline case

2.online case
L2

L3

3.abort case

CI Result

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
(Optional) Release notes update. If your change is user facing, please update the release notes draft.

BEFORE SUBMITTING, PLEASE READ https://github.com/vllm-project/vllm-omni/blob/main/CONTRIBUTING.md (anything written below this line will be removed by GitHub Actions)

Signed-off-by: wangyu31577 <[email protected]>

…ci-qwen3

Signed-off-by: wangyu31577 <[email protected]>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 66a1f1ef67

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

tests/conftest.py

hsliuustc0106 · 2026-02-09T03:15:08Z

.buildkite/test-merge.yml

+                  path: /mnt/hf-cache
+                  type: DirectoryOrCreate
+
+  # - label: "Bagel Text2Img Model Test with H100"


will this be included in PR-merge?

Yes, I will contact the use case author to see how this use case can be split.

hsliuustc0106 · 2026-02-09T03:15:41Z

tests/e2e/offline_inference/test_qwen2_5_omni.py

 from vllm.envs import VLLM_USE_MODELSCOPE
 from vllm.multimodal.image import convert_image_mode

+from tests.conftest import OmniRunner


why move it here?

Because the original conftest only had the vllmrunner class, it seemed unnecessary to keep it as a separate file. Moreover, after merging it into the unified conftest, the functions for validating online use cases have been reused.

hsliuustc0106 · 2026-02-09T03:18:08Z

tests/engine/test_async_omni_engine_abort.py

@@ -8,13 +8,13 @@
 from vllm import SamplingParams


change file name to test_async_omni_engine_abort

already rename

Signed-off-by: yenuo26 <[email protected]>

tjtanaa · 2026-02-09T06:30:55Z

.buildkite/pipeline.yml

+       - export VLLM_TEST_CLEAN_GPU_MEMORY="1"
+       - pytest -s -v tests/e2e/offline_inference/test_qwen3_omni.py
+       - pytest -s -v tests/e2e/online_serving/test_qwen3_omni.py -m "core_model" --run-level "core_model"
+       - pytest -s -v tests/engine/test_abort.py


This line needs to change as well?

test_abort.py to test_async_omni_engine_abort.py

yes, you're right, It has been modified.

…ons to use the new async engine abort test. Signed-off-by: yenuo26 <[email protected]>

yenuo26 · 2026-02-09T10:24:11Z

@hsliuustc0106 @david6666666 please help add ready label

Signed-off-by: yenuo26 <[email protected]>

…ci-qwen3 Signed-off-by: yenuo26 <[email protected]>

hsliuustc0106 · 2026-02-09T11:44:21Z

fix precommit & resolve conflicts

Signed-off-by: yenuo26 <[email protected]>

…ci-qwen3

Signed-off-by: yenuo26 <[email protected]>

yenuo26 · 2026-02-09T12:46:38Z

fix precommit & resolve conflicts

fixed

…fline inference tests. Updated pytest command to focus on specific tests and removed unnecessary import statements. Signed-off-by: yenuo26 <[email protected]>

…orker_type', and reduce max model length and batched tokens to 25000 for improved performance. Signed-off-by: yenuo26 <[email protected]>

…rt test, adjust synthetic video/image dimensions, and reduce max model length and batched tokens for Qwen2_5 Omni CI settings. Signed-off-by: yenuo26 <[email protected]>

Signed-off-by: wangyu <[email protected]>

Signed-off-by: yenuo26 <[email protected]>

This commit introduces a `kill_process_tree` function in `tests/conftest.py` to handle the termination of a process and its children, ensuring all processes are properly killed and verified. The existing `_kill_process_tree` method in the `OmniServer` class has been replaced with this new function for improved clarity and reusability. Additionally, the function is now utilized in the context manager exit methods of both `OmniServer` and `OmniRunner` classes to ensure proper cleanup of resources. Signed-off-by: yenuo26 <[email protected]>

This commit moves the `kill_process_tree` function into the `OmniServer` class as a private method `_kill_process_tree`, enhancing encapsulation. The method is now used in the context manager exit to ensure proper cleanup of resources. The previous standalone function has been removed to streamline the code. Signed-off-by: yenuo26 <[email protected]>

Signed-off-by: yenuo26 <[email protected]>

This commit adds a new private method `_cleanup_process` to the `OmniRunner` class, which iterates through running processes to terminate any related to "vllm" or "core". This method is called during the context manager exit to ensure proper resource cleanup. Signed-off-by: yenuo26 <[email protected]>

This commit modifies the `_cleanup_process` method in the `OmniRunner` class to remove the "vllm" keyword from the process termination logic, focusing solely on processes related to "core". This change streamlines the cleanup process during context manager exit. Signed-off-by: yenuo26 <[email protected]>

This commit modifies the `_cleanup_process` method in the `OmniRunner` class to change the process keyword from "core" to "enginecore". This adjustment refines the process filtering during cleanup operations. Signed-off-by: yenuo26 <[email protected]>

Signed-off-by: yenuo26 <[email protected]>

Signed-off-by: wangyu <[email protected]>

Signed-off-by: Hongsheng Liu <[email protected]>

hsliuustc0106 · 2026-02-11T12:09:58Z

@gcanlin @zhenwei-intel PTAL

gcanlin · 2026-02-11T12:20:59Z

tests/e2e/stage_configs/rocm/qwen2_5_omni_ci.yaml

@@ -13,8 +13,8 @@ stage_args:
      model_arch: Qwen2_5OmniForConditionalGeneration
      worker_type: ar
      scheduler_cls: vllm_omni.core.sched.omni_ar_scheduler.OmniARScheduler
-      max_model_len: 896
-      max_num_batched_tokens: 896
+      max_model_len: 2400


Why not also move rocm directory into tests/e2e/stage_configs/? we'd like to create the npu directory here as well.

i will move it

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Signed-off-by: wangyu31577 <[email protected]>

…ci-qwen3

hsliuustc0106 · 2026-02-11T14:08:18Z

.buildkite/test-merge.yml

+    agents:
+      queue: "cpu_queue_premerge"
+
+  # - label: "Test on NPU"


rm these comments please

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Signed-off-by: wangyu31577 <[email protected]>

wangyu31577 and others added 7 commits February 5, 2026 19:50

split case

55b8128

Signed-off-by: wangyu31577 <[email protected]>

Merge branch 'vllm-project:main' into ci-qwen3

66851f7

split case

e310649

Signed-off-by: wangyu31577 <[email protected]>

Merge branch 'ci-qwen3' of https://github.com/yenuo26/vllm-omni into …

4c16595

…ci-qwen3

split case

f0e9aa2

Signed-off-by: wangyu31577 <[email protected]>

add L3 case

5d0f0a4

Signed-off-by: wangyu31577 <[email protected]>

Merge branch 'vllm-project:main' into ci-qwen3

66a1f1e

yenuo26 requested a review from hsliuustc0106 as a code owner February 9, 2026 02:57

chatgpt-codex-connector bot reviewed Feb 9, 2026

View reviewed changes

tests/conftest.py Show resolved Hide resolved

hsliuustc0106 reviewed Feb 9, 2026

View reviewed changes

This was referenced Feb 9, 2026

[RFC]: L2 & L3 Test Case Stratification Design for Omni Model JiusiServe/vllm-omni#99

Open

[RFC]: vllm-omni CI/CD plan #400

Open

fix copilot

4563ebe

Signed-off-by: yenuo26 <[email protected]>

yenuo26 force-pushed the ci-qwen3 branch from 79c6359 to 4563ebe Compare February 9, 2026 04:53

tjtanaa reviewed Feb 9, 2026

View reviewed changes

Update test commands in Buildkite pipeline and test merge configurati…

c47f8ab

…ons to use the new async engine abort test. Signed-off-by: yenuo26 <[email protected]>

yenuo26 mentioned this pull request Feb 9, 2026

[CI] Reduce the time for Diffusion Sequence Parallelism Test #1283

Merged

5 tasks

yenuo26 and others added 3 commits February 9, 2026 19:07

fix conflicts

8551744

Signed-off-by: yenuo26 <[email protected]>

Merge branch 'ci-qwen3' of https://github.com/yenuo26/vllm-omni into …

0464136

…ci-qwen3 Signed-off-by: yenuo26 <[email protected]>

Merge branch 'main' into ci-qwen3

9a53e7f

yenuo26 added 4 commits February 9, 2026 20:31

Merge remote-tracking branch 'upstream/main' into ci-qwen3

433a2d1

Signed-off-by: yenuo26 <[email protected]>

fix conflicts

d4df3cd

Signed-off-by: yenuo26 <[email protected]>

Merge branch 'ci-qwen3' of https://github.com/yenuo26/vllm-omni into …

45462c3

…ci-qwen3

fix conflicts

e34748a

Signed-off-by: yenuo26 <[email protected]>

yenuo26 force-pushed the ci-qwen3 branch from 9312fa3 to e34748a Compare February 9, 2026 12:45

hsliuustc0106 added the ready label to trigger buildkite CI label Feb 9, 2026

Refactor test commands in CI configuration and clean up imports in of…

a52af4f

…fline inference tests. Updated pytest command to focus on specific tests and removed unnecessary import statements. Signed-off-by: yenuo26 <[email protected]>

yenuo26 and others added 16 commits February 10, 2026 16:59

Update Qwen2_5 Omni CI configuration: change worker class names to 'w…

35354c0

…orker_type', and reduce max model length and batched tokens to 25000 for improved performance. Signed-off-by: yenuo26 <[email protected]>

Update CI configuration and tests: replace async test with engine abo…

ed4c200

…rt test, adjust synthetic video/image dimensions, and reduce max model length and batched tokens for Qwen2_5 Omni CI settings. Signed-off-by: yenuo26 <[email protected]>

Merge branch 'main' into ci-qwen3

50f173b

Signed-off-by: wangyu <[email protected]>

fix confllict

08e2939

Signed-off-by: yenuo26 <[email protected]>

Merge branch 'main' into ci-qwen3

e0e1100

add log

bbcad34

Signed-off-by: yenuo26 <[email protected]>

add log

64f5d07

Signed-off-by: yenuo26 <[email protected]>

add log

83e2a15

Signed-off-by: yenuo26 <[email protected]>

retry CI

a08f2d4

Signed-off-by: yenuo26 <[email protected]>

Merge branch 'main' into ci-qwen3

8c34689

Signed-off-by: wangyu <[email protected]>

Merge branch 'main' into ci-qwen3

c29c420

Signed-off-by: Hongsheng Liu <[email protected]>

hsliuustc0106 requested a review from Copilot February 11, 2026 12:09

gcanlin reviewed Feb 11, 2026

View reviewed changes

Copilot AI reviewed Feb 11, 2026

View reviewed changes

wangyu31577 and others added 4 commits February 11, 2026 20:42

move rocm stage configs

b1720ef

Signed-off-by: wangyu31577 <[email protected]>

Merge branch 'main' into ci-qwen3

81103ae

retry ci

6aaa415

Signed-off-by: wangyu31577 <[email protected]>

Merge branch 'ci-qwen3' of https://github.com/yenuo26/vllm-omni into …

ca072e3

…ci-qwen3

hsliuustc0106 requested a review from Copilot February 11, 2026 14:06

hsliuustc0106 reviewed Feb 11, 2026

View reviewed changes

Copilot AI reviewed Feb 11, 2026

View reviewed changes

retry ci

fc92aad

Signed-off-by: wangyu31577 <[email protected]>

Copilot started reviewing on behalf of hsliuustc0106 February 11, 2026 17:46 View session

Merge branch 'main' into ci-qwen3

a3fcb6f

[Test] L2 & L3 Test Case Stratification Design for Omni Model #1272

Are you sure you want to change the base?

[Test] L2 & L3 Test Case Stratification Design for Omni Model #1272

Uh oh!

Conversation

yenuo26 commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

CI Result

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yenuo26 commented Feb 9, 2026

Uh oh!

hsliuustc0106 commented Feb 9, 2026

Uh oh!

yenuo26 commented Feb 9, 2026

Uh oh!

hsliuustc0106 commented Feb 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yenuo26 commented Feb 9, 2026 •

edited

Loading