[RFC]: vllm-omni CI/CD plan

[yenuo26]

Motivation.

Build E2E CI for vllm-omni to strengthen quality protection.Currently, the CI pipeline needs to be expanded to cover the latest omni-modal and diffusion-based models. This update ensures robust validation for both online (real-time inference) and offline (batch/development) scenarios.

Proposed Change.

This testing system aims to build a complete, efficient, and well-structured quality assurance framework for the development, integration, and release of model services. It draws on the concept of the test pyramid from modern software engineering, progressively expanding testing activities from basic code logic verification to complex end-to-end (E2E) functionality, performance, accuracy, and even long-term stability validation.

Tiered testing structure:

Level	Scope & Focus	Time Cost	Test Dir	Doc	Frequency	Hardware
Common	Contribution Guideline & PR checklist	/	/	docs/contributing/ci/README.md .github/PULL_REQUEST_TEMPLATE.md docs/contributing/ci/tests_style.md	/	/
	CI Failure Description	/	/	docs/contributing/ci/failures.md	/	/
L1 (Unit & Logic)	Unit tests for components like entrypoints, models	<15min	/tests/{component_name}/test_xxx	docs/contributing/ci/CI_5levels.md	PR with ready label (also can run locally)	CPU
L2 (E2E across models & GPU-required UT)	Online & Offline (basic deployment scenarios): dummy, normal inference function (output format, stream), some instance startup UT		/tests/e2e/online_serving/test_{model_name}.py /tests/e2e/offline_inference/test_{model_name}.py	docs/contributing/ci/CI_5levels.md Section 1 L1&L2: Purpose, Test Content, Directory Location, Example	PR with ready label	GPU
L3 (Important Perf & Integration & Accuracy)	Online & Offline (multiple deployment scenarios): real model, normal inference function, normal accuracy	<30min	/tests/e2e/online_serving/test_{model_name}_expansion.py /tests/e2e/offline_inference/test_{model_name}_expansion.py	docs/contributing/ci/CI_5levels.md Section 2 L3: Purpose, Test Content, Directory Location, Example	PR Merged (Also run L1&L2 Tests)	GPU
L4 (Perf & Integration & Accuracy)	Online & Offline: full functional scenarios + performance test + doc test	<3 hour	Full Function: /tests/e2e/online_serving/test_{model_name}_expansion.py /tests/e2e/offline_inference/test_{model_name}_expansion.py Performance: /tests/e2e/perf/nightly.json Doc Test: tests/example/online_serving/test_{model_name}.py tests/example/offline_inference/test_{model_name}.py	docs/contributing/ci/CI_5levels.md Section 3 L4: Purpose, Test Content, Directory Location, Example	Nightly	GPU
L5 (Stability & Reliability)	Online & Offline: long-term stability test + reliability test	Depends on reality	Stability: tests/e2e/stability/weekly.json Reliability: tests/e2e/reliability/test_{model_name}.py	docs/contributing/ci/CI_5levels.md Section 4 L5: Purpose, Test Content, Directory Location, Example	Weekly / Days before Release	GPU

Detailed Design for Each Level

Common Specifications

Before entering specific testing levels, the project establishes two common specifications aimed at standardizing the development process and quickly locating issues.

1. PR Checklist (Tests Style): This template defines the self-check items that must be completed before submitting a code review (Pull Request). It ensures that each code change meets basic requirements such as code style, dependency updates, and documentation synchronization before entering the automated testing pipeline, serving as the first manual line of defense for quality assurance.
2. CI Failure Explanation (CI Failures): This document archives and explains common failure patterns in the Continuous Integration (CI) pipeline, error log interpretation, and preliminary troubleshooting steps. It helps developers and testers quickly diagnose the causes of automated test failures, improving problem-solving efficiency.

L1 & L2 Level Testing - Unit Testing and Basic End-to-End Verification

1.1 Testing Purpose

L1 and L2 level testing form the foundation of the quality assurance system. L1 level testing focuses on verifying the internal logic correctness of code units (e.g., functions, classes), ensuring each independent component behaves as designed.

L2 level testing builds upon L1 by introducing GPU resources and verifying that the end-to-end (E2E) process of the model in basic deployment scenarios is smooth. For example, it uses dummy models to confirm that core interfaces like the inference pipeline, output format, and streaming response work properly. The common goal of these two levels is to provide developers with rapid feedback, discovering and fixing issues early in the development cycle .

1.2 Testing Content and Scope

• L1 (Unit & Logic Testing):
• • Scope: Tests internal functions and methods of core components such as entrypoints, models.
  • Focus: Branch coverage, exception handling, algorithm logic correctness. Does not involve external dependencies or the complete service stack.
  • Time Cost: Execution time is controlled within 15 minutes to ensure fast feedback.
• L2 (Basic End-to-End Testing):
• • Scope: Covers two basic deployment scenarios: online (serving) and offline (inference).
  • Focus: Uses dummy models or lightweight real models to verify that the entire chain from request input to result output works normally, including output data structure, streaming (stream) support, etc. Also includes some unit tests that require launching independent service instances.
  • Characteristic: Requires GPU resources to perform model computations.

1.3 Test Directory and Execution Files

A clear directory structure is key to managing test cases efficiently.

• L1 Test Directory: /tests/{component_name}/test_xxx.py
• • Here, {component_name} corresponds to modules in the source code, such as distributed, entrypoints, etc., and test_xxx.py is the specific test file.
• L2 Test Directory:
• • Online Serving: /tests/e2e/online_serving/test_{model_name}.py
  • Offline Inference: /tests/e2e/offline_inference/test_{model_name}.py

L3 Level Testing - Core Integration, Performance, and Accuracy Verification

2.1 Testing Purpose

L3 level testing executes after code is merged into the main branch. Its core purpose is to verify the integration effect, key performance indicators, and output accuracy of real models in multiple deployment scenarios

. It acts as the "quality gatekeeper" for the main branch, ensuring that no merge breaks the core capabilities of the model service. Testing needs to provide clear conclusions within a relatively short time (<30min), balancing test depth with feedback speed.

2.2 Testing Content and Scope

• Deployment Scenarios: Covers richer online and offline deployment configurations, which may include different hardware configurations, batch sizes, concurrency levels, etc.
• Core Verification:
• 1. Inference Functionality: Ensures real models can perform forward computation normally and return results.
  2. Accuracy Compliance: Verifies that the model's evaluation metrics (e.g., accuracy) meet the expected baseline, preventing code changes from introducing accuracy issues.
  3. Important Performance: Verifies whether performance (e.g., P99 latency, throughput) in core scenarios meets preset thresholds.

2.3 Test Directory and Execution Files

• Functional Testing:
• • Online Serving: /tests/e2e/online_serving/test_{model_name}_expansion.py
  • Offline Inference: /tests/e2e/offline_inference/test_{model_name}_expansion.py
  • (Note: _expansion.py likely means it contains more comprehensive scenario cases compared to L2 tests).

L4 Level Testing - Full Functionality, Performance, and Documentation Testing

3.1 Testing Purpose

L4 level testing is a comprehensive quality audit before a version release. It expands upon L3, executing full functional scenarios, conducting systematic performance stress tests, and simultaneously verifying the correctness of accompanying example documentation. Its purpose is to perform deep validation of the system during off-peak nighttime hours, providing quality trend reports for daytime development and data support for release decisions.

3.2 Testing Content and Scope

• Full Functionality Testing: Executes all test cases defined in test_{model_name}_expansion.py, covering all implemented features, positive flows, boundary conditions, and exception handling.
• Performance Testing: Uses the /tests/e2e/perf/nightly.json configuration file to drive performance testing tools for stress, load, and endurance tests, collecting metrics like throughput, response time, and resource utilization.
• Documentation Testing: Verifies whether the example code provided to users is runnable and its results match the description.

3.3 Test Directory and Execution Files

• Functional Testing: Same directories as L3.
• Performance Test Configuration: /tests/e2e/perf/nightly.json
• Documentation Example Tests:
• • tests/example/online_serving/test_{model_name}.py
  • tests/example/offline_inference/test_{model_name}.py

L5 Level Testing - Stability and Reliability Testing

4.1 Testing Purpose

L5 level testing focuses on the performance of model services under long-running and abnormal fault scenarios. It aims to uncover deep-seated issues that only manifest under sustained pressure or extreme conditions, such as memory leaks, resource contention, gradual performance degradation, and lack of fault tolerance mechanisms. This is the final, yet crucial, line of defense for ensuring service high availability and production environment robustness.

4.2 Testing Content and Scope

• Long-term Stability (Stability) Testing: Uses the tests/e2e/stability/weekly.json configuration to run the service under moderate load for an extended period (e.g., over 12 hours), monitoring whether metrics like memory/VRAM usage, response time, and throughput degrade over time, and whether the service process remains stable.
• Reliability Testing: Uses tests/e2e/reliability/test_{model_name}.py to actively simulate various fault and abnormal scenarios, such as: dependent service interruption, abnormal input data, network flicker, hardware resource preemption, etc., to verify the system's fault tolerance, self-healing, and graceful degradation capabilities.

4.3 Test Directory and Execution Files

• Stability Test Configuration: tests/e2e/stability/weekly.json
• Reliability Test Suite: tests/e2e/reliability/test_{model_name}.py

Detailed Implementation Roadmap & To-Do

Priority	Category	Task	Description
P0	Documentation	Five-Level CI Test Documentation	Update documentation [#1167]
P0	Build & Automation	Nightly Build Script Implementation	Create test-nightly build script[#867]
P0	Build & Test Organization	L2/L3 Test Case Refactoring	Configure test-merge build script and Split existing test cases into L2 and L3 levels [RFC: #1218] [PR: #1272]
P0	Test Capabilities	Performance Test Framework	Develop public framework for performance test [RFC: #1313 ] [PR: #1321 ]
P1	Test Capabilities	Stability Test Framework	Develop public framework for stability test
P1	Test Capabilities	Add E2E, Example Test cases	Supplementing E2E (End-to-End) Example Use Cases for Various Models
P1	Test Capabilities	Add UT Test Cases	Supplementing Unit Tests for Various Components

---

Feedback Period.

No response

CC List.

No response

Any Other Things.

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[hsliuustc0106]

please add your detailed design in this doc: https://docs.google.com/document/d/18SQUBSMeq-2Zp6jgfmkRn7Xha7Ai5JmuoTWwaaGo6sk/edit?tab=t.0

[hsliuustc0106]

under this RFC, please only display the general design

[ZJY0516]

cc @congw729

[ZJY0516]

I was wondering how you plan to monitor the generation quality of diffusion models.

[congw729]

Maybe you can list a table? This is easier to read.

[congw729]

I was wondering how you plan to monitor the generation quality of diffusion models.

Personally, I think it's hard to verify the quality, the same for the audio result.

[hsliuustc0106]

Maybe you can list a table? This is easier to read.

me too, very hard to understand the hierachy of the test plan, I think LLM can help you make it better to present your plan @yenuo26
In 2025, it's hard to believe we cannot explain our ideas in a very accurate, concise and readable description with the help of LLM

[hsliuustc0106]

here is an optimized version from gemini @yenuo26 , please modify accordingly:

Motivation

Build E2E CI for vllm-omni to strengthen quality protection.
Currently, the CI pipeline needs to be expanded to cover the latest omni-modal and diffusion-based models. This update ensures robust validation for both online (real-time inference) and offline (batch/development) scenarios.

---

Proposed Changes

1. Test Matrix & Coverage

The following models and configurations will be integrated into the E2E test suite:

Category	Models	Input Modalities	Output Modalities
Omni-Modal	Qwen2.5-Omni, Qwen3-Omni	Audio, Video, Text, Mixed	Text + Audio, Text, Audio
Diffusion	Qwen-Image-Edit, Qwen-Image, Wan2.2 (T2V)	Text, Image	Image, Video

Test Parameters:

• Sampling: top_k, top_p, temperature, ignore_eos, max_tokens.
• Streaming: Validation for both stream: true and stream: false.
• Deployment: * Stage: 3
• Parallelism: TP (1, 2, 4), EP (1, 2, 4)

2. Validation Checkpoints

To ensure high-quality inference, the following metrics will be tracked:

1. Functional: Successful generation of text, audio, and video assets.
2. Performance: Request Latency (P99) and throughput.
3. Accuracy: Audio-to-Text consistency (verifying the generated audio matches the text output).

---

Additional Information

• Detailed Design: View Design Doc

Implementation Roadmap & To-Do

Phase 1: Model Integration

• Add support for Diffusion-based model architectures.
• Integrate Image-to-Image support (Qwen-Image-Edit).
• Integrate Text-to-Image support (Qwen-Image).
• Integrate Text-to-Video support (Wan2.2 Text-to-Video).

Phase 2: Test Suite Development

• Develop E2E test scripts for Qwen2.5-Omni & Qwen3-Omni.

  • Cover mixed modality inputs (Image/Audio/Video/Text).
  • Cover mixed modality outputs (Audio/Text).
  • Cover TP/EP parallelism configurations.

• Implement test cases for Diffusion generation workflows.

  • Configure online (streaming) and offline (batch) test runners.

Phase 3: Validation & Quality Gate

• Implement Audio-Text Consistency check (verify generated audio matches text).
• Add Latency Monitoring to the CI pipeline for performance regression tracking.
• Verify success/failure checkpoints for all modal outputs (Text, Audio, Image, Video).
• Integrate CI results into the PR status check workflow.

---