Audio: add Qwen3-ASR adapter

[mohammadaaftabv]

Summary

Add Qwen3-ASR as a second implementation of Curator's shared ASRAdapter contract.

The PR keeps both adapters:

• QwenASRAdapter for Qwen/Qwen3-ASR-0.6B, using qwen-asr's vLLM-only Qwen3ASRModel.LLM path.
• The existing QwenOmniASRAdapter, Qwen3-Omni pipeline, dependency contract, tests, and vLLM behavior remain functionally unchanged.

There is no standalone Qwen-ASR runner. Manifest transcription uses the generic ASRStage adapter path.

Qwen3-ASR adapter

• Implements the shared lifecycle names and signatures used by Qwen-Omni: download_weights_on_node, load_model, unload_model, and transcribe_batch.
• Constructs Qwen3ASRModel.LLM with the same engine arguments as nkoluguri/integration-test: model, GPU-memory utilization, inference batch size, max new tokens, remote-code trust, eager execution, prefix caching, and xxhash.
• Contains no Transformers .from_pretrained model loading or inference implementation.
• Includes the same scoped transformers.check_model_inputs compatibility shim as the nkoluguri reference because qwen-asr 0.0.6 still invokes the decorator-factory form.
• Maps one Curator adapter batch to one qwen-asr transcribe call while preserving input order and skipped positions.
• Forwards revision to weight download and vLLM construction.
• Exposes a copied vllm_kwargs dictionary; adapter-owned engine arguments cannot be overridden through it.
• Requires exactly one integer GPU per ASR actor and validates integer/resource knobs.
• Cleans up after partial model-load failure and marks empty/whitespace/None output as skipped.
• Returns detected language through ASRResult.extras.

audio_cuda12 pins qwen-asr==0.0.6. Qwen-ASR installations compose it with Curator's shared vllm extra. The dependency intentionally does not use qwen-asr[vllm], whose vLLM 0.14.0 pin conflicts with Curator's Qwen-Omni-compatible vLLM 0.22.0 stack.

Generic adapter extras

ASRStage can persist arbitrary adapter metadata without introducing dynamic top-level columns:

• ASRResult.extras remains one generic adapter-owned dictionary.
• Set ASRStage.extras_key to write a shallow copy under one nested manifest field.
• extras_key=None is the default, preserving the existing Qwen-Omni output contract and existing ASRStage callers.
• Qwen-ASR configures extras_key: asr_extras, producing "asr_extras": {"detected_language": "English"}.
• Empty extras remove stale metadata for processed rows.
• Empty names and collisions with prediction, _skipme, or additional_notes columns are rejected.
• The nested dictionary is declared through outputs(), so executor/schema planning remains deterministic.

Shared stage, tests, and tutorial

• Adds tutorials/audio/qwen_asr/ using ManifestReader -> ResampleAudioStage -> ASRStage(QwenASRAdapter) -> ManifestWriterStage.
• Uses SoundFile to decode resampled PCM WAVs in the shared ASR stage.
• Uses Qwen's 30-code language order:
  zh, en, yue, ar, de, fr, es, pt, id, it, ko, ru, th, vi, ja, tr, hi, ms, nl, sv, da, fi, pl, cs, fil, fa, el, hu, mk, ro.
• Reuses tests/fixtures/audio/qwen_omni/audio_1_5s_16khz_mono.wav; this PR adds no duplicate WAV fixture.
• Keeps the GPU tests in tests/models/asr/test_qwen_asr.py under @pytest.mark.gpu and adds the audio_qwen_asr GPU test group.
• Uses tests/fixtures/audio/tagging/sample_input.jsonl as the documented two-file smoke input.
• Adds the tutorial-only vllm_kwargs.max_model_len=8192 override for 12 GB GPUs; the adapter's production/reference default is unchanged.
• Makes no change to benchmarking/README.md.

Shared validate_vllm_kwargs and merge_vllm_kwargs helpers now live in nemo_curator.utils.vllm_utils. Both Qwen adapters describe their actual caller-owned constructor kwargs to merge_vllm_kwargs, and the duplicate adapter-local _RESERVED_VLLM_KWARGS constants are removed. Protected-key collision checking now occurs once, inside the shared merge helper when load_model() assembles engine kwargs; the adapters no longer repeat validation in __post_init__. Qwen-Omni's protected-key behavior, engine arguments, and output contract remain unchanged.

Current-main integration

• PR head: 4c8ac6f6604ec71ea5919ee014e4137ee1d0e560
• Integrated main: 09ae28c41cfb7704d66eb4d6cbffc33fc6f05fe3
• History: three DCO-signed commits on current main

Validation

Current head

• 173 focused Qwen-ASR, Qwen-Omni, generic ASRStage, tutorial-contract, configuration, lazy-import, and shared-vLLM utility tests passed; 2 GPU tests were deselected.
• Exact Qwen-ASR and Qwen-Omni constructor-call tests pass after the shared kwargs merge refactor.
• Ruff formatting and lint checks passed.
• uv lock --check passed with the current-main-required uv 0.12.2.
• git diff --check passed.
• GPU-test-group coverage passed for all 46 GPU test files.
• GitHub DCO passes for all three signed commits.

Local GPU and tutorial smoke

The earlier head ddfb74cdc423dec0ffe64dae3e05fcf210b14617 passed:

• Real one-GPU vLLM smoke on an RTX 3080 Ti using qwen-asr 0.0.6, vLLM 0.22.0, and the existing Qwen-Omni WAV fixture.
• The documented Ray Data tutorial command on both bundled OPUS inputs: 2/2 nonempty transcripts, no skips, and asr_extras.detected_language == "English".

The current head has not repeated that GPU smoke after the main rebase and kwargs-assembly refactor. The exact constructor tests confirm that the refactor preserves the engine arguments used by both Qwen adapters.

Latest paired Qwen3-ASR parity

The latest paired run compared target commit ddfb74cdc423dec0ffe64dae3e05fcf210b14617 with nkoluguri/integration-test commit fff467c8d2e3632f9a96c96ec34cd0e6614503da, using the same frozen eight-row English cohort (569.486 seconds), one node, and one L40 per arm.

• Target run: 1dc99ea4-4b0f-438d-832d-5d902384453b
• Reference run: 0443d025-95c7-42f8-a68b-ee510c9ce0e1
• Both independently submitted arms and all tasks succeeded on their first attempt.
• 8/8 expected identities and nonempty predictions in each arm.
• 8/8 target asr_extras.detected_language values matched reference asr_language; all were English.
• No missing, unexpected, duplicate, skipped, malformed, empty, forbidden-payload, or published-WAV cases.
• Mean normalized token similarity: 0.9699208113126543 (gate: at least 0.95).
• Rows with normalized similarity at least 0.95: 7/8 (gate: at least 7/8).
• Normalized exact matches: 2/8; raw exact matches: 1/8.
• Lowest row similarity: 0.8516129032258064.

The target used qwen-asr 0.0.6 with Curator's vLLM 0.22.0 stack; the remote-exact reference used qwen-asr 0.0.6 with its pinned vLLM 0.14.0 stack. Timing is therefore a canary observation, not a performance comparison.

Since the parity-tested head, the branch was rebased onto current main and the host-side vLLM kwargs validation/assembly was centralized. The actual Qwen-ASR engine values, input contract, output mapping, and inference path are unchanged; live parity has not been rerun after this review refactor.

Diff

• 18 files changed
• 1,490 insertions / 32 deletions

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

The PR adds a Qwen3-ASR implementation of the shared ASR adapter and extends the generic stage to persist nested adapter metadata while retaining Qwen-Omni support.

• Adds worker-local Qwen3-ASR model loading, batched transcription, skipped-position preservation, and detected-language metadata.
• Adds optional ASRStage.extras_key output and SoundFile-based waveform decoding.
• Centralizes protected vLLM keyword validation and merging.
• Adds dependency configuration, tutorial assets, and focused CPU/GPU test coverage.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains in the available follow-up review scope.

No blocking failure remains.

Important Files Changed

Filename	Overview
nemo_curator/models/asr/qwen_asr.py	Adds the Qwen3-ASR adapter with validated model construction, ordered batch-result mapping, cleanup, and language metadata.
nemo_curator/stages/audio/inference/asr/stage.py	Adds optional nested extras persistence and switches file-backed audio decoding to SoundFile while preserving normalized waveform preparation.
nemo_curator/models/asr/qwen_omni.py	Refactors Qwen-Omni engine-argument assembly to use the shared protected-key merge helper.
nemo_curator/utils/vllm_utils.py	Adds reusable validation and deep-copy merging for caller-owned and user-supplied vLLM arguments.
pyproject.toml	Adds the pinned qwen-asr package to the CUDA audio dependency set while retaining Curator's shared vLLM extra.
tutorials/audio/qwen_asr/pipeline.yaml	Defines the manifest-to-resample-to-Qwen-ASR-to-writer tutorial pipeline with nested ASR metadata output.
uv.lock	Updates resolved dependencies for the new qwen-asr installation contract.

Sequence Diagram

sequenceDiagram
  participant Manifest as ManifestReader
  participant Resample as ResampleAudioStage
  participant Stage as ASRStage
  participant Adapter as QwenASRAdapter
  participant Model as Qwen3ASRModel.LLM
  participant Writer as ManifestWriterStage
  Manifest->>Resample: AudioTask with source path
  Resample->>Stage: Resampled PCM WAV path
  Stage->>Stage: Decode, mono-convert, resample
  Stage->>Adapter: waveform, sample rate, language
  Adapter->>Model: transcribe(audio, language)
  Model-->>Adapter: text and detected language
  Adapter-->>Stage: Ordered ASRResult list
  Stage->>Stage: Write prediction, skip state, and asr_extras
  Stage->>Writer: Updated AudioTask

Loading

_{Reviews (10): Last reviewed commit: "Address latest Qwen ASR review feedback" | Re-trigger Greptile}

[sarahyurick]

Since this is not specific to Qwen ASR can we move it to a more generic utils file, and find out if there are other stages that can use it?

[sarahyurick]

Can we move most if not all of these imports to the top of the file please?

[sarahyurick]

Let's remove this block, if the user is running it correctly in a pipeline then it should never reach.

[sarahyurick]

Can we remove the claude comments please.

[sarahyurick]

The GPU tests can be with the other tests in test_qwen_asr with @pytest.mark.gpu.

[sarahyurick]

Any reason why we can't just add this to audio_cuda12?

[virajkarandikar]

lgtm. vllm kwargs can be moved to utils.

[sarahyurick]

Instead of patching should we just use the actual compatible Transformers version? We could do it via the stage's runtime_env maybe?

[sarahyurick]

Can this list be imported from somewhere?

[sarahyurick]

Nit but can we not have 2 separate test classes for validate/merge please.

[mohammadaaftabv]

/ok to test f5d42df

[sarahyurick]

Thanks!