Whisper models in Foundry Local SDK: v1 models return empty text, v2 models only transcribe first 30 seconds

[yuvalshmaryahu]

Hi,

I’m trying to use the Whisper-CPU models through the Foundry Local SDK, but I’m encountering some unexpected behavior.
There appear to be two model versions: v1 and v2.
For v1 models (e.g., base, small, medium), the API consistently returns empty text, even for audio files that produce correct transcriptions when using whisper-tiny.
For v2 models (e.g., tiny, large), the transcription is returned, but only for the first ~30 seconds of the audio. The rest of the audio is not transcribed.
Because of this, I’m currently unable to get a full transcription using the available models.

Could you please help clarify:
Whether this is a known issue with these models in the Foundry Local SDK?
If there is a recommended configuration or workaround to obtain full transcriptions?

Thank you!

[leestott]

Thanks @yuvalshmaryahu

Issue #517 confirmed! The SDK's AudioClient only transcribed the first ~30 seconds of the 4m12s audio file. The transcription cuts off at "The floor was old cracked" — exactly matching the reported behavior.

Completed: tested a 4 min audio .wav file

Validation results

Runtime	Audio length	Transcribed	Result
JavaScript (SDK AudioClient)	4m 12s	~30 sec	Truncated — cuts off at "The floor was old cracked"
Python (ONNX Runtime direct)	4m 12s	~30 sec	Truncated — identical cutoff point

Root cause: Whisper's encoder processes a fixed 30-second mel spectrogram window (3000 frames at the standard hop length). Neither the SDK's AudioClient nor the Python sample implements audio chunking — splitting longer audio into overlapping ~30 second segments, transcribing each, and concatenating the results. This is a limitation in the Foundry Local SDK / ONNX model pipeline, not a model quality issue.

Fix

I have provide a sample at https://github.com/microsoft-foundry/Foundry-Local-Lab

Solution implemented: All three language samples now split long audio into ≤30-second chunks, transcribe each chunk independently, and concatenate the results there is a sample of over 4mins available

Changes by language:

1. Python (foundry-local-whisper.py) — A transcribe() function splits audio using librosa into CHUNK_SAMPLES (30 × 16000 samples), runs _transcribe_chunk() on each segment via direct ONNX Runtime encoder/decoder inference, and joins the text with spaces.

2. JavaScript (foundry-local-whisper.mjs) — A splitWavIntoChunks() function parses the WAV header, splits the PCM data into 30-second segments, writes each to a temporary WAV file with a valid RIFF header, then calls audioClient.transcribe() on each chunk. Temporary files are cleaned up after processing.

3. C# (WhisperTranscription.cs) — A SplitWavIntoChunks() method performs the same WAV-header-aware split, writes chunk files to a temp directory, then calls audioClient.TranscribeAudioAsync() on each. The temp directory is cleaned up afterwards.

Supporting changes:

• Test audio: A longer sample (zava-full-project-walkthrough.wav, ~4 min) was added specifically for testing the chunking workaround.
• All three files include a comment linking back to https://github.com/microsoft/Foundry-Local/issues/517 for traceability.

Pattern used (all languages):

1. Load the WAV file
2. If duration ≤ 30 s → single-pass transcription (no change)
3. If duration > 30 s → split PCM data into 30-second chunks
4. Transcribe each chunk independently
5. Concatenate chunk texts with spaces
6. Clean up any temporary files

This is a client-side workaround — the underlying Whisper encoder limitation (30-second window) still exists in the model architecture and the Foundry Local SDK. The fix ensures the workshop samples produce complete transcriptions for audio of any length.

[yuvalshmaryahu]

Hey,
I’m part of the Foundry Local team enabled by Arc. As part of our service, we serve the models without performing any preprocessing on the input.

The mitigation you suggested, which involves splitting the file, is not acceptable for us. We expect a mitigation to be implemented on your side.

[baijumeswani]

For v1 models (e.g., base, small, medium), the API consistently returns empty text, even for audio files that produce correct transcriptions when using whisper-tiny.

By default, the AudioClient does not inject the language token. Could you explicitly set it?

var audioClient = await model.GetAudioClientAsync();
audioClient.Settings.Language = "en";

[yuvalshmaryahu]

Thanks for the update — I have a few follow-up questions:

1. In v1 models, we need to explicitly set the language, while in v2 this isn’t required. What’s the reason for this difference? Does v2 automatically detect the language, or does it default to English?
2. Are there plans for v1 models to transition to v2 behavior? In other words, should we invest in adapting to v1 specifics, or will this become unnecessary soon?
3. What are the key differences between v1 and v2? Specifically, what defines whether a model is considered v1 vs. v2?
4. Is there any update regarding the 30-second limitation in v2 models? Also, does this limitation apply to v1 models as well?

[yuvalshmaryahu]

Hi, could you please provide an update on any progress regarding this issue?

[samuel100]

This issue has been automatically closed due to extended inactivity. If this is still a problem, please re-open the issue or file a new one with updated details. Thank you for your contributions!

[samuel100]

Re-opening — this issue was closed prematurely as part of a stale-issue sweep. Apologies for the noise.