MusicgenMelody ignores audio conditioning (regression between 4.48 and 4.57)

[audiodude]

(Original Note): Claude removed this line when editing, but I wanted to fully disclose that this issue was discovered and written up by Claude code

Update (corrected): the regression is wider than the title suggests — it already exists in transformers 4.57.6, the latest 4.x. So this is not a v5 regression; it broke somewhere between 4.48 and 4.57.

System info

• transformers versions tested:
  • 4.48.3 — works (audio conditioning active)
  • 4.57.6 — broken (audio ignored)
  • 5.5.4 — broken (audio ignored)
• Python 3.12.13, PyTorch 2.11.0+cu130, CUDA on RTX 3080 Ti, fp16
• Model: facebook/musicgen-melody

Description

MusicgenMelodyForConditionalGeneration.generate() ignores the audio reference (input_features) in transformers ≥ 4.57. Two reference audios with different chroma classes produce byte-identical generated audio for the same text prompt and seed. The same code in 4.48 produces meaningfully different output.

Minimal reproducer

import numpy as np, torch
from transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration

device, dtype = "cuda", torch.float16
proc = AutoProcessor.from_pretrained("facebook/musicgen-melody")
model = MusicgenMelodyForConditionalGeneration.from_pretrained(
    "facebook/musicgen-melody", torch_dtype=dtype
).to(device)

sr = 32000
t = np.linspace(0, 4, sr * 4)
A  = (0.4 * np.sin(2 * np.pi * 440.00 * t)).astype(np.float32)  # A4
Eb = (0.4 * np.sin(2 * np.pi * 311.13 * t)).astype(np.float32)  # Eb4 — different chroma class

def gen(audio):
    inputs = proc(text=["jazz"], audio=audio, sampling_rate=sr,
                  padding=True, return_tensors="pt").to(device)
    inputs["input_features"] = inputs["input_features"].to(dtype)
    torch.manual_seed(42)
    return model.generate(**inputs, max_new_tokens=100,
                          do_sample=True, guidance_scale=3.0)

a, e = gen(A), gen(Eb)
diff = (a[0, 0].float() - e[0, 0].float()).abs().mean().item()
print(f"output diff (A vs Eb): {diff:.4f}")

Results

transformers	output diff (A vs Eb)	audio conditioning
4.48.3	0.1610	works
4.57.6	0.0000	ignored
5.5.4	0.0000	ignored

Where the chain breaks (per v5.5.4 tracing)

• The processor extracts chroma features correctly. input_features is shape (1, N, 12) and values differ between A and Eb (chroma abs diff ≈ 0.17).
• _prepare_encoder_hidden_states_kwargs_for_generation does receive input_features in model_kwargs (verified by hooking).
• The returned encoder_hidden_states (audio prefix concatenated with text encoder output) differs between A and Eb — mean abs diff ≈ 0.015 against mean abs ≈ 0.034, i.e. the audio-prefix portion is meaningfully different.
• But per-step logits returned by model.generate are byte-identical between the two audios.

So the audio conditioning reaches the encoder-hidden-states side correctly, but the decoder produces identical logits regardless — suggesting the audio prefix is not actually being attended to in the decoder. dtype was not the cause; promoting audio_enc_to_dec_proj to fp32 with cast hooks did not change the result.

Bisection range

Broken: 4.57.6, 5.5.4. Last-known-good: 4.48.3. Haven't bisected within the 4.48 → 4.57 range.

Disclosure

This regression was identified by Claude Code during a debugging session — disclosing for transparency.

[voodoovampire]

Hi, this PR updates the SDPA inference tolerances used in the MPS backend by routing "mps" through the same tolerance branch as "xpu" in the SDPA tests. On MPS, the default tolerances were too strict for fp16 and caused flaky failures even though the numerical differences were within expected fp16 ranges. All CI checks are passing; happy to adjust anything if you’d like a different tolerance strategy. I’m happy to propose a minimal PR plus a unit test that checks different input_features do not produce identical outputs for a fixed seed

[geurinmccuan345-maker]

Ok tänks bro في السبت، 25 أبريل 2026 19:59 Travis Briggs ***@***.***> كتب:

…

*audiodude* created an issue (huggingface/transformers#45647) <#45647> *Full disclosure: This issue was found and written up by Claude Code* System info - transformers version (broken): *5.5.4* - transformers version (working): *4.48* - Python 3.12.13, PyTorch 2.11.0+cu130, CUDA on RTX 3080 Ti, fp16 - Model: facebook/musicgen-melody Description MusicgenMelodyForConditionalGeneration.generate() ignores the audio reference (input_features) in transformers v5.5.4. Two reference audios with different chroma classes produce *byte-identical* generated audio for the same text prompt and seed. The same code in v4.48 produces meaningfully different output. This is a regression — src/transformers/models/musicgen_melody/modeling_musicgen_melody.py changed by ~1666 lines between v4.48 and v5.5.4. Minimal reproducer import numpy as np, torchfrom transformers import AutoProcessor, MusicgenMelodyForConditionalGeneration device, dtype = "cuda", torch.float16proc = AutoProcessor.from_pretrained("facebook/musicgen-melody")model = MusicgenMelodyForConditionalGeneration.from_pretrained( "facebook/musicgen-melody", torch_dtype=dtype ).to(device) sr = 32000t = np.linspace(0, 4, sr * 4)A = (0.4 * np.sin(2 * np.pi * 440.00 * t)).astype(np.float32) # A4Eb = (0.4 * np.sin(2 * np.pi * 311.13 * t)).astype(np.float32) # Eb4 — different chroma class def gen(audio): inputs = proc(text=["jazz"], audio=audio, sampling_rate=sr, padding=True, return_tensors="pt").to(device) inputs["input_features"] = inputs["input_features"].to(dtype) torch.manual_seed(42) return model.generate(**inputs, max_new_tokens=100, do_sample=True, guidance_scale=3.0) a, e = gen(A), gen(Eb)diff = (a[0, 0].float() - e[0, 0].float()).abs().mean().item()print(f"output diff (A vs Eb): {diff:.4f}") Results transformers output diff (A vs Eb) audio conditioning 4.48 *0.1466* works 5.5.4 *0.0000* ignored Where the chain breaks (per v5.5.4 tracing) - The processor extracts chroma features correctly. input_features is shape (1, N, 12) and values differ between A and Eb (chroma abs diff ≈ 0.17). - _prepare_encoder_hidden_states_kwargs_for_generation does receive input_features in model_kwargs (verified by hooking). - The returned encoder_hidden_states (audio prefix concatenated with text encoder output) differs between A and Eb — mean abs diff ≈ 0.015 against mean abs ≈ 0.034, i.e. the audio-prefix portion is meaningfully different. - But per-step logits returned by model.generate are byte-identical between the two audios. So the audio conditioning reaches the encoder-hidden-states side correctly, but the decoder produces identical logits regardless — suggesting the audio prefix is not actually being attended to in the decoder under v5. dtype was not the cause; promoting audio_enc_to_dec_proj to fp32 with cast hooks did not change the result. Suspected commits Significant changes landed in modeling_musicgen_melody.py between v4 and v5; candidates include: - #43590 <#43590> — "Remove many output_attentions and other traced outputs on 100+ models" - #44759 <#44759> — "Remove cache_position in more models (3)" - #43916 <#43916> — "Harmonize input_embeds to inputs_embeds everywhere" I haven't bisected to a single commit. Disclosure This regression was identified by Claude Code during a debugging session — disclosing for transparency. — Reply to this email directly, view it on GitHub <#45647>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CBUSR6HJT4H5QEE6Y3NEXW34XUDHZAVCNFSM6AAAAACYGKZ4QKVHI2DSMVQWIX3LMV43ASLTON2WKOZUGMZDQOJWHA3TIMQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/CBUSR6CAMH5JUJJ4FYCADID4XUDHZA5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBTGI4DSNRYG42DFJTSMVQXG33OVJZXKYTTMNZGSYTFMSSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/CBUSR6ECSYFF3LQL2WNUMMD4XUDHZA5CNFSL4Z3JMQ5C6L3HNF2C22DVMIXUS43TOVSS6NBTGI4DSNRYG42DFJTSMVQXG33OVJZXKYTTMNZGSYTFMSSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Rocketknight1]

cc @ebezzam @eustlb