Finetuning Base results in progressively faster speech with every epoch

[dragonfyre13]

Description

See pull request #178

Now that finetuning is basically working (given the change to sft_12hz.py in commit 680d4e9), it seems there's only a few things to hammer out the rest of the way. This is the big one for me personally, without it finetuning isn't fully functional IMHO.

Each successive epoch of finetuning is skewing how fast generated audio is

Reproduction

Run normal finetuning with a few hundred audio files, setting --num_epochs to 20.

python ./finetuning/sft_12hz.py --init_model_path ./Qwen3-TTS-12Hz-1.7B-Base --output_model_path output --train_jsonl output-with-codes.jsonl --num_epochs 20

Then run the following to perform basic inference using each of the checkpoints:

import torch
import soundfile as sf
from qwen_tts import Qwen3TTSModel

for i in range(50):
    wavs, sr = Qwen3TTSModel.from_pretrained(f"output/checkpoint-epoch-{i}", device_map="cuda:0", dtype=torch.bfloat16, attn_implementation="flash_attention_2").generate_custom_voice(text="She said she would be here by noon.", language="english", speaker="speaker_test")
    sf.write(f"output/output_file_{i}.wav", wavs[0], sr)

Listen to each checkpoint and hear the difference (see attached for a few examples prior to pull request #178, which resolves this):
output_file_5.wav
output_file_7.wav
output_file_10.wav
output_file_12.wav
output_file_15.wav
output_file_19.wav

Logs

Environment Information

Tested on Debian 12, with nvidia driver 535.129.03, CUDA version 12.2, RTX 3090.

Known Issue

• The issue hasn't been already addressed in Documentation, Issues, and Discussions.

[Yusasif-AI]

which of the epoch is the best ?

[dragonfyre13]

That depends on what you consider "best". 19 is closest to the original audio, though it's very much sped up to the degree it's practically unintelligible. The actual speed of speech is closest in 5 or 7, but it's furthest from the original audio pitch and timbre.

[Yusasif-AI]

what about inference . It takes 40 seconds for a 30-word text on an L4 GPU. Were you able to get a lower inference?

[rekuenkdr]

Description

See pull request #178

Now that finetuning is basically working (given the change to sft_12hz.py in commit 680d4e9), it seems there's only a few things to hammer out the rest of the way. This is the big one for me personally, without it finetuning isn't fully functional IMHO.

Each successive epoch of finetuning is skewing how fast generated audio is

Reproduction

Run normal finetuning with a few hundred audio files, setting --num_epochs to 20.

python ./finetuning/sft_12hz.py --init_model_path ./Qwen3-TTS-12Hz-1.7B-Base --output_model_path output --train_jsonl output-with-codes.jsonl --num_epochs 20

Then run the following to perform basic inference using each of the checkpoints:

import torch
import soundfile as sf
from qwen_tts import Qwen3TTSModel

for i in range(50):
wavs, sr = Qwen3TTSModel.from_pretrained(f"output/checkpoint-epoch-{i}", device_map="cuda:0", dtype=torch.bfloat16, attn_implementation="flash_attention_2").generate_custom_voice(text="She said she would be here by noon.", language="english", speaker="speaker_test")
sf.write(f"output/output_file_{i}.wav", wavs[0], sr)

Listen to each checkpoint and hear the difference (see attached for a few examples prior to pull request #178, which resolves this): output_file_5.wav output_file_7.wav output_file_10.wav output_file_12.wav output_file_15.wav output_file_19.wav

Logs

Environment Information

Tested on Debian 12, with nvidia driver 535.129.03, CUDA version 12.2, RTX 3090.

Known Issue

* [x]  The issue hasn't been already addressed in Documentation, Issues, and Discussions.

I have encounter the same issue with finetuning since day one, the less epochs you train the better, early training keeps the pace while more epochs just derail the voice.

[humblenginr]

hey @dragonfyre13 , did you also notice that the script does double-shifting to the labels? The forward function uses (https://github.com/QwenLM/Qwen3-TTS/blob/main/qwen_tts/core/models/modeling_qwen3_tts.py#L1731) uses the HF PreTrainedModel class's default loss_function, which is ForCausalLMLoss (https://github.com/huggingface/transformers/blob/main/src/transformers/loss/loss_utils.py#L57) which does internal shifting of labels. And the finetuning data preparation script also does label shifting.

[fumyou13]

@humblenginr I agree with you that existing SFT codes indeed have two label-shifting operations (one is Line#114-125 in sft_12hz.py, and the other one is the HF loss function). This bug is highly relevant to the phenomenon that the generated speech goes faster after SFT. Here are my results:

Setting: 8 GPU+bs 16+ga 16+lr 1e-6+1.7B model+5k samples + epoch 4

version	Loss_before_sft	loss_after_sft	audio
base	22	13	very fast
fixed	8.3	7.8	correct

How to fix:

1. In sft_12hz.py:

outputs = model.talker(
    inputs_embeds=input_embeddings[:, :-1, :],
    attention_mask=attention_mask[:, :-1],
    labels=codec_0_labels[:, 1:],
    output_hidden_states=True
)

hidden_states = outputs.hidden_states[0][-1]
talker_hidden_states = hidden_states[codec_mask[:, 1:]]
talker_codec_ids = codec_ids[codec_mask]

->

outputs = model.talker(
    inputs_embeds=input_embeddings,
    attention_mask=attention_mask,
    labels=codec_0_labels,
    output_hidden_states=True
)
# Sub-talker loss: hidden_states shape [B, T, D]，codec_mask shape [B, T]
hidden_states = outputs.hidden_states[0][-1]
target_codec_mask = codec_mask[:, 1:]
talker_hidden_states = hidden_states[:, :-1][target_codec_mask]
talker_codec_ids = codec_ids[:, 1:][target_codec_mask]

2. In qwen_tts/core/models/modeling_qwen3_tts.py, line 1243:

loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs)

->

import torch.nn.functional as F
loss = F.cross_entropy(
    logits.reshape(-1, self.config.vocab_size),
    labels.reshape(-1),
    ignore_index=-100,
)

[humblenginr]

Hi @fumyou13
Thanks for sharing this. Were you able to get progressive improvements with more epochs? I feel that this fine-tuning script is fundamentally broken. Even though this double shifting is a prominent issue, I think there's more to it.

Also, I don't agree with @dragonfyre13 that the summing of codec embeddings should be removed.
During inference, the sub-talker generates levels 1-15 before the talker's next step (https://github.com/QwenLM/Qwen3-TTS/blob/main/qwen_tts/core/models/modeling_qwen3_tts.py#L1671). The talker never sees a level-0-only input. Every autoregressive step feeds in the sum of all 16 levels. So they do exist at inference time. Removing them from training would create a train-test mismatch where the model trains on level-0-only inputs but receives 16-level inputs at inference.

Did you get these results with removed codec embedding summing or with it intact?
And did the model capture the finer nuances in the target audio? How is the fine-tuning model generation compared to simple inference time voice cloning?

[fumyou13]

Hi @humblenginr
Thanks for sharing the insight.

I got these results by maintaining the multi-layer codec embedding summing so that the training and inference procedure is consistent. By increasing the training epochs (about 20 epochs), the speaker similarity gets progressively improved and performs better than zero-shot voice cloning. As for capturing the finer nuances in the target audio, it depends on many factors such as SFT data size, quality, and so on.

Based on my experiments, by fixing two bugs-the missing text_projection and double shifting in labels-the tuning code works properly.

[jungao1106]

@humblenginr I agree with you that existing SFT codes indeed have two label-shifting operations (one is Line#114-125 in sft_12hz.py, and the other one is the HF loss function). This bug is highly relevant to the phenomenon that the generated speech goes faster after SFT. Here are my results:

Setting: 8 GPU+bs 16+ga 16+lr 1e-6+1.7B model+5k samples + epoch 4

version Loss_before_sft loss_after_sft audio
base 22 13 very fast
fixed 8.3 7.8 correct
How to fix:

1. In sft_12hz.py:

outputs = model.talker(
inputs_embeds=input_embeddings[:, :-1, :],
attention_mask=attention_mask[:, :-1],
labels=codec_0_labels[:, 1:],
output_hidden_states=True
)

hidden_states = outputs.hidden_states[0][-1]
talker_hidden_states = hidden_states[codec_mask[:, 1:]]
talker_codec_ids = codec_ids[codec_mask]
->

outputs = model.talker(
inputs_embeds=input_embeddings,
attention_mask=attention_mask,
labels=codec_0_labels,
output_hidden_states=True
)

Sub-talker loss: hidden_states shape [B, T, D]，codec_mask shape [B, T]

hidden_states = outputs.hidden_states[0][-1]
target_codec_mask = codec_mask[:, 1:]
talker_hidden_states = hidden_states[:, :-1][target_codec_mask]
talker_codec_ids = codec_ids[:, 1:][target_codec_mask]
2. In qwen_tts/core/models/modeling_qwen3_tts.py, line 1243:

loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs)
->

import torch.nn.functional as F
loss = F.cross_entropy(
logits.reshape(-1, self.config.vocab_size),
labels.reshape(-1),
ignore_index=-100,
)

import torch.nn.functional as F
loss = F.cross_entropy(
logits.reshape(-1, self.config.vocab_size),
labels.reshape(-1),
ignore_index=-100,
)

Hello @fumyou13 , if I understand correctly, the logits and labels here should be offset, right?

[fumyou13]

@jungao1106 The forward function uses the HF PreTrainedModel class's default loss_function (ForCausalLMLoss), which has internal label-shifting. Therefore, for the talker, the logits and labels inputs should be aligned. For the sub-talker, the logits and labels inputs are offset since I directly use the standard CE loss implementation. It is worth noticing that the standard CE loss is only used by the sub-talker while the talker uses the ForCausalLMLoss.

[MrMuhannadObeidat]

I am using the updated files posted in the repo here:
https://github.com/rekuenkdr/Qwen3-TTS-streaming/tree/main/finetuning
I still have the code without the latest fix of "revert: restore codec levels 1-15 embedding summing in finetuning"

I am training with Arabic and using a 10K data set for starters. The output produced is pretty good from the first epoch (doing 5 runs in total). I did not update my repo yes for reverting the summing of additional codec levels. Will try that and report back.

One thing though, my data set has very good examples for text containing numbers, however, the fine tuned model does not read that properly if it is surrounded by Arabic text. If it is surrounded by English text then no issue.

Do I have to do something for data set prep to address issues related to reading numbers correctly?