Adding different text tokenizers

[SmittyB00p]

Hello all,
I have been working with this model for a few weeks and was wondering if there could be an option of adding different text tokenizers, such as Qwen 2.5. I have manually experimented with using the Qwen 2.5 1.5B tokenizer and the quality of speech is different, but it would be nice to be able to run the different text tokenizers to hear the difference in a notebook.
Thanks!

[ZackHodari]

A model is paired to it's tokenizer, the tokenizer cannot be changed without updating the model weights. I'd actually be surprised if it sounds like speech at all (unless you've trained the model for the new tokenizer)

[SmittyB00p]

So by adding another tokenizer like so:

'''
def load_qwen2_5_tokenizer():
tokenizer_name = "Qwen/Qwen2.5-1.5B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
bos = tokenizer.bos_token
eos = tokenizer.eos_token
tokenizer._tokenizer.post_processor = TemplateProcessing(
single=f"{bos}:0 $A:0 {eos}:0",
pair=f"{bos}:0 $A:0 {eos}:0 {bos}:1 $B:1 {eos}:1",
special_tokens=[(f"{bos}", tokenizer.bos_token_id), (f"{eos}", tokenizer.eos_token_id)],
)

return tokenizer

'''

and then setting the tokenizer to self._text_tokenizer = load_qwen2_5_tokenizer(), that will make for un-discernible speech? Am I understanding that correctly?

[ZackHodari]

The model is only compatible with the tokenizer it was trained with. How you configure it won't matter it, how you train it is more important