Can I Fine-Tune the Diarization Model to Recognize a Specific Individual's Voice?

[shivamtawari]

Hi @MahmoudAshraf97

I'm curious to know if it's possible to customize the diarization output. Specifically, can we assign a custom name, such as 'Mr. XYZ', to dialogues spoken by a particular person, while the rest are labeled as 'Person 0', 'Person 1', etc.?

Thanks!

[MahmoudAshraf97]

It's doable but not through finetuning, you will use the intermediate embeddings generated from MSDD model and compare them to reference embeddings that you generated to identify which speaker is XYZ

[shivamtawari]

Thanks for getting back to me!

I get the idea of using intermediate embeddings from the MSDD model, but I'm not too sure how to actually create reference embeddings for someone like XYZ. Could you point me in the right direction on how to set that up? Also, what would be the best way to compare those embeddings to figure out who's speaking?

Any advice would be really helpful!