Generate songs based on a reference

[MJ54pwnz]

Hi! Thank you so much for this incredible work, you just put an end to closed source platforms like Suno. I have a question about using reference as a base for a song. Can we take a reference song for example and change lyrics somehow? Or generate a brand new song with new sound and etc?

[a43992899]

Can we take a reference song for example and change lyrics somehow?

• Are you talking about the stemgen mode? See issue Generate vocals on top of an existing instrumental, or an instrumental on top of existing vocals? #21
  We do plan to support it.
• For ICL, it mostly allows you to generate new songs conditioned on the style of the ref song. We deliberately train it to not directly copying the ref song melody.

Or generate a brand new song with new sound and etc?

Could you elaborate?

[MJ54pwnz]

@a43992899
Oh yeah, now I get it, yes, that's the feature I was talking about, ICL. I was also thinking about the fact that you could copy the style and melody with vocals, but change the lyrics while keeping the same style and voice. And about the elaborating, I was thinking about taking a song and changing its genre but trying to keep the style, like turning a pop song into a rock song. Again, thank you so much for this work, if you need any help with data collection I'd be happy to help!

[a43992899]

It is hard for ICL to copy melody for now.

If you're referring to editing lyrics while preserving the melody, our current model does not support this directly. While there might be complex workarounds to achieve music continuation that aligns with lyrics editing, they are quite tricky to realize.

A more elegant solution would be to continue pretraining the stage 1 checkpoint to enable song-level Fill-In-the-Middle (FIM). However, this approach would require substantial computational resources. We may consider adding this feature in YuE v2.

[MJ54pwnz]

Okay that's makes sense, we'll you guys don't have to rush, take as much time as you need!^^ and the last question. Will it be possible to fine tune the model on 24gb gpu? Also I think it would be wonderful for you guys to make an discord server so all people could share their samples and talk about your amazing models

[frink]

It would be great if there was a flag that could be flipped to copy the melody instead of the style. (Kinda thinking like Mureka...)

[a43992899]

I plan to add melody control in the future. However, the open-source community currently lacks high-quality melody transcription models.

Music Information Retrieval (MIR) progress lags behind advancements in generative AI. That said, I’ve noticed the use of chroma conditioning in MusicGen, which could be a promising direction. Perhaps someone could fine-tune a model to enable MusicGen-style melody control.

[LCF2764]

I plan to add melody control in the future. However, the open-source community currently lacks high-quality melody transcription models.

Music Information Retrieval (MIR) progress lags behind advancements in generative AI. That said, I’ve noticed the use of chroma conditioning in MusicGen, which could be a promising direction. Perhaps someone could fine-tune a model to enable MusicGen-style melody control.

Great idea to include melody control! I specialize in singing melody transcription and have gotten solid results in this area. Happy to collaborate if you need support!!

[a43992899]

@LCF2764

Thanks! That sounds great—do you have any models on Hugging Face that you plan to open source? I'd love to try them out.