More than one individual person fine-tuning makes fine-tuning useless

[Marcophono2]

Hello!

Today I first trained SD 1.5 with 7 of my pictures and was absolutely impressed of the results! There was nothing I could not let my alter ego do. Wow! Then I repeated this with the pictures of a friend. As the source model I took the updated SD 1.5 after my training. But afterwards - what a surprise: When I wanted to create an image with him and with me there were always one older and one younger asian guy displayed. My friend is from Vietnam. But I am not. He is much younger - okay, that worked! :-) So obviously I kind of mixed the two training results. Is there a way to prevent this? I would like to train my SD model with 10 or more friends' images but I even lost with the second.
I trained also the not-touched SD 1.5 model with my friend's pictures and that worked also perfect. Like the other SD model which was trained with my pictures. But how to bring that together?

Best regards
Marc

P.S. Another annoying point is that I have two Ubuntu computers and on none of them Dreambooth is working correctly. On the one the preview images are not displayed since I installed (and also removed) an extension (mask2mask or similar) and on the other the Dreambooth tab is not displayed. Yes, I restarted the gui and everything. Also I reinstalled everything from the scratch today with the latest 1111 version but nothing helps. Is that normal that I have such problems or is the technical api between 1111 and the Dreambooth extension a bit limited?

[ExponentialML]

Hey. Sure, when you finetune one model then train something else on it, the two features from both training sessions will mix.
What you have to do is train both of your subjects at the same time.

As of the time of this comment, training multiple concepts doesn't work in the UI. The workaround is to put all of your images in one directory, then use [filewords] as the instance prompt.

Create a matching .txt file with the same name as your image, then caption each one with their specific token eg. a revs man, a lun man, etc. There should be some extensions to help with this if you look for it in the webui extensions tab.

[Marcophono2]

Thank you very much, @ExponentialML ! Does this mean that is impossible to add new concepts time be time due to the technical circumstanzas of the AI model? And what if I would want to trains 100 concepts all together in form of the txt files as you mentioned? Would that also work, aside from a huge training time? Or is then a VRAM limitation a problem? And finally: Will all the informations of the main model be untouched or could fine-training processes dilute or re-network it to lose quality?
Sorry for all this questions! 😄

[ExponentialML]

@Marcophono2 Not a problem! 🙂 .

You can train as many concepts as you like. Let me fill you in on the difference.

1. Dreambooth = Fine tuning the entire model with prior preservation (using class images and a rare token for you subject)
2. Fine tuning the entire model (like you did) without class images and text + image captions is well, fine tuning!

Fine tuning the model will almost always result in some sort of language drift or moving towards the dataset you've trained on, but Dreambooth really helps mitigate that.

It's really useful for fitting a concept into a model that already has a lot of information (like Stable Diffusion).
How you want to train your model is entirely up to you, and what effect you're looking after.

[Marcophono2]

Thak you very much, @ExponentialML ! I feel that I am about to understand now. 😄
If I understand you correct then it is possible to train more than only one concept BUT only if they are trained together. I think it is not possible to train 10 concepts today, 10 more concepts tomorrow and so on. Sure, I could also train 30 concepts on the third day but if the amount of new concepts increase it will also increase the rendering time in the same linear way, probably.
You said, Dreambooth would help to mitigate the language shift. Are there experiences what happens if you train 1,000 concepts?

[Marcophono2]

@ExponentialML Damn! With the update from today I saw that the DB extension for 1111 now offers up to 3 concepts per GUI. What a perfect timing! 😃
But I am still interested if there are experiences about the maximum number of concepts which could be trained without risking noticable quality loss of the model. Also I am still unsure if I also could use SD 2.1 (768) for training concepts. In every documentation or description I found about DB the editor pointed out that the user can choose from the model list (including 2.1 768) but not even one of them is using it. They all use 1.5. I think there will be a reason for it, or not? I could not find any performance comparation matrix.

[vinzenzuhr]

@Marcophono2 Have you ever read something about the maximum number of concepts? You're not the only one wondering ;)