Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Therories about my failed trainings. Please Read. #3036

Open
Erlandsson opened this issue Jan 4, 2025 · 1 comment
Open

Therories about my failed trainings. Please Read. #3036

Erlandsson opened this issue Jan 4, 2025 · 1 comment

Comments

@Erlandsson
Copy link

Erlandsson commented Jan 4, 2025

I have noticed that when installing, it often does not find files or scripts.
If installation is run one or more times, the files are found and installed.
The same when starting a train, first it complains a bout scripts or file not found, but when pressing train again it goes further.
And if pressed a third or fourth time it starts to train.

My guess is now:
when training, it needs to access files constantly? If so, maybe it then does not find them and therefore stops training with error about "runpy.py" , "accelerate.EXE", "accelerate_cli.py" and "launch.py" and some other stuff sometimes.

It does not help installing on other drives, or running on a fresh installed machine.

@Erlandsson
Copy link
Author

Problem is solved. Everything works now (automatic1111,Forge ui, Kohya)

I found out it was INTEL CPU BOOST !!!!. I noticed that when running kohya, automatic etc, that cpu frequency spiked around 5.5-5.8Ghz (i9 1490k 3.2Ghz), and began to think it if was that somehow files read back and forth just got corrupted in that speed?
Even though NO OTHER program i run have or show any problems. Like Blender for example. But otn the other hand, it does not access drives that much)..

So i tried turning off INTEL TURBO BOOST in the bios. That helped RIGHT AWAY. No problems at all installing kohya, no more failed files, no more failed downloads etc. I have now run several 4-5 hour trains with no hickup.

Now i just need to see if i can get windows OWN cpu limiter to work , maybe i can try my way up to a higher clock where it is stable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant