CUDA out of memory Error

[muhdusman]

I am getting CUDA out of memory error while running preload_models. On ipython I am running these commands as well, to lower the requirements

`import os
os.environ["SUNO_USE_SMALL_MODELS"] = "True"
os.environ["SUNO_OFFLOAD_CPU"] = "True"

from bark import SAMPLE_RATE, generate_audio, preload_models
from IPython.display import Audio

preload_models(
text_use_small=True,
coarse_use_small=True,
fine_use_gpu=False,
fine_use_small=True,
)

`

It initially downloads the file "41a4361a0bb21b667a35fbf76a309a77.pt" in this directory .cache/suno/bark_v0/ which is 2.2 GB

But the commands exits with this error

OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.58 GiB total capacity; 644.06 MiB already allocated; 15.38 MiB free; 682.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I have checked other threads as well, and adding some of the info which was asked over their

print(torch.cuda.mem_get_info())
(16121856, 15653470208)

print(torch.cuda.device_count())
4

[DarkParasite]

@muhdusman Did you fix it? Im getting the same problem

[muhdusman]

Nope, unable to fix it still

[fnrcum]

os.environ["SUNO_USE_SMALL_MODELS"] = "True"
os.environ["SUNO_OFFLOAD_CPU"] = "True"

Try making these actual env variables

[freckletonj]

It works for me if I just set the env vars before the import of bark:

import os
os.environ["SUNO_USE_SMALL_MODELS"] = "True"
os.environ["SUNO_OFFLOAD_CPU"] = "True"

import bark

it appears that bark reads these upon import, not upon usage.

[eusoubrasileiro]

@freckletonj thanks that works!

[JonathanFly]

Just to make sure you know you can use large models with 6GB as long as cpu offloading is on, no problem.

[boringtaskai]

Just to make sure you know you can use large models with 6GB as long as cpu offloading is on, no problem.

Yes, thanks it works! Does this still using GPU or just CPU only?

[asterocean]

I figured out a solution, load modules on demand rather than load them all at the same time, try this pull request: #531