RuntimeError: unable to mmap 23802932552 bytes from file <./flux1-dev.safetensors>: Cannot allocate memory (12)

[sakura-nyaa]

System Info

• transformers version: 4.44.2
• Platform: Linux-6.10.10-arch1-1-x86_64-with-glibc2.40
• Python version: 3.12.6
• Huggingface_hub version: 0.24.6
• Safetensors version: 0.4.6-dev.0
• Accelerate version: 0.34.2
• Accelerate config: not found
• PyTorch version (GPU?): 2.3.1 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: No.
• Using GPU in script?: No.
• GPU type: AMD Radeon RX 7600 XT

Information

• The official example scripts
• My own modified scripts

Reproduction

1. Download https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/flux1-dev.safetensors
2. Try to load it with ws = safe_open("./flux1-dev.safetensors", framework="pt")
3. Receive "RuntimeError: unable to mmap 23802932552 bytes from file <./flux1-dev.safetensors>: Cannot allocate memory (12)"

Expected behavior

The tensors should be lazy loaded.
I should be able to access metadata and choose to actually load a tensor into RAM/VRAM when needed.
The fact that the .safetensors file is 24 GB and I have 16 GB of RAM shouldn't affect the loading.

(I can safely lazy load a safetensors file that is around 5 GB with the exact same script, and no memory is actually allocated and everything works as expected. The issue only arises when attempting to load a safetensors file that is larger than system RAM.)

[Narsil]

Is it possibly you are on 32bit system : https://stackoverflow.com/questions/10561368/i-have-enough-memory-but-mmap-keeps-failing-saying-cannot-allocate-memory ?

The tensors are lazily loaded, but here it seems mmap itself is failing.

For torch (pt) we are not doing the memory mapping ourselves, instead letting pytorch to the heavy lifting (this sidesteps a few constraints of lifetimes).

If you memory map the file manually, does it work ? https://docs.python.org/3/library/mmap.html

[shelvinskyi]

hey @Narsil, many thanks for the follow up, I have the same issue.

from safetensors import safe_open
safe_open('/lm/mistral-large-2407/consolidated.safetensors', framework='pt', device='cpu')

raises

RuntimeError: unable to mmap 245220228152 bytes from file </lm/mistral-large-2407/consolidated.safetensors>: Cannot allocate memory (12)

the consolidated.safetensors file is 229G in this case. following code with mmap works fine

import mmap

with open('/lm/mistral-large-2407/consolidated.safetensors', 'r+b') as f:
    mm = mmap.mmap(f.fileno(), 0)
    print(mm.read(1024))

any ideas what could be the reason? thanks!

adding some system info:

> uname -a
Linux mlgpu 6.8.0-51-generic #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec  5 13:09:44 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

> pip freeze
filelock==3.16.1
fsspec==2024.12.0
Jinja2==3.1.5
MarkupSafe==3.0.2
mpmath==1.3.0
networkx==3.4.2
nvidia-cublas-cu12==12.4.5.8
nvidia-cuda-cupti-cu12==12.4.127
nvidia-cuda-nvrtc-cu12==12.4.127
nvidia-cuda-runtime-cu12==12.4.127
nvidia-cudnn-cu12==9.1.0.70
nvidia-cufft-cu12==11.2.1.3
nvidia-curand-cu12==10.3.5.147
nvidia-cusolver-cu12==11.6.1.9
nvidia-cusparse-cu12==12.3.1.170
nvidia-nccl-cu12==2.21.5
nvidia-nvjitlink-cu12==12.4.127
nvidia-nvtx-cu12==12.4.127
safetensors==0.5.2
sympy==1.13.1
torch==2.5.1
triton==3.1.0
typing_extensions==4.12.2