OSError: Consistency check failed

[Nirbhay2727]

Describe the bug

OSError: Consistency check failed: file should be of size 9860464979 but has size 4277965032 ((…)l-00006-of-00007.bin).
We are sorry for the inconvenience. Please retry download and pass force_download=True, resume_download=False as argument.
If the issue persists, please let us know by opening an issue on https://github.com/huggingface/huggingface_hub.
Downloading (…)l-00006-of-00007.bin: 43%|███████████████▌ | 4.28G/9.86G [03:25<04:27, 20.8MB/s]

Reproduction

No response

Logs

model = AutoModelForCausalLM.from_pretrained(checkpoint,force_download=True, resume_download=False).to(device)
#device='cuda'

System info

- huggingface_hub version: 0.15.1
- Platform: Linux-5.15.0-1033-oracle-x86_64-with-glibc2.35
- Python version: 3.10.11
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Token path ?: /home/ubuntu/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: Nirbhay2727
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 1.13.1
- Jinja2: N/A
- Graphviz: N/A
- Pydot: N/A
- Pillow: 9.4.0
- hf_transfer: N/A
- gradio: N/A
- numpy: 1.24.3
- ENDPOINT: https://huggingface.co
- HUGGINGFACE_HUB_CACHE: /home/ubuntu/.cache/huggingface/hub
- HUGGINGFACE_ASSETS_CACHE: /home/ubuntu/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/ubuntu/.cache/huggingface/token
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False

[Wauplin]

Hi @Nirbhay2727, I'm sorry for the inconvenience. Can you tell me which model and file you are trying to instantiate ? Thanks in advance

[mateoriosglb]

Hello, I'm having the same issue with models:

• stabilityai/stable-diffusion-2-1
• runwayml/stable-diffusion-v1-5
• SG161222/Realistic_Vision_V1.4
• prompthero/openjourney

[ParseDark]

same issue. I want to use the whisper

[matthew-hippocraticai]

Same issue with falcon-40b-instruct

[Wauplin]

Sorry not taking care of it earlier. @matthew-hippocraticai could you post here:

• the output of huggingface-cli env (run in your machine)
• the script you used to download the weight
• the exact stacktrace you get (the error message + trace)

If you can't reproduce the error please let me know as well. And just in case, could you update to latest huggingface_hub version and retry?

Thanks a lot in advance! I'll investigate this thoroughly in the coming days so more information will help.

[matthew-hippocraticai]

I believe the issue was actually just running out of storage. Has been fixed now, thanks!

[Wauplin]

I believe the issue was actually just running out of storage.

Ah! That's a really good insight for us. How did you found out in the end?
@mateoriosglb @ParseDark could it be the same for you?

If confirmed, I'll add a message to the exception to suggest to the user to check disk usage and retry.

[matthew-hippocraticai]

It was simply OSError: [Errno 28] No space left on device but I didn't see it at first and just assumed this error was the only one

[julien-c]

note that we could maybe explore checking space left on device before starting downloading (but – and it's a big but – i'm not sure if there's any reliable way to do this)

[Wauplin]

@julien-c Given #1498 (comment), I don't think space left on device was the root cause of the problem for everyone in this issue (ping @mateoriosglb @ParseDark @Nirbhay2727 if you can confirm?). But anyway, checking for free space is still an interesting nice to have IMO so I opened an issue for it #1551. There is a cross-platform Python's built-in to check disk_usage. I would only trigger a warning instead of raising an exception.

[Wauplin]

Also related, I've got a user getting repeatedly this issue (#1549) but not when downloading with snapshot_download. I'm trying to investigate it more.

[mateoriosglb]

I still don't know the source of the problem but I agree with @Wauplin. I have more than enough space left so I rebooted the kernel, reloaded everything and then they started working fine. However I would like to find the source because it is possible that it will happen again.