The file structure of model folder (/data)

[Starlento]

I would like to know the folder structure of /data to arranage my first launch.
Simply because I do not have internet access inside my docker env, I tried

MyFolder
   - HuggingFaceH4
         -  zephyr-7b-beta

Using command

docker run --gpus all --shm-size 1g -p 8080:80 -v MyFolder:/data ghcr.io/huggingface/text-generation-inference:1.1.0 --model-id HuggingFaceH4/zephyr-7b-beta

It still shows INFO download: text_generation_launcher: Starting download process. and then failed.
I am wondering whether it is mandatory to have internet access? Or I should follow something like ~/.cache/huggingface to arrange my folder.

Here is the args printed:

Args { model_id: "HuggingFaceH4/zephyr-7b-beta", revision: None, validation_workers: 2, sharded: None, num_shard: None, quantize: None, dtype: None, trust_remote_code: false, max_concurrent_requests: 128, max_best_of: 2, max_stop_sequences: 4, max_top_n_tokens: 5, max_input_length: 1024, max_total_tokens: 2048, waiting_served_ratio: 1.2, max_batch_prefill_tokens: 4096, max_batch_total_tokens: None, max_waiting_tokens: 20, hostname: "18f548370d8d", port: 80, shard_uds_path: "/tmp/text-generation-server", master_addr: "localhost", master_port: 29500, huggingface_hub_cache: Some("/data"), weights_cache_override: None, disable_custom_kernels: false, cuda_memory_fraction: 1.0, rope_scaling: None, rope_factor: None, json_output: false, otlp_endpoint: None, cors_allow_origin: [], watermark_gamma: None, watermark_delta: None, ngrok: false, ngrok_authtoken: None, ngrok_edge: None, env: false }

[Starlento]

I also tried mounting the cache where I do have models--HuggingFaceH4--zephyr-7b-beta
While according to https://huggingface.co/docs/huggingface_hub/guides/manage-cache. It should be ~/.cache/huggingface/hub.
But no matter it is ~/.cache, ~/.cache/huggingface, ~/.cache/huggingface/hub. It seems that it is still accessing Internet.

(base) starlento@DESKTOP-HELIOS:~/.cache/huggingface/hub$ ll
total 10336
drwxr-xr-x 20 starlento starlento     4096 Nov  4 11:27 ./
drwxr-xr-x  9 starlento starlento     4096 Nov  4 11:35 ../
-rw-r--r--  1 starlento starlento     2814 Apr  3  2023 066329cdbf0bb3f3816e6f84e2d4876143fd4e26a11ec3ed7c9837e33d9fe41e.30cb82e2121eb5ce659a35437f3a58bdceac7b5cd7c0aaaa1f10345719259ba9
-rw-r--r--  1 starlento starlento      156 Apr  3  2023 066329cdbf0bb3f3816e6f84e2d4876143fd4e26a11ec3ed7c9837e33d9fe41e.30cb82e2121eb5ce659a35437f3a58bdceac7b5cd7c0aaaa1f10345719259ba9.json
-rwxr-xr-x  1 starlento starlento        0 Apr  3  2023 066329cdbf0bb3f3816e6f84e2d4876143fd4e26a11ec3ed7c9837e33d9fe41e.30cb82e2121eb5ce659a35437f3a58bdceac7b5cd7c0aaaa1f10345719259ba9.lock*
drwxr-xr-x  5 starlento starlento     4096 Apr  5  2023 datasets--hf-internal-testing--fixtures_ade20k/
drwxr-xr-x  5 starlento starlento     4096 Apr  5  2023 datasets--segments--sidewalk-semantic/
drwxr-xr-x  5 starlento starlento     4096 Apr 22  2023 datasets--shi-labs--oneformer_demo/
drwxr-xr-x  5 starlento starlento     4096 Oct 18 11:31 models--bennyguo--zero123-xl-diffusers/
drwxr-xr-x  6 starlento starlento     4096 Apr 22  2023 models--bert-base-cased/
drwxr-xr-x  5 starlento starlento     4096 Oct 13 22:34 models--DionTimmer--controlnet_qrcode-control_v1p_sd15/
drwxr-xr-x  6 starlento starlento     4096 Nov  4 11:27 models--HuggingFaceH4--zephyr-7b-beta/
drwxr-xr-x  6 starlento starlento     4096 Jul  1 14:46 models--jonatasgrosman--whisper-large-zh-cv11/
drwxr-xr-x  2 starlento starlento     4096 Jul  1 12:44 models--models--mask2former-swin-small-ade-semantic/
drwxr-xr-x  6 starlento starlento     4096 Jun 13 20:24 models--openai--clip-vit-large-patch14/
drwxr-xr-x  5 starlento starlento     4096 Oct 13 22:36 models--runwayml--stable-diffusion-v1-5/
drwxr-xr-x  5 starlento starlento     4096 Oct 18 11:40 models--stabilityai--stable-diffusion-2-1-base/
drwxr-xr-x  5 starlento starlento     4096 Oct 13 20:59 models--stabilityai--stable-diffusion-xl-base-1.0/
drwxr-xr-x  5 starlento starlento     4096 Sep 28 19:23 models--timm--convnextv2_large.fcmae_ft_in22k_in1k_384/
drwxr-xr-x  5 starlento starlento     4096 Sep 28 16:22 models--timm--resnet18.a1_in1k/
drwxr-xr-x  5 starlento starlento     4096 Sep 28 10:03 models--timm--tf_efficientnetv2_xl.in21k_ft_in1k/
drwxr-xr-x  5 starlento starlento     4096 Sep 28 16:17 models--timm--wide_resnet50_2.racm_in1k/
drwxr-xr-x  5 starlento starlento     4096 Jul  1 14:25 models--ydshieh--wav2vec2-large-xlsr-53-chinese-zh-cn-gpt/
-rw-------  1 starlento starlento 10485760 Oct 18 11:30 tmpktidcoxa
-rw-r--r--  1 starlento starlento        1 Oct 13 20:57 version_diffusers_cache.txt
-rw-r--r--  1 starlento starlento        1 Mar  7  2023 version.txt

[Starlento]

According to #815
I modified a docker-compose.yml which makes it work:
The model-id is actually a dir if you want to use the local model.

#Huggingface Inference
version: '3.5'

services:
  huggingface-inference:
    container_name: huggingface-inference
    image: ghcr.io/huggingface/text-generation-inference:latest
    shm_size: 8g
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    # runtime: nvidia
    volumes:
      - ./models:/data
    ports:
      - "8000:80"
    environment:
      # - NVIDIA_VISIBLE_DEVICES=0
      # - CUDA_LAUNCH_BLOCKING=1
      - MODEL_ID=/data/zephyr-7b-alpha
      # - SHARDED=true
      # - QUANTIZE=bitsandbytes