From d47819dae18f16bd8394e72e7b27b8f0c80f0c53 Mon Sep 17 00:00:00 2001 From: Steven Liu <59462357+stevhliu@users.noreply.github.com> Date: Tue, 25 Feb 2025 14:19:01 -0800 Subject: [PATCH 1/7] Create model-repo-layout.md --- docs/hub/model-repo-layout.md | 60 +++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 docs/hub/model-repo-layout.md diff --git a/docs/hub/model-repo-layout.md b/docs/hub/model-repo-layout.md new file mode 100644 index 000000000..9acf7fd56 --- /dev/null +++ b/docs/hub/model-repo-layout.md @@ -0,0 +1,60 @@ +# Model repository files + +A model repository holds all the files required to initialize a pretrained model for inference or training. The repository directory structure and files may vary depending on the library integration, but this guide covers what to expect in a Transformers or Diffusers model repository. + +## Transformers + +A [Transformers](https://hf.co/docs/transformers/index) model repository generally contains model files and preprocessor files. + +
+ +
+ +### Model + +- The **`config.json`** file stores details about the model architecture such as the number of hidden layers, vocabulary size, number of attention heads, the dimensions of each head, and more. This metadata is the model blueprint. +- The **`model.safetensors`** file stores the models pretrained layers and weights. For large models, the safetensors file is sharded to limit the amount of memory required to load it. Browse the **`model.safetensors.index.json`** file to see which safetensors file the model weights are being loaded from. + + ```json + { + "metadata": { + "total_size": 16060522496 + }, + "weight_map": { + "lm_head.weight": "model-00004-of-00004.safetensors", + "model.embed_tokens.weight": "model-00001-of-00004.safetensors", + ... + } + } + ``` + + You can also visualize this mapping by clicking on the ↗ button on the model card. + +
+ +
+ + [Safetensors](https://hf.co/docs/safetensors/index) is a safer and faster serialization format - compared to [pickle](./security-pickle#use-your-own-serialization-format) - for storing model weights. You may encounter weights pickled in formats such as **`bin`**, **`pth`**, or **`ckpt`**, but **`safetensors`** is increasingly adopted in the model ecosystem as a better alternative. + +- A model may also have a **`generation_config.json`** file which stores details about how to generate text, such as whether to sample, the top tokens to sample from, the temperature, and the special tokens for starting and stopping generation. + +### Preprocessor + +- The **`tokenizer_config.json`** file stores the special tokens added by a model. These special tokens signal many things to a model such as the beginning of a sentence, specific formatting for chat templates, or indicating an image. This file also shows the maximum input sequence length the model can accept, the preprocessor class, and the outputs it returns. +- The **`tokenizer.json`** file stores the model's learned vocabulary. +- The **`special_tokens_map.json`** is a mapping of the special tokens. For example, in [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/special_tokens_map.json), the beginning of string token is `"<|begin_of_text|>"`. + +> [!TIP] +> For other modalities, the `tokenizer_config.json` file is replaced by `preprocessor_config.json`. + +## Diffusers + +A [Diffusers](https://hf.co/docs/diffusers/index) model repository contains all the required model sub-components such as the variational autoencoder for encoding images and decoding latents, text encoder, transformer model, and more. These sub-components are organized into a multi-folder layout. + +
+ +
+ +Each subfolder contains the weights and configuration - where applicable - for each component similar to a Transformers model. + +Weights are usually stored as safetensors files and the configuration is usually a json file with information about the model architecture. From 5749844e98c99187bbf0d7973cd3a5346feeb1b5 Mon Sep 17 00:00:00 2001 From: Steven Liu <59462357+stevhliu@users.noreply.github.com> Date: Tue, 25 Feb 2025 14:21:07 -0800 Subject: [PATCH 2/7] Update _toctree.yml --- docs/hub/_toctree.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/hub/_toctree.yml b/docs/hub/_toctree.yml index 611d832f0..0fafff234 100644 --- a/docs/hub/_toctree.yml +++ b/docs/hub/_toctree.yml @@ -54,8 +54,11 @@ title: Gated Models - local: models-uploading title: Uploading Models + - local: model-repo-layout + title: Model repository files - local: models-downloading title: Downloading Models + - local: model-repo-layout - local: models-libraries title: Integrated Libraries sections: From 8685dfe50253ee0eaba0f8f5927e86676a2a1bca Mon Sep 17 00:00:00 2001 From: Steven Liu <59462357+stevhliu@users.noreply.github.com> Date: Tue, 25 Feb 2025 14:21:53 -0800 Subject: [PATCH 3/7] fix --- docs/hub/_toctree.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/hub/_toctree.yml b/docs/hub/_toctree.yml index 0fafff234..62ad4c9e0 100644 --- a/docs/hub/_toctree.yml +++ b/docs/hub/_toctree.yml @@ -58,7 +58,6 @@ title: Model repository files - local: models-downloading title: Downloading Models - - local: model-repo-layout - local: models-libraries title: Integrated Libraries sections: From e1bd6e5f8ec077e98a383b4678a18067d7876097 Mon Sep 17 00:00:00 2001 From: Steven Liu <59462357+stevhliu@users.noreply.github.com> Date: Wed, 26 Feb 2025 09:52:45 -0800 Subject: [PATCH 4/7] move to transformers.md --- docs/hub/transformers.md | 45 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/docs/hub/transformers.md b/docs/hub/transformers.md index 6b2914d5b..4c2fb0a46 100644 --- a/docs/hub/transformers.md +++ b/docs/hub/transformers.md @@ -26,6 +26,51 @@ You can try out the models directly in the browser if you want to test them out +## Transformers repository files + +A [Transformers](https://hf.co/docs/transformers/index) model repository generally contains model files and preprocessor files. + +
+ +
+ +### Model + +- The **`config.json`** file stores details about the model architecture such as the number of hidden layers, vocabulary size, number of attention heads, the dimensions of each head, and more. This metadata is the model blueprint. +- The **`model.safetensors`** file stores the models pretrained layers and weights. For large models, the safetensors file is sharded to limit the amount of memory required to load it. Browse the **`model.safetensors.index.json`** file to see which safetensors file the model weights are being loaded from. + + ```json + { + "metadata": { + "total_size": 16060522496 + }, + "weight_map": { + "lm_head.weight": "model-00004-of-00004.safetensors", + "model.embed_tokens.weight": "model-00001-of-00004.safetensors", + ... + } + } + ``` + + You can also visualize this mapping by clicking on the ↗ button on the model card. + +
+ +
+ + [Safetensors](https://hf.co/docs/safetensors/index) is a safer and faster serialization format - compared to [pickle](./security-pickle#use-your-own-serialization-format) - for storing model weights. You may encounter weights pickled in formats such as **`bin`**, **`pth`**, or **`ckpt`**, but **`safetensors`** is increasingly adopted in the model ecosystem as a better alternative. + +- A model may also have a **`generation_config.json`** file which stores details about how to generate text, such as whether to sample, the top tokens to sample from, the temperature, and the special tokens for starting and stopping generation. + +### Preprocessor + +- The **`tokenizer_config.json`** file stores the special tokens added by a model. These special tokens signal many things to a model such as the beginning of a sentence, specific formatting for chat templates, or indicating an image. This file also shows the maximum input sequence length the model can accept, the preprocessor class, and the outputs it returns. +- The **`tokenizer.json`** file stores the model's learned vocabulary. +- The **`special_tokens_map.json`** is a mapping of the special tokens. For example, in [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/special_tokens_map.json), the beginning of string token is `"<|begin_of_text|>"`. + +> [!TIP] +> For other modalities, the `tokenizer_config.json` file is replaced by `preprocessor_config.json`. + ## Using existing models All `transformer` models are a line away from being used! Depending on how you want to use them, you can use the high-level API using the `pipeline` function or you can use `AutoModel` for more control. From 9d24532d1d7049f848604a97bd94dc2d1f851f70 Mon Sep 17 00:00:00 2001 From: Steven Liu <59462357+stevhliu@users.noreply.github.com> Date: Wed, 26 Feb 2025 09:54:32 -0800 Subject: [PATCH 5/7] move to diffusers.md --- docs/hub/diffusers.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/hub/diffusers.md b/docs/hub/diffusers.md index 9370b3668..e66dfd59f 100644 --- a/docs/hub/diffusers.md +++ b/docs/hub/diffusers.md @@ -19,6 +19,18 @@ You can try out the models directly in the browser if you want to test them out +## Diffusers repository files + +A [Diffusers](https://hf.co/docs/diffusers/index) model repository contains all the required model sub-components such as the variational autoencoder for encoding images and decoding latents, text encoder, transformer model, and more. These sub-components are organized into a multi-folder layout. + +
+ +
+ +Each subfolder contains the weights and configuration - where applicable - for each component similar to a [Transformers](./transformers) model. + +Weights are usually stored as safetensors files and the configuration is usually a json file with information about the model architecture. + ## Using existing pipelines All `diffusers` pipelines are a line away from being used! To run generation we recommended to always start from the `DiffusionPipeline`: From 58273e89f21a00afe29a0b480a8f5e73d0743ded Mon Sep 17 00:00:00 2001 From: Steven Liu <59462357+stevhliu@users.noreply.github.com> Date: Wed, 26 Feb 2025 09:55:00 -0800 Subject: [PATCH 6/7] remove from toctree --- docs/hub/_toctree.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/hub/_toctree.yml b/docs/hub/_toctree.yml index 62ad4c9e0..611d832f0 100644 --- a/docs/hub/_toctree.yml +++ b/docs/hub/_toctree.yml @@ -54,8 +54,6 @@ title: Gated Models - local: models-uploading title: Uploading Models - - local: model-repo-layout - title: Model repository files - local: models-downloading title: Downloading Models - local: models-libraries From 0faf9af598f8a52c2a42c24eea89605d1f3a3c2a Mon Sep 17 00:00:00 2001 From: Steven Liu <59462357+stevhliu@users.noreply.github.com> Date: Wed, 26 Feb 2025 09:56:23 -0800 Subject: [PATCH 7/7] remove --- docs/hub/model-repo-layout.md | 60 ----------------------------------- 1 file changed, 60 deletions(-) delete mode 100644 docs/hub/model-repo-layout.md diff --git a/docs/hub/model-repo-layout.md b/docs/hub/model-repo-layout.md deleted file mode 100644 index 9acf7fd56..000000000 --- a/docs/hub/model-repo-layout.md +++ /dev/null @@ -1,60 +0,0 @@ -# Model repository files - -A model repository holds all the files required to initialize a pretrained model for inference or training. The repository directory structure and files may vary depending on the library integration, but this guide covers what to expect in a Transformers or Diffusers model repository. - -## Transformers - -A [Transformers](https://hf.co/docs/transformers/index) model repository generally contains model files and preprocessor files. - -
- -
- -### Model - -- The **`config.json`** file stores details about the model architecture such as the number of hidden layers, vocabulary size, number of attention heads, the dimensions of each head, and more. This metadata is the model blueprint. -- The **`model.safetensors`** file stores the models pretrained layers and weights. For large models, the safetensors file is sharded to limit the amount of memory required to load it. Browse the **`model.safetensors.index.json`** file to see which safetensors file the model weights are being loaded from. - - ```json - { - "metadata": { - "total_size": 16060522496 - }, - "weight_map": { - "lm_head.weight": "model-00004-of-00004.safetensors", - "model.embed_tokens.weight": "model-00001-of-00004.safetensors", - ... - } - } - ``` - - You can also visualize this mapping by clicking on the ↗ button on the model card. - -
- -
- - [Safetensors](https://hf.co/docs/safetensors/index) is a safer and faster serialization format - compared to [pickle](./security-pickle#use-your-own-serialization-format) - for storing model weights. You may encounter weights pickled in formats such as **`bin`**, **`pth`**, or **`ckpt`**, but **`safetensors`** is increasingly adopted in the model ecosystem as a better alternative. - -- A model may also have a **`generation_config.json`** file which stores details about how to generate text, such as whether to sample, the top tokens to sample from, the temperature, and the special tokens for starting and stopping generation. - -### Preprocessor - -- The **`tokenizer_config.json`** file stores the special tokens added by a model. These special tokens signal many things to a model such as the beginning of a sentence, specific formatting for chat templates, or indicating an image. This file also shows the maximum input sequence length the model can accept, the preprocessor class, and the outputs it returns. -- The **`tokenizer.json`** file stores the model's learned vocabulary. -- The **`special_tokens_map.json`** is a mapping of the special tokens. For example, in [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/special_tokens_map.json), the beginning of string token is `"<|begin_of_text|>"`. - -> [!TIP] -> For other modalities, the `tokenizer_config.json` file is replaced by `preprocessor_config.json`. - -## Diffusers - -A [Diffusers](https://hf.co/docs/diffusers/index) model repository contains all the required model sub-components such as the variational autoencoder for encoding images and decoding latents, text encoder, transformer model, and more. These sub-components are organized into a multi-folder layout. - -
- -
- -Each subfolder contains the weights and configuration - where applicable - for each component similar to a Transformers model. - -Weights are usually stored as safetensors files and the configuration is usually a json file with information about the model architecture.