[docs] Model repo files (#1617)

* Create model-repo-layout.md

* Update _toctree.yml

* fix

* move to transformers.md

* move to diffusers.md

* remove from toctree

* remove

Author: stevhliu

docs/hub/diffusers.md

@@ -19,6 +19,18 @@ You can try out the models directly in the browser if you want to test them out
 <img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/refs%2Fpr%2F35/hub/libraries-diffusers_widget.png"/>
 </div>
 
+## Diffusers repository files
+
+A [Diffusers](https://hf.co/docs/diffusers/index) model repository contains all the required model sub-components such as the variational autoencoder for encoding images and decoding latents, text encoder, transformer model, and more. These sub-components are organized into a multi-folder layout.
+
+<div class="flex justify-center">
+  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/diffusers-model-files-repo.png"/>
+</div>
+
+Each subfolder contains the weights and configuration - where applicable - for each component similar to a [Transformers](./transformers) model.
+
+Weights are usually stored as safetensors files and the configuration is usually a json file with information about the model architecture.
+
 ## Using existing pipelines
 
 All `diffusers` pipelines are a line away from being used! To run generation we recommended to always start from the `DiffusionPipeline`: 

docs/hub/transformers.md

@@ -26,6 +26,51 @@ You can try out the models directly in the browser if you want to test them out
 <img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/libraries-transformers_widget-dark.png"/>
 </div>
 
+## Transformers repository files
+
+A [Transformers](https://hf.co/docs/transformers/index) model repository generally contains model files and preprocessor files.
+
+<div class="flex justify-center">
+  <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/model-files-repo.png"/>
+</div>
+
+### Model
+
+- The **`config.json`** file stores details about the model architecture such as the number of hidden layers, vocabulary size, number of attention heads, the dimensions of each head, and more. This metadata is the model blueprint.
+- The **`model.safetensors`** file stores the models pretrained layers and weights. For large models, the safetensors file is sharded to limit the amount of memory required to load it. Browse the **`model.safetensors.index.json`** file to see which safetensors file the model weights are being loaded from.
+
+  ```json
+  {
+  "metadata": {
+    "total_size": 16060522496
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00004-of-00004.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
+    ...
+    }
+  }
+  ```
+
+  You can also visualize this mapping by clicking on the ↗ button on the model card.
+
+  <div class="flex justify-center">
+    <img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/hub/model-files-safetensors-button.png"/>
+  </div>
+
+  [Safetensors](https://hf.co/docs/safetensors/index) is a safer and faster serialization format - compared to [pickle](./security-pickle#use-your-own-serialization-format) - for storing model weights. You may encounter weights pickled in formats such as **`bin`**, **`pth`**, or **`ckpt`**, but **`safetensors`** is increasingly adopted in the model ecosystem as a better alternative.
+
+- A model may also have a **`generation_config.json`** file which stores details about how to generate text, such as whether to sample, the top tokens to sample from, the temperature, and the special tokens for starting and stopping generation.
+
+### Preprocessor
+
+- The **`tokenizer_config.json`** file stores the special tokens added by a model. These special tokens signal many things to a model such as the beginning of a sentence, specific formatting for chat templates, or indicating an image. This file also shows the maximum input sequence length the model can accept, the preprocessor class, and the outputs it returns.
+- The **`tokenizer.json`** file stores the model's learned vocabulary.
+- The **`special_tokens_map.json`** is a mapping of the special tokens. For example, in [Llama 3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct/blob/main/special_tokens_map.json), the beginning of string token is `"<|begin_of_text|>"`.
+
+> [!TIP]
+> For other modalities, the `tokenizer_config.json` file is replaced by `preprocessor_config.json`.
+
 ## Using existing models
 
 All `transformer` models are a line away from being used! Depending on how you want to use them, you can use the high-level API using the `pipeline` function or you can use `AutoModel` for more control.