docs

Author: YasenT

PROJECT_OVERVIEW.md

@@ -0,0 +1,52 @@
+# Project Overview
+
+**Pulp Hugging Face Plugin** - A Pulp plugin for managing Hugging Face Hub content with pull-through caching support.
+
+## Key Features
+- Pull-through caching for Hugging Face content (models, datasets, spaces)
+- Authentication support via HF tokens for private repositories
+- API proxying to forward requests to Hugging Face Hub
+- File download caching and serving
+
+## Technology Stack
+- **Framework**: Django-based Pulp plugin
+- **Python**: 3.9-3.12
+- **Core Dependencies**: pulpcore (3.100.0-3.115), httpx
+- **Version**: 0.4.0.dev
+
+## Project Structure
+
+```
+pulp_hugging_face/
+├── app/
+│   ├── models.py         - Core models (Content, Remote, Repository, Distribution)
+│   ├── viewsets.py       - REST API viewsets
+│   ├── serializers.py    - API serializers
+│   ├── handler.py        - Custom content handler
+│   ├── tasks/            - Async tasks (sync, publish)
+│   └── migrations/       - Database migrations
+└── tests/                - Test suite structure (functional, unit, performance)
+```
+
+## Main Components
+
+1. **HuggingFaceContent** - Represents cached files from HF Hub
+2. **HuggingFaceRemote** - Configuration for fetching from HF Hub
+3. **HuggingFaceDistribution** - Serves content with pull-through caching
+4. **HuggingFaceRepository** - Groups cached content
+
+## Current Status
+- REST API fully functional
+- Pull-through caching implemented
+- CLI support planned but not yet implemented
+- Active development (version 0.4.0.dev)
+
+## Workflow
+1. Create a remote pointing to huggingface.co
+2. Create a distribution with the remote for pull-through caching
+3. Access HF content through Pulp (automatically cached on first request)
+4. Subsequent requests served from cache
+
+## Implementation Details
+
+The plugin uses custom handlers to inject HF-compatible headers and manage the caching lifecycle. The `HuggingFaceDistribution` model includes a custom `content_headers_for()` method that adds Hugging Face Hub compatible headers like `X-Repo-Commit`, `X-Linked-ETag`, and proper `Content-Disposition` headers to ensure compatibility with HF CLI tools.

docs/admin/guides/.gitkeep

@@ -0,0 +1 @@
+

docs/admin/index.md

@@ -0,0 +1,25 @@
+# Welcome to Pulp Hugging Face for Admins!
+
+Here you'll find information about Hugging Face-specific admin workflows.
+
+If you just got here, consider following the top [Admin Manual](site:pulpcore/#admin) links, as it provides the common ground for setting up and configuring your Pulp deployment.
+
+## Configuration
+
+The pulp_hugging_face plugin currently uses standard pulpcore configuration. Key settings to consider:
+
+- **CONTENT_ORIGIN**: The base URL for serving content
+- **REMOTE_USER_ENVIRON_NAME**: For external authentication setups
+
+See the [pulpcore settings documentation](site:pulpcore/docs/admin/reference/settings/) for details on these and other core settings.
+
+## Access Control
+
+The plugin uses pulpcore's standard Role Based Access Control (RBAC) system. Users with appropriate permissions can:
+
+- Create and manage Hugging Face remotes
+- Create and manage distributions for pull-through caching
+- View cached content
+
+Refer to the [pulpcore RBAC documentation](site:pulpcore/docs/admin/guides/rbac/) for information on managing permissions.
+

docs/admin/learn/.gitkeep

@@ -0,0 +1 @@
+

docs/admin/reference/.gitkeep

@@ -0,0 +1 @@
+

docs/dev/guides/.gitkeep

@@ -0,0 +1 @@
+

docs/dev/index.md

@@ -0,0 +1,72 @@
+# Welcome to Pulp Hugging Face for Developers!
+
+Here you'll find information useful for the Hugging Face plugin developers.
+
+If you just got here, consider exploring Pulpcore's [Developer Manual](site:pulpcore/docs/dev/), as it provides the common ground for developers for contributing to docs, to code and getting basic background on plugin development.
+
+## Development Setup
+
+### Prerequisites
+
+- Python 3.9+
+- A running Pulp development environment
+- Git
+
+### Clone and Install
+
+```bash
+git clone https://github.com/pulp/pulp_hugging_face.git
+cd pulp_hugging_face
+pip install -e .
+```
+
+### Running Tests
+
+```bash
+# Run unit tests
+pytest pulp_hugging_face/tests/unit/
+
+# Run functional tests (requires running Pulp)
+pytest pulp_hugging_face/tests/functional/
+```
+
+## Architecture Overview
+
+The pulp_hugging_face plugin implements a pull-through caching system for Hugging Face Hub content.
+
+### Key Components
+
+| Component | File | Description |
+|-----------|------|-------------|
+| **HuggingFaceContent** | `models.py` | Represents cached files from HF Hub |
+| **HuggingFaceRemote** | `models.py` | Configuration for fetching from HF Hub |
+| **HuggingFaceRepository** | `models.py` | Groups cached content |
+| **HuggingFaceDistribution** | `models.py` | Serves content with pull-through caching |
+| **Handler** | `handler.py` | Custom content handler for HF-compatible responses |
+
+### Pull-through Caching Flow
+
+1. Request arrives at the content handler
+2. Handler checks if content exists locally
+3. If not, fetches from Hugging Face Hub via the configured remote
+4. Content is saved and streamed to the client
+5. Future requests served from cache
+
+### HF-Compatible Headers
+
+The `HuggingFaceDistribution.content_headers_for()` method injects headers required by Hugging Face CLI tools:
+
+- `X-Repo-Commit` - Git commit hash
+- `X-Linked-ETag` - ETag for the content
+- `Content-Disposition` - Filename header
+
+## Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Make your changes
+4. Run tests and linting
+5. Submit a pull request
+
+See [CONTRIBUTING.md](https://github.com/pulp/pulp_hugging_face/blob/main/CONTRIBUTING.md) for detailed guidelines.
+

docs/dev/learn/.gitkeep

@@ -0,0 +1 @@
+

docs/dev/reference/.gitkeep

@@ -0,0 +1 @@
+

docs/index.md

@@ -1,12 +1,66 @@
-# Welcome to Pulp 
+# Welcome to Pulp Hugging Face
 
-The `` plugin extends pulpcore to support hosting  packages.
+The `pulp_hugging_face` plugin extends pulpcore to support hosting and caching Hugging Face Hub content.
 This plugin is a part of the Pulp Project, and assumes some familiarity with the
 [pulpcore documentation](site:pulpcore/).
 
 If you are just getting started, we recommend:
 
-- [Getting Started with ](site:pulp_hugging_face/docs/tutorials/getting-started.md),
-  for a starting out with a common use case.
+- [Getting Started with Hugging Face](site:pulp_hugging_face/docs/user/tutorials/getting_started/),
+  for setting up your first pull-through cache for Hugging Face models.
+- [Core Concepts](site:pulp_hugging_face/docs/user/learn/concepts/),
+  to understand the plugin architecture and key terminology.
 
+## Features
+
+- **Pull-through Caching**: Automatically fetch and cache models, datasets, and spaces from Hugging Face Hub on first access
+- **Authentication Support**: Use Hugging Face tokens to access private repositories
+- **Full Compatibility**: Works seamlessly with `huggingface-cli`, transformers library, and other HF tools
+- **All Content Types**: Support for models, datasets, and spaces
+- **On-demand Downloads**: Reduce storage by only downloading content when requested
+- **Versioned Repositories**: Every operation creates a restorable snapshot
+
+## Documentation Sections
+
+### For Users
+
+- [User Guide](site:pulp_hugging_face/docs/user/) - Getting started and feature documentation
+- [Tutorials](site:pulp_hugging_face/docs/user/tutorials/getting_started/) - Step-by-step guides
+- [Guides](site:pulp_hugging_face/docs/user/guides/configuration/) - Detailed workflow documentation
+
+### For Administrators
+
+- [Admin Guide](site:pulp_hugging_face/docs/admin/) - Installation and configuration
+
+### For Developers
+
+- [Developer Guide](site:pulp_hugging_face/docs/dev/) - Contributing to the plugin
+
+## Quick Start
+
+1. Create a remote pointing to Hugging Face Hub:
+
+```bash
+curl -X POST https://pulp.example.com/pulp/api/v3/remotes/hugging_face/hugging-face/ \
+  -H "Content-Type: application/json" \
+  -u admin:password \
+  -d '{"name": "hf-remote", "url": "https://huggingface.co", "policy": "on_demand"}'
+```
+
+2. Create a distribution for pull-through caching:
+
+```bash
+curl -X POST https://pulp.example.com/pulp/api/v3/distributions/hugging_face/hugging-face/ \
+  -H "Content-Type: application/json" \
+  -u admin:password \
+  -d '{"name": "hf-cache", "base_path": "huggingface", "remote": "<remote_href>"}'
+```
+
+3. Access Hugging Face content through Pulp:
+
+```bash
+# Use with huggingface-cli
+export HF_ENDPOINT="https://pulp.example.com/pulp/content/huggingface"
+huggingface-cli download bert-base-uncased
+```