Contributing to AI Hub Models

This guide covers how to add new models to the repository and contribute changes.

Prerequisites

Before contributing:

Legal approval is required before publishing new models. Submit a request at go/genairequest.
Join the access lists (required for GitHub repo access): qai-hub-users and qai-hub-models-contributors.

Clone the repository and set up your environment:

git clone https://github.com/qualcomm/ai-hub-models
cd ai-hub-models
python scripts/build_and_test.py install_deps
source qaihm-dev/bin/activate
pre-commit install

Create a branch using the format: dev/<username>/<branch_name>

Terminology

model_id - The folder name (e.g., yolov7, ddrnet23_slim)
model_name - The published display name in manifest.yaml (e.g., "YOLOv7", "DDRNet23-Slim")

Model Directory Structure

Each model lives in qai_hub_models/models/<model_id>/ and requires:

File	Purpose
`model.py`	PyTorch model inheriting from `BaseModel`
`app.py`	End-to-end application with pre/post-processing (can be inherited from `models/templates/`)
`demo.py`	CLI demo running the app on sample data
`test.py`	Unit tests for the model
`manifest.yaml`	Website metadata + codegen/export options (single unified schema)

Note: Many models extend template implementations from models/templates/. For example, segmentation models can inherit from CityscapesSegmentor and use the template SegmentationApp. Check existing models solving similar tasks before writing everything from scratch.

Sample models to reference:

Regular model: ddrnet23_slim - segmentation model using a template base class
Collection model: whisper_tiny - encoder-decoder with separately compiled components

Optional Files

File	Purpose
`requirements.txt`	Model-specific dependencies (pinned versions required)
`scorecard-config.yaml`	CI/scorecard-only knobs (e.g., `global_requirements_incompatible`, `skip_scorecard`, `test_split`)

Auto-generated Files

These are created by running codegen scripts:

File	Generated By
`README.md`, `export.py`, `evaluate.py`, `test_generated.py`	`python qai_hub_models/scripts/run_codegen.py -m <model_id>` (`evaluate.py` is skipped for precompiled models or when `skip_evaluate` is set in `scorecard-config.yaml`)
`perf.yaml`, `numerics.yaml`, `release-assets.yaml`	Scorecard / CI (generated weekly, not included in your PR)

1. requirements.txt

Model-specific dependencies not in the base environment.

All packages must be pinned to exact versions (e.g., torch==2.0.1).

When adding dependencies:

Check qai_hub_models/requirements.txt first - don't duplicate
Check global_requirements.txt - use the same version if possible
If a different version is required, set global_requirements_incompatible: true in the model's scorecard-config.yaml
After adding new packages, run python qai_hub_models/scripts/generate_global_requirements.py
If the package is not in global_requirements.txt, you can choose any version as long as its dependencies are compatible with existing global requirements

Additional options for complex dependency scenarios:

global_requirements_incompatible (in scorecard-config.yaml) - Set when versions differ from global requirements
pip_pre_build_reqs (in manifest.yaml) - Packages that must be installed before the model's requirements (for broken build dependencies)
pip_install_flags (in manifest.yaml) - Extra pip flags needed when installing (requires global_requirements_incompatible: true in scorecard-config.yaml)

2. model.py

The model file defines the PyTorch module that will be compiled and run on-device.

Basic Structure

All models must:

Inherit from BaseModel (which inherits torch.nn.Module)
Include MODEL_ID = __name__.split(".")[-2] before the class definition
Implement the required methods listed below

Required Methods

Method	Description
`from_pretrained(cls)`	Classmethod to load pretrained weights. All arguments must have defaults.
`get_input_spec()`	Returns `InputSpec` dict of `{input_name: (shape, dtype)}`. For image inputs, follow the standard: RGB format with values in range [0, 1].
`get_output_spec()`	Returns dict of outputs

Optional Methods

These have default implementations but can be overridden:

Method	Description
`_sample_inputs_impl()`	Provide real sample inputs instead of random data. Important for more accurate PSNR data, since `export.py` runs a single sample inference and reports the PSNR difference between torch and device.
`get_hub_compile_options()` / `get_hub_profile_options()`	Custom AI Hub Workbench flags
`get_unsupported_reason()`	Mark specific device attributes that can't be supported. Rarely defined in practice; only necessary if a specific Hexagon version is required for advanced models.
`get_eval_dataset_classes()`	List of `BaseDataset` classes on which this model can be evaluated. If unset, model will not support eval.
`get_evaluator()`	Return evaluator instance for accuracy measurement. If unset, model will not support eval.
`get_calibration_dataset_cls()`	`BaseDataset` class used for quantization calibration. If unset, model will not support quantization.
`get_hub_litemp_percentage(precision)`	Percentage (0-100) of layers to keep in higher precision for mixed precision

Loading External Model Code

When your model depends on external code (e.g., from a GitHub repo or third-party package), follow this preference order:

Add to requirements.txt (Preferred) - Add the dependency to requirements.txt. Prefer PyPI packages when available:
```
timm==0.9.2
```
If not on PyPI, install directly from GitHub by adding to manifest.yaml:
```
additional_pip_requirements:
  - git+https://github.com/owner/repo.git@v1.0.0
```
Monkeypatching - If you need to modify external code behavior, use monkeypatching rather than copying source (see Appendix: Monkeypatching).
SourceAsRoot (Last Resort) - Only use SourceAsRoot when the above approaches are not possible (e.g., the code is not installable, requires heavy modifications, or has incompatible dependencies). This utility clones the external repo to the user's machine and sets up the Python environment as if running from that repo's source root. It also allows you to apply patches to the cloned source. Should be avoided when possible.

For pretrained weights:

Library weights - Many libraries provide pretrained options
Download links - Use URLs from model READMEs
S3 upload - Upload to qaihub-public-assets bucket (see Uploading Assets to S3)

Special Model Types

CollectionModel - For models compiled as separate assets (e.g., encoder-decoder):

Main class inherits from CollectionModel
Each component inherits from BaseModel
See whisper_tiny for an example

PrecompiledWorkbenchModel - For models where only compiled assets are published (no PyTorch source).

3. app.py

The app enables end-to-end usage with user-friendly I/O formats.

Requirements

Define an App class that takes a Callable in __init__. The callable should represent a model. We use Callable (rather than a specific type) because the PyTorch implementation may be replaced with an equivalent implementation that uses a different runtime.
Implement a predict() method for inference

Useful Utilities

Check these files in qai_hub_models/utils/ before implementing your own:

File	Purpose
`image_processing.py`	Image preprocessing and conversion (e.g., `app_to_net_image_inputs`)
`bounding_box_processing.py`	NMS and bounding box utilities for object detection
`draw.py`	Drawing utilities for visualization (points, connections, boxes)
`display.py`	Display and save image utilities
`asset_loaders.py`	Loading assets from URLs, S3, Google Drive
`transpose_channel.py`	Channel transposition utilities (NCHW <-> NHWC)

Also check qai_hub_models/models/templates/ for template app implementations for common tasks (segmentation, classification, object detection, etc.).

I/O Conventions

Follow existing patterns for consistency. For image inputs to the model, follow the standard: RGB format with values normalized to range [0, 1].

Task	Input Format	Output Format
Image processing	`PIL.Image.Image`	`PIL.Image.Image`
Video processing	filepath string	filepath string
Classification	image/tensor	tensor of probabilities (post-softmax)
Object detection	image	list of bounding boxes (post-NMS)

Check qai_hub_models/utils/ for existing utilities like app_to_net_image_inputs before implementing your own.

4. demo.py

The demo runs the app on sample data and presents results. See ddrnet23_slim/demo.py for an example.

Standard Pattern

Parse input arguments (model options, --eval-mode, --device, --output-dir, etc.)
Initialize the model
Load and preprocess inputs
Initialize and run the app
Display or save results (behind is_test flag for unit tests)

Sample data is typically stored in S3 and loaded using asset utilities.

5. test.py

Standard Tests

Test	Required	Purpose
`test_task`	Yes	PyTorch model accuracy on sample input
`test_demo`	Yes	Demo runs without exceptions

6. manifest.yaml

The manifest is a single unified schema holding both website-facing metadata and codegen/export options. It replaces the earlier split between info.yaml and code-gen.yaml.

See ddrnet23_slim/manifest.yaml for an example
See QAIHMModelManifest in qai_hub_models/configs/manifest_yaml.py for field details

Website metadata fields:

name, id, status - Model identity (id must match folder name). Set status: pending for new models. The scorecard will automatically update this to public once the model passes validation.
headline, description, use_case, domain - Public-facing description
tags, applicable_scenarios, related_models - Categorization
form_factors - Target devices (Phone, Tablet, IoT)
technical_details - Model specs (some auto-filled by script)
license_type, source_repo, research_paper - Attribution
labels_file - Path to labels file for classification models
has_static_banner - Whether the model has a static web banner (required for publishing)

Banner assets: If you set has_static_banner: true, a static banner must exist at the model's web-asset URL, so upload one to the web-asset S3 bucket for this model. If you don't have a banner, either set has_static_banner: false (the model stays pending and won't be promoted to public), or reuse an existing banner from another model in the same category/domain if one is relevant.

License restrictions: Before onboarding a model, check the license of its source weights and set license_type accordingly. The license decides whether a model can be onboarded and published (see MODEL_LICENSE in qai_hub_models/configs/_info_yaml_enums.py):

Tier	Licenses	Onboard?
Non-restrictive	`apache-2.0`, `mit`, `bsd-3-clause`, `cc-by-4.0`	Preferred. Permit redistribution and commercial use freely.
Copyleft / model-specific	`agpl-3.0`, `gpl-3.0`, `creativeml-openrail-m`, `llama2`, `llama3`, `gemma`, `falcon3`, `taide`	Allowed, but carry redistribution / same-license obligations. Set `restrict_model_sharing: true` if sharing the weights is restricted.
Non-commercial	`other-non-commercial`, `cc-by-non-commercial-4.0`	Avoid. Cannot be promoted to `public`.

Codegen / export options:

supported_precisions - List of precisions to enable (float, w8a8, w8a16, w4a16, etc.)
has_on_target_demo - Set to true if the model supports on-device demo
disabled_paths - Disable specific precision/runtime combinations with reason
is_collection_model - True for multi-component (encoder-decoder) models

Auto-fill size and parameter count:

python qai_hub_models/scripts/autofill_manifest_yaml.py -m <model_id>

CI-only knobs (skip_scorecard, test_split, global_requirements_incompatible, numerics_threshold_override, ...) live in a separate scorecard-config.yaml under scorecard/models/<model_id>/. See QAIHMModelScorecardConfig in qai_hub_models/scorecard/scorecard_config_yaml.py.

Adding Quantization Support

To enable quantized precisions (w8a8, w8a16, etc.) for a model:

1. Add a Dataset

Create or reuse a dataset. Inherit from BaseDataset and implement the required methods.

Where to put it: if the dataset is specific to a single model, place it in that model's own folder (e.g. qai_hub_models/models/<model_id>/dataset.py). Only put it in the shared qai_hub_models/datasets/ folder if it is reusable across multiple models (e.g. ImageNet, COCO). This mirrors the convention for evaluators below. Do not add model-specific dataset code under the shared datasets/ folder.

Dataset licensing: A dataset is used in two distinct ways, and each has different licensing requirements:

Evaluation datasets are only read to measure accuracy; they are never redistributed or baked into a shipped artifact. Any non-restrictive license is acceptable here (e.g. a dataset that is free for research or evaluation use).
Calibration datasets must be open source. Calibration data is used to quantize the model, and the resulting quantized weights are published as part of the reproducible quantization flow, so the data effectively becomes part of what we ship. A restrictive or non-commercial license on the calibration set would carry those restrictions into the released model.

Practically: never use a dataset with a restrictive or non-commercial license for calibration. If the only dataset that fits your task is restrictive, you can still use it for evaluation, but pick a separate open-source dataset (or a public subset) for calibration.

Required methods:

Method	Description
`__init__(self, split, ...)`	Initialize with `DatasetSplit.TRAIN` or `DatasetSplit.VAL`
`__len__(self)`	Return the number of samples
`__getitem__(self, idx)`	Return `(input_tensor, ground_truth)` for a sample
`_download_data(self)`	Download dataset to `self.dataset_path`
`default_samples_per_job()`	Static method returning default batch size for inference jobs
`get_dataset_metadata()`	Return `DatasetMetadata(link, split_description)` for website

Optional methods:

Method	Description
`_validate_data(self)`	Validate downloaded data (default: check path exists)
`collate_fn(batch)`	Custom collation for DataLoader

See imagenette.py for an example of a shared dataset, and foot_track_net/dataset.py for a model-specific dataset that lives in the model's own folder.

2. Add an Evaluator

Create or reuse an evaluator. Place it in the model's own folder if it's only used by that model, or in the appropriate template folder under qai_hub_models/models/templates/<name>/ if it's used by multiple models. Inherit from BaseEvaluator and implement the required methods.

Required methods:

Method	Description
`add_batch(self, output, gt)`	Accumulate metrics for a batch of model outputs vs ground truth
`reset(self)`	Reset accumulated state
`get_accuracy_score(self)`	Return single float accuracy (higher is better by default; for an error metric like WER or NME, set `higher_is_better=False` on the metric returned by `get_metric_metadata`)
`formatted_accuracy(self)`	Return formatted string with accuracy and units

Optional methods:

Method	Description
`get_metric_metadata(self)`	Return `MetricMetadata` for website publishing (name, unit, range, and whether higher or lower is better)

See classification_evaluator.py for an example evaluator implementation.

3. Update the Model

Add these methods to model.py. They return the dataset/evaluator classes directly (import them from wherever you placed them above):

@classmethod
def get_eval_dataset_classes(cls) -> list[type[BaseDataset]]:
    return [YourDataset]

def get_calibration_dataset_cls(self) -> type[BaseDataset]:
    return YourDataset

def get_evaluator(self) -> BaseEvaluator:
    return YourEvaluator(...)  # Return an instance

4. Update manifest.yaml

Add supported precisions (defaults to float only if not specified):

supported_precisions:
  - float
  - w8a8
  - w8a16

5. Run Codegen and Test

python qai_hub_models/scripts/run_codegen.py -m <model_id>
python -m qai_hub_models.models.<model_id>.evaluate --precision w8a8

Accuracy drop from float should be reasonable (10 points or less). Consider mixed precision (e.g., w8a8_mixed_int16) if accuracy is too low.

Uploading Assets to S3

For model checkpoints or test data not available via public URLs:

Authenticate: Run python scripts/build_and_test.py validate_aws_credentials (prompts for password)

Upload: Use AWS profile qaihm:

aws s3 cp <local_file> s3://qaihub-public-assets/qai-hub-models/models/<model_id>/v1/ --profile qaihm

Set permissions: Grant public-read access when uploading
Reference in code: Set MODEL_ASSET_VERSION = 1 in model.py
Versioning: Assets cannot be deleted. For new versions, create v2/, v3/, etc. and update MODEL_ASSET_VERSION.

Verification Workflow

Before submitting a PR:

# Run codegen for your model
python qai_hub_models/scripts/run_codegen.py -m <model_id>

# Auto-fill manifest.yaml
python qai_hub_models/scripts/autofill_manifest_yaml.py -m <model_id>

# Run all pre-commit hooks
pre-commit run --all-files

# Run package unit tests
python scripts/build_and_test.py test_qaihm

Run model-specific tests:

# Test export (install model dependencies first per README)
python -m qai_hub_models.models.<model_id>.export --target-runtime tflite --chipset qualcomm-snapdragon-8gen3

# Test evaluation (if available)
python -m qai_hub_models.models.<model_id>.evaluate

export.py should produce a model that profiles successfully on device (for all added precisions)
evaluate.py should produce good results in torch and on-device for all supported precisions

Code Quality

Linting, Formatting, and Type Checking

You can run these tools manually, but they also run automatically via pre-commit hooks:

Ruff - Linting and formatting: ruff check --fix and ruff format
mypy - Type checking: mypy qai_hub_models/

Pre-commit Hooks

Hooks run automatically and include:

License header insertion (BSD-3)
YAML validation, trailing whitespace, large file detection
Ruff check + format
mypy type checking

Import Restrictions

Don't import directly: numba, xtcocotools, git - use qai_hub_models.extern.* wrappers instead.

Support

For questions, reach out to the default PR reviewers or ping the Teams channel.

Appendix: Monkeypatching

When external model code needs modification for on-device compilation, prefer monkeypatching over copying source. This keeps the codebase maintainable and makes it clear what was changed.

Common pattern:

Create a model_patches.py file with replacement functions
Import the original class/module
Replace the method: OriginalClass.method = patched_method

Example - replacing a forward method (gkt):

# In model_patches.py - define the patched function
def KernelAttention_forward(self, q, k, v, skip=None, mask=None):
    # Modified implementation for on-device compatibility
    ...

# In model.py - apply the patch after importing
from external_repo import KernelAttention
from .model_patches import KernelAttention_forward

KernelAttention.forward = KernelAttention_forward

Example - patching multiple components (sam2):

# Patch module-level functions
import sam2.modeling.backbones.hieradet as hieradet
hieradet.window_partition = window_partition_5d

# Patch class methods with functools.partial
sam2.sam_prompt_encoder._embed_points = functools.partial(
    patched_embed_points, sam2.sam_prompt_encoder
)

# Replace entire submodules
for block in model.blocks:
    block.mlp = PatchedMLP(block.mlp)

When to monkeypatch:

Replacing operations unsupported by QNN (e.g., certain einsum patterns, dynamic shapes)
Optimizing tensor layouts for on-device performance (e.g., 6D → 5D tensors)
Removing GPU-specific code paths

Navigation Menu

Uh oh!

Contributing to AI Hub Models

Prerequisites

Terminology

Model Directory Structure

Optional Files

Auto-generated Files

1. requirements.txt

2. model.py

Basic Structure

Required Methods

Optional Methods

Loading External Model Code

Special Model Types

3. app.py

Requirements

Useful Utilities

I/O Conventions

4. demo.py

Standard Pattern

5. test.py

Standard Tests

6. manifest.yaml

Adding Quantization Support

1. Add a Dataset

2. Add an Evaluator

3. Update the Model

4. Update manifest.yaml

5. Run Codegen and Test

Uploading Assets to S3

Verification Workflow

Code Quality

Linting, Formatting, and Type Checking

Pre-commit Hooks

Import Restrictions

Support

Appendix: Monkeypatching

Uh oh!

FilesExpand file tree

CONTRIBUTING.md

Latest commit

History

CONTRIBUTING.md

File metadata and controls

Contributing to AI Hub Models

Prerequisites

Terminology

Model Directory Structure

Optional Files

Auto-generated Files

1. requirements.txt

2. model.py

Basic Structure

Required Methods

Optional Methods

Loading External Model Code

Special Model Types

3. app.py

Requirements

Useful Utilities

I/O Conventions

4. demo.py

Standard Pattern

5. test.py

Standard Tests

6. manifest.yaml

Adding Quantization Support

1. Add a Dataset

2. Add an Evaluator

3. Update the Model

4. Update manifest.yaml

5. Run Codegen and Test

Uploading Assets to S3

Verification Workflow

Code Quality

Linting, Formatting, and Type Checking

Pre-commit Hooks

Import Restrictions

Support

Appendix: Monkeypatching