Skip to content

DEVELOPMENT.md

Latest commit

 

History

History
294 lines (215 loc) · 8.47 KB

DEVELOPMENT.md

File metadata and controls

294 lines (215 loc) · 8.47 KB
Raw

Development Guide

This guide covers the development setup and workflow for the MLPerf Inference Endpoint Benchmarking System. For contribution guidelines, see CONTRIBUTING.md.

Getting Started

Prerequisites

  • Python: 3.12+ (3.12 recommended)
  • Git: Latest version
  • OS: Linux or macOS (Windows is not supported)

Development Environment Setup

# 1. Fork https://github.com/mlcommons/endpoints on GitHub, then clone your fork
git clone https://github.com/YOUR_USERNAME/endpoints.git
cd endpoints

# 2. Add the upstream repo as a remote
git remote add upstream https://github.com/mlcommons/endpoints.git

# 3. Install development dependencies
uv sync --extra dev --extra test

# 4. Install pre-commit hooks
uv run pre-commit install

# 5. Verify installation
uv run inference-endpoint --version
uv run pytest --version
Using pip + venv instead (backward-compatible)

Note: Does not use uv.lock — dependency versions may differ from the lockfile.

python3.12 -m venv venv && source venv/bin/activate
pip install -e ".[dev,test]"
pre-commit install

After activating the venv, commands work without the uv run prefix.

Project Structure

endpoints/
├── src/inference_endpoint/     # Main package source
│   ├── main.py                 # Entry point and CLI app
│   ├── exceptions.py           # Project-wide exception types
│   ├── async_utils/            # Event loop, ZMQ transport, pub/sub
│   ├── commands/               # CLI command implementations
│   ├── config/                 # Configuration and schema management
│   ├── core/                   # Core types and orchestration
│   ├── dataset_manager/        # Dataset handling and loading
│   ├── endpoint_client/        # HTTP/ZMQ endpoint communication
│   ├── evaluation/             # Accuracy evaluation and scoring
│   ├── load_generator/         # Load generation and scheduling
│   ├── metrics/                # Performance measurement and reporting
│   ├── openai/                 # OpenAI API compatibility
│   ├── plugins/                # Plugin system
│   ├── profiling/              # Performance profiling tools
│   ├── sglang/                 # SGLang API adapter
│   ├── testing/                # Test utilities (echo server, etc.)
│   └── utils/                  # Common utilities
├── tests/                      # Test suite
│   ├── unit/                   # Unit tests
│   ├── integration/            # Integration tests
│   ├── performance/            # Performance benchmarks
│   └── datasets/               # Test data (dummy_1k.jsonl, squad_pruned/)
├── docs/                       # Documentation
├── examples/                   # Usage examples
└── scripts/                    # Utility scripts

Testing

Running Tests

# All tests (excludes slow/performance)
uv run pytest

# Unit tests only
uv run pytest -m unit

# Integration tests
uv run pytest -m integration

# Single file with verbose output
uv run pytest -xvs tests/unit/path/to/test_file.py

# With coverage
uv run pytest --cov=src --cov-report=html

# Performance suites (CI-skipped; run on demand — see docs/CLIENT_PERFORMANCE_TUNING.md)
uv run pytest -vs -m performance --no-cov tests/performance

Test Markers

Every test function must have a marker:

import pytest

@pytest.mark.unit
def test_something():
    ...

@pytest.mark.unit
@pytest.mark.asyncio  # strict mode is configured globally in pyproject.toml
async def test_async_something():
    ...

Available markers: unit, integration, slow, performance, run_explicitly

Key Fixtures

Defined in tests/conftest.py — use these instead of mocking:

  • mock_http_echo_server — real HTTP echo server on dynamic port
  • mock_http_oracle_server — dataset-driven response server
  • dummy_dataset — in-memory test dataset
  • events_db — pre-populated SQLite events database

Coverage

Target >90% coverage for all new code.

Code Quality

Pre-commit Hooks

All of these run automatically on commit:

  • trailing-whitespace, end-of-file-fixer, check-yaml, check-merge-conflict, debug-statements
  • ruff (lint + autofix) and ruff-format
  • mypy type checking
  • prettier for YAML/JSON/Markdown
  • License header enforcement
  • YAML template validation and regeneration

IMPORTANT: Always run pre-commit run --all-files before every commit. Hooks may modify files. If files are modified, stage the changes and commit once.

# Run all hooks
uv run pre-commit run --all-files

# Install hooks (done during setup)
uv run pre-commit install

Code Style

Markdown prose wraps at 80 columns, configured in .prettierrc.yaml. The existing Prettier pre-commit hook applies this automatically, and the Pre-commit GitHub Actions workflow enforces the same formatting across all tracked files. Tables, code blocks, and indivisible links or inline code may exceed the target width.

To reformat Markdown after editing, run:

uv run pre-commit run prettier --all-files
  • Formatter/Linter: ruff (line-length 88, target Python 3.12)
  • Type checking: mypy
  • Formatting: ruff-format (double quotes, space indent)
  • License headers: Required on all Python files (auto-added by pre-commit)
  • Commit messages: Conventional commits — feat:, fix:, docs:, test:, chore:, perf:
  • Comments: Only where the why isn't obvious from the code

Development Workflow

Feature Development

# Sync your fork with upstream before starting
git fetch upstream
git checkout main
git merge upstream/main

# Create a feature branch on your fork
git checkout -b feat/your-feature-name

# Make changes and test
uv run pytest
uv run pre-commit run --all-files

# Commit changes
git add <specific files>
git commit -m "feat: add your feature description"

# Push to your fork and open a PR against mlcommons/endpoints
git push origin feat/your-feature-name

Branch Naming

feat/short-description
fix/short-description
docs/short-description

YAML Config Templates

Config templates in src/inference_endpoint/config/templates/ are auto-generated from schema defaults. When you change config/schema.py, regenerate them:

uv run python scripts/regenerate_templates.py

The pre-commit hook auto-regenerates templates when schema.py, config.py, or regenerate_templates.py change. CI validates templates are up to date via --check mode.

Two variants are generated per mode (offline, online, concurrency):

  • _template.yaml — minimal: only required fields + placeholders
  • _template_full.yaml — all fields with schema defaults + inline # options: comments

Package Management

Adding Dependencies

Use uv add <package>==<version> to update both pyproject.toml and uv.lock atomically (always pin to exact versions with ==):

  • Runtime dependencies: [project.dependencies]
  • Optional groups (dev, test, etc.): [project.optional-dependencies]

After adding a dependency, run pip-audit to verify it has no known vulnerabilities:

uv run pip-audit --ignore-vuln PYSEC-2026-3740

PYSEC-2026-3740 is an nltk advisory with no fixed release whose affected APIs this repo doesn't use; CI ignores it too. Drop the flag once #520 is resolved.

Performance Considerations

Code in load_generator/, endpoint_client/worker.py, and async_utils/transport/ is latency-critical. In these paths:

  • No match statements — use dict dispatch
  • Use dataclass(slots=True) or msgspec.Struct for frequently instantiated classes
  • Minimize async suspends
  • Use msgspec over json/pydantic for serialization
  • The HTTP client uses custom ConnectionPool with httptools parser — not aiohttp/requests

Debugging

# Run with verbose logging
uv run inference-endpoint -v benchmark offline ...

# Run tests with stdout visible
uv run pytest -xvs tests/unit/path/to/test.py

# Use Python debugger
uv run python -m pdb -m pytest tests/unit/path/to/test.py

Getting Help