Skip to content

on_boarding.md

Latest commit

 

History

History
116 lines (79 loc) · 11.7 KB

on_boarding.md

File metadata and controls

116 lines (79 loc) · 11.7 KB
Raw
graph LR
    Model_Management_Definition["Model Management & Definition"]
    ML_Compilation_Pipeline["ML Compilation Pipeline"]
    MLCEngine_Inference_Runtime_["MLCEngine (Inference Runtime)"]
    API_Server_Protocol["API Server & Protocol"]
    Deployment_Tooling["Deployment & Tooling"]
    Benchmarking_Evaluation["Benchmarking & Evaluation"]
    Model_Management_Definition -- "Provides model weights and architectural definitions." --> ML_Compilation_Pipeline
    ML_Compilation_Pipeline -- "Outputs compiled model artifacts." --> MLCEngine_Inference_Runtime_
    MLCEngine_Inference_Runtime_ -- "Processes inference requests." --> API_Server_Protocol
    Deployment_Tooling -- "Initiates model preparation." --> Model_Management_Definition
    Deployment_Tooling -- "Invokes compilation." --> ML_Compilation_Pipeline
    Deployment_Tooling -- "Manages runtime deployment." --> MLCEngine_Inference_Runtime_
    Benchmarking_Evaluation -- "Submits inference benchmarks." --> API_Server_Protocol
    Benchmarking_Evaluation -- "Collects runtime performance data." --> MLCEngine_Inference_Runtime_
    API_Server_Protocol -- "References model schemas." --> Model_Management_Definition
    click Model_Management_Definition href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/mlc-llm/Model_Management_Definition.md" "Details"
    click ML_Compilation_Pipeline href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/mlc-llm/ML_Compilation_Pipeline.md" "Details"
    click API_Server_Protocol href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/mlc-llm/API_Server_Protocol.md" "Details"
    click Deployment_Tooling href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/mlc-llm/Deployment_Tooling.md" "Details"
    click Benchmarking_Evaluation href "https://github.com/CodeBoarding/GeneratedOnBoardings/blob/main/mlc-llm/Benchmarking_Evaluation.md" "Details"
Loading

CodeBoardingDemoContact

Details

The MLC LLM project is structured around a core set of components designed for efficient large language model (LLM) compilation, serving, and deployment. The Model Management & Definition component acts as the initial entry point, handling the acquisition and structural definition of LLM models and their associated tokenizers. These model definitions are then fed into the ML Compilation Pipeline, which optimizes and transforms them into deployable artifacts through processes like quantization and memory estimation. The resulting compiled models are consumed by the MLCEngine (Inference Runtime), the central component responsible for loading models, managing the KV cache, and executing inference requests across various hardware targets.

For external interaction, the API Server & Protocol exposes the MLCEngine's capabilities via a standardized API, primarily an OpenAI-compatible interface, handling request parsing, response formatting, and routing of inference requests to the MLCEngine. The Deployment & Tooling component provides command-line interfaces and utilities for developers, enabling model conversion, library delivery, compilation invocation, and server management. Finally, the Benchmarking & Evaluation component offers tools and datasets for systematically assessing the performance and correctness of the MLCEngine and compiled models, submitting inference benchmarks to the API Server & Protocol and collecting performance data directly from the MLCEngine. This architecture ensures a streamlined workflow from model definition and compilation to efficient serving and rigorous evaluation.

Model Management & Definition [Expand]

Manages the acquisition, loading, and structural definition of LLM models, including tokenizer information and neural network architectures. It serves as the initial entry point for raw model assets and their blueprints.

Related Classes/Methods:

ML Compilation Pipeline [Expand]

The core compiler that transforms raw or preprocessed LLM models into optimized, deployable artifacts. This includes quantization, memory estimation, and various compiler passes, leveraging highly optimized low-level operations.

Related Classes/Methods:

MLCEngine (Inference Runtime)

The central runtime engine responsible for loading compiled models, managing KV cache, and executing inference requests efficiently across various hardware targets. It serves as the unified inference layer.

Related Classes/Methods:

API Server & Protocol [Expand]

Exposes the MLCEngine's capabilities through a standardized API, primarily an OpenAI-compatible interface. It handles request parsing, response formatting, and routing of inference requests.

Related Classes/Methods:

Deployment & Tooling [Expand]

Provides command-line interfaces and utilities for developers to interact with the ML Toolkit, including model conversion, library delivery, compilation invocation, server management, and packaging for deployment on various target platforms.

Related Classes/Methods:

Benchmarking & Evaluation [Expand]

Contains tools and datasets for systematically evaluating the performance, correctness, and efficiency of the MLCEngine and compiled models across different configurations and hardware.

Related Classes/Methods:

FAQ