Summary
disclaimer: i used ai to write this feture request
Support running large Mixture-of-Experts models whose total size exceeds available RAM/NPU memory by keeping only the dense components (attention, router, shared layers) plus a bounded cache of "hot" routed experts resident, and streaming the remaining experts from storage on demand — the same approach used by [Colibrì](https://github.com/JustVugg/colibri) for GLM-5.2 (744B MoE) on desktop hardware.
Motivation
Only a small subset of experts activates per token in a MoE model, so most expert weights sit idle at any given moment. On memory-constrained devices this makes MoE models in the 60B+ range currently out of reach even though only a few billion parameters are active per token. Expert streaming would let GenieX run these models on hardware that can't hold the full weight set resident.
Current behavior
As far as I can tell from the docs/README, neither GenieX runtime supports this today:
llama_cpp runtime (GGUF, NPU/GPU/CPU via Hexagon HTP kernels): loads the full model; no expert-level lazy loading or disk streaming.qairt runtime (pre-compiled AI Hub bundles, NPU only): ships an ahead-of-time compiled, per-chipset graph, which by design fixes memory layout before inference — no obvious hook for runtime-decided expert paging.
Proposed feature
For the llama_cpp backend specifically: expose expert-level streaming analogous to what's landing upstream in llama.cpp itself —
- resident: attention, router, embeddings, norms, KV cache, plus an LRU (or similar) cache of N expert slots per layer
- on-demand: fetch an expert from the GGUF file when the router selects one not already cached
- prefetch: overlap I/O with compute where possible
Relevant upstream prior art:
- ggml-org/llama.cpp PR #25294 — optional SSD streaming of MoE routed experts, O_DIRECT reads, async I/O worker pool
- ggml-org/llama.cpp Discussion #23324 — on-demand expert paging PoC (Metal/Apple Silicon)
- ggml-org/llama.cpp Discussion #27149 — expert-aware SSD streaming discussion thread
Open questions for maintainers
- Does the Hexagon HTP GGML backend have any architectural blocker to supporting the custom op(s) the upstream streaming PR introduces, or is this mainly a "wire it up once upstream merges" situation?
- For the
qairt/NPU path: is runtime expert paging fundamentally incompatible with the AOT-compiled bundle model, or is there a compilation mode that could support a bounded expert cache?
- Is device storage (UFS on mobile, or whatever the Dragonwing QCS9075 IoT target uses) expected to be a hard bottleneck here, independent of software support? Curious whether the IoT/Linux ARM64 target has meaningfully better storage throughput than phone-class UFS.
Summary
disclaimer: i used ai to write this feture request
Support running large Mixture-of-Experts models whose total size exceeds available RAM/NPU memory by keeping only the dense components (attention, router, shared layers) plus a bounded cache of "hot" routed experts resident, and streaming the remaining experts from storage on demand — the same approach used by [Colibrì](https://github.com/JustVugg/colibri) for GLM-5.2 (744B MoE) on desktop hardware.
Motivation
Only a small subset of experts activates per token in a MoE model, so most expert weights sit idle at any given moment. On memory-constrained devices this makes MoE models in the 60B+ range currently out of reach even though only a few billion parameters are active per token. Expert streaming would let GenieX run these models on hardware that can't hold the full weight set resident.
Current behavior
As far as I can tell from the docs/README, neither GenieX runtime supports this today:
llama_cppruntime (GGUF, NPU/GPU/CPU via Hexagon HTP kernels): loads the full model; no expert-level lazy loading or disk streaming.qairtruntime (pre-compiled AI Hub bundles, NPU only): ships an ahead-of-time compiled, per-chipset graph, which by design fixes memory layout before inference — no obvious hook for runtime-decided expert paging.Proposed feature
For the
llama_cppbackend specifically: expose expert-level streaming analogous to what's landing upstream in llama.cpp itself —Relevant upstream prior art:
Open questions for maintainers
qairt/NPU path: is runtime expert paging fundamentally incompatible with the AOT-compiled bundle model, or is there a compilation mode that could support a bounded expert cache?