Skip to content

ROADMAP.md

Latest commit

 

History

History
125 lines (100 loc) · 6.76 KB

ROADMAP.md

File metadata and controls

125 lines (100 loc) · 6.76 KB
Raw

model-xl8r — Roadmap

Phased plan, ordered by leverage. Each phase is "ship-and-test before moving on".

Phase 0 — Migration (✅ done)

  • Project scaffold (pyproject.toml, src/xl8r/, CLI dispatcher)
  • Migrate nemotron-encode/scripts/export_nemotron_gguf.py → xl8r/safetensor2gguf.py
  • Generic Llama-3-family path (Llama 3, 3.1, 3.2, 3.3, Nemotron-VL, Hermes, etc.)
  • Multimodal-VL split (mmproj + language) for Nemotron-VL pattern
  • Tokenizer write (gpt2 model + llama-bpe pre + merges + bos/eos/pad + chat_template)
  • Vocab off-by-one handling (matches llama.cpp convert_hf_to_gguf.py policy)

Phase 1 — Round-trip & validation (next)

  • gguf2safetensor — migrate nemotron-encode/scripts/gguf_to_safetensors.py
    • Add --also-pytorch-bin flag for legacy HF format
    • Test round-trip (HF → GGUF → HF) numerical equivalence within quant noise
  • validate — migrate nemotron-encode/scripts/validate_gguf.py
    • STS-B correlation vs FP32 reference
    • Per-quant quality table
  • quantize — migrate nemotron-encode/scripts/quantize_model.py
    • Expose IQ-quants (IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S, IQ3_M, IQ4_NL, IQ4_XS)
    • Expose K-quants (Q2_K, Q3_K_S/M/L, Q4_K_S/M, Q5_K_S/M, Q6_K)
    • iMatrix calibration + HAWQ-V2 + BRECQ recipes
  • safetensor2safetensor — cross-format conversion
    • regular ↔ MLX (mlx_lm.utils.dequantize/quantize)
    • regular ↔ AWQ (autoawq unpack/pack)
    • regular ↔ CompressedTensors (compressed-tensors unpack/pack)
    • regular ↔ GPTQ (via optimum-quanto or auto-gptq)
    • regular ↔ BitsAndBytes (8-bit / NF4)

Phase 2 — Pre-quantized inputs

Goal: accept already-quantized HF safetensors and emit GGUF without going through FP32 → quant ourselves.

  • xl8r/formats/awq.py — detect AWQ, dequant via autoawq.utils.packing.unpack_awq, then re-quant to target GGUF format
  • xl8r/formats/mlx.py — detect MLX (group-wise affine), dequant via mlx.core.dequantize
  • xl8r/formats/compressed_tensors.py — detect CT (w4a16, w8a8, etc.), dequant via compressed-tensors
  • xl8r/formats/gptq.py — detect GPTQ, dequant via auto-gptq
  • Auto interim-F16 pipeline: AWQ/MLX/CT → F16 safetensors → target GGUF quant
    • One CLI flag: safetensor2gguf --auto-format detects input format + emits requested quant
  • Test models on disk:
    • Qwen_Qwen2.5-3B-Instruct-AWQ (AWQ)
    • RedHatAI_Qwen3-0.6B-quantized.w4a16 (CompressedTensors)
    • Any MLX model from HF (need to download — small 1B class)

Phase 3 — Architecture coverage

Test the safetensor2gguf path against varied architectures, fix per-arch quirks (xl8r/formats/llama.py, gemma.py, qwen.py, gpt_oss.py, mistral.py).

Model Arch Test plan
google_gemma-4-E2B-it gemma3 dense Convert → llama.cpp loads → matches HF eval
google_gemma-4-E4B-it gemma3 dense Same as E2B
Qwen_Qwen3-Embedding-0.6B qwen3 (encoder) Embed test
Qwen_Qwen3-Embedding-8B qwen3 (encoder) Embed test (large)
Qwen_Qwen3.5-9B qwen3 dense Generation test
gpt-oss 20B / 120B gpt-oss MoE Confirm 20b vs 120b packing identical (the user's open question)
nomic-ai_modernbert-embed-base modernbert Embed test
BAAI_bge-base-en-v1.5 bert Embed test

Phase 4 — Modality coverage

  • Audio mmproj — Whisper-style audio encoder + adapter
    • Detect via audio_config in config.json
    • Test: Qwen2-Audio-7B, Whisper Large-v3
  • Video mmproj — Qwen2.5-VL temporal patches, InternVideo
    • Detect via temporal patch params
  • Diffusion models — SD3, Flux, separate code path (UNet/MMDiT, not LLM)
    • Test: unsloth_FLUX.2-klein-9B-GGUF (already on disk in GGUF — round-trip back?)
    • Output: GGUF compatible with stable-diffusion.cpp
  • ColBERT — late-interaction encoder, custom head
    • Test: colbert-ir_colbertv2.0
  • ColPali — multimodal ColBERT (vision tokens)
    • Test: vidore_colpali-v1.3
  • Factcheck — T5-based aggregation
    • Test: Linuss4880_MiniCheck-Flan-T5-Large-Q8_0-GGUF (already GGUF — round-trip)

Phase 5 — Cross-format converters

  • xl8r/crossfmt/onnx.py — HF Optimum forward + custom reverse
  • xl8r/crossfmt/openvino.py — optimum-intel for IR conversion + NNCF for INT8/INT4
  • xl8r/crossfmt/coreml.py — coremltools for .mlpackage (Apple ANE-friendly)
  • xl8r/crossfmt/pytorch_bin.py — legacy torch.save({}, 'pytorch_model.bin')
  • Obscure NPU/accelerator formats (research first, implement on demand):
    • Qualcomm Snapdragon NPU (QNN)
    • Intel NPU (OpenVINO + IR)
    • AMD XDNA (XCLBIN)
    • Google Edge TPU (.tflite + edgetpu_compiler)
    • Hailo (.hef)

Phase 6 — TurboQuant integration

Vendored research lives under _research/turboquant-* (gitignored). Clean re-implementations land here:

  • xl8r/quants/turboquant.py — base TurboQuant compression
  • xl8r/quants/turboquant_plus.py — TurboQuant+ variant
  • xl8r/quants/back2matching.py — back-to-matching variant
  • xl8r/quants/lloyd_max.py — Lloyd-Max scalar quantizer
  • Integrate into quantize subcommand as --recipe turboquant|tq+|b2m|lloydmax

Testing methodology

For each conversion class:

  1. Convert a real HF model from ~/Projects/models/ or ~/Projects/CLAUDE/ConfluenceSearch2/models/
  2. Load the artifact into the upstream, unmodified runtime (vanilla llama.cpp, onnxruntime, openvino, coremltools, mlx_lm)
  3. Compare output (logits / embeddings / generated tokens) to HF reference within quant noise
  4. Document results in tests/results/<class>.md
  5. Discard the converted artifact (we have limited disk) and move to the next class
  6. Keep a tiny canonical set on-hand: Xenova_all-MiniLM-L6-v2, google_gemma-4-E2B-it, Qwen_Qwen3-Embedding-0.6B, nvidia_llama-nemotron-embed-vl-1b-v2

Non-goals

  • No llama.cpp / onnxruntime / openvino native binary build dependency
  • No custom CUDA / Metal kernels — pure Python + NumPy + Torch
  • No model retraining — converter only, never trains
  • No hosted service — this is a library/CLI, not a SaaS

Open questions

  • gpt-oss packing: is the 20B model packed identically to 120B (same MoE expert layout, same num_experts_per_tok)? Need to inspect both configs side-by-side once a 20B is on disk.
  • TurboQuant license: vendored repos (turboquant-pytorch, turboquant_plus, turboquant-back2matching) are MIT/Apache. Clean re-implementation should keep attribution intact in module headers.
  • HF Optimum vs direct ONNX: Optimum's exports sometimes split into multiple ONNX files (encoder_model.onnx, decoder_model.onnx). Round-tripping back to safetensors needs to recombine. May skip multi-file export.