cocoindex-code supports a variety of embedding models and providers. This guide helps you choose the right path for your hardware, privacy requirements, and codebase size.
- Which Path Should I Choose?
- Understanding Speed, Context, and Performance
- The
ccc initWizard - Local Sentence-Transformers
- LiteLLM Remote (Cloud Providers)
- LiteLLM Local
- Choosing Based on Your Content
- Pacing & Rate Limits
| Path | Best For... | Key Advantage | Trade-off |
|---|---|---|---|
| Local Sentence-Transformers | Most users, laptops, quick setup. | Fastest, private, offline. | Larger initial pip install ([full]). |
| Cloud LiteLLM Remote | Large codebases, weak local hardware. | Top performance, zero local resource usage. | Per-token costs, data leaves machine. |
| Local LiteLLM | Power users, shared GPU resources. | Flexibility, unified model management. | Requires managing a separate server. |
- Local Sentence-Transformers: Typically the fastest option for small-to-medium encoder models. It avoids network latency and the server overhead of Local Servers (Ollama). On Apple Silicon, MPS inference runs in a supervised child process so Metal allocations can be reclaimed independently; other devices use the normal in-process path. Decoder-based models (e.g.
harrier-oss-v1) process tokens sequentially — expect 3–10× slower indexing per chunk on CPU unless you have a GPU. - Local Servers (Ollama): Ideal for running heavy models (like
mxbai-embed-large) on a GPU. While it has slight overhead compared to in-process execution, it is much faster than running large models on a CPU. - Cloud APIs: Slower per-request due to network latency, but highly parallel. Best for the initial indexing of massive repositories.
Most local models have a 512-token context window, while cloud models (OpenAI, Voyage) support 8k to 32k.
In cocoindex-code, this matters less than you might expect due to our Language-Aware Chunking strategy:
- Logical Boundaries: The tool uses Tree-Sitter to understand code structure. It tries to split files at logical boundaries like functions, classes, or methods.
- Target Size: While respecting boundaries, it targets a chunk size of ~1,000 characters (~300 tokens).
- Compatibility: This hybrid approach ensures code snippets are contextually coherent while remaining small enough to fit perfectly within even the smallest 512-token context windows.
Indexing speed depends on both parameter count and model architecture:
- Encoder models (BERT/ModernBERT-based, e.g. Snowflake, LateOn-Code, CodeSearch-ModernBERT): Process all tokens in parallel — fast on any modern CPU. The default
xsand all models marked Fast or Very Fast in the table below are encoder-based. - Decoder models (LLM-based, e.g.
harrier-oss-v1): Process tokens sequentially, making them 3–10× slower than an encoder of comparable parameter count on CPU. A GPU is strongly recommended to achieve acceptable indexing throughput with these models.
For encoder models at medium or large sizes, a GPU will still accelerate indexing. Add device: cuda (or mps on Mac) to global_settings.yml.
On Apple Silicon, SentenceTransformer calls run through CocoIndex's isolated
GPU subprocess.
This keeps the model warm across batches while separating Metal allocations
from the daemon. cocoindex-code configures PyTorch to begin adaptive allocation
cleanup at 40% of its recommended maximum working set and to enforce a hard
limit at 50%; tune these values with mps_low_watermark_ratio and
mps_high_watermark_ratio. At the end of each index run, unused allocator cache
is released inside the GPU subprocess. CocoIndex also retries MPS OOM failures
with smaller batches.
See the PyTorch MPS environment variable reference
for the allocator semantics. Explicit environment variables take precedence
over the YAML defaults.
The easiest way to configure embeddings is by running ccc init. On first run, it will guide you through an interactive wizard. To reconfigure later, delete ~/.cocoindex_code/global_settings.yml and re-run.
- Provider Selection: Choose between
sentence-transformers(local, free) orlitellm(cloud/local server). - Model Selection: Enter a HuggingFace ID or a LiteLLM model string (e.g.,
voyage/voyage-code-3). - Automatic Tuning:
ccc initwill automatically apply curated defaults (likeinput_typeorprompt_name) and test the connection.
This option runs embedding models directly on your machine using the library.
These are based on MTEB datasets as of 15-Jun-2026. All listed models have been verified to work with the sentence-transformers provider in cocoindex-code.
| Tier | Model | Params | Dims | Code Score | Arch | CPU Speed | Best For |
|---|---|---|---|---|---|---|---|
| Default | Snowflake/arctic-embed-xs |
22M | 384 | 0.67 | Enc | Very Fast | Smallest, most compatible default |
| Micro | lightonai/LateOn-Code-edge |
17M | 256 | 0.82 | Enc | Crazy Fast (231 cps) | Efficiency King. Incredible code performance for its size. |
| Small | lightonai/LateOn-Code |
149M | 768 | 0.85 | Enc | Fast (7 cps) | Great balance of speed and accuracy on modern laptops. |
| Medium | Shuu12121/CodeSearch-ModernBERT-Crow-Plus |
152M | 0.89 | Enc | Fast | High accuracy with encoder speed; best for CPU-only indexing at this tier. | |
| Larger | microsoft/harrier-oss-v1-270m |
270M | 640 | 0.90 | Dec | Slow | Highest local accuracy; GPU strongly recommended for acceptable speed. |
| Multi-Lingual | ibm-granite/granite-embedding-97m-multilingual-r2 |
97M | 384 | 0.80 | Enc | Fast | Multilingual codebases (e.g. Code + Docs in different languages). |
The default of Snowflake/arctic-embed-xs is a good choice in most situations, but if you want other options...
- Discovery Script: The easiest way is to run our included script to find the current best models for your hardware:
uv run scripts/find_best_models.py. - MTEB v3 Leaderboard: For manual discovery, visit the MTEB v3 Leaderboard:
- Go to the Benchmarks tab
- Select Code Information Retrieval (CoIR).
- Filter Model Type to Dense.
- Enable the Sentence-Transformers Compatible toggle.
- Adjust Model Size to fit your hardware (e.g.,
< 500Mfor CPUs).
- Compatibility: Look for Bi-encoders with a fixed dimension size (e.g., 384, 768, 1024). Avoid "Late Interaction" (ColBERT) or "Cross-Encoders".
- Architecture: Encoder models (BERT/ModernBERT-based) are much faster on CPUs than Decoder models (LLM-based). A decoder model of the same parameter count can be 3–10× slower per chunk due to sequential token processing. Prefer encoders for CPU-only workloads.
Install with the full extra: pip install "cocoindex-code[full]".
Example global_settings.yml:
embedding:
provider: sentence-transformers
model: lightonai/LateOn-Code-edge
device: cpu # Use 'cuda' or 'mps' if you have a GPUFor more information, see the Sentence-Transformers Documentation.
Use external API providers for high-quality embeddings via the LiteLLM bridge.
Well ranked in the MTEB v3 benchmarks
- Voyage AI (
voyage-4-large): Current #1 for code (Score: 0.97). - Gemini (
text-embedding-004): Top-tier performance (Score: 0.97) with a generous free tier. - OpenAI (
text-embedding-3-small): Reliable and very cost-effective for large codebases.
Example for Voyage AI:
embedding:
provider: litellm
model: voyage/voyage-4-large
envs:
VOYAGE_API_KEY: your-api-key-hereFor more information, see the LiteLLM Providers Documentation.
Connect to a local embedding server (Ollama, llama.cpp or compatible) for privacy and flexibility.
Ensure Ollama is running and you have pulled the model (ollama pull jina/jina-embeddings-v5).
Suggested Models:
- Low-end:
ollama/all-minilm - Mid-range:
ollama/jina-embeddings-v5 - High-end:
ollama/mxbai-embed-large
Configuration:
embedding:
provider: litellm
model: ollama/jina-embeddings-v5See the Ollama Model Library for more options.
If you prefer running llama.cpp directly (e.g., using llama-server), you can connect via the OpenAI-compatible interface.
Configuration:
- Start your server:
llama-server --embedding -m your_model.gguf - Configure
global_settings.yml:
embedding:
provider: litellm
model: openai/your-model-name
envs:
OPENAI_API_BASE: http://localhost:8080/v1
OPENAI_API_KEY: "not-needed"- Heavy Source Code, CPU-only: Use LateOn-Code (Micro/Small) or CodeSearch-ModernBERT-Crow-Plus (Medium Encoder). Both are encoder-based and score >0.85 on code search benchmarks with good CPU throughput.
- Heavy Source Code, GPU available: Harrier 270m (Medium Decoder, 0.90) gives the highest local accuracy and is much faster with a GPU.
- Large Documentation / Files: Models with large context windows like Voyage 4 Large (Cloud) or OpenAI v3 Large (8k).
- Multilingual Projects: Granite 97m (Small Local) or Cohere Multilingual v3 (Cloud).
The ccc init wizard will automatically apply recommended defaults for known models.
Example for LiteLLM (Voyage, Gemini):
embedding:
provider: litellm
model: voyage/voyage-4-large
indexing_params:
input_type: document
query_params:
input_type: queryExample for Sentence-Transformers (symmetric models — Harrier, CodeSearch-ModernBERT, etc.):
Most local models don't require asymmetric prompts. If ccc init doesn't auto-detect defaults
for your model, an explicit null disables any inherited prompt:
embedding:
provider: sentence-transformers
model: microsoft/harrier-oss-v1-270m # or Shuu12121/CodeSearch-ModernBERT-Crow-Plus
indexing_params:
prompt_name: null
query_params:
prompt_name: nullWhen using cloud providers, you often encounter rate limits (number of requests per minute). cocoindex-code provides several mechanisms to manage this:
min_interval_ms(Pacing): Introduces a mandatory delay between requests (e.g.,500for 2 req/sec).- Automatic Retries: The daemon automatically retries rate-limited requests (429 errors) with exponential backoff (up to 6 times).
- Batching:
cocoindex-codeautomatically batches up to 64 text chunks into a single API request to maximize throughput.