Performance Guide

Performance Metrics

Typical Performance Characteristics

Metric	Small Project (<1k files)	Medium Project (1k-10k files)	Large Project (10k+ files)
Indexing Speed	500+ files/second	200-400 files/second	100-200 files/second
Search Latency	<50ms	<100ms	<200ms
Memory Usage	50-100MB	100-500MB	500MB-2GB
Storage Size	1-10MB	10-100MB	100MB-1GB
Startup Time	<1s	1-3s	3-10s

Factors Affecting Performance

File Size: Larger files take longer to parse and embed
Language Complexity: Complex languages (C++, TypeScript) slower than simple ones (JSON, Markdown)
Embedding Model: Local models faster than cloud APIs
Hardware: CPU, RAM, and storage speed impact performance
Network: Cloud embedding providers depend on network latency

Recent Performance Improvements

2. Vector Index Optimization (v0.9.1+)

New Feature: Automatic vector index optimization based on dataset characteristics.

How it works:

Smart Index Creation: Skips indexing for small datasets (< 1K rows) where brute force is faster
Optimal Parameters: Automatically calculates partitions, sub-vectors, and search parameters
Growth-Aware: Recreates indexes with better parameters as datasets grow
Consistent Distance: Always uses Cosine distance for semantic similarity

Performance Impact:

Small datasets (< 1K rows): Brute force search (fastest)
Medium datasets (1K-100K rows): Optimized IVF_PQ index with intelligent parameters
Large datasets (> 100K rows): Growth-aware optimization with enhanced recall
Search queries: Automatic nprobes (5-15% of partitions) + refine_factor for better accuracy

Technical Details:

// Automatic optimization - no configuration required
let params = VectorOptimizer::calculate_index_params(row_count, vector_dimension);
if params.should_create_index {
    // Creates optimized index with calculated parameters
    table.create_index(&["embedding"], Index::IvfPq(IvfPqIndexBuilder::default()
        .num_partitions(params.num_partitions)
        .num_sub_vectors(params.num_sub_vectors)
        .num_bits(params.num_bits)
        .distance_type(DistanceType::Cosine)
    )).await?;
}

3. File Discovery Optimization (v0.8.1+)

Issue Fixed: The indexing process was extremely slow during the "Found X files..." counting phase, especially for large codebases.

Root Cause: The NoindexWalker::has_noindex_files() function was performing expensive full directory tree traversal to check for .noindex files on every indexing run.

Solution:

Replaced O(n) tree traversal with O(1) targeted file system checks
Added thread-safe caching using OnceLock<RwLock<HashMap>>
Now checks only common directories: src, lib, tests, docs, target, etc.

Performance Impact:

Before: Could take minutes for large codebases
After: Instant startup, 100x-1000x improvement
Compatibility: 100% backward compatible with all .gitignore and .noindex functionality

Technical Details:

// Fast targeted checks instead of full tree traversal
let common_paths = ["src", "lib", "tests", "docs", "target", "build", "dist"];
for subdir in &common_paths {
    if current_dir.join(subdir).join(".noindex").exists() {
        return true;
    }
}

Optimization Strategies

1. Embedding Model Selection

For Speed (Local Models)

# FastEmbed - Fastest local option
# Model validation happens at startup - invalid models fail fast

octocode config \
  --code-embedding-model "fastembed:BAAI/bge-small-en-v1.5" \
  --text-embedding-model "fastembed:multilingual-e5-small"

# Optimized configuration
[embedding]
code_model = "fastembed:BAAI/bge-small-en-v1.5"  # 384 dim, fast
text_model = "fastembed:multilingual-e5-small"   # 384 dim, multilingual

For Maximum Quality (Cloud)

# High-quality cloud models (requires API keys)
# Default models - no configuration needed

octocode config \
  --code-embedding-model "voyage:voyage-code-3" \
  --text-embedding-model "voyage:voyage-3.5-lite"

# Alternative: Jina models
[embedding]
code_model = "jina:jina-embeddings-v4"
text_model = "jina:jina-embeddings-v3"

Note: FastEmbed models require the fastembed feature flag. Cloud models (voyage, jina, google) work with --no-default-features.

Speed-Optimized Settings

[index]
chunk_size = 1000                # Smaller chunks process faster
embeddings_batch_size = 64       # Larger batches for efficiency
quantization = true              # RaBitQ 32x compression (faster than IVF_HNSW_SQ)
contextual_descriptions = false  # Disable for faster indexing

[search]
max_results = 20                 # Limit results for faster response
similarity_threshold = 0.65      # Higher threshold = fewer, more relevant results

Quality-Optimized Settings

[index]
chunk_size = 2000                # Larger chunks for better context
embeddings_batch_size = 32       # Smaller batches for stability
quantization = true              # Still uses compression, just more data
contextual_descriptions = true   # Enable AI-enriched chunks (requires LLM config)
contextual_model = "openrouter:openai/gpt-4o-mini"

[search]
max_results = 50                 # More comprehensive results
similarity_threshold = 0.3       # Lower threshold = more results

3. Vector Database Optimization

Automatic Optimization (Default)

Octocode automatically optimizes vector indexes based on your dataset size and characteristics. No configuration required.

What happens automatically:

Small datasets (< 1K rows): Uses brute force search (fastest)
Medium datasets (1K-100K rows): Creates optimized IVF_PQ indexes
Large datasets (> 100K rows): Recreates indexes at growth milestones
Search parameters: Automatically calculated for best recall/latency balance

Manual Tuning (Advanced)

For specific use cases, you can influence performance through configuration:

[search]
max_results = 20                 # Limit results for faster response
similarity_threshold = 0.65      # Higher threshold = fewer, more relevant results

[index]
embeddings_batch_size = 16       # Batch size for embedding generation
flush_frequency = 2              # How often to flush to disk

Note: Vector index parameters (partitions, sub-vectors, etc.) are automatically calculated and cannot be manually configured.

4. Hardware Optimization

CPU Optimization

Multi-core: Embedding generation uses multiple cores
CPU Type: Modern CPUs with AVX2 support perform better
Recommended: 4+ cores for optimal performance

Search Optimization

[search]
max_results = 30                 # Reduce for lower memory usage

Storage Optimization

SSD: Significantly faster than HDD for database operations
NVMe: Best performance for large codebases
Network Storage: Avoid for database files

5. Network Optimization (Cloud Providers)

Reduce API Calls

# Use local models when possible
octocode config --code-embedding-model "fastembed:all-MiniLM-L6-v2"

# Batch operations
octocode clear && octocode index  # Index all at once vs incremental

API Rate Limiting & LLM Batching

[embedding]
# Adjust batch sizes for API limits
embeddings_batch_size = 16       # Smaller batches for cloud APIs

[graphrag.llm]
# Optimize AI call costs and rate limits
ai_batch_size = 8               # Process multiple files per AI call
max_batch_tokens = 16384        # Stay within model context limits
fallback_to_individual = true   # Reliability if batch processing fails

LLM Cost Optimization: The ai_batch_size parameter significantly reduces API costs by processing multiple files in a single request. With the default value of 8, you get ~87% fewer API calls compared to individual processing. Increase for more savings, decrease if hitting rate limits.

Performance Monitoring

Built-in Metrics

# Enable debug logging for performance metrics
RUST_LOG=debug octocode index

# Monitor indexing progress
octocode clear && octocode index 2>&1 | grep "Processed"

# Check database size
ls -lh ~/.local/share/octocode/

Custom Monitoring

#!/bin/bash
# Performance monitoring script

echo "=== Octocode Performance Report ==="
echo "Date: $(date)"
echo

# Database size
echo "Database size:"
du -sh ~/.local/share/octocode/

# Index timing
echo "Indexing performance:"
time (octocode clear && octocode index)

# Search timing
echo "Search performance:"
time octocode search "authentication" > /dev/null

# Memory usage
echo "Memory usage during search:"
/usr/bin/time -v octocode search "database" > /dev/null 2>&1 | grep "Maximum resident"

Troubleshooting Performance Issues

Slow Indexing

Symptoms: Indexing takes much longer than expected

Solutions:

Reduce chunk size: chunk_size = 1000
Use faster embedding model: Switch to FastEmbed
Disable contextual descriptions: contextual_descriptions = false
Check disk space: Ensure sufficient free space
Monitor CPU usage: Ensure no other heavy processes

# Quick fix for slow indexing
octocode config --code-embedding-model "fastembed:BAAI/bge-small-en-v1.5"
octocode config --contextual-descriptions false
octocode clear && octocode index

Symptoms: Search queries take several seconds

Solutions:

Increase similarity threshold: similarity_threshold = 0.3
Reduce max results: max_results = 20
Check database corruption: octocode clear && octocode index
Optimize query: Use more specific search terms

# Quick fix for slow search
octocode config --max-results 20
octocode config --similarity-threshold 0.3

High Memory Usage

Symptoms: Octocode uses excessive RAM

Solutions:

Clear old data: octocode clear
Use smaller embedding models: Switch to 384-dim models
Limit search results: max_results = 20

# Quick fix for memory issues
octocode config --max-results 20
octocode clear
octocode config --code-embedding-model "fastembed:all-MiniLM-L6-v2"

API Rate Limiting

Symptoms: Errors from cloud embedding providers

Solutions:

Reduce batch size: embeddings_batch_size = 8
Add delays: Use local models for development
Switch providers: Try different cloud providers
Use local models: Switch to FastEmbed/SentenceTransformer

# Quick fix for rate limiting
octocode config --code-embedding-model "fastembed:all-MiniLM-L6-v2"
octocode config --text-embedding-model "fastembed:multilingual-e5-small"

Benchmarking

Standard Benchmark

#!/bin/bash
# Octocode benchmark script

PROJECT_PATH="/path/to/test/project"
cd "$PROJECT_PATH"

echo "=== Octocode Benchmark ==="
echo "Project: $PROJECT_PATH"
echo "Files: $(find . -type f -name "*.rs" -o -name "*.py" -o -name "*.js" | wc -l)"
echo

# Clear previous data
octocode clear

# Benchmark indexing
echo "Indexing benchmark:"
time octocode index

# Benchmark search
echo "Search benchmark (10 queries):"
queries=("authentication" "database" "API" "error handling" "configuration" "testing" "middleware" "validation" "logging" "security")

for query in "${queries[@]}"; do
    echo -n "Query '$query': "
    time octocode search "$query" > /dev/null
done

# Database size
echo "Final database size:"
du -sh ~/.local/share/octocode/

Performance Comparison

Configuration	Indexing (1000 files)	Search Latency	Memory Usage	Quality Score
FastEmbed	45s	50ms	200MB	7/10
Jina (cloud)	60s	80ms	250MB	8.5/10
Voyage (cloud)	75s	90ms	300MB	9/10
Google (cloud)	60s	85ms	280MB	8.5/10

Note: Cloud model times include API latency. FastEmbed runs locally but requires the fastembed feature flag.

Best Practices

Development Workflow

Use local models during development
Enable cloud models for production/final indexing
Regular cleanup: octocode clear periodically
Monitor performance: Track indexing and search times

Production Deployment

Optimize for your use case: Speed vs quality tradeoff
Monitor resource usage: CPU, memory, storage
Plan for scaling: Consider hardware requirements
Backup strategy: Regular database backups

Configuration Templates

Development (Speed Focus)

[embedding]
code_model = "fastembed:BAAI/bge-small-en-v1.5"
text_model = "fastembed:multilingual-e5-small"

[index]
chunk_size = 1000
contextual_descriptions = false

[search]
max_results = 20
similarity_threshold = 0.65

Production (Quality Focus)

[embedding]
code_model = "voyage:voyage-code-3"
text_model = "voyage:voyage-3.5-lite"

[index]
chunk_size = 2000
contextual_descriptions = true
contextual_model = "openrouter:openai/gpt-4o-mini"

[search]
max_results = 50
similarity_threshold = 0.3

Large Scale (Balanced)

[embedding]
code_model = "fastembed:BAAI/bge-small-en-v1.5"
text_model = "fastembed:multilingual-e5-small"

[index]
chunk_size = 1500
contextual_descriptions = true

[search]
max_results = 30
similarity_threshold = 0.5

Uh oh!

FilesExpand file tree

PERFORMANCE.md

Latest commit

History

PERFORMANCE.md

File metadata and controls

Performance Guide

Performance Metrics

Typical Performance Characteristics

Factors Affecting Performance

Recent Performance Improvements

2. Vector Index Optimization (v0.9.1+)

3. File Discovery Optimization (v0.8.1+)

Optimization Strategies

1. Embedding Model Selection

For Speed (Local Models)

For Maximum Quality (Cloud)

Speed-Optimized Settings

Quality-Optimized Settings

3. Vector Database Optimization

Automatic Optimization (Default)

Manual Tuning (Advanced)

4. Hardware Optimization

CPU Optimization

Search Optimization

Storage Optimization

5. Network Optimization (Cloud Providers)

Reduce API Calls

API Rate Limiting & LLM Batching

Performance Monitoring

Built-in Metrics

Custom Monitoring

Troubleshooting Performance Issues

Slow Indexing

High Memory Usage

API Rate Limiting

Benchmarking

Standard Benchmark

Performance Comparison

Best Practices

Development Workflow

Production Deployment

Configuration Templates

Development (Speed Focus)

Production (Quality Focus)

Large Scale (Balanced)