Vector Database Benchmark Tool

Benchmarks and compares vector database performance for MLPerf Storage. Currently supports Milvus with DiskANN, HNSW, AISAQ, FLAT, and IVF-style indexes.

The benchmark can be run in two ways:

Directly with the scripts in vdb_benchmark/vdbbench/
Through the MLPerf Storage CLI with ./mlpstorage <closed|open|whatif> vectordb

The mlpstorage path is recommended for standard benchmark workflows.

The modular backend-agnostic runner is currently a standalone preview. It is invoked with python -m vdbbench.benchmark. The existing ./mlpstorage <closed|open|whatif> vectordb command continues to use the Milvus-oriented scripts until the modular runner is integrated.

1. Prerequisites
2. Deploy Milvus
- Option A: Local Storage with MinIO
- Option B: S3 Storage
3. Quick Start — First Benchmark in 10 Minutes
4. Recommended Path: mlpstorage CLI
5. Alternative Path: Direct Scripts
6. CLI Reference
7. Distributed Execution (Multi-Node)
8. End-to-End Examples
9. Enhanced Benchmark Full Reference
10. Metrics and Measurement
11. Testing and Validation
12. Troubleshooting
13. Contributing

1. Prerequisites

System Requirements

Requirement	Version	Notes
Python	≥ 3.12	Required
Docker Engine	≥ 20.10	For running Milvus containers
Docker Compose	v2+	`docker compose` (v2 CLI plugin) preferred
Git	Any	To clone the repository
`uv`	Latest	Recommended package manager (install)
MPI (MPICH or OpenMPI)	Any	Only for distributed/multi-node runs; requires `mpi4py ≥ 4.0.0`

Python Dependencies

Package	Version	Purpose
`pymilvus`	≥ 2.4.0	Milvus client
`numpy`	≥ 1.24.3	Vector generation and recall math
`pandas`	≥ 2.0.3	Latency/statistics aggregation
`pyyaml`	≥ 6.0	YAML config support
`tabulate`	≥ 0.9.0	Collection info table display

The datasize command does not require Milvus or pymilvus. Load and run commands require a running Milvus server.

Clone the Repository

git clone https://github.com/mlcommons/storage.git
cd storage

2. Deploy Milvus

A running Milvus instance is required for all load (datagen) and benchmark (run) commands. This section applies to both the mlpstorage CLI and direct script paths.

Standalone Milvus stacks are available in the vdb_benchmark/stacks directory:

vdb_benchmark/stacks/
└── milvus/
    ├── cluster/
    └── standalone/
        ├── minio/
        │   ├── .env.example
        │   └── docker-compose.yml
        └── s3/
            ├── .env.example
            └── docker-compose-s3.yml

For each specific instance, copy the .env.example file to .env and update the values as needed.

Option A: Local Storage with MinIO

cp vdb_benchmark/stacks/milvus/standalone/minio/.env.example \
   vdb_benchmark/stacks/milvus/standalone/minio/.env

The compose file uses /mnt/vdb as the root directory for Docker volumes. Set DOCKER_VOLUME_DIRECTORY in the .env file or edit the compose file to point to your target storage location.

The stack creates three containers:

Milvus database
MinIO object storage
etcd metadata store

Start:

docker compose -f vdb_benchmark/stacks/milvus/standalone/minio/docker-compose.yml up -d

or:

docker-compose -f vdb_benchmark/stacks/milvus/standalone/minio/docker-compose.yml up -d

Option B: S3 Storage

Copy and configure environment (fill in your S3 credentials):

cp vdb_benchmark/stacks/milvus/standalone/s3/.env.example \
   vdb_benchmark/stacks/milvus/standalone/s3/.env

Start:

docker compose -f vdb_benchmark/stacks/milvus/standalone/s3/docker-compose-s3.yml up -d

or:

docker-compose -f vdb_benchmark/stacks/milvus/standalone/s3/docker-compose-s3.yml up -d

Verify Milvus is Healthy

docker ps -a

All three containers (milvus-etcd, milvus-minio, milvus-standalone) should show healthy/running.

The default Milvus endpoint is:

127.0.0.1:19530

3. Quick Start — First Benchmark in 10 Minutes

This section gets you from zero to a working benchmark result on a standalone-system. Assumes Milvus is set up as per section #2 instructions.

Step 1 — Install

cd storage
uv sync --extra vectordb
uv pip install -e ./vdb_benchmark

Verify:

./mlpstorage open vectordb --help

Step 2 — Start Milvus

cp vdb_benchmark/stacks/milvus/standalone/minio/.env.example \
   vdb_benchmark/stacks/milvus/standalone/minio/.env

docker compose -f vdb_benchmark/stacks/milvus/standalone/minio/docker-compose.yml up -d

Wait for healthy status:

docker ps -a

All three containers (milvus-etcd, milvus-minio, milvus-standalone) should show healthy/running.

The default endpoint is 127.0.0.1:19530.

Step 3 — Initialize the results directory

./mlpstorage init MLCommons /tmp/vdb_results

Step 4 — Load 50K vectors (smoke test)

./mlpstorage open vectordb datagen file \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_smoke \
  --num-vectors 50000 \
  --dimension 1536 \
  --num-shards 1 \
  --force \
  --systemname vdb_smoke_system \
  --results-dir /tmp/vdb_results

Step 5 — Run a 30-second benchmark

./mlpstorage open vectordb run file \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_smoke \
  --benchmark-mode timed \
  --runtime 30 \
  --num-query-processes 2 \
  --batch-size 10 \
  --systemname vdb_smoke_system \
  --results-dir /tmp/vdb_results

Step 6 — Check results

python - <<'PY'
import json
from pathlib import Path

stats_files = sorted(
    Path("/tmp/vdb_results").glob(
        "**/vector_database/milvus/DISKANN/run/**/statistics.json"
    )
)
assert stats_files, "No statistics.json found"
stats = json.loads(stats_files[-1].read_text())
print(f"Throughput: {stats['throughput_qps']:.1f} QPS")
print(f"P95 latency: {stats['p95_latency_ms']:.2f} ms")
print(f"Total queries: {stats['total_queries']}")
PY

If you see QPS and latency numbers, your setup is working.

Continue to Section 4 for full documentation.

4. Recommended Path: mlpstorage CLI

This section covers the complete workflow using the mlpstorage CLI. This is the recommended approach for standard benchmark workflows.

4.1 Installation

From the repository root:

cd storage

# Install MLPerf Storage with VectorDB dependencies.
uv sync --extra vectordb

# Install the vdbbench package into the uv-managed environment.
uv pip install -e ./vdb_benchmark

This makes the following commands available:

./mlpstorage open vectordb --help
./mlpstorage open vectordb datasize --help
./mlpstorage open vectordb datagen --help
./mlpstorage open vectordb run --help
./mlpstorage closed vectordb run --help
./mlpstorage whatif vectordb run --help

Before running commands that write benchmark output, initialize the results directory once:

./mlpstorage init MLCommons /tmp/vdb_results

The distributed VectorDB launcher additionally provides:

vdb-mpi-wrapper
vdb-aggregate

These are installed from vdb_benchmark/pyproject.toml.

Verify installation:

uv run vdb-mpi-wrapper --help
uv run vdb-aggregate --help

4.2 Estimate Storage (datasize)

This step is optional. It is pure math and does not require a running Milvus instance.

./mlpstorage open vectordb datasize \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --dimension 1536 \
  --num-vectors 10000000 \
  --num-shards 10

Example output:

Vectors: 10,000,000 x dim=1536 x 4B
Raw data: 61.44 GB
Index type: DISKANN (130% overhead)
Shards: 10
Estimated total: 798.72 GB

4.3 Load Vectors (datagen)

Load using the default config (1M vectors)

./mlpstorage open vectordb datagen file \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_1m_1536dim_uniform_diskann \
  --force \
  --systemname vdb_system \
  --results-dir /tmp/vdb_results

Load using the 10M config

./mlpstorage open vectordb datagen file \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config 10m \
  --collection mlps_10m_1536dim_uniform_diskann \
  --force \
  --systemname vdb_system \
  --results-dir /tmp/vdb_results

Override vector count for quick testing

./mlpstorage open vectordb datagen file \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_smoke_50k \
  --num-vectors 50000 \
  --dimension 1536 \
  --num-shards 1 \
  --force \
  --systemname vdb_system \
  --results-dir /tmp/vdb_results

Notes

The --config argument refers to YAML files in configs/vectordbbench/ without the .yaml extension.
The --force flag drops and recreates the collection if it already exists.
See Dimension Consistency for important rules about keeping dimensions aligned between load and run.
If a distributed load fails with rate limit exceeded[rate=0.1], see Troubleshooting — this is Milvus's per-collection flush rate limiter (issue #705), handled automatically by current benchmark versions.

4.4 Compact

The load script performs compaction automatically when enabled in the config or when --compact is passed. Compaction runs as part of the datagen workflow. No separate command is needed unless the load command exits early.

See Section 5.3 for manual compaction if needed.

4.5 Run Benchmarks

Simple benchmark modes

`--benchmark-mode` value	Script	Purpose
`timed`	`vdbbench`	Run for a fixed duration
`query_count`	`vdbbench`	Run exactly N total queries

Timed mode

./mlpstorage open vectordb run file \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_1m_1536dim_uniform_diskann \
  --benchmark-mode timed \
  --runtime 120 \
  --num-query-processes 4 \
  --batch-size 10 \
  --report-count 100 \
  --systemname vdb_system \
  --results-dir /tmp/vdb_results

Query-count mode

./mlpstorage open vectordb run file \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_1m_1536dim_uniform_diskann \
  --benchmark-mode query_count \
  --queries 10000 \
  --num-query-processes 4 \
  --batch-size 10 \
  --report-count 100 \
  --systemname vdb_system \
  --results-dir /tmp/vdb_results

Enhanced benchmark / sweep mode

Use enhanced mode for:

parameter sweeps
warm/cold cache comparisons
recall-target optimization
richer disk and memory reporting
comparing index/search configurations

Enhanced mode is selected with --benchmark-mode sweep:

./mlpstorage open vectordb run file \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_1m_1536dim_uniform_diskann \
  --benchmark-mode sweep \
  --queries 10000 \
  --num-query-processes 4 \
  --systemname vdb_system \
  --results-dir /tmp/vdb_results

4.6 View Results

./mlpstorage history show

5. Alternative Path: Direct Scripts

This section covers the complete workflow using the Python scripts directly, without the mlpstorage CLI wrapper.

5.1 Installation

cd storage/vdb_benchmark

# For development, use editable installation.
pip3 install -e ./

Or with uv:

cd storage
uv pip install -e ./vdb_benchmark

Verify:

uv run load-vdb --help
uv run vdbbench --help
uv run enhanced-bench --help

Note: You still need a running Milvus instance. Follow the Docker setup in Section 2.

5.2 Load Vectors

Working directory: All python vdbbench/... commands in this section assume you are in storage/vdb_benchmark/. Console scripts (uv run load-vdb, etc.) work from any directory.

Load using a YAML config

python vdbbench/load_vdb.py \
  --config vdbbench/configs/10m_diskann.yaml

Or via the console script:

uv run load-vdb \
  --config vdbbench/configs/10m_diskann.yaml

Load with overrides for quick testing

python vdbbench/load_vdb.py \
  --config vdbbench/configs/10m_diskann.yaml \
  --collection-name mlps_500k_10shards_1536dim_uniform_diskann \
  --num-vectors 500000

Key parameters

--collection-name
--dimension
--num-vectors
--chunk-size
--distribution
--batch-size

Config file location

Direct script configs live in vdbbench/configs/ (relative to storage/vdb_benchmark/):

vdbbench/configs/
├── 10m_diskann.yaml      (10M vectors, 10 shards, 1536 dim)
├── 10m_hnsw.yaml
├── 1m_diskann.yaml       (1M vectors, 1 shard, 1536 dim)
├── 1m_diskann_512dim.yaml
├── 1m_hnsw.yaml
└── 1m_aisaq_512dim.yaml

5.3 Compact

The load script performs compaction automatically when enabled in the config or when --compact is passed.

If the load command exits early, run compaction manually:

python vdbbench/compact_and_watch.py \
  --config vdbbench/configs/10m_diskann.yaml \
  --interval 5

Or via the console script:

uv run compact-and-watch \
  --config vdbbench/configs/10m_diskann.yaml \
  --interval 5

5.4 Run Simple Benchmark

python vdbbench/simple_bench.py \
  --host 127.0.0.1 \
  --port 19530 \
  --collection-name mlps_1m_1536dim_uniform_diskann \
  --processes 4 \
  --batch-size 10 \
  --runtime 120 \
  --output-dir /tmp/vdbbench_results

Or via the console script:

uv run vdbbench \
  --host 127.0.0.1 \
  --port 19530 \
  --collection-name mlps_1m_1536dim_uniform_diskann \
  --processes 4 \
  --batch-size 10 \
  --runtime 120 \
  --output-dir /tmp/vdbbench_results

5.5 Run Enhanced Benchmark

uv run enhanced-bench \
  --host 127.0.0.1 \
  --port 19530 \
  --collection mlps_1m_1536dim_uniform_diskann \
  --sweep \
  --queries 10000 \
  --processes 4 \
  --out-dir /tmp/vdbbench_results

See Section 9 for full parameter reference and execution paths.

Optional: planted queries and tie-aware recall

Both simple_bench.py and enhanced_bench.py accept three optional flags that improve recall measurement on synthetic datasets (see Query Modes and Recall Semantics):

uv run vdbbench \
  --host 127.0.0.1 \
  --port 19530 \
  --collection-name mlps_1m_1536dim_uniform_diskann \
  --processes 4 \
  --batch-size 10 \
  --runtime 120 \
  --query-mode planted \
  --query-noise 0.05 \
  --recall-epsilon 1e-4 \
  --output-dir /tmp/vdbbench_results

Defaults (--query-mode independent, --recall-epsilon 0) reproduce the historical behavior exactly.

6. CLI Reference

Available Commands

./mlpstorage open vectordb --help
./mlpstorage open vectordb datasize --help
./mlpstorage open vectordb datagen --help
./mlpstorage open vectordb run --help
./mlpstorage closed vectordb run --help
./mlpstorage whatif vectordb run --help

Important Terminology

VectorDB uses two similar-looking host flags with different meanings.

Flag / positional	Meaning
`open`, `closed`, `whatif`	Top-level benchmark mode before `vectordb`
`file` / `object`	Positional storage mode selector for `datagen` and `run`
`--vdb-engine`	Vector database engine identity; currently `milvus`
`--vdb-index`	Result identity index family, for example `DISKANN`, `HNSW`, or `AISAQ`
`--index-type`	Milvus implementation index type for `datasize` and `datagen`; defaults to `--vdb-index` when omitted
`--host` / `-s`	Milvus database endpoint host
`--port` / `-p`	Milvus database endpoint port
`--hosts`	Benchmark client hosts used by MPI
`--npernode`	MPI ranks to start on each benchmark client host
`--num-query-processes`	Local Python query workers inside each MPI rank
`--benchmark-mode`	VectorDB run mode: `timed`, `query_count`, or `sweep`
`--systemname`	System-under-test directory name under `results/`
`--results-dir`	Root directory for benchmark output

Do not confuse --host and --hosts.

--host 10.0.0.10        # Milvus server endpoint
--hosts node01 node02   # benchmark client hosts

Effective distributed query workers:

effective_workers = len(--hosts) * --npernode * --num-query-processes

Example:

--hosts node01 node02 --npernode 2 --num-query-processes 4

starts:

2 hosts * 2 MPI ranks per host * 4 Python workers per rank = 16 query workers

Config Files

VectorDB mlpstorage configs live in:

configs/vectordbbench/

The --config flag takes the filename without .yaml.

Example:

--config default

loads:

configs/vectordbbench/default.yaml

Available configs:

Config	Vectors	Dimension	Shards	Index
`default`	1M	1536	1	DiskANN
`10m`	10M	1536	10	DiskANN

Custom configs can be added to the same directory.

Dimension Consistency

The vector dimension must be consistent between data loading and benchmarking.

If you override --dimension during datagen, the config YAML used for run must specify the same dimension. Otherwise, Milvus will reject queries with a vector dimension mismatch.

The safest approach is to use the same --config for both datagen and run, or create a dedicated config YAML for non-default dimensions.

7. Distributed Execution (Multi-Node)

7.1 Prerequisites

For multi-node runs:

Run ./mlpstorage open vectordb ... from one launcher host.
The launcher host participates in the benchmark.
Passwordless SSH must work from the launcher to all hosts listed in --hosts.
The repository path must be identical on every benchmark client host.
The same uv environment must be installed on every benchmark client host.
mpiexec must be installed and available on every benchmark client host.
The --results-dir path must be visible at the same path from every host.
The Milvus endpoint given by --host and --port must be reachable from every benchmark client host.

Install on every benchmark client host

cd /path/to/storage
uv sync --extra vectordb
uv pip install -e ./vdb_benchmark

Verify MPICH launch

mpiexec -n 2 -hosts node01,node02 hostname

Verify VectorDB package import

mpiexec -n 2 -hosts node01,node02 \
  uv run python -c "import vdbbench; print('vdbbench import ok')"

Verify MPI rank detection

mpiexec -n 2 -hosts node01,node02 \
  uv run python -c "from vdbbench.mpi_common import get_mpi_context; print(get_mpi_context())"

7.2 Distributed Load

Distributed load uses MPI to start one or more VectorDB loader ranks across benchmark client hosts.

Before using /shared/vdb_results, initialize it once:

./mlpstorage init MLCommons /shared/vdb_results

Rank behavior

rank 0:
  create/drop collection if --force
  create index
  write collection-ready marker

all ranks:
  wait for collection-ready marker
  insert disjoint vector ID ranges
  flush
  write rank-local load summary

rank 0:
  wait for all rank completion markers
  monitor index build
  compact if requested
  aggregate global load metrics

Command

./mlpstorage open vectordb datagen file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 1 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_1m_1536dim_uniform_diskann \
  --dimension 1536 \
  --num-vectors 1000000 \
  --num-shards 4 \
  --vector-dtype FLOAT_VECTOR \
  --distribution uniform \
  --batch-size 1000 \
  --chunk-size 10000 \
  --force \
  --systemname vdb_multinode_system \
  --results-dir /shared/vdb_results

Example with two MPI ranks per host

./mlpstorage open vectordb datagen file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 2 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_4rank_load \
  --dimension 1536 \
  --num-vectors 2000000 \
  --num-shards 4 \
  --batch-size 1000 \
  --chunk-size 10000 \
  --force \
  --systemname vdb_multinode_system \
  --results-dir /shared/vdb_results

With --hosts node01 node02 and --npernode 2, the distributed load starts four MPI ranks.

Output structure

/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/datagen/<timestamp>/
├── load_statistics.json
├── vdb_multi_node_summary.json
├── rank_0/
│   ├── rank_metadata.json
│   └── load_rank_0.json
├── rank_1/
│   ├── rank_metadata.json
│   └── load_rank_1.json
└── ...

Global load metrics

inserted_vectors
total_time_seconds
vectors_per_second
rank_file_count
rank_stats
mpi.rank_count
mpi.ranks_seen
mpi.expected_ranks
mpi.missing_ranks
mpi.partial_failure

Aggregation rules

inserted_vectors = sum(rank inserted vectors)
total_time_seconds = max(rank end time) - min(rank start time)
vectors_per_second = inserted_vectors / total_time_seconds

7.3 Distributed Simple Benchmark

Distributed simple benchmark mode starts one vdbbench instance per MPI rank.

Each rank writes rank-local CSV, recall, and statistics files. The launcher then aggregates the rank outputs.

Timed mode

In timed mode, every MPI rank runs for the requested runtime.

./mlpstorage open vectordb run file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 2 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_1m_1536dim_uniform_diskann \
  --benchmark-mode timed \
  --runtime 120 \
  --num-query-processes 2 \
  --batch-size 10 \
  --report-count 100 \
  --systemname vdb_multinode_system \
  --results-dir /shared/vdb_results

This starts:

2 hosts * 2 MPI ranks per host * 2 query processes per rank = 8 query workers

Query-count mode

In query-count mode, --queries is interpreted as the global query count. The MPI wrapper splits the query count across ranks.

For example:

--queries 100000 --hosts node01 node02 --npernode 2

starts four MPI ranks, and each rank receives approximately 25,000 queries.

./mlpstorage open vectordb run file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 2 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_1m_1536dim_uniform_diskann \
  --benchmark-mode query_count \
  --queries 100000 \
  --num-query-processes 2 \
  --batch-size 10 \
  --report-count 100 \
  --systemname vdb_multinode_system \
  --results-dir /shared/vdb_results

Output structure

/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run/<timestamp>/
├── statistics.json
├── vdb_multi_node_summary.json
├── rank_0/
│   ├── rank_metadata.json
│   ├── config.json
│   ├── recall_stats.json
│   ├── statistics.json
│   └── milvus_benchmark_p0.csv
├── rank_1/
│   ├── rank_metadata.json
│   ├── config.json
│   ├── recall_stats.json
│   ├── statistics.json
│   └── milvus_benchmark_p0.csv
└── ...

Global metrics

total_queries
total_time_seconds
throughput_qps
min_latency_ms
max_latency_ms
mean_latency_ms
median_latency_ms
p95_latency_ms
p99_latency_ms
p999_latency_ms
p9999_latency_ms
batch_count
successful_batches
failed_batches
recall
disk_io
mpi

Aggregation rules

global_start_time = min(timestamp)
global_end_time = max(timestamp + batch_time_seconds)
total_time_seconds = global_end_time - global_start_time
total_queries = sum(batch_size)
throughput_qps = total_queries / total_time_seconds

Latency percentiles are computed from all rank-local CSV rows:

rank_*/milvus_benchmark_p*.csv

Recall is aggregated exactly when rank-local recall_stats.json files include:

per_query_recall
recall_by_query

7.4 Distributed Enhanced / Sweep Benchmark

Distributed enhanced benchmark mode starts one enhanced-bench instance per MPI rank. Each rank writes enhanced-bench output under a rank-local output directory. The launcher then groups and aggregates rank outputs by parameter set.

Command

./mlpstorage open vectordb run file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 1 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_1m_1536dim_uniform_diskann \
  --benchmark-mode sweep \
  --queries 10000 \
  --num-query-processes 4 \
  --systemname vdb_multinode_system \
  --results-dir /shared/vdb_results

Example with four total MPI ranks

./mlpstorage open vectordb run file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 2 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_1m_1536dim_uniform_diskann \
  --benchmark-mode sweep \
  --queries 20000 \
  --num-query-processes 2 \
  --systemname vdb_multinode_system \
  --results-dir /shared/vdb_results

With --hosts node01 node02, --npernode 2, and --num-query-processes 2:

2 hosts * 2 MPI ranks per host * 2 query processes per rank = 8 query workers

Output structure

/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run/<timestamp>/
├── enhanced_statistics.json
├── vdb_multi_node_summary.json
├── rank_0/
│   ├── rank_metadata.json
│   ├── combined_bench_rank_0.json
│   ├── combined_bench_rank_0.csv
│   └── combined_bench_rank_0.sweep.csv
├── rank_1/
│   ├── rank_metadata.json
│   ├── combined_bench_rank_1.json
│   ├── combined_bench_rank_1.csv
│   └── combined_bench_rank_1.sweep.csv
└── ...

Global metrics

benchmark_phase
aggregation
json_file_count
results
mpi.rank_count
mpi.ranks_seen
mpi.expected_ranks
mpi.missing_ranks
mpi.partial_failure

Enhanced aggregation groups rank outputs by benchmark parameter set, including:

mode
cache_state
k
index_type
metric_type
search parameters
index parameters

Aggregation rules

total_queries = sum(rank queries)
throughput_qps = sum(rank throughput_qps)
mean_latency_ms = query-count-weighted mean
p95_latency_ms = max(rank p95_latency_ms)
p99_latency_ms = max(rank p99_latency_ms)
recall_mean = query-count-weighted mean, or exact when per-query recall is present

For simple-bench, p95/p99 latency percentiles are exact because raw per-batch CSV rows are available. For enhanced-bench, p95/p99 are conservative max-rank values unless raw latency samples are also emitted by the enhanced output.

7.5 Metrics Aggregation

Distributed VectorDB runs use rank-local output directories and a final aggregation step. The aggregation script is:

uv run vdb-aggregate

It is normally invoked automatically by ./mlpstorage open vectordb. It can also be run manually.

Manual load aggregation

uv run vdb-aggregate \
  --phase load \
  --base-output-dir /shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/datagen/<timestamp> \
  --expected-ranks 2

Manual simple aggregation

uv run vdb-aggregate \
  --phase simple \
  --base-output-dir /shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run/<timestamp> \
  --expected-ranks 2

Manual enhanced aggregation

uv run vdb-aggregate \
  --phase enhanced \
  --base-output-dir /shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run/<timestamp> \
  --expected-ranks 2

7.6 Disk I/O Deduplication

Disk I/O counters are node-local. If multiple MPI ranks run on the same host, summing every rank's /proc/diskstats delta would double-count that host's disk I/O.

Distributed aggregation counts disk I/O only once per benchmark client host, using the rank where:

local_rank == 0

The aggregated disk_io field records this policy.

7.7 Ground Truth and Recall

Recall is computed outside the timed query loop so it does not inflate latency measurements.

The benchmark uses a FLAT ground-truth collection for exact nearest-neighbor results.

Recommended ground-truth collection name:

_flat_gt

Distributed wrappers should avoid multiple ranks racing to create/drop the same FLAT ground-truth collection.

The orchestration flow is:

rank 0: create or validate FLAT ground-truth collection
non-rank-0: validate existing FLAT ground-truth collection
run with --no-create-flat

Rank-local recall files include:

recall_stats.json

Recall fields include:

mean_recall
median_recall
min_recall
max_recall
p5_recall
p95_recall
p99_recall
num_queries_evaluated
per_query_recall
recall_by_query

The per_query_recall and recall_by_query fields are used for exact multi-rank recall aggregation.

7.8 Open MPI Alternative

The distributed VectorDB path defaults to MPICH-style launch syntax:

--mpi-impl mpich
--mpi-bin mpiexec

Open MPI can be selected with:

--mpi-impl openmpi
--mpi-bin mpirun

Example:

./mlpstorage open vectordb run file \
  --distributed \
  --mpi-impl openmpi \
  --mpi-bin mpirun \
  --hosts node01 node02 \
  --npernode 1 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_1m_1536dim_uniform_diskann \
  --benchmark-mode timed \
  --runtime 120 \
  --num-query-processes 2 \
  --batch-size 10 \
  --systemname vdb_multinode_system \
  --results-dir /shared/vdb_results

Additional MPI arguments can be passed with --mpi-params:

./mlpstorage open vectordb run file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 1 \
  --mpi-params="-env UCX_TLS tcp" \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_1m_1536dim_uniform_diskann \
  --benchmark-mode query_count \
  --queries 10000 \
  --num-query-processes 2 \
  --batch-size 10 \
  --systemname vdb_multinode_system \
  --results-dir /shared/vdb_results

8. End-to-End Examples

Single-Node Example

# Initialize output root once.
./mlpstorage init MLCommons ~/vdb_results

# 1. Estimate storage.
./mlpstorage open vectordb datasize \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --dimension 1536 \
  --num-vectors 1000000

# 2. Load vectors.
./mlpstorage open vectordb datagen file \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_single_1m \
  --force \
  --systemname vdb_single_system \
  --results-dir ~/vdb_results

# 3. Run simple benchmark.
./mlpstorage open vectordb run file \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_single_1m \
  --benchmark-mode timed \
  --num-query-processes 2 \
  --runtime 60 \
  --batch-size 10 \
  --systemname vdb_single_system \
  --results-dir ~/vdb_results

# 4. Run enhanced benchmark.
./mlpstorage open vectordb run file \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_single_1m \
  --benchmark-mode sweep \
  --queries 10000 \
  --num-query-processes 2 \
  --systemname vdb_single_system \
  --results-dir ~/vdb_results

# 5. View history.
./mlpstorage history show

Distributed MPICH Example

# 1. Verify MPI.
mpiexec -n 2 -hosts node01,node02 hostname

# 2. Initialize output root once.
./mlpstorage init MLCommons /shared/vdb_results

# 3. Load vectors across two benchmark client hosts.
./mlpstorage open vectordb datagen file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 1 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_dist_1m \
  --num-vectors 1000000 \
  --dimension 1536 \
  --num-shards 4 \
  --force \
  --systemname vdb_dist_system \
  --results-dir /shared/vdb_results

# 4. Run distributed simple benchmark.
./mlpstorage open vectordb run file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 1 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_dist_1m \
  --benchmark-mode timed \
  --runtime 120 \
  --num-query-processes 2 \
  --batch-size 10 \
  --systemname vdb_dist_system \
  --results-dir /shared/vdb_results

# 5. Run distributed enhanced benchmark.
./mlpstorage open vectordb run file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 1 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_dist_1m \
  --benchmark-mode sweep \
  --queries 10000 \
  --num-query-processes 2 \
  --systemname vdb_dist_system \
  --results-dir /shared/vdb_results

9. Enhanced Benchmark Full Reference

Working directory: All commands below assume you are in storage/vdb_benchmark/.

enhanced_bench.py merges the operational features of simple_bench.py with advanced features for parameter sweeps, warm/cold cache regimes, budget mode, YAML config, and memory estimation.

Two Execution Paths

The script automatically selects the path based on the flags provided.

Path	Trigger	Best for
Runtime / query-count	`--runtime` or `--batch-size` present	Sustained load, CI gating, storage testing
Sweep / cache	Neither `--runtime` nor `--batch-size` present, or explicit `--sweep`	Parameter tuning, recall target sweep, warm/cold analysis

Path A — Runtime / Query-Count Mode

This path mimics simple_bench.py. It runs workers for a fixed duration or query count, writes per-process CSV files, and aggregates latency and recall stats.

Create the FLAT ground-truth collection (first run only):

python vdbbench/enhanced_bench.py \
  --host 127.0.0.1 \
  --collection mlps_10m_10shards_1536dim_uniform_diskann \
  --auto-create-flat \
  --runtime 1 \
  --batch-size 1 \
  --processes 1

Runtime-based run:

python vdbbench/enhanced_bench.py \
  --host 127.0.0.1 \
  --collection mlps_10m_10shards_1536dim_uniform_diskann \
  --runtime 120 \
  --batch-size 10 \
  --processes 4 \
  --search-limit 10 \
  --search-ef 200

Query-count-based run:

python vdbbench/enhanced_bench.py \
  --host 127.0.0.1 \
  --collection mlps_10m_10shards_1536dim_uniform_diskann \
  --queries 50000 \
  --batch-size 10 \
  --processes 4

With explicit FLAT GT collection:

python vdbbench/enhanced_bench.py \
  --host 127.0.0.1 \
  --collection mlps_10m_10shards_1536dim_uniform_diskann \
  --gt-collection mlps_10m_10shards_1536dim_uniform_diskann_flat_gt \
  --runtime 120 \
  --batch-size 10 \
  --processes 4

Path B — Sweep / Cache / Budget Mode

Single-thread, both warm and cold cache, recall sweep targeting 0.95:

python vdbbench/enhanced_bench.py \
  --host 127.0.0.1 \
  --collection mlps_10m_10shards_1536dim_uniform_diskann \
  --gt-collection mlps_10m_10shards_1536dim_uniform_diskann_flat_gt \
  --mode single \
  --sweep \
  --target-recall 0.95 \
  --cache-state both \
  --queries 1000 \
  --k 10

Multi-process default parameters:

python vdbbench/enhanced_bench.py \
  --host 127.0.0.1 \
  --collection mlps_10m_10shards_1536dim_uniform_diskann \
  --gt-collection mlps_10m_10shards_1536dim_uniform_diskann_flat_gt \
  --mode mp \
  --processes 8 \
  --cache-state warm \
  --queries 1000 \
  --k 10

Multiple recall targets, optimized for latency:

python vdbbench/enhanced_bench.py \
  --host 127.0.0.1 \
  --collection mlps_10m_10shards_1536dim_uniform_diskann \
  --gt-collection mlps_10m_10shards_1536dim_uniform_diskann_flat_gt \
  --mode both \
  --sweep \
  --recall-targets 0.90 0.95 0.99 \
  --optimize latency \
  --cache-state warm

Key Parameters

Parameter	Default	Description
`--collection`	required	ANN-indexed collection name
`--runtime`	`None`	Benchmark duration in seconds
`--queries`	`1000`	Total query count
`--batch-size`	required for runtime path	Queries per batch
`--processes`	`8`	Worker processes
`--search-limit`	`10`	Top-k results per query
`--search-ef`	`200`	Search parameter override
`--num-query-vectors`	`1000`	Pre-generated query vectors for recall
`--recall-k`	`--search-limit`	k for recall@k
`--query-mode`	`independent`	`independent` or `planted` — see Query Modes
`--query-noise`	`0.05`	L2 displacement of planted queries from their base vectors
`--recall-epsilon`	`0.0`	Tie tolerance for recall@k; `0` = exact set-intersection recall
`--gt-collection`	`_flat_gt`	FLAT GT collection name
`--auto-create-flat`	`False`	Auto-create FLAT GT collection from source
`--no-create-flat`	`False`	Validate and reuse existing FLAT GT collection
`--vector-dim`	`1536`	Vector dimension
`--output-dir` / `--out-dir`	`vdbbench_results/`	Output directory
`--json-output`	`False`	Print summary as JSON
`--report-count`	`10`	Batches between progress logs
`--host` / `--port`	`localhost:19530`	Milvus connection
`--config`	`None`	YAML config file

Output Files

Runtime path

config.json
milvus_benchmark_p0.csv
milvus_benchmark_p1.csv
recall_hits_p0.jsonl
recall_hits_p1.jsonl
recall_stats.json
statistics.json

Sweep path

combined_bench_.json
combined_bench_.csv
combined_bench_.sweep.csv

10. Metrics and Measurement

Recall Measurement

Recall is computed outside the timed benchmark loop so it does not inflate latency measurements.

The benchmark uses a FLAT ground-truth collection for exact nearest-neighbor results.

Simple benchmark output includes:

recall_stats.json
statistics.json

Recall fields include:

recall_at_k              # normative recall (equals recall_at_k_exact when epsilon = 0)
recall_at_k_exact        # strict set-intersection recall, always reported
recall_epsilon           # tie tolerance used for recall_at_k (0 = exact)
mean_recall
median_recall
min_recall
max_recall
p5_recall
p95_recall
p99_recall
num_queries_evaluated
per_query_recall
recall_by_query

The per_query_recall and recall_by_query fields are used for exact multi-rank aggregation. query_mode, query_noise, and recall_epsilon are also recorded in each run's config.json / benchmark_meta.json, so every result is self-describing about which recall definition produced it.

Query Modes and Recall Semantics (Issue #625)

The benchmark generates synthetic vectors. With the historical defaults — i.i.d. random database vectors and i.i.d. random, independent query vectors — recall@k is computed correctly (ground truth is exact brute force) but carries almost no signal at high dimension: at 1536-d the query-to-corpus cosine similarity concentrates so tightly (relative contrast ≈ 1.1) that the true top-k boundary sits within float32 noise, recall barely responds to search effort, and results do not transfer to real embedding workloads (issue #625).

Two opt-in mechanisms address this. Both default off, and the defaults are identical to the previous behavior.

Planted queries (--query-mode planted / query_mode: planted). Each query is a small deterministic perturbation (--query-noise, default 0.05 L2 displacement) of a stored database vector, so every query has a genuine planted near neighbor. Stored vectors, ingest, index build, and the load-phase I/O profile are unchanged — only the query set differs. Measured effect at 1536-d: nearest-neighbor relative contrast rises from ~1.1 to >100, giving a recall/QPS operating curve that actually responds to search parameters. Requires the standard vdbbench data layout (dense INT64 primary keys 0..N-1); the benchmark fails loudly, never silently falls back, if the collection does not conform.

Tie-aware epsilon recall (--recall-epsilon / recall_epsilon). Following the big-ann-benchmarks convention, a returned neighbor whose ground-truth score is within epsilon of the k-th neighbor's is credited rather than scored as a miss, so float32-level ties do not add noise to the metric. recall_at_k_exact is always reported alongside, and recall_epsilon is recorded with every result.

Note on search_list_size / --search-ef: raising the search effort is not an equivalent workaround. On structureless random data with independent queries, even large search-list values move recall only marginally while inflating read I/O — and at the extreme the graph search degenerates toward an exhaustive scan, distorting the very storage access pattern this benchmark exists to measure. Search-effort parameters tune a run within a recall definition; query_mode and recall_epsilon change the definition itself, and are labeled accordingly.

Guidance for Submitters: Comparability and Reruns

Classification: these features are an opt-in methodology enhancement. They are not a bug fix — the previous pipeline computed recall correctly against exact ground truth, and no previously published result is wrong or invalidated. When the new modes are enabled, they are a metric definition change, and results are treated the way MLPerf treats any definition change between rounds:

Existing and in-flight submissions: no rerun and no restatement required. The defaults are identical to prior behavior (verified by regression test down to the query RNG stream), so this change can merge without affecting anyone.
Current round: planted / epsilon runs are available as a diagnostic mode. Official results remain on the existing definition until the Working Group rules say otherwise.
Comparing runs: only compare recall numbers produced with the same query_mode and recall_epsilon. Check config.json / benchmark_meta.json when in doubt; when epsilon recall is enabled, recall_at_k_exact provides the strict metric for cross-checking.

Disk I/O Metrics

Disk I/O is measured by reading /proc/diskstats before and after each benchmark run.

Fields include:

bytes_read
bytes_written
read_mbps
write_mbps
read_iops
write_iops

In distributed mode, disk I/O is aggregated once per benchmark client host to avoid double-counting multiple MPI ranks on the same host.

Scope and validity of `disk_io`

/proc/diskstats only accounts for local block devices on the benchmark client node. The disk_io figures are therefore only valid when the storage under test is backed by a local block device on that node (e.g. local NVMe hosting the Milvus data directory).

When the storage under test is a network / remote filesystem (NFS, CIFS/SMB, CephFS, GlusterFS, Lustre, GPFS, BeeGFS, PanFS, virtiofs, FUSE-based remote clients, etc.), no corresponding local block device exists, so diskstats deltas do not describe the storage under test. In that case the benchmark marks disk_io as not applicable:

"disk_io": {
  "applicable": false,
  "status": "N/A",
  "not_applicable_reason": "storage under test is a network/remote filesystem (nfs4 mounted at /mnt/vdb from filer:/export/vdb); /proc/diskstats only accounts for local block devices, so disk_io is not applicable.",
  "storage_target": { "fstype": "nfs4", "mountpoint": "/mnt/vdb", "...": "..." },
  "client_local_io": { "total_bytes_read": 0, "...": "..." }
}

client_local_io preserves the raw client-local counters for debugging and audit, but must not be interpreted as I/O to the storage under test.

Pass --data-path <path> (the mount point or directory backing the storage under test on the client node) for exact classification. If --data-path is omitted, classification is heuristic: disk_io is still reported, storage_target.confidence is set to heuristic, and any detected network mounts are listed so reviewers can judge scope.

disk_io is informational only. The benchmark score (QPS, latency, recall) is never derived from disk_io, so applicable: false does not affect scoring or submission validity.

In distributed mode, hosts reporting applicable: false are excluded from the aggregated totals and listed under disk_io.hosts_not_applicable; the aggregated disk_io.applicable flag is false when every sampled host was N/A.

11. Testing and Validation

1. MPI launch smoke test

mpiexec -n 2 -hosts node01,node02 hostname

Expected result:

node01
node02

2. Rank detection smoke test

mpiexec -n 2 -hosts node01,node02 \
  uv run python -c "from vdbbench.mpi_common import get_mpi_context; print(get_mpi_context())"

Expected result:

MpiContext(rank=0, world_size=2, local_rank=0, hostname='node01')
MpiContext(rank=1, world_size=2, local_rank=0, hostname='node02')

3. `mlpstorage` dry run

Initialize the shared results directory once:

./mlpstorage init MLCommons /shared/vdb_results

Use top-level whatif plus --dry-run to inspect the generated command without running it:

./mlpstorage whatif vectordb run file \
  --dry-run \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 1 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_smoke \
  --benchmark-mode query_count \
  --queries 100 \
  --num-query-processes 1 \
  --batch-size 10 \
  --systemname vdb_smoke_system \
  --results-dir /shared/vdb_results

4. Single-host distributed smoke test

This uses MPI on localhost and is useful before testing multiple nodes.

./mlpstorage init MLCommons /tmp/vdb_results

./mlpstorage open vectordb datagen file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts localhost \
  --npernode 2 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_smoke_10k \
  --dimension 128 \
  --num-vectors 10000 \
  --num-shards 2 \
  --batch-size 500 \
  --chunk-size 1000 \
  --force \
  --systemname vdb_smoke_system \
  --results-dir /tmp/vdb_results

Then run a query-count benchmark:

./mlpstorage open vectordb run file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts localhost \
  --npernode 2 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 127.0.0.1 \
  --port 19530 \
  --config default \
  --collection mlps_smoke_10k \
  --benchmark-mode query_count \
  --queries 200 \
  --vector-dim 128 \
  --num-query-processes 1 \
  --batch-size 10 \
  --systemname vdb_smoke_system \
  --results-dir /tmp/vdb_results

Validate aggregated results:

python - <<'PY'
import json
from pathlib import Path

stats_files = sorted(
    Path("/tmp/vdb_results").glob(
        "**/vector_database/milvus/DISKANN/run/**/statistics.json"
    )
)
assert stats_files, "No distributed statistics.json found"
stats = json.loads(stats_files[-1].read_text())
assert stats["total_queries"] == 200
assert stats["mpi"]["partial_failure"] is False
print(json.dumps(stats, indent=2)[:2000])
PY

5. Multi-node load test

Initialize the shared results directory once:

./mlpstorage init MLCommons /shared/vdb_results

Load data:

./mlpstorage open vectordb datagen file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 1 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_multinode_1m \
  --dimension 1536 \
  --num-vectors 1000000 \
  --num-shards 4 \
  --batch-size 1000 \
  --chunk-size 10000 \
  --force \
  --systemname vdb_multinode_system \
  --results-dir /shared/vdb_results

Post-check:

python - <<'PY'
import json
from pathlib import Path

load_files = sorted(
    Path("/shared/vdb_results").glob(
        "**/vector_database/milvus/DISKANN/datagen/**/load_statistics.json"
    )
)
assert load_files, "No load_statistics.json found"
stats = json.loads(load_files[-1].read_text())
assert stats["inserted_vectors"] == 1000000
assert stats["mpi"]["partial_failure"] is False
print(json.dumps(stats, indent=2)[:2000])
PY

6. Multi-node simple benchmark test

./mlpstorage open vectordb run file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 1 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_multinode_1m \
  --benchmark-mode timed \
  --runtime 120 \
  --num-query-processes 2 \
  --batch-size 10 \
  --systemname vdb_multinode_system \
  --results-dir /shared/vdb_results

7. Multi-node enhanced benchmark test

./mlpstorage open vectordb run file \
  --distributed \
  --mpi-impl mpich \
  --mpi-bin mpiexec \
  --hosts node01 node02 \
  --npernode 1 \
  --vdb-engine milvus \
  --vdb-index DISKANN \
  --host 10.0.0.10 \
  --port 19530 \
  --config default \
  --collection mlps_multinode_1m \
  --benchmark-mode sweep \
  --queries 10000 \
  --num-query-processes 2 \
  --systemname vdb_multinode_system \
  --results-dir /shared/vdb_results

Expected files:

/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/datagen/<timestamp>/load_statistics.json
/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run/<timestamp>/statistics.json
/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run/<timestamp>/enhanced_statistics.json
/shared/vdb_results/open/MLCommons/results/vdb_multinode_system/vector_database/milvus/DISKANN/run/<timestamp>/vdb_multi_node_summary.json

12. Troubleshooting

`vector dimension mismatch`

The dimension used for load and run does not match. Use the same config for both datagen and run, or update the config to match the dimension passed to datagen.

`rate limit exceeded[rate=0.1]` during datagen

Symptom (typically at high rank counts, e.g. --npernode 4 across many hosts):

pymilvus.exceptions.MilvusException: <MilvusException: (code=8, message=...
failed to flush collection: ... rate limit exceeded[rate=0.1], request is
rejected by grpc RateLimiter middleware, please retry later)>

Milvus 2.4+ ships a per-collection flush rate limiter enabled by default:

quotaAndLimits:
  flushRate:
    collection:
      max: 0.1   # one flush() per 10 seconds per collection

Older benchmark versions called flush() once per MPI rank on the same collection, so many concurrent ranks exhausted the pymilvus retry budget (~210 seconds) and failed with the error above, while smaller rank counts squeaked through. Since issue #705 was fixed, datagen flushes each collection exactly once (from rank 0, after all ranks finish inserting) and retries any rate-limited flush while respecting the limiter period, so no Milvus configuration change is needed.

If you still hit this error:

Update to a benchmark version that includes the issue #705 fix. The load summary of a fixed version contains a top-level collection_flush_seconds field.
If external tooling flushes the benchmark collections concurrently, raise or disable the limiter via stacks/milvus/user.yaml.example (MinIO stack, mounted as /milvus/configs/user.yaml) or the commented QUOTAANDLIMITS_FLUSHRATE_COLLECTION_MAX environment variable (S3 stack).

The Milvus configuration is part of the system under test: keep the defaults for official submissions unless the rules state otherwise, and record any override in your system description.

MPI launches only on one host

Check:

mpiexec -n 2 -hosts node01,node02 hostname

If both lines show the same host, inspect the MPICH/Hydra host configuration and SSH setup.

Rank output is missing

Check:

rank_*.error.json
rank_*/rank_metadata.json
vdb_multi_node_summary.json

The aggregated summary reports:

ranks_seen
missing_ranks
partial_failure

Distributed aggregation cannot find files

Ensure --results-dir is visible at the same path from all benchmark client hosts.

For example, use a shared filesystem path such as:

/shared/vdb_results

Recall is zero

Check that the FLAT ground-truth collection exists and contains the same vectors as the ANN collection.

Also check that rank-local recall_stats.json files contain non-empty:

per_query_recall
recall_by_query

Recall is low and does not improve with `search_list` / `ef`

This is expected with the default --query-mode independent on high-dimensional random data — the recall metric is correct but poorly conditioned, not broken. See Query Modes and Recall Semantics and re-run with --query-mode planted (optionally --recall-epsilon 1e-4) to obtain a discriminative recall/QPS curve.

Milvus is not reachable from worker hosts

Run this from every benchmark client host:

uv run python - <<'PY'
from pymilvus import connections
connections.connect(alias="default", host="10.0.0.10", port="19530")
print("Milvus connection ok")
connections.disconnect("default")
PY

`vdb-mpi-wrapper` is not found

Install the VectorDB package in the uv-managed environment on every client host:

cd /path/to/storage
uv pip install -e ./vdb_benchmark

Verify:

uv run vdb-mpi-wrapper --help
uv run vdb-aggregate --help

13. Contributing

Contributions are welcome. Pull requests that add or modify distributed VectorDB behavior should include:

implementation changes
unit tests
single-host MPI smoke test results
multi-node test results when applicable
README updates

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Vector Database Benchmark Tool

Table of Contents

1. Prerequisites

System Requirements

Python Dependencies

Clone the Repository

2. Deploy Milvus

Option A: Local Storage with MinIO

Option B: S3 Storage

Verify Milvus is Healthy

3. Quick Start — First Benchmark in 10 Minutes

Step 1 — Install

Step 2 — Start Milvus

Step 3 — Initialize the results directory

Step 4 — Load 50K vectors (smoke test)

Step 5 — Run a 30-second benchmark

Step 6 — Check results

4. Recommended Path: mlpstorage CLI

4.1 Installation

4.2 Estimate Storage (datasize)

4.3 Load Vectors (datagen)

Load using the default config (1M vectors)

Load using the 10M config

Override vector count for quick testing

Notes

4.4 Compact

4.5 Run Benchmarks

Simple benchmark modes

Timed mode

Query-count mode

Enhanced benchmark / sweep mode

4.6 View Results

5. Alternative Path: Direct Scripts

5.1 Installation

5.2 Load Vectors

Load using a YAML config

Load with overrides for quick testing

Key parameters

Config file location

5.3 Compact

5.4 Run Simple Benchmark

5.5 Run Enhanced Benchmark

Optional: planted queries and tie-aware recall

6. CLI Reference

Available Commands

Important Terminology

Config Files

Dimension Consistency

7. Distributed Execution (Multi-Node)

7.1 Prerequisites

Install on every benchmark client host

Verify MPICH launch

Verify VectorDB package import

Verify MPI rank detection

7.2 Distributed Load

Rank behavior

Command

Example with two MPI ranks per host

Output structure

Global load metrics

Aggregation rules

7.3 Distributed Simple Benchmark

Timed mode

Query-count mode

Output structure

Global metrics

Aggregation rules

7.4 Distributed Enhanced / Sweep Benchmark

Command

Example with four total MPI ranks

Output structure

Global metrics

Aggregation rules

7.5 Metrics Aggregation

Scope and validity of `disk_io`

3. `mlpstorage` dry run

`vector dimension mismatch`

`rate limit exceeded[rate=0.1]` during datagen

Recall is low and does not improve with `search_list` / `ef`

`vdb-mpi-wrapper` is not found