feat(cpp): gaia::VectorIndex — flat vector index with persistence - #2807
feat(cpp): gaia::VectorIndex — flat vector index with persistence#2807kovtcharov wants to merge 1 commit into
Conversation
The C++ framework had no vector search of any kind, so a native binary could not do RAG, code indexing, or memory recall. This adds an exhaustive flat index over float32 embeddings with save/load, which is all the Python SDK actually needs: every FAISS call site in gaia uses IndexFlatL2 or IndexFlatIP, so the rankings match vector-for-vector with no BLAS-linked dependency added to three platform builds. Scores follow the Python convention exactly — 1/(1+d²) over the squared L2 distance FAISS reports, and the raw dot product for inner product, which is cosine similarity when normalize-on-add is enabled. Ties break by insertion order so rankings are reproducible across runs and platforms. Mismatches raise instead of returning misleading results: a wrong-sized vector on add/search names both dimensions, and loading an index built by a different embedder names both models. The .vec parser validates every field against the actual file length before allocating, and rejects non-finite payloads, so a corrupt or hostile cache file cannot trigger a huge allocation, an OOB read, or NaN-poisoned rankings. C++ cannot read Python's index.faiss and does not try to; the runtimes use separate cache directories and share only metadata.json.
|
Verdict: Approve ✅ This adds The bottom line: this is a genuinely well-built change. Every failure path raises an actionable error (what failed, why, how to fix) rather than returning a plausible-but-wrong ranking — right in the spirit of GAIA's fail-loudly rule — and the load path guards against a corrupt header ballooning into a multi-terabyte allocation before it reserves anything. No blocking issues found. Real-world evidenceN/A — this is an internal C++ library component with no user-facing CLI / REST / Agent UI / MCP surface, so there's no runnable surface for the evidence lane to exercise. Correctness rests on the bundled gtest suite (hand-computed L2/IP scores, tie-stability, save/load round-trips, corrupt-file rejection, and numeric parity against faiss 1.13.2) plus static review. The suite couldn't be compiled/run in this review environment ( 🔍 Technical detailsCorrectness review — no issues found. Spot-checks that held up:
Strengths:
Nits: none worth posting. |
The failing
|
…xe (amd#2818) Every `cpp/**` PR has been failing the `C++ Integration Tests (STX)` check before it compiles anything — amd#2807, amd#2809 and amd#2816 are all red for a reason unrelated to their diffs. The self-hosted runner cached CMake under `$env:TEMP` and re-downloaded it only when `bin\cmake.exe` was missing; Windows Temp cleanup deleted `share\cmake-3.31\Modules` and left `bin\`, so the job kept trusting a CMake that cannot resolve `CMAKE_ROOT` and every run died the same way until someone cleared Temp by hand. Now each candidate toolchain is probed for the thing the build actually depends on, a failed probe falls through to a clean re-download, and tools live in the runner tool cache instead of a directory the OS sweeps. One measurement drove the design and is worth flagging for review: **exit codes cannot detect this failure.** A CMake missing its Modules tree prints `Could not find CMAKE_ROOT` to stderr and still exits 0 — for `--version` and for `--help-module-list` (measured on 4.4.2). So the issue's suggested `cmake --version` exit-0 check would not have caught it on its own; validity requires the Modules tree on disk *and* a probe that does not report a broken root. The stale `%TEMP%\cmake` tree on the runner is now inert — nothing reads it — so no manual cleanup is needed to make this work; deleting it just reclaims disk. Closes amd#2817 ## Test plan - [ ] `pwsh -File .github/scripts/tests/CppBuildTools.Tests.ps1` passes (21/21). It asserts the exact regression: `bin/cmake.exe` present + `share/` absent reports **invalid**, both present reports **valid**, and includes negative controls showing the old `Test-Path cmake.exe` check and an exit-code-only check would both have accepted the broken install. - [ ] New `C++ toolchain script tests` job is green (parse-checks every `.github/scripts/*.ps1`, then runs the unit tests). - [ ] `C++ Integration Tests (STX)` on this PR gets past `Ensure C++ build tools are available` and reaches compilation. The step log should name which CMake it accepted and, if it rejected one, why. - [ ] Reproduce the root cause on any machine: copy a `cmake` binary alone into an empty directory and run `--version` — it prints the `CMAKE_ROOT` error and exits 0. - [ ] After merge, re-run CI on amd#2807, amd#2809 and amd#2816 with no changes to their diffs and confirm the STX check goes green.
1 participant
The C++ framework has no vector search of any kind today, so a native GAIA binary cannot do RAG, code indexing, or memory recall — the tools that make an agent domain-specific rather than general-purpose. This adds an exhaustive flat index over float32 embeddings with save/load persistence, which is all the Python SDK actually needs: every FAISS call site in the repo uses
IndexFlatL2orIndexFlatIP(brute-force scans — no IVF, HNSW, or PQ anywhere), so a C++ agent now gets the same rankings Python gets, with zero new dependencies across the Windows/Linux/macOS builds. Scores match Python's convention exactly —1/(1+d²)for L2 and the raw dot product for inner product — and mismatches raise instead of quietly returning wrong results: a wrong-sized vector names both dimensions, and loading an index built by a different embedder names both models.Unblocks P2.2 (RAG) and P2.3 (code index). C++ cannot read Python's
index.faissand does not try to; the two runtimes use separate cache directories and share onlymetadata.json.Test plan
cmake -S cpp -B cpp/build -DGAIA_BUILD_TESTS=ON && cmake --build cpp/build && ctest --test-dir cpp/build --output-on-failure— 501/501 pass (38 new)VectorIndexTest.MatchesPythonIndexFlatL2TopK/MatchesPythonIndexFlatIPTopKassert the same top-5 ordering and scores that faiss 1.13.2 produces for the same vectors (the regeneration snippet is in the test file)-fsanitize=address,undefinedCloses #2792