feat(cpp): text extraction and chunking

[kovtcharov]

The C++ runtime could not turn a document into retrieval-ready text, so a native agent had nothing to index and any splitter a consumer hand-rolled produced chunks incomparable to the Python SDK's. gaia::chunkFile() now extracts plain text, Markdown, and source files and splits them into sentence-aware chunks whose boundaries match RAGSDK._split_text_into_chunks byte-for-byte — an index built by one runtime is directly comparable to one built by the other. PDF, DOCX, XLSX and PPTX stay out of scope and are refused by name; a silent empty extraction would produce an agent answering confidently from nothing.

Getting parity right turned out to hinge on Unicode: a differential run against the Python implementation found real mismatches on documents with accented capitals, because Python treats Ü as a section-title start and an ASCII check does not. Hence the generated str.isupper() table, guarded by a Python test so it cannot silently drift.

Test plan

• cmake -B cpp/build -S cpp -DCMAKE_BUILD_TYPE=Release && cmake --build cpp/build -j
• ctest --test-dir cpp/build --output-on-failure — 485/485 pass, 22 of them new:

19/22 Test #459: ChunkingTest.RegisteredExtractorHandlesOutOfScopeFormats ...............   Passed    0.00 sec
20/22 Test #460: ChunkingTest.RegisteredExtractorFailurePropagates ......................   Passed    0.00 sec
21/22 Test #461: ChunkingTest.ChunkFileExtractsAndSplits ................................   Passed    0.00 sec
22/22 Test #462: ChunkingTest.ChunkBoundariesMatchPythonRagSdk ..........................   Passed    0.00 sec

100% tests passed out of 22

• ChunkingTest.ChunkBoundariesMatchPythonRagSdk replays 11 fixture cases over 6 documents (headers, paragraph fallback, both sides of the sections <= 3 branch, title-line heuristic, non-ASCII capitals, NBSP, CRLF, zero overlap) against chunks generated by the Python splitter
• pytest tests/unit/test_cpp_unicode_table.py — fails if the uppercase table drifts from this interpreter's str.isupper()
• cpp/build/tests_mock --gtest_repeat=3 --gtest_shuffle — clean, so the process-global extractor registry does not leak between runs
• Unsupported types: extractFile("x.pdf") throws naming the type, the supported set, and the registerExtractor() hook; a directory or binary file is refused rather than extracted as empty text
• Install round-trip: cmake --install ships gaia/chunking.h and a consumer translation unit compiles and links against it

Beyond the committed tests, the port was checked against the Python implementation on ~1,700 generated documents (random headings, rules, abbreviations, mixed scripts, Unicode whitespace, CRLF, chunk sizes 1–500, overlaps 0–100) plus a set of adversarial shapes — zero divergences.

Closes #2795

[github-actions]

Verdict: Approve

This adds a native C++ text-extraction + semantic-chunker (gaia/chunking.h) that is a faithful port of RAGSDK._split_text_into_chunks, so a document chunked by the C++ coding-agent runtime lands on the same boundaries as the Python runtime. I checked the port against the real Python implementation (src/gaia/rag/sdk.py) — section detection, paragraph fallback, sentence splitting, overlap trimming, and the token estimate all match — and the divergences that remain (no LLM chunking, no VLM-block atomicity, binary files rejected instead of read as mojibake, unusable configs throw) are each deliberate and documented in the header. The bar for a "parity" claim is that it actually holds; here it's pinned by a fixture test that replays Python-generated chunks and requires byte-identical output, plus a Python-side guard that catches drift in the Unicode uppercase table. That's the right harness for this kind of change.

The one thing worth a follow-up (non-blocking): the extractor's UTF-8 check is more lenient than Python's strict decoder, so a handful of malformed byte sequences would be kept as UTF-8 here but decoded as latin-1 by Python — a narrow, adversarial-input gap rather than something real documents hit. Details below.

Real-world evidence

N/A — this is a native C++ library with no GAIA Agent-UI / CLI / MCP / HTTP surface; its proof is the bundled gtest suite (cpp/tests/test_chunking.cpp, incl. the cross-runtime parity case), which runs under ctest. No evidence-bundle.md was produced (the automated evidence lane targets the Python surfaces, none of which this touches), and gh/Bash were unavailable in this run so I couldn't pull the PR description — the verdict rests on static review of the diff against the Python source it ports.

🔍 Technical details

Parity spot-checks against src/gaia/rag/sdk.py (all match):

• Section boundaries: # prefix, ^[\-=_]{3,}$ rule, and the title-line heuristic (<100 code points, prev-empty/next-non-empty, isupper() first char, no trailing .!?,;) — splitSections at cpp/src/chunking.cpp:497 vs sdk.py:2035-2073. Uses code-point length and str.isupper()-equivalent table, not ASCII.
• sections.size() <= 3 → paragraph fallback on re.split(r"\n\s*\n", text) (splitParagraphs:455 vs sdk.py:2076-2082).
• Overlap: _get_last_n_tokens semantics incl. the first_space > 0 word-boundary trim (lastNTokens:429 vs sdk.py:2225-2237), and both overlap recomputations (sentence branch sdk.py:2124-2134, paragraph branch sdk.py:2159-2166).
• Sentence splitter abbreviation list + (?<=[.!?])\s+(?=[A-Z]) with ASCII-only capital lookahead (splitIntoSentences:703 vs sdk.py:2205-2223). The test even pins the known Python quirk that a real trailing etc. swallows the next sentence — good, diverging there would move boundaries.

🟢 Minor — UTF-8 validation is more lenient than Python's decoder (cpp/src/chunking.cpp:624-653). decodeText accepts any structurally-well-formed lead/continuation pattern, but Python's bytes.decode("utf-8") (the first link in the utf-8 → latin-1 chain at sdk.py:1461) rejects overlong encodings, surrogate halves (ED A0..), and code points > U+10FFFF (F4 90..). For those inputs Python falls through to latin-1 while this passes the bytes through as UTF-8, so chunk text diverges — contradicting the header's "decoded as latin-1, matching the Python extractor's encoding fallback chain." It only bites on malformed/adversarial files, not real documents, so it's a doc-accuracy nit more than a functional one: either tighten the validator to reject those sequences (then the latin-1 branch matches Python) or soften the header claim to note the malformed-UTF-8 edge.

Strengths:

• The parity harness is the highlight: a committed parity_expected.json generated from the Python method, replayed for byte-identical output across section/prose/mixed/unicode/sections<=3 fixtures, and a Python test_cpp_unicode_table.py that regenerates str.isupper() and fails on table drift (with a version-pinned skip) — closing the gap the C++ fixtures can't see. That's exactly how you keep a cross-language port honest.
• Fail-loudly throughout, matching GAIA conventions: unsupported doc formats, structured (CSV/JSON) formats, unknown extensions, binary/NUL content, and missing files/directories all raise actionable errors naming the type and the extractPlainTextFile / registerExtractor escape hatches — no silent empty-text returns. Error paths are covered by tests.
• Correct code-point (not byte) handling for lengths, strip, and last-N-tokens; ASCII-only lower-casing avoids locale corruption of UTF-8; the extractor registry is mutex-guarded and copies the callback out before invoking it. CMake wiring is sound — GAIA_TEST_FIXTURES_DIR and nlohmann_json are both already available to tests_mock.

[kovtcharov]

Windows was red on the first push. The cause was not a splitter bug: the runner checks out with core.autocrlf, which rewrote the fixture documents to CRLF, so the parity test compared chunks against a file that no longer matched the one the expectations were generated from.

I ruled out a parity bug before fixing it — converted a fixture to CRLF locally, reran the Python splitter against that same CRLF copy, and got byte-identical chunks from both runtimes at every chunk size. Both sides read binary and keep \r; the contract held, only the fixture bytes moved.

Fixed by pinning the fixture directory -text in .gitattributes (next to the existing rule protecting the image fixtures) rather than normalizing on read. Normalizing would have quietly diverged from Python, which does not normalize either, and parity_unicode.md carries CRLF on purpose to cover that path. Each case also records the line endings it was generated from, so a future translation fails with that reason instead of a wall of chunk diffs.

Worth carrying into #2798 (SKILL.md parser): the frontmatter parser has the same exposure, and the lesson is that a fixture asserting exact bytes needs a .gitattributes pin — a test that merely reads in binary mode still fails when git rewrote the file on checkout.

[kovtcharov]

Both remaining red checks are infrastructure, not this diff

Your Windows C++ build fix worked — C++ (windows-latest) is green. The two still red are neither yours:

1. Test GAIA CLI on Windows (Full Integration) — fails at the step Run FedericoCarboni/setup-ffmpeg@v3. A third-party action failing to fetch ffmpeg. A C++ chunking change cannot affect it; this is a transient setup/network failure.
2. C++ Integration Tests (STX) — the poisoned CMake cache on xsj-aimlab-stxp-08, one machine in the STX pool (fix(ci): STX C++ integration job trusts a partially-deleted CMake cache #2817, fix in PR fix(ci): validate the STX CMake cache instead of trusting bin/cmake.exe #2818). Machine-dependent, so it will look flaky.

Do not spend any more time on either. If you have already pushed the CRLF fix, you are done — please report what the Windows build failure actually was so it is captured for #2798, which has to handle the same CRLF class of bug in the SKILL.md parser.

[kovtcharov]

Windows is green after the .gitattributes pin — C++ (windows-latest), C++ Install Test (windows-latest) and C++ Shared Library (windows-latest) all pass, along with every Linux C++ job, CodeQL and the packaging matrix.

The one remaining red check is C++ Integration Tests (STX), which is the known self-hosted runner problem in #2817, not this change:

CMake Error: Could not find CMAKE_ROOT !!!
CMake Error: Error executing cmake::LoadCache(). Aborting.

It fails in the "Configure and build integration tests" step, before any code from this PR is compiled. It is also intermittent rather than persistent here — the same job passed on the previous push to this branch and failed on this one with no change to anything it touches.