|
1 | 1 | { |
2 | | - "_note": "Expected chunk boundaries produced by RAGSDK._split_text_into_chunks (src/gaia/rag/sdk.py). Regenerate by calling that method on the fixture text with the chunk_size/chunk_overlap of each case.", |
| 2 | + "_note": "Expected chunk boundaries produced by RAGSDK._split_text_into_chunks (src/gaia/rag/sdk.py) for the documents in this directory. file_has_crlf records the line endings each case was generated from; see README.md.", |
3 | 3 | "cases": [ |
4 | 4 | { |
5 | 5 | "file": "parity_sections.md", |
6 | 6 | "chunk_size": 60, |
7 | 7 | "chunk_overlap": 15, |
| 8 | + "file_has_crlf": false, |
8 | 9 | "chunks": [ |
9 | 10 | "# GAIA Retrieval Notes The retrieval stack indexes documents locally so a question can be answered\nwithout sending the source text to a hosted model. Everything below is a\ndescription of that pipeline as it exists in the Python runtime today.", |
10 | 11 | "of that pipeline as it exists in the Python runtime today. ## Extraction\n\nText files are read as UTF-8, with a latin-1 fallback for legacy encodings.\nMarkdown is treated as text; the headers are what the splitter later uses to\nfind section boundaries. Source files go down the same path.", |
|
22 | 23 | "file": "parity_sections.md", |
23 | 24 | "chunk_size": 500, |
24 | 25 | "chunk_overlap": 100, |
| 26 | + "file_has_crlf": false, |
25 | 27 | "chunks": [ |
26 | 28 | "# GAIA Retrieval Notes The retrieval stack indexes documents locally so a question can be answered\nwithout sending the source text to a hosted model. Everything below is a\ndescription of that pipeline as it exists in the Python runtime today. ## Extraction\n\nText files are read as UTF-8, with a latin-1 fallback for legacy encodings.\nMarkdown is treated as text; the headers are what the splitter later uses to\nfind section boundaries. Source files go down the same path. ## Chunking Chunks are sized in estimated tokens, where one token is approximated as four\ncharacters. A paragraph that fits inside the budget is kept whole so a single\nidea does not straddle two chunks. A paragraph that does not fit is split on\nsentence boundaries instead of on words. --- ## Overlap Each chunk carries the tail of its predecessor. Without that overlap a question\nwhose answer spans a boundary retrieves half an answer, which reads as a\nconfident but incomplete response. The overlap is trimmed to a word boundary. ## Encoding Notes A café menu, a naïve reader, and a résumé with accents all round-trip through\nthe extractor unchanged. Counting is done in code points, not bytes, so\nmulti-byte text does not shift chunk boundaries between the two runtimes. ## Limits The index is bounded by a maximum file count and a maximum chunk count. When\neither limit is reached the SDK refuses to index more rather than silently\nevicting content a user still expects to be searchable." |
27 | 29 | ] |
|
30 | 32 | "file": "parity_prose.txt", |
31 | 33 | "chunk_size": 40, |
32 | 34 | "chunk_overlap": 10, |
| 35 | + "file_has_crlf": false, |
33 | 36 | "chunks": [ |
34 | 37 | "Retrieval quality is decided long before the embedding model runs. It is decided by where the text was cut.", |
35 | 38 | "is decided by where the text was cut. A chunk that ends mid-argument gives the reranker two halves that each look irrelevant, and the answer that comes back is confidently wrong.", |
|
45 | 48 | "file": "parity_prose.txt", |
46 | 49 | "chunk_size": 120, |
47 | 50 | "chunk_overlap": 30, |
| 51 | + "file_has_crlf": false, |
48 | 52 | "chunks": [ |
49 | 53 | "Retrieval quality is decided long before the embedding model runs. It is decided by where the text was cut. A chunk that ends mid-argument gives the reranker two halves that each look irrelevant, and the answer that comes back is confidently wrong. Dr. Ada Lovelace would have called that a machine doing exactly what it was told. The splitter therefore prefers sentence boundaries over word boundaries, e.g. it will overshoot the token budget slightly rather than cut a clause in half.", |
50 | 54 | "boundaries over word boundaries, e.g. it will overshoot the token budget slightly rather than cut a clause in half. Mr. Babbage, vs. the alternative, would probably agree. That preference costs a little index size and buys a lot of answer quality, i.e. it is the trade every retrieval system eventually makes. Overlap is the second half of the same idea, etc. Without it, a fact that straddles two chunks is retrievable from neither. The second paragraph is short.", |
|
55 | 59 | "file": "parity_mixed.txt", |
56 | 60 | "chunk_size": 45, |
57 | 61 | "chunk_overlap": 12, |
| 62 | + "file_has_crlf": false, |
58 | 63 | "chunks": [ |
59 | 64 | "Release Checklist", |
60 | 65 | "Release Checklist This file exercises the title-line heuristic: a short capitalized line with no\ntrailing punctuation, surrounded by blank lines, is treated as a section break.\n\nBuild The Wheel", |
|
73 | 78 | "file": "parity_mixed.txt", |
74 | 79 | "chunk_size": 200, |
75 | 80 | "chunk_overlap": 40, |
| 81 | + "file_has_crlf": false, |
76 | 82 | "chunks": [ |
77 | 83 | "Release Checklist This file exercises the title-line heuristic: a short capitalized line with no\ntrailing punctuation, surrounded by blank lines, is treated as a section break.\n\nBuild The Wheel Run the packaging job and confirm the artifact hash matches the one recorded in\nthe release notes. If it does not, stop and investigate before tagging. ===\n\nlowercase heading is not a boundary A line that starts lowercase is left inside the previous section, and so is a\nline that ends with punctuation, because the heuristic requires a capital start\nand no trailing period. This line is far too long to be treated as a section title because it runs well\npast the one hundred character limit the heuristic enforces on candidate lines. # Hash Marks Start A Section", |
78 | 84 | "be treated as a section title because it runs well past the one hundred character limit the heuristic enforces on candidate lines. # Hash Marks Start A Section Any line beginning with a hash is a boundary regardless of length or case. That\ncovers Markdown headers and shell comments alike, which is why a source file\nfull of comments fragments into many sections.\n\nFinal Notes Dr. Knuth wrote about literate programming. He argued that a program should be\nwritten for a human reader first, e.g. as an essay, and only incidentally for\nthe machine. That argument applies to chunking too: the cut points should fall\nwhere a human would pause, not where a byte counter happens to land." |
|
82 | 88 | "file": "parity_unicode.md", |
83 | 89 | "chunk_size": 35, |
84 | 90 | "chunk_overlap": 8, |
| 91 | + "file_has_crlf": true, |
85 | 92 | "chunks": [ |
86 | 93 | "# Überblick Über die Chunking-Strategie", |
87 | 94 | "Über die Chunking-Strategie Die Zeile darüber beginnt mit einem Großbuchstaben und endet ohne Satzzeichen, also behandelt die Heuristik sie als Abschnittstitel.", |
|
97 | 104 | "file": "parity_unicode.md", |
98 | 105 | "chunk_size": 150, |
99 | 106 | "chunk_overlap": 25, |
| 107 | + "file_has_crlf": true, |
100 | 108 | "chunks": [ |
101 | 109 | "# Überblick Über die Chunking-Strategie Die Zeile darüber beginnt mit einem Großbuchstaben und endet ohne Satzzeichen, also behandelt die Heuristik sie als Abschnittstitel. Ελληνικά κείμενα Greek and Cyrillic headings must start a section too, exactly as they do in the Python runtime. Привет is a heading below. Привет мир A paragraph separated by a non-breaking space line follows this one. That line held only a non-breaking space, which Python treats as whitespace, so the two paragraphs are separated rather than joined. Đường dẫn tài liệu", |
102 | 110 | "treats as whitespace, so the two paragraphs are separated rather than joined. Đường dẫn tài liệu Vietnamese capitals are also cased, so this heading opens its own section and the chunk boundaries stay identical between the two runtimes." |
|
106 | 114 | "file": "parity_sections.md", |
107 | 115 | "chunk_size": 40, |
108 | 116 | "chunk_overlap": 0, |
| 117 | + "file_has_crlf": false, |
109 | 118 | "chunks": [ |
110 | 119 | "# GAIA Retrieval Notes The retrieval stack indexes documents locally so a question can be answered\nwithout sending the source text to a hosted model.", |
111 | 120 | "GAIA Retrieval Notes The retrieval stack indexes documents locally so a question can be answered without sending the source text to a hosted model. Everything below is a\ndescription of that pipeline as it exists in the Python runtime today.", |
|
125 | 134 | "file": "parity_three_sections.txt", |
126 | 135 | "chunk_size": 200, |
127 | 136 | "chunk_overlap": 40, |
| 137 | + "file_has_crlf": false, |
128 | 138 | "chunks": [ |
129 | 139 | "# Alpha The first section body is one line that ends with a period. # Beta The second section body is also a single line ending in a period. # Gamma The third section body closes this fixture out with one more line." |
130 | 140 | ] |
|
133 | 143 | "file": "parity_four_sections.txt", |
134 | 144 | "chunk_size": 200, |
135 | 145 | "chunk_overlap": 40, |
| 146 | + "file_has_crlf": false, |
136 | 147 | "chunks": [ |
137 | 148 | "# Alpha\n\nThe first section body is one line that ends with a period. # Beta\n\nThe second section body is also a single line ending in a period. # Gamma\n\nThe third section body closes this fixture out with one more line. # Delta\n\nThe fourth section body only exists in the four-section fixture." |
138 | 149 | ] |
|
0 commit comments