You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
TableChunker.chunk() returns source ranges relative to an extracted substring rather than to the original input. Oversized table chunks omit the table's starting offset, and prose after a table also starts again at zero. A consumer using text.slice(chunk.start, chunk.end) for citations or highlighting consequently selects different text from chunk.content.
Steps To Reproduce
On unchanged upstream main at 72a46c76b4507e1f98b41d5050a92bd30587eb70:
import{TableChunker,whitespaceTokenizer}from"@voltagent/rag";consttext="Intro.\n| name | value |\n| apple | green |\nClosing sentence.";constchunks=newTableChunker(whitespaceTokenizer).chunk(text,{maxTokens: 4});for(constchunkofchunks){console.log(chunk.content,chunk.start,chunk.end,text.slice(chunk.start,chunk.end));}
Examples of the actual output:
content='| name | value', start=0, end=14, source='Intro.\n| name '
content='Closing sentence.', start=0, end=17, source='Intro.\n| name | v'
The short-table path does apply the original starting offset; the token fallback and prose paths do not.
Expected behavior
Returned start/end offsets should address the original input for both table chunks and surrounding prose, so source extraction returns the chunk text. Position metadata should agree with those offsets. Contents, grouping and token limits should be unchanged.
Packages
@voltagent/rag 1.0.2 source on current main; Node.js 24.19.0, Windows. Executed the actual public exports with the declared js-tiktoken dependency. No model or external service is involved.
Additional Context
I plan to carry the original prose-buffer offset into the delegated sentence chunks, translate delegated table-token offsets by the table start, and add public-API regressions for leading/trailing prose, split tables and unchanged short tables. I searched existing issues and PRs for table chunk offsets and did not find a matching fix. This is separate from recursive chunk ID handling in #1465 and JSON changes in #1459/#1463.
AI disclosure: Codex investigated and reproduced this under LJK-ikun's direction. This report states observed local output, not independent human validation.
Activity