Skip to content

[BUG] rag: table chunks report source ranges from extracted substrings #1467

Open

Description

@LJK-ikun

Describe the bug

TableChunker.chunk() returns source ranges relative to an extracted substring rather than to the original input. Oversized table chunks omit the table's starting offset, and prose after a table also starts again at zero. A consumer using text.slice(chunk.start, chunk.end) for citations or highlighting consequently selects different text from chunk.content.

Steps To Reproduce

On unchanged upstream main at 72a46c76b4507e1f98b41d5050a92bd30587eb70:

import { TableChunker, whitespaceTokenizer } from "@voltagent/rag";

const text = "Intro.\n| name | value |\n| apple | green |\nClosing sentence.";
const chunks = new TableChunker(whitespaceTokenizer).chunk(text, { maxTokens: 4 });
for (const chunk of chunks) {
  console.log(chunk.content, chunk.start, chunk.end, text.slice(chunk.start, chunk.end));
}

Examples of the actual output:

content='| name | value', start=0, end=14, source='Intro.\n| name '
content='Closing sentence.', start=0, end=17, source='Intro.\n| name | v'

The short-table path does apply the original starting offset; the token fallback and prose paths do not.

Expected behavior

Returned start/end offsets should address the original input for both table chunks and surrounding prose, so source extraction returns the chunk text. Position metadata should agree with those offsets. Contents, grouping and token limits should be unchanged.

Packages

@voltagent/rag 1.0.2 source on current main; Node.js 24.19.0, Windows. Executed the actual public exports with the declared js-tiktoken dependency. No model or external service is involved.

Additional Context

I plan to carry the original prose-buffer offset into the delegated sentence chunks, translate delegated table-token offsets by the table start, and add public-API regressions for leading/trailing prose, split tables and unchanged short tables. I searched existing issues and PRs for table chunk offsets and did not find a matching fix. This is separate from recursive chunk ID handling in #1465 and JSON changes in #1459/#1463.

AI disclosure: Codex investigated and reproduced this under LJK-ikun's direction. This report states observed local output, not independent human validation.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions