Skip to content

feat(agent): extend chat context picker#988

Closed
Nikhil (shadowfax92) wants to merge 1 commit intodevfrom
polecat/pearl/bosmain-d4u@moxnrd5t
Closed

feat(agent): extend chat context picker#988
Nikhil (shadowfax92) wants to merge 1 commit intodevfrom
polecat/pearl/bosmain-d4u@moxnrd5t

Conversation

@shadowfax92
Copy link
Copy Markdown
Contributor

Summary

  • Add a protected context API for fuzzy file and memory lookup.
  • Extend the @ picker to support tabs, files, and memories with attached-context chips.
  • Carry attached context through BrowserOS chat/agent requests and inject it outside USER_QUERY on the server.

Fixes #951

Test plan

  • git diff --check origin/dev...HEAD
  • bun --env-file=apps/server/.env.development test apps/agent/entrypoints/sidepanel/index/useChatSession.test.ts apps/server/tests/agent/format-message.test.ts apps/server/tests/api/services/chat-service.test.ts
  • bunx biome check on touched files
  • bun run typecheck from packages/browseros-agent/apps/server (blocked locally by existing tsc resolution issue tracked in bosmain-woi)
  • bun run typecheck from packages/browseros-agent/apps/agent (blocked locally by existing wxt script resolution issue tracked in bosmain-090)
  • bun run build from packages/browseros-agent/apps/agent (blocked locally by existing graphql-codegen script resolution issue tracked in bosmain-4xe)

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 9, 2026

✅ Tests passed — 1236/1240

Suite Passed Failed Skipped
agent 81/81 0 0
build 9/9 0 0
eval 93/93 0 0
server-agent 268/268 0 0
server-api 205/205 0 0
server-browser 4/4 0 0
server-integration 9/10 0 1
server-lib 245/245 0 0
server-root 60/63 0 3
server-skills 31/31 0 0
server-tools 231/231 0 0

View workflow run

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 9, 2026

Greptile Summary

This PR extends the BrowserOS chat @ picker to support files and memories alongside tabs, adds a protected /context/files and /context/memories server API with fuzzy search, carries attached context through the request pipeline, and injects it outside <USER_QUERY> in the formatted prompt.

  • New context.ts server routes walk the workspace directory and read memory files, surfacing results via Fuse.js fuzzy ranking; format-message.ts serialises attachments as <attached_context> blocks with tag-stripping sanitisation.
  • The client adds attachedContexts state, ChatAttachedContexts chip display, and wires everything through useChatActions, buildChatRequestBody, and both the LLM and ACP request paths.

Confidence Score: 3/5

The core prompt formatting and sanitisation are solid, but a live-filtering bug in memory ID construction will cause visually selected items to silently deselect during typing in the @ picker.

Memory entry IDs embed a positional index that shifts whenever Fuse re-ranks results as the user types, causing selected items to appear unselected mid-interaction. The double-indexing of memory content (sections + individual lines) also produces overlapping results that would confuse users. Both issues live on the newly added context routes that are central to the feature.

context.ts (memory ID stability, double indexing), agents.ts (missing title/source length caps), use-available-context.ts (unused items export)

Important Files Changed

Filename Overview
packages/browseros-agent/apps/server/src/api/routes/context.ts New /context/files and /context/memories endpoints. Memory entry IDs are positionally unstable (breaking live-filter selection), and each memory file is double-indexed at section and line level producing overlapping search results.
packages/browseros-agent/apps/agent/components/elements/use-available-context.ts New hook that fetches files and memories with 150ms debounce. Returns an unused items computed value that duplicates logic in the popover component.
packages/browseros-agent/apps/server/src/api/routes/agents.ts Extended to accept and parse contextAttachments for sidepanel agent chat. Manual parser is missing title/source length caps that exist in the Zod schema path.
packages/browseros-agent/apps/server/src/agent/format-message.ts Adds formatContextAttachments with sanitization of prompt-delimiter tags in both content and attributes. Sanitization logic looks correct.
packages/browseros-agent/apps/agent/components/elements/tab-picker-popover.tsx Extended to show Files and Memories groups in addition to Tabs. Keyboard navigation updated to span all item groups correctly.
packages/browseros-agent/apps/agent/lib/messaging/server/buildChatRequestBody.ts Strips id/truncated from ContextAttachment before sending to server, correctly limiting payload to what the server schema requires.
packages/browseros-agent/apps/server/src/api/types.ts Adds ContextAttachmentSchema with proper field bounds and wires it into ChatRequestSchema with max(10) array constraint.
packages/browseros-agent/apps/agent/entrypoints/sidepanel/index/Chat.tsx Adds attachedContexts state alongside attachedTabs with toggle/remove helpers; context cleared on message submit.
packages/browseros-agent/apps/server/tests/agent/format-message.test.ts New tests covering attachment injection and prompt-delimiter sanitisation. Good coverage of the injection-stripping path.

Sequence Diagram

sequenceDiagram
    participant U as User at-picker
    participant H as useAvailableContext
    participant S as Server /context
    participant CP as ChatFooter
    participant BR as buildChatRequestBody
    participant FS as format-message
    participant LLM as LLM/ACP agent

    U->>H: open picker, type filter
    H->>S: "GET /context/files?cwd=X&q=filter"
    H->>S: "GET /context/memories?q=filter"
    S-->>H: files and memories arrays
    H-->>U: render Files + Memories groups
    U->>CP: toggle item, attachedContexts updated
    U->>CP: submit message
    CP->>BR: buildChatRequestBody with contextAttachments
    BR-->>CP: strip id and truncated fields
    CP->>S: POST /chat or /agents/id/sidepanel/chat
    S->>FS: formatUserMessage with contextAttachments
    FS-->>S: attached_context blocks prepended outside USER_QUERY
    S->>LLM: formatted prompt with context
Loading

Comments Outside Diff (1)

  1. packages/browseros-agent/apps/server/src/api/routes/context.ts, line 1416-1427 (link)

    P1 Unstable memory IDs break selection state during live filtering

    Memory IDs are built with a positional index from the ranked/sliced result: `memory:${entry.source}:${index}:${title}`. When the user types additional characters into the @ filter, Fuse re-ranks entries and the same entry lands at a different position, so its ID changes. The selectedContextIds set in the popover still holds the old ID, causing a just-selected memory to appear unselected after each additional keystroke.

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: packages/browseros-agent/apps/server/src/api/routes/context.ts
    Line: 1416-1427
    
    Comment:
    **Unstable memory IDs break selection state during live filtering**
    
    Memory IDs are built with a positional index from the ranked/sliced result: `` `memory:${entry.source}:${index}:${title}` ``. When the user types additional characters into the `@` filter, Fuse re-ranks entries and the same entry lands at a different position, so its ID changes. The `selectedContextIds` set in the popover still holds the old ID, causing a just-selected memory to appear unselected after each additional keystroke.
    
    How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
Fix the following 4 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 4
packages/browseros-agent/apps/server/src/api/routes/context.ts:1416-1427
**Unstable memory IDs break selection state during live filtering**

Memory IDs are built with a positional index from the ranked/sliced result: `` `memory:${entry.source}:${index}:${title}` ``. When the user types additional characters into the `@` filter, Fuse re-ranks entries and the same entry lands at a different position, so its ID changes. The `selectedContextIds` set in the popover still holds the old ID, causing a just-selected memory to appear unselected after each additional keystroke.

### Issue 2 of 4
packages/browseros-agent/apps/agent/components/elements/use-available-context.ts:98-107
**Dead code: `items` is never consumed by callers**

`useAvailableContext` computes and returns an `items` array, but every caller (`tab-picker-popover.tsx`) immediately recomputes its own `items` memo from the individual `tabs`, `files`, and `memories` arrays. The exported `items` is never referenced. Per the team's rule against leaving dead code, it should be removed from both the hook and its return type.

```suggestion
  return {
    tabs,
    files,
    memories,
    allTabs,
    isLoading: isLoadingTabs || isLoadingContext,
    hasWorkspace: Boolean(selectedFolder?.path),
  }
}
```

### Issue 3 of 4
packages/browseros-agent/apps/server/src/api/routes/context.ts:149-159
**Double-indexing memory content creates duplicate/overlapping search results**

`loadMemoryEntries` adds every memory file's content twice: once per H2 section (full section block) and again once per individual non-heading line. For a file with 3 sections containing 10 lines each, this produces 3 + 30 = 33 entries. Because section content contains the same lines as the line-level entries, Fuse can return both `## Section\nLine A\nLine B` and `Line A` as top hits for the same query, showing confusing duplicates to the user. Consider choosing one granularity (section-level is usually more useful) or explicitly separating the two strategies.

```suggestion
      const sections = content.split(/^## /m).filter(Boolean)
      for (const section of sections) {
        entries.push({ source: file, content: `## ${section}`.trim() })
      }
```

### Issue 4 of 4
packages/browseros-agent/apps/server/src/api/routes/agents.ts:883-920
**Missing title and source length caps in manual `contextAttachments` parser**

The Zod `ContextAttachmentSchema` in `types.ts` validates `title.max(500)` and `source.max(1000)`, but the hand-written `parseContextAttachments` function here only checks that `title` is non-empty and `content` is ≤ 50,000 bytes — `title` and `source` are unbounded. A caller can send an arbitrarily long title or source string that passes validation and is then injected into prompt attributes via `sanitizeAttribute`.

Reviews (1): Last reviewed commit: "feat: extend chat context picker (bosmai..." | Re-trigger Greptile

Comment on lines +149 to +159
const sections = content.split(/^## /m).filter(Boolean)
for (const section of sections) {
entries.push({ source: file, content: `## ${section}`.trim() })
}

for (const line of content.split('\n')) {
const trimmed = line.trim()
if (trimmed && !trimmed.startsWith('#')) {
entries.push({ source: file, content: trimmed })
}
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Double-indexing memory content creates duplicate/overlapping search results

loadMemoryEntries adds every memory file's content twice: once per H2 section (full section block) and again once per individual non-heading line. For a file with 3 sections containing 10 lines each, this produces 3 + 30 = 33 entries. Because section content contains the same lines as the line-level entries, Fuse can return both ## Section\nLine A\nLine B and Line A as top hits for the same query, showing confusing duplicates to the user. Consider choosing one granularity (section-level is usually more useful) or explicitly separating the two strategies.

Suggested change
const sections = content.split(/^## /m).filter(Boolean)
for (const section of sections) {
entries.push({ source: file, content: `## ${section}`.trim() })
}
for (const line of content.split('\n')) {
const trimmed = line.trim()
if (trimmed && !trimmed.startsWith('#')) {
entries.push({ source: file, content: trimmed })
}
}
const sections = content.split(/^## /m).filter(Boolean)
for (const section of sections) {
entries.push({ source: file, content: `## ${section}`.trim() })
}
Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/browseros-agent/apps/server/src/api/routes/context.ts
Line: 149-159

Comment:
**Double-indexing memory content creates duplicate/overlapping search results**

`loadMemoryEntries` adds every memory file's content twice: once per H2 section (full section block) and again once per individual non-heading line. For a file with 3 sections containing 10 lines each, this produces 3 + 30 = 33 entries. Because section content contains the same lines as the line-level entries, Fuse can return both `## Section\nLine A\nLine B` and `Line A` as top hits for the same query, showing confusing duplicates to the user. Consider choosing one granularity (section-level is usually more useful) or explicitly separating the two strategies.

```suggestion
      const sections = content.split(/^## /m).filter(Boolean)
      for (const section of sections) {
        entries.push({ source: file, content: `## ${section}`.trim() })
      }
```

How can I resolve this? If you propose a fix, please make it concise.

Comment on lines 883 to 920
: { error: 'Invalid selectedTextSource' }
}

function parseContextAttachments(
value: unknown,
): { value?: UserContextAttachment[] } | { error: string } {
if (value === undefined) return { value: undefined }
if (!Array.isArray(value) || value.length > 10) {
return { error: 'Invalid contextAttachments' }
}

const attachments: UserContextAttachment[] = []
for (const item of value) {
if (!item || typeof item !== 'object' || Array.isArray(item)) {
return { error: 'Invalid contextAttachments' }
}
const record = item as Record<string, unknown>
const kind = record.kind
const title = readOptionalTrimmedString(record, 'title')
const source = readOptionalTrimmedString(record, 'source')
const content = readOptionalString(record, 'content')
if (
(kind !== 'file' && kind !== 'memory') ||
!title ||
!content ||
content.length > 50_000
) {
return { error: 'Invalid contextAttachments' }
}
attachments.push({ kind, title, source, content })
}

return { value: attachments }
}

function readOptionalString(
record: Record<string, unknown>,
key: string,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Missing title and source length caps in manual contextAttachments parser

The Zod ContextAttachmentSchema in types.ts validates title.max(500) and source.max(1000), but the hand-written parseContextAttachments function here only checks that title is non-empty and content is ≤ 50,000 bytes — title and source are unbounded. A caller can send an arbitrarily long title or source string that passes validation and is then injected into prompt attributes via sanitizeAttribute.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/browseros-agent/apps/server/src/api/routes/agents.ts
Line: 883-920

Comment:
**Missing title and source length caps in manual `contextAttachments` parser**

The Zod `ContextAttachmentSchema` in `types.ts` validates `title.max(500)` and `source.max(1000)`, but the hand-written `parseContextAttachments` function here only checks that `title` is non-empty and `content` is ≤ 50,000 bytes — `title` and `source` are unbounded. A caller can send an arbitrarily long title or source string that passes validation and is then injected into prompt attributes via `sanitizeAttribute`.

How can I resolve this? If you propose a fix, please make it concise.

@shadowfax92
Copy link
Copy Markdown
Contributor Author

Refinery rejected this merge request after Greptile reported a branch-caused P1: memory IDs use ranked result positions, so live @ picker filtering can change IDs and break selected memory state. Source issue bosmain-d4u has been reopened with details for rework. CI otherwise passed after rerunning an unrelated server-tools flake.

@shadowfax92 Nikhil (shadowfax92) deleted the polecat/pearl/bosmain-d4u@moxnrd5t branch May 9, 2026 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feat] Extend @ context picker to attach files, memories, bookmarks & more

1 participant