feat(memory): add bounded cursor pagination for Memory entries

[knqiufan]

Feature description

Add an authorized, bounded Server query for Memory entries, implementing the directory portion of D2 in RFC #1455. Track this as a focused deliverable related to the broader capacity problem in #1321.

Problem and proposed solution

At aa697c5315204249e090acdf1a00d0582e851c5f, POST /v1/memory/entries/list accepts Scope, inactive-entry, and tag-filter inputs but has no page size or cursor. Its response contains the complete selected entries. The persistence implementation loads entry versions referenced by the complete Memory manifest.

Client-side pagination or slicing the fully assembled HTTP response leaves the Server work unbounded. Generic Artifact pagination also does not paginate entries inside one Memory Artifact.

Specify and implement a bounded query with an explicit default/maximum page size, deterministic ordering, an opaque continuation cursor, and clear consistency/error semantics. Agree on whether this is an additive operation or a versioned evolution; do not silently change existing clients from complete results to an undisclosed first page.

Acceptance criteria

• Publish request/response schemas, ordering, page-size bounds, end-of-list behavior, and cursor invalidation/expiry rules.
• Apply current authorization and supported inactive/tag filtering before page selection; continuation requests reauthorize access.
• Bind continuation to the relevant resource/query context. Reject tampered, mismatched, and invalid/expired cursors without revealing other Scopes or private data.
• Define concurrent append/revise/retire behavior: traverse a pinned revision or explicitly require restart when the consistency boundary changes. Do not silently skip or duplicate entries.
• Preserve exact Memory and entry/citation identities so a listed item can be inspected correctly.
• Bound entry-body loading and response assembly, not just the final response slice. State and measure any remaining whole-manifest cost; if it prevents the promised bound, agree the necessary storage/index work before claiming bounded browsing.
• Cover empty and multi-page results, the final page, inactive/tag filters, unauthorized requests, permission changes, invalid cursors, and concurrent mutations through observable behavior tests.
• Include a scale fixture or measurement demonstrating the declared bounds without freezing private call counts.
• Document compatibility/migration for existing clients; keep old full-list semantics explicit while supported. Regenerate OpenAPI-derived code and run contract/relevant behavior checks.

Alternatives considered

• Fetch everything and page in the UI: does not bound Server work or network payload.
• Truncate responses without continuation: loses data and misrepresents completeness.
• Reuse Artifact-directory pagination: addresses a different level of the data model.

Additional context

• Refs bug: Memory append storage and latency grow superlinearly with entry history #1321, docs(rfc): propose Tauri desktop control center #1455 (D2/AC-19), Tracking Issue: deliver a secure PowerContext desktop control center #1428, and the Web viewer work in feat(web): add a read-only Memory entry viewer #1454.
• Current request/response contract.
• Current entry loading.
• Scope: entry directory queries. Memory history pagination, full storage-growth remediation, split/compaction policy, and Desktop UI are separate work.
• Open for community implementation in a focused PR targeting master. Coordinate shared cursor/consistency and compatibility choices with the history work.
• This gates complete Memory browsing, not Desktop's initial explicit-save/search/exact-detail workflow.

Desktop consumer: #1654.

Related history-pagination deliverable: #1657. Both issues cover focused portions of #1321 and should agree on shared consistency/compatibility decisions.

[wutongyuonce]

Coordinated pagination SPEC proposal for #1656 and #1657

@knqiufan I would like to implement #1656 first and #1657 second as independently reviewable PRs. I reviewed the existing cursor, authentication, OpenAPI, and migration patterns; those established conventions do not need separate maintainer decisions.

Shared public contract

• Add paged query operations rather than silently changing the existing full-result operations.
• Proposed limit: default 50, maximum 100. A documented response-byte budget may end a page earlier and return continuation.
• Reuse the existing signed opaque cursor codec, TTL, and stable cursor errors. Bind it to endpoint, Scope, filters, limit, ordering version, and pinned Memory revision.
• Reauthorize every page. Permission changes stop continuation even when the cursor remains cryptographically valid.
• Pin the Memory revision on page one; later pages traverse that immutable boundary, so concurrent writes do not skip or duplicate results.
• Missing next_cursor means completion. Tampered, expired, cross-Scope, cross-endpoint, or query-mismatched cursors fail without disclosing private data.

#1656 entry query

• Inputs retain Scope, inactive-entry, and tag filters, plus limit and optional cursor.
• New paged order: entry_id ASC; legacy full-list behavior remains unchanged.
• Filtering occurs before page selection.
• Results preserve exact Memory revision, entry/version identity, state, and citation.
• Response assembly loads only page-sized entry bodies plus bounded lookahead.

#1657 history query

• Preserve the existing exclusive since_revision semantics, including 0, omitted/null, and positive nonexistent revisions.
• Order by (revision ASC, change_ordinal ASC).
• Count changes, not revisions, against the page limit and allow continuation within one revision.
• Preserve change metadata and exact references without expanding entry bodies.

Required persistence boundary

The current entry path decodes the complete manifest, and the current history path loads Artifact revisions before filtering. HTTP-level slicing would not satisfy either Issue.

I propose:

1. A normalized entry-directory query index with revision validity (valid_from_revision / valid_to_revision, or an equivalent representation), permitting keyset reads at the pinned revision without copying a complete manifest per revision.
2. Normalized change rows keyed by (scope, memory_artifact_id, revision, change_ordinal), permitting continuation inside a large revision.
3. Atomic index updates with the authoritative Memory Artifact commit. Artifact/Memory content remains authoritative; the new tables are rebuildable query indexes.
4. An explicit, resumable offline plan/apply/verify backfill for existing databases, following the repository processing-migration pattern. Startup does not silently perform an unbounded rebuild or fall back to unbounded reads.
5. No retention, split/routing, tombstone compaction, or broader feat(memory): define capacity contract for long-lived Memories #1718 capacity policy in these PRs.

Verification and PR split

#1656 will cover the additive entries contract, shared cursor context, directory index/backfill, generated code, observable behavior tests, and a scale measurement based on loaded rows/items and serialized bytes. #1657 will reuse that seam and add change-row/backfill support plus large-single-revision coverage.

The existing repository already determines cursor implementation, authorization, migration workflow shape, generated-code handling, and the two-PR split. The only coordination decision is whether the additive pinned-revision contract and normalized query-index boundary above are acceptable. Please confirm that boundary; unless maintainers identify a conflict, I will apply the existing conventions directly rather than ask for further implementation choices.

[knqiufan]

@wutongyuonce Thanks for coordinating both issues. I support additive paged operations with a pinned Memory revision and rebuildable query indexes, keeping the existing full-result operations unchanged. Separate PRs for #1656 and #1657 make sense.

Before freezing the contract, please clarify three points:

1. Tag assignments can change without advancing the Memory revision. Please define how tag-filtered continuation handles those changes, such as explicit cursor invalidation/restart, so revision pinning does not imply a snapshot guarantee it cannot provide.
2. For existing databases, document the behavior while backfill is incomplete, including whether only the new operations are unavailable or Server startup is blocked.
3. Define page-byte and oversized-item behavior. Scale evidence should cover database query work and incremental index-write costs, in addition to loaded items and serialized bytes, preserving the improvements from perf(memory): update search projections incrementally per revision #1709.

The normalized-index approach is acceptable in principle, provided authoritative records and exact historical references remain preserved. Shared decisions can stay in this issue and be reused by #1657.

[wutongyuonce]

@knqiufan Thanks. I rechecked these points against the current tag transaction, cursor/error, migration, exact-detail, and #1709 projection boundaries. I propose the following contract refinements.

1. Tag-filtered continuation

A Memory revision alone cannot snapshot mutable tag assignments, so tag-filtered pagination will use a separate durable tag generation owned by the tag persistence authority. It will not reuse governance_generation or reinterpret a Memory revision.

• Every effective normalized Memory-entry tag-set mutation increments the owning Memory tag generation in the same transaction. No-op replacements do not increment it.
• Tag writes already serialize on the owning Artifact head; the generation update stays under that lock.
• Page one records both the pinned Memory revision and current tag generation in the signed cursor.
• A continuation validates the generation before and after its bounded tag/index read. A mismatch returns the existing invalid-cursor class with a documented tag_state_changed reason; the client restarts from page one.
• Unfiltered cursors are not invalidated by tag-only changes.
• We will not collect per-entry tag etags into the cursor or materialize an unbounded tag snapshot.

This gives the pinned-revision guarantee for Memory state while explicitly making tag changes a restart boundary, so the contract does not imply a tag snapshot that the current model cannot provide.

2. Existing databases while backfill is incomplete

The migration is feature-scoped rather than a Server-wide startup gate.

• Fresh databases initialize the query-index marker as complete.
• Existing databases use an explicit, resumable offline plan/apply/verify backfill in bounded batches for SQLite and OceanBase. The procedure requires quiesced writes and rejects mixed old/new writers.
• Until verification marks the index complete, only the new paged Memory directory/history operations return 503 memory_query_index_unavailable, after normal authentication and authorization. They never fall back to the old unbounded read path.
• Existing full-result operations, exact detail, search, and ordinary Memory writes remain available outside the maintenance window. New-version writes maintain the new index even while historical backfill is incomplete, but the paged operations remain unavailable until full verification.
• Failed or interrupted apply is resumable and does not advertise readiness. Existing authoritative Artifact/Memory data is never rewritten by the backfill.

This keeps an optional new read capability from taking down unrelated Server behavior while still failing closed for the bounded-work claim.

3. Page bytes, oversized items, and scale evidence

I will narrow the new #1656 operation to a directory response. Each item contains the exact citation (memory_ref, entry_id, entry_version_id), logical version, kind, and state. It does not expand text, source_refs, or artifact_refs; callers use the existing exact-detail operation for the body. The legacy full-list response remains unchanged.

• Item limit remains default 50, maximum 100.
• The Server also applies a fixed 4 MiB encoded item-payload budget, consistent with the existing bounded Source listing pattern, plus a separately bounded response envelope/cursor.
• Selection stops before the next item would exceed either limit and returns continuation from the last emitted key.
• All legal v1 directory fields already have bounded stored identities/kind/state, so one legal directory item cannot approach the byte cap. If corrupt legacy data or a future incompatible shape produces a single item that cannot fit, the operation returns 413 memory_directory_item_too_large, with no partial page and no advanced cursor. Entry-body size does not affect directory traversal.
• feat(memory): add bounded pagination for Memory changes and history #1657 reuses the same page budget. Its compact change item is also bounded by identifier and reason limits, and it still supports continuation within a revision.

Scale evidence will cover 200, 1,000, and 5,000-entry fixtures on SQLite and OceanBase where available, recording:

• query rows examined/returned as limit + bounded lookahead, independent of total directory/history size;
• zero entry-body expansion for the directory operation;
• encoded page bytes and final-page behavior;
• incremental directory/change-index row mutations per Memory write, proportional to the changed entries/changes rather than the total manifest/history;
• bounded backfill batch work separately from steady-state reads/writes; and
• unchanged-entry projection rows remaining untouched, preserving the perf(memory): update search projections incrementally per revision #1709 guarantee.

The normalized index remains rebuildable query state. Artifact Revisions, Memory manifests, entry versions, tag rows, and exact historical references remain authoritative.

With these refinements, the shared contract remains additive and pinned by Memory revision, tag-filtered cursors have an explicit restart boundary, incomplete migration cannot masquerade as bounded browsing, and both response assembly and persistence work have measurable bounds. If this resolves the three points, I will implement #1656 first and reuse these shared rules in the separate #1657 PR.

[PsiACE]

Note: We'll revisit next steps as the migration and Memory redesign become clearer; see this comment.