Skip to content

feat(benchmark): add LoCoMo-Plus evaluation - #1693

Merged
Teingi merged 3 commits into
oceanbase:masterfrom
sunlight798:locomo-plus
Sep 21, 2026
Merged

Teingi merged 3 commits into
oceanbase:masterfrom
sunlight798:locomo-plus

Conversation

@sunlight798

Copy link
Copy Markdown
Contributor

Which issue or RFC does this PR close?

Closes #1692.

Related to #1601 and #1263.

Rationale for this change

LoCoMo-Plus extends factual recall evaluation with cognitive memory: whether an agent retains implicit user constraints and applies them to later requests. It covers causal, state, goal, and value constraints.

This benchmark evaluates PowerContext’s complete memory workflow, from conversation ingestion and Memory extraction to retrieval, answer generation, and scoring.

What changes are included in this PR?

  • Add a LoCoMo-Plus adapter using public Runtime APIs and isolated SQLite state.
  • Support smoke and full profiles. Smoke reduces only the sample count; selected cases retain the same complete histories and execution settings as full runs.
  • Include ten cognitive samples with their complete conversation histories.
  • Support memory, memory-source, full-context, query-only, and oracle-cue modes.
  • Download full data from a fixed upstream revision, validate its structure, and document malformed-cue exclusions and timestamp handling. Dataset hash validation is not required.
  • Add category-specific scoring, stage-level failure reporting, usage accounting, checkpoint recovery, and offline report replay.
  • Add documentation, .env.example, benchmark-local ignore rules, and automated tests.

Are there any user-facing changes?

Contributors can run the benchmark through python -m benchmark.locomo_plus, inspect workloads with --dry-run, and evaluate the bundled samples without downloading the full dataset.

No changes to public PowerContext APIs or the existing LoCoMo workflow are required. Runtime artifacts remain local and are excluded from version control.

How was this change tested?

Automated validation

uv run pytest tests/test_locomo_plus*.py -q
uv run prek run -a

All 49 tests and all pre-commit checks passed. Coverage includes dataset loading, complete-history preservation, scoring, failure diagnostics, configured request limits, checkpoint recovery, and offline replay.

Fresh real-service smoke test

Ran cognitive:0000 through the memory mode with all 20 historical sessions and 371 dialogue turns.

  • Generator and judge: anthropic:MiniMax-M3
  • Embedding: openai:text-embedding-v4
  • All stages completed successfully: ingestion → extraction → retrieval → generation → judging.
  • No execution failures; the target evidence was retrieved, and the judge marked the answer correct.
  • Recorded duration: 147.6 seconds.
  • Recorded usage: 45 model requests and 141,353 tokens, including ingestion and embedding.
  • Offline replay reproduced the saved summary.

Ten-case smoke comparison

Both modes used the same ten cognitive cases and all 253 historical sessions, with identical answer-model and judge settings.

Metric PowerContext memory full-context baseline
Completed evaluations 10/10 10/10
Correct answers 3/10 0/10
Execution failures after recovery 0 0

These are fixed-subset smoke results. The full benchmark was not executed, and generation and judging used the same model.

AI usage statement

OpenAI Codex assisted with implementation, documentation and smoke validation.

Add public Runtime integration, full-history smoke and full profiles, comparison modes, resumable runs, and offline reporting. Include ten complete cognitive samples, configuration examples, and benchmark tests.

Refs oceanbase#1601, oceanbase#1263.
Comment thread benchmark/locomo_plus/README.md Outdated
@Teingi
Teingi merged commit 32be40d into oceanbase:master Sep 21, 2026
22 checks passed
@Alanxtl Alanxtl linked an issue Sep 21, 2026 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

evaluation: implement the LoCoMo-Plus benchmark adapter evaluation: add a bounded LoCoMo-Plus (cognitive memory) workload

2 participants