|
| 1 | +# Daksh Agent Memory Benchmarker PR |
| 2 | + |
| 3 | +Date: 2026-07-08 |
| 4 | + |
| 5 | +Status: submitted as upstream PR #1. |
| 6 | + |
| 7 | +URL: https://github.com/dakshjain-1616/agent-memory-benchmarker/pull/1 |
| 8 | + |
| 9 | +Evidence comment: https://github.com/TerminallyLazy/Tree-Ring-Memory/issues/26#issuecomment-4917005318 |
| 10 | + |
| 11 | +## Fit |
| 12 | + |
| 13 | +Daksh Jain's Agent Memory Benchmarker compares memory backends across common |
| 14 | +factual recall and contradiction detection tasks. Tree Ring Memory fits as an |
| 15 | +optional local backend because it exposes write, recall, and clear behavior |
| 16 | +through the public `tree-ring` CLI without requiring cloud credentials. |
| 17 | + |
| 18 | +## Submission |
| 19 | + |
| 20 | +The PR adds an optional Tree Ring Memory backend: |
| 21 | + |
| 22 | +- `agent_memory_benchma/backends/tree_ring_backend.py` implements the local |
| 23 | + CLI-backed backend. |
| 24 | +- `tree_ring` is registered only when `tree-ring` is on `PATH` or |
| 25 | + `TREE_RING_BIN` points to a binary, so default installs retain the original |
| 26 | + backend set. |
| 27 | +- Each backend instance uses an isolated temporary Tree Ring root by default. |
| 28 | +- `TREE_RING_ROOT` can point maintainers at an explicit root; in that mode, |
| 29 | + clear removes only memories the backend instance added. |
| 30 | +- `TREE_RING_PROJECT` controls the Tree Ring project name. |
| 31 | +- Search uses direct recall plus a query-derived fallback so benchmark queries |
| 32 | + can recover fact-oriented entries without using expected answers. |
| 33 | +- README usage docs describe the optional backend and environment variables. |
| 34 | +- Tests were updated to keep the default registry stable while also accepting |
| 35 | + the optional Tree Ring backend when installed. |
| 36 | + |
| 37 | +## Validation |
| 38 | + |
| 39 | +- `.venv/bin/python -m compileall agent_memory_benchma` |
| 40 | +- `env -u TREE_RING_BIN PATH=/usr/bin:/bin .venv/bin/python -m pytest tests/test_backends.py::test_backend_registry_contains_all tests/test_benchmarker.py::TestBackendRegistry -q` |
| 41 | +- `TREE_RING_BIN=/Users/lazy/Projects/Tree_Ring_Memory/target/debug/tree-ring .venv/bin/python -m pytest tests/test_backends.py::test_backend_registry_contains_all tests/test_benchmarker.py::TestBackendRegistry -q` |
| 42 | +- Direct Tree Ring backend smoke: add, query, and clear passed; query returned |
| 43 | + `Alice likes cerulean blue.` with score `0.72`. |
| 44 | +- `TREE_RING_BIN=/Users/lazy/Projects/Tree_Ring_Memory/target/debug/tree-ring .venv/bin/python demo.py --mock --backends tree_ring --tasks factual_recall contradiction_detection --no-pdf --output-dir outputs-tree-ring` |
| 45 | + |
| 46 | +Local demo result with the mock evaluator: |
| 47 | + |
| 48 | +- Factual recall accuracy: 0.334 |
| 49 | +- Contradiction detection accuracy: 0.205 |
| 50 | +- Mean accuracy: 0.269 |
| 51 | +- Average latency was about 30.8ms for factual recall and 23.2ms for |
| 52 | + contradiction detection. |
| 53 | + |
| 54 | +These are recorded as smoke evidence for the submitted adapter, not as public |
| 55 | +leaderboard claims. |
| 56 | + |
| 57 | +## GitHub State |
| 58 | + |
| 59 | +- Fork: `TerminallyLazy/agent-memory-benchmarker` |
| 60 | +- Branch: `add-tree-ring-backend` |
| 61 | +- Commit: `ec0b2fe12ae10fc0bce2e353ac144bdcde625b2d` |
| 62 | +- PR state at submission: open, ready, mergeable, no status checks reported. |
0 commit comments