Skip to content

Commit d8bbef4

Browse files
OsherElhadadOsherElhadad
authored
Merge pull request #5 from kagenti/feat/results-preset
docs: per-component SWE-bench results + agent preset
2 parents 4d357ce + ebbb700 commit d8bbef4

4 files changed

Lines changed: 219 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,7 @@ Details in [docs/integrations.md](docs/integrations.md).
9999
- [docs/components.md](docs/components.md) — every registered component: how it works, before→after, lossiness, config, best use.
100100
- [docs/integrations.md](docs/integrations.md) — proxy gateway vs AuthBridge plugin, with request paths.
101101
- [docs/setup.md](docs/setup.md) — setup + a concrete SWE-bench run through the eval-containers gateway.
102+
- [docs/RESULTS.md](docs/RESULTS.md) — per-component SWE-bench benchmark (Claude Code, claude-sonnet-4-6): `mask` ≈27% token savings, no reward loss.
102103

103104
## License
104105

‎config/config.go‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,14 @@ var presets = map[string][]string{
7878
"aggressive": {"format", "dedup", "failed_run", "cmdfilter", "smartcrush", "extract", "cacheinject"},
7979
"coding": {"format", "skeleton", "cmdfilter", "cacheinject"},
8080
"mcp": {"format", "smartcrush", "cacheinject"},
81+
// agent: tuned for long agentic sessions (e.g. Claude Code on SWE-bench),
82+
// where the dominant cost is the transcript of tool outputs (file reads)
83+
// re-sent every turn. mask (drop old tool outputs) is the biggest lever
84+
// there — ~27% content-token savings with no task-reward loss in the
85+
// eval-containers SWE-bench sweep (see docs/RESULTS.md); extract + failed_run
86+
// + dedup add relevance/supersession/dup wins; cacheinject keeps the prefix
87+
// cacheable. Order: lossless first, then offload old-then-large, cache last.
88+
"agent": {"format", "dedup", "failed_run", "mask", "extract", "cacheinject"},
8189
}
8290

8391
// Build constructs the ordered pipeline from the config, wiring each named

‎docs/RESULTS.md‎

Lines changed: 207 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,207 @@
1+
# Benchmark results
2+
3+
Per-component evaluation of context-guru on **SWE-bench** with the **Claude Code** agent and model
4+
**`claude-sonnet-4-6`**, through the context-guru gateway in [eval-containers](setup.md). Baseline = the
5+
same gateway with an empty pipeline (passthrough); each `cg-*` = that one component alone (default config,
6+
**no task-specific tuning**); `cg-balanced` = the preset. 10 tasks × 13 configs = 130 runs.
7+
8+
## Headline
9+
10+
- **Zero task-reward change, every component.** Of the 10 tasks, the baseline resolved 6; on those 6 every
11+
config still resolved (6/6), and none of the 4 the baseline failed were rescued. context-guru changes
12+
tokens, not agent capability — no task broken, none fixed.
13+
- **`mask` is the biggest lever** (drop tool outputs older than the keep-recent window): **26.8% mean
14+
content-token savings on the resolved tasks, up to 93.5% on a long session**, reward preserved.
15+
- **Component overhead is single-digit milliseconds** — the token savings cost effectively no compute.
16+
17+
## Per-component summary
18+
19+
| component | fires on | mean save% (all 10) | mean save% (resolved 6) | reward kept | compute/req |
20+
|---|---|---:|---:|:---:|---:|
21+
| `cg-format` | pretty-printed JSON | 0.0 | 0.0 | 6/6 | ~1–7 ms |
22+
| `cg-dedup` | exact dup tool outputs | 0.3 | 0.2 | 6/6 | ~1–7 ms |
23+
| `cg-cmdfilter` | shell-cmd banners | 0.0 | 0.0 | 6/6 | ~1–7 ms |
24+
| `cg-cacheinject` | provider cache (not content) | 0.0 | 0.0 | 6/6 | ~1–7 ms |
25+
| `cg-failed_run` | superseded test/build runs | 7.3 | 5.8 | 6/6 | ~6 ms |
26+
| `cg-skeleton` | fenced code blocks | 0.0 | 0.0 | 6/6 | ~1–7 ms |
27+
| `cg-collapse` | >2000-tok outputs | 0.6 | 0.0 | 6/6 | ~1–7 ms |
28+
| `cg-mask` | old tool outputs | 30.9 | 26.8 | 6/6 | ~2 ms |
29+
| `cg-smartcrush` | JSON arrays | 0.0 | 0.0 | 6/6 | ~1–7 ms |
30+
| `cg-extract` | large outputs (relevance) | 11.4 | 11.1 | 6/6 | ~6 ms |
31+
| `cg-phi_evict` | transcript > budget | 0.0 | 0.0 | 6/6 | ~1–7 ms |
32+
| `cg-balanced` | combined preset | 9.0 | 12.2 | 6/6 | ~1–7 ms |
33+
34+
Compute/req measured directly for mask/failed_run/extract; the rest are the same order of magnitude
35+
(regex / JSON / hashing over the message array). All negligible vs. the agent's seconds-to-minutes per step.
36+
37+
## Content-token savings % per task (within-run before→after)
38+
39+
| config | sympy-13647 | sympy-16766 | sympy-20438 | sphinx-7910 | sphinx-9320 | scikit-learn-12973 | scikit-learn-25931 | xarray-4629 | django-11820 | django-14089 |
40+
|---|---|---|---|---|---|---|---|---|---|---|
41+
| `baseline` | — | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | — | 0.0 | 0.0 |
42+
| `cg-format` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
43+
| `cg-dedup` | 1.5 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 1.5 | 0.0 |
44+
| `cg-cmdfilter` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
45+
| `cg-cacheinject` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
46+
| `cg-failed_run` | 9.0 | 19.7 | 1.4 | 5.4 | 0.0 | 0.0 | 0.8 | 0.0 | 36.8 | 0.0 |
47+
| `cg-skeleton` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
48+
| `cg-collapse` | 0.0 | 0.0 | 6.2 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
49+
| `cg-mask` | 19.6 | 42.9 | 93.5 | 19.0 | 39.0 | 0.0 | 40.6 | 0.0 | 50.0 | 4.2 |
50+
| `cg-smartcrush` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
51+
| `cg-extract` | 14.9 | 5.0 | 28.8 | 19.3 | 3.8 | 11.2 | 12.1 | 0.0 | 19.1 | 0.0 |
52+
| `cg-phi_evict` | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
53+
| `cg-balanced` | 14.6 | 14.1 | 2.6 | 21.0 | 0.0 | 0.0 | 23.4 | 0.0 | 14.1 | 0.0 |
54+
55+
## Reward per task (1 = resolved)
56+
57+
| config | sympy-13647 | sympy-16766 | sympy-20438 | sphinx-7910 | sphinx-9320 | scikit-learn-12973 | scikit-learn-25931 | xarray-4629 | django-11820 | django-14089 |
58+
|---|---|---|---|---|---|---|---|---|---|---|
59+
| `baseline` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | — | 0 | 0 |
60+
| `cg-format` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
61+
| `cg-dedup` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
62+
| `cg-cmdfilter` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
63+
| `cg-cacheinject` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
64+
| `cg-failed_run` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
65+
| `cg-skeleton` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
66+
| `cg-collapse` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
67+
| `cg-mask` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
68+
| `cg-smartcrush` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
69+
| `cg-extract` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
70+
| `cg-phi_evict` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
71+
| `cg-balanced` | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 |
72+
73+
## Agent wall-clock (seconds) — single run each; **dominated by agent nondeterminism, not component cost**
74+
75+
| config | sympy-13647 | sympy-16766 | sympy-20438 | sphinx-7910 | sphinx-9320 | scikit-learn-12973 | scikit-learn-25931 | xarray-4629 | django-11820 | django-14089 |
76+
|---|---|---|---|---|---|---|---|---|---|---|
77+
| `baseline` | 67 | 59 | 333 | 56 | 53 | 28 | 138 | — | 200 | 22 |
78+
| `cg-format` | 74 | 60 | 342 | 60 | 52 | 31 | 129 | 25 | 113 | 23 |
79+
| `cg-dedup` | 71 | 59 | 461 | 108 | 38 | 35 | 154 | 21 | 172 | 34 |
80+
| `cg-cmdfilter` | 82 | 70 | 251 | 74 | 47 | 33 | 96 | 22 | 93 | 25 |
81+
| `cg-cacheinject` | 58 | 49 | 1352 | 119 | 37 | 32 | 130 | 22 | 118 | 28 |
82+
| `cg-failed_run` | 86 | 95 | 375 | 126 | 53 | 32 | 91 | 22 | 219 | 23 |
83+
| `cg-skeleton` | 80 | 82 | 253 | 60 | 45 | 36 | 106 | 24 | 233 | 22 |
84+
| `cg-collapse` | 76 | 82 | 1047 | 106 | 46 | 30 | 121 | 29 | 168 | 25 |
85+
| `cg-mask` | 63 | 87 | 1833 | 109 | 59 | 33 | 155 | 33 | 116 | 27 |
86+
| `cg-smartcrush` | 65 | 89 | 705 | 125 | 38 | 30 | 132 | 24 | 240 | 23 |
87+
| `cg-extract` | 70 | 96 | 517 | 109 | 61 | 42 | 123 | 23 | 218 | 22 |
88+
| `cg-phi_evict` | 79 | 93 | 276 | 111 | 37 | 35 | 140 | 26 | 191 | 38 |
89+
| `cg-balanced` | 91 | 71 | 633 | 129 | 59 | 36 | 156 | 29 | 454 | 22 |
90+
91+
## Real message rewrites (captured on `sphinx-doc__sphinx-7910`)
92+
93+
Actual before→after content the components produced (via `CONTEXT_GURU_DUMP`). Offloaded originals are
94+
recoverable from the store by the `<<cg:HASH>>` marker.
95+
96+
### `mask` — drops tool outputs older than the keep-recent window
97+
98+
**606 → 27 tokens (saved 579):**
99+
100+
```text
101+
BEFORE:
102+
407- `obj` will be `A.__init__`.
103+
408- skip : bool
104+
409- A boolean indicating if autodoc will skip this member if `_skip_member`
105+
410- does not override the decision
106+
411- options : sphinx.ext.autodoc.Options
107+
412- …
108+
AFTER:
109+
[older tool output masked] <<cg:b162e82de872a202>> [full output: call context_guru_expand]
110+
```
111+
112+
**223 → 31 tokens (saved 192):**
113+
114+
```text
115+
BEFORE:
116+
426 if name != '__weakref__' and has_doc and is_member:
117+
427 cls_is_owner = False
118+
428 if what == 'class' or what == 'exception':
119+
429 qualname = getattr(obj, '__qualname__', '')
120+
430 cls_path, _, _ = …
121+
AFTER:
122+
[older tool output masked] <<cg:f7d9263d63d37c8e>> [full output: call context_guru_expand]
123+
```
124+
125+
### `failed_run` — collapses a run superseded by a later one
126+
127+
**423 → 33 tokens (saved 390):**
128+
129+
```text
130+
BEFORE:
131+
Exit code 1
132+
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7ffffd4f3fa0>: Failed to establi …
133+
AFTER:
134+
[superseded by a later run] <<cg:58242760d7f8bf6f>> [full output: call context_guru_expand]
135+
```
136+
137+
**201 → 32 tokens (saved 169):**
138+
139+
```text
140+
BEFORE:
141+
__wrapped__: True
142+
__qualname__: MyClass.__init__
143+
cls_path: MyClass
144+
wrapper __globals__ keys: ['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__builtins__', 'functools', 'my_decorator']
145+
Old way FAILED with KeyError - BUG CO …
146+
AFTER:
147+
[superseded by a later run] <<cg:3beb3a2148120db7>> [full output: call context_guru_expand]
148+
```
149+
150+
### `extract` — keeps query-relevant lines, elides the rest
151+
152+
**455 → 434 tokens (saved 21):**
153+
154+
```text
155+
BEFORE:
156+
415 directive.
157+
416
158+
417 Returns
159+
418 -------
160+
419 bool
161+
420 True if the member should be skipped during creation of the docs,
162+
421 False if it should be included in the docs.
163+
422
164+
423 """
165+
424 has_doc …
166+
AFTER:
167+
415 directive.
168+
416
169+
417 Returns
170+
418 -------
171+
419 bool
172+
420 True if the member should be skipped during creation of the docs,
173+
421 False if it should be included in the docs.
174+
…
175+
424 has_doc = getattr(obj, '__doc__', False)
176+
425 is_member = (what == 'class' or wh
177+
```
178+
179+
## Methodology & caveats
180+
181+
- Harness: [`deploy/eval-containers/sweep.py`](../deploy/eval-containers/sweep.py) (resumable) → `aggregate.py`.
182+
One cell = one `(task, config)` run through the compose stack; reward from `output/task/result.json`,
183+
savings from the gateway `/stats` (within-run `tokens_before → tokens_after`), component time from the
184+
gateway's per-component `duration_ms` logs, example rewrites from `CONTEXT_GURU_DUMP`.
185+
- **Within-run savings %** is the honest metric — before/after are the same request stream, immune to
186+
run-to-run agent nondeterminism. Cumulative token totals across a growing transcript overstate per-task
187+
savings and are not used here.
188+
- **Single run per cell.** Agent trajectories are stochastic, so both wall-clock and per-task savings vary
189+
run-to-run (e.g. `extract` on sphinx-7910 was 19% in the sweep, 0.7% on the capture re-run). Robust
190+
numbers need N repeats per cell; treat single-task figures as indicative, the per-component means as the
191+
signal.
192+
- **Reward parity** is judged only on the 6 tasks the baseline resolved; the other 4 neither baseline nor
193+
context-guru resolved (SWE-bench resolve rates vary).
194+
- **$ cost** is not measured directly — Claude Code streams (SSE) and the gateway would need to parse the
195+
streaming `usage` for exact provider-billed tokens. Savings are content tokens; at a representative
196+
~$3/M Sonnet input price the message-portion input cost drops roughly in proportion to the savings %
197+
each turn, compounding across a session.
198+
- Host was arm64 (SWE-bench images run under emulation); functional results, not latency benchmarks.
199+
- **Known follow-up:** `extract`/`collapse`/`mask`/`failed_run`/`skeleton` check their per-message size
200+
win *before* appending the `<<cg:HASH>>` marker, so an individual message can occasionally grow by the
201+
marker's ~10–15 tokens (the pipeline's overall never-worse guard still reverts any net regression). The
202+
marker-inclusive check already added to `cmdfilter` should be applied to the other offloaders too.
203+
204+
## Tasks
205+
206+
`sympy-13647/16766/20438`, `sphinx-7910/9320`, `scikit-learn-12973/25931`, `xarray-4629`,
207+
`django-11820/14089`.

‎docs/components.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,9 @@ messages (`role:"tool"`; for Anthropic, `tool_result` blocks normalized to that
2323

2424
Presets (`config`): `off` `[]` · `safe` `[format, cacheinject]` · `balanced`
2525
`[format, dedup, failed_run, cmdfilter, cacheinject]` · `aggressive` adds `smartcrush, extract` ·
26-
`coding` `[format, skeleton, cmdfilter, cacheinject]` · `mcp` `[format, smartcrush, cacheinject]`.
26+
`coding` `[format, skeleton, cmdfilter, cacheinject]` · `mcp` `[format, smartcrush, cacheinject]` ·
27+
**`agent`** `[format, dedup, failed_run, mask, extract, cacheinject]` — for long agentic sessions;
28+
`mask` is the biggest lever there (~27% content-token savings, no reward loss — see [RESULTS.md](RESULTS.md)).
2729

2830
Common gates every Offload respects: skip non-text (`Rewritable`) messages, skip content already
2931
carrying a marker (no double-offload), and skip if the rewrite (marker + hint included) isn't

0 commit comments

Comments
 (0)