Skip to content

Commit eb79880

Browse files
author
Ovtcharov
committed
docs: environment discovery is switched off, and procedures are never recalled
Two things the live store proved, both cheap to fix and both load-bearing for the "personalized from day one" goal. Environment discovery exists and is unreachable. init_system_context writes system-category memories, but the enable flag defaults false and the only opt-in named in the code is a Memory Dashboard panel the TUI does not have. The store holds zero system rows out of 19, so the agent has no stored self-knowledge and no way for a user to grant it any. Closing that is a consent surface plus a discovery pass over tools that already exist — the best value-per-line item in the document, and it belongs in the first-run setup being built now. Procedural memory is worse than missing, it is silently idle: the store reports five procedures, four active, last_recalled None. The write path has run and recall has never fired once. That is a bug to chase, not a feature to design.
1 parent 6c94a83 commit eb79880

1 file changed

Lines changed: 117 additions & 5 deletions

File tree

GAIA_AGENT_V2_SCOPE.md

Lines changed: 117 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -228,12 +228,31 @@ Design notes:
228228
- **Review is not a second draft.** The valuable form is a specific check —
229229
"does this claim follow from what I observed?" — not "make it better", which
230230
invites padding.
231-
- **It must be measured, not assumed.** Extra steps cost latency, and on Gemma-4
232-
a second pass costs roughly a whole turn. I have not yet measured whether it
233-
improves accuracy on this agent; that measurement belongs in
234-
`TESTING_STATUS.md` before the state ships.
231+
**Measured, not assumed.** I ran the two arms against a question with known
232+
ground truth — "can you reliably run Python scripts on this machine?", where
233+
memory held a stale claim that it could not.
235234

236-
**Size: M. Recommend prototyping behind a flag and measuring before committing.**
235+
| | Steps | Tools | Wall clock | ttft | Answer |
236+
|---|---|---|---|---|---|
237+
| No critique | 1 | 0 | 13.0s | 11.8s | "Not fully reliably… intermittently timed out" — **wrong**, recited from stale memory, nothing verified |
238+
| With critique | 4 | 3 | **11.2s** | **2.2s** | Ran a test script (0.09s), then "simple Python scripts run reliably here" — **right**, and corrected its own recollection out loud |
239+
240+
The critique arm was **more accurate and not slower**. The step and tool counts
241+
tripled, but wall clock fell, because time-to-first-token dropped from 11.8s to
242+
2.2s: instead of composing a long hedged paragraph, it went and checked. That is
243+
a real effect and it argues the cost model for this state is not "one extra model
244+
call" but "trades generation for verification".
245+
246+
Caveats: n=1, on Claude, and the critique was delivered in the *user's* prompt
247+
rather than as an agent state. A real implementation has to reproduce it from the
248+
loop, and must be re-measured on Gemma-4-E4B where a second pass is proportionally
249+
much more expensive. But it moves this from a plausible idea to one with evidence.
250+
251+
The specific instruction that worked is worth keeping, because it is narrow: *is
252+
the claim you are about to make something you observed just now, or something you
253+
are recalling? If recalled, verify before asserting.* Not "review your answer".
254+
255+
**Size: M. Prototype behind a flag, then measure on Lemonade before committing.**
237256

238257
### 4.7 GAIA as a super-agent — routing across models, hardware, and agents
239258

@@ -269,6 +288,45 @@ a stale note and redact a secret without asking the agent to do it.
269288

270289
---
271290

291+
### 4.8b Environment discovery — the personalization that is free on day one
292+
293+
The user's point, and it is the strongest cheap win in this document: GAIA runs on
294+
your machine, so it can become **personalized immediately** by learning about that
295+
machine and storing what it finds. Not "after you have taught it for a month" —
296+
on first boot.
297+
298+
**The machinery exists and is switched off.** `init_system_context` writes
299+
`system`-category memories from `system_context.py`, but
300+
`_system_context_is_enabled()` defaults to `False` and the only documented opt-in
301+
is *"the Memory Dashboard settings panel"* — which the TUI does not have. Checked
302+
the live store: **zero `system` rows**, 19 knowledge rows total. So today the
303+
agent has no stored self-knowledge and **no reachable way for a user to consent to
304+
it**.
305+
306+
That is the gap to close, and it is small:
307+
308+
1. **Ask during `/setup` / first boot.** One clear opt-in, stating what will be
309+
read and that it stays on the machine. Re-runnable, and revocable — the setup
310+
task already being built is the natural home.
311+
2. **Then actually discover, and store it.** OS, hardware, GPU/NPU, shells and
312+
runtimes available, project directories, the tools that are installed. This is
313+
what makes the second session better than the first.
314+
3. **OS-dependent probes, reusing what is there.** `shell`, `filesystem`,
315+
`file_search` and `system_context.py` already cover most of it; the missing
316+
piece is a *discovery pass* that runs them deliberately and writes conclusions
317+
to memory, rather than each tool answering one question and forgetting.
318+
4. **Tiered scope, opt-in per tier.** Machine specs and installed tooling are
319+
uncontroversial. Project folders are a step further. Browser history and
320+
personal documents are a different conversation and must be their own explicit
321+
choice, defaulting off — see §6.2 on secrets, and the Recall backlash for why
322+
this is the tier that decides whether users trust the product.
323+
5. **Show what it learned, and let it be deleted.** The read-only `/memory` view
324+
is the start; discovery makes edit/delete (§4.8) a requirement rather than a
325+
nicety.
326+
327+
**Size: S–M, and the best value-per-line in this document.** It needs no new
328+
architecture — a consent surface, a discovery pass, and the existing store.
329+
272330
### 4.9 The stated aim: the most human agent there is — and why that is an architecture problem
273331

274332
The user's goal for GAIA is to be **the most human agent on the planet**, and to
@@ -315,6 +373,52 @@ capability against a frontier model, and does not need to — it can own
315373
continuity, initiative, and running on your own hardware. That is a design
316374
position, and it is the one every recommendation in §4 is pointed at.
317375

376+
#### The main loop is the thing to redesign
377+
378+
The user's sharpest point: the agent **should not be just a task executor**
379+
unless a skill has deliberately programmed it to be one. Today's loop is
380+
literally receive-query → plan → execute → answer → wait. Everything human about
381+
an agent lives in states that loop does not have.
382+
383+
Borrowing from how minds actually work, the loop needs at least four things it
384+
lacks. Each is a state, not a prompt:
385+
386+
- **Curiosity / exploration.** A drive to reduce its own uncertainty. Concretely:
387+
when the agent notices a gap in what it knows about you or your environment, it
388+
investigates or asks — rather than proceeding on an assumption and being
389+
confidently wrong. This is the same machinery as §4.6's self-critique pointed
390+
outward instead of inward, which is a strong argument for building them
391+
together.
392+
- **Self-generated goals.** §4.4's queue, but the *origin* matters as much as the
393+
queue: goals derived from observation ("you have opened this file four times
394+
today") not only from instruction.
395+
- **Challenging the user.** Disagreeing, asking the question behind the question,
396+
saying "that is not what you actually want" — a *tool for pushing back* rather
397+
than a politeness setting. Note the tension with the honesty rules already in
398+
the gaia-voice skill: pushing back requires being *right*, so this depends on
399+
§4.6 landing first. A sycophantic agent is annoying; a confidently wrong
400+
contrarian is unusable.
401+
- **Volition about when to speak.** The hard one. A curious agent that narrates
402+
every thought is exhausting. This needs an explicit model of the user's
403+
attention — what is worth an interruption, what waits for the next turn, what is
404+
never worth saying. Voice (§4.3) makes this sharper: an interruption you have to
405+
listen to costs far more than a line you can skim.
406+
407+
Two guardrails worth stating now, before any of this is built:
408+
409+
1. **Personality must not be able to overwrite truth.** Curiosity, initiative and
410+
pushback all increase the number of *unprompted assertions* the agent makes,
411+
and this session shows its unprompted assertions are the unreliable ones. Every
412+
one of these states needs to sit behind self-critique, not in front of it.
413+
2. **Human-like is not human-imitating.** Claiming feelings it does not have, or
414+
inventing shared history, is the failure mode — and memory poisoning (§6.1)
415+
plus the unexplained "Jordan" persona (§6.3) show the machinery already drifts
416+
that way on its own. The honesty rules in gaia-voice are load-bearing here, not
417+
decoration.
418+
419+
This is the section to design against. Everything else in §4 is a component of
420+
it.
421+
318422
---
319423

320424
## 5. Architectural limitations observed
@@ -397,6 +501,14 @@ The problems are all about what *else* it holds.
397501
is exactly how findings 1–3 nearly went unnoticed. Direct observability is not
398502
a nice-to-have for a system whose value proposition is accumulated state.
399503

504+
5. **Stored procedures have never once been recalled.** The live store reports
505+
`procedures: {total: 5, active: 4, last_recalled: None}`. So the write path
506+
(`skill_synthesis.py``put_skill`) has run and produced four active
507+
procedures, and `recall_skill` has never fired for any of them. Whatever the
508+
flagship has learned about how to do things, it has not read back a single
509+
time. This is a concrete, testable bug hiding inside §4.5, not just a design
510+
gap — worth chasing before designing anything new on top of procedural memory.
511+
400512
### What memory has that nothing uses
401513

402514
Reading the store's API surface against what the flagship actually calls, several

0 commit comments

Comments
 (0)