Showcasing Bitterbot at ARC-AGI-3: biological memory in agents

[VGIL77]

Bitterbot is entering the ARC-AGI-3 Kaggle leaderboard. We're using it as a public test of our biological-memory architecture: a knowledge graph of learned rules, hybrid keyword + vector retrieval, hormonal modulation of exploration breadth, and count-based curiosity, all running offline on a single H100 alongside an open-weights 72B reasoning model.

This post is the architecture writeup and the three falsifiable claims we want the leaderboard to test.

TL;DR

We're entering the ARC-AGI-3 Kaggle leaderboard with an agent that pairs an open-weights reasoning LLM (Qwen2.5-72B-Instruct, AWQ-quantized to 4-bit) with Bitterbot, a biological-memory substrate that maintains a knowledge graph of learned game rules, scores action novelty via count-based curiosity, tracks per-game hypotheses, and modulates exploration breadth through a hormonal state model with the same homeostatic dynamics we use in our production memory system.

The agent runs entirely inside the Kaggle scoring container (single H100, 9 hour wall clock, no internet) and is fully open source as required by the competition rules.

We don't expect to win the Grand Prize. The current top of the leaderboard sits around 0.5 percent, and frontier hosted models from the major labs cluster below that on the public benchmark. Our goal is more specific: show that a small, self-contained system that explicitly carries rule structure across levels can match or exceed approaches that either rely on heavy local search or treat each level as an isolated reasoning task.

What ARC-AGI-3 is, briefly

ARC-AGI-3 is an interactive game-playing benchmark. Each game is a 64x64 grid world with unstated rules. The agent emits one of seven discrete actions per turn (four directional, one contextual, one click with coordinates, one undo) and observes the next frame. Games have multiple levels that share mechanics. Scoring rewards efficiency: level_score = (human_actions / ai_actions)^2, capped at 1.15x. Halving your action count quadruples your score.

Internal reasoning and tool calls don't count toward the score. Only game-side actions do. That detail matters a lot for our design.

Why we picked the Kaggle track

There are two visible ARC-AGI-3 leaderboards. The Verified Testing track at three.arcprize.org accepts hosted-LLM agents. The Kaggle competition is the prize-eligible one, with milestone pots in June and September and a $700K Grand Prize. The Kaggle scoring container has no internet, which means none of the OpenAI-, Anthropic-, or other API-backed agents in the shipped templates can compete there. That constraint reshapes the field. To submit, you need a self-contained agent: model weights, code, and any auxiliary services all baked into Kaggle Datasets the notebook attaches.

That gap is the lane we're entering.

Architecture in one diagram

+----------------------------------------------------------+
|  Kaggle scoring container (H100 80GB, 9h, no internet)   |
|                                                          |
|  arcprize/ARC-AGI-3-Agents framework                     |
|     |                                                    |
|     v                                                    |
|  BitterbotARCAgent  ---> calls vLLM at localhost:8000    |
|     |                                                    |
|     | choose_action(frames, latest)                      |
|     v                                                    |
|  BitterbotMemory                                         |
|   - KnowledgeGraph (sqlite)                              |
|   - SAGE-style hybrid retrieval (BM25 + BGE embeddings)  |
|   - HormonalState (dopamine, cortisol, oxytocin)         |
|   - CuriosityEngine (count-based novelty)                |
|   - EpistemicDirectives (per-game hypothesis)            |
|   - RuleStore (arc_rule entities, reinforced on repeat)  |
|                                                          |
|  vLLM serving Qwen2.5-72B-Instruct-AWQ (~40 GB on H100)  |
+----------------------------------------------------------+

The memory layer is a Python port of biological-memory subsystems we already use elsewhere in Bitterbot. We kept the TypeScript implementation as the primary source and mirrored its semantics in Python so the Kaggle agent is a faithful slice of the larger system, not a one-off.

What we want to demonstrate

Three claims, each falsifiable on the leaderboard:

1. Cross-level rule transfer is the actual edge on ARC-AGI-3

The most consistently documented failure mode of frontier LLMs on this benchmark is that they treat every level independently. Rules learned in level 1 don't apply themselves in level 2. We log every (state_hash, action, next_state_hash) tuple the agent observes into a knowledge graph as arc_rule entities, reinforce them on identical text repeats, and surface them in the next-turn prompt. The agent should converge faster on each subsequent level of the same game and faster on subsequent games that share mechanics. If the first level always takes the same number of actions as a frontier baseline but later levels take fewer, the memory layer is doing its job. If we see no improvement, it isn't.

2. Hormonal modulation of exploration breadth has a measurable effect on action efficiency

Bitterbot's hormonal state model has three axes (dopamine, cortisol, oxytocin) with explicit half-lives and homeostatic baselines. We stimulate curiosity_high when an action visibly changes the world, achievement when a level resolves, error on GAME_OVER. The prompt template surfaces the current hormonal state with a breadth hint ("high cortisol leads to exploit, high dopamine leads to explore"). This is a direct port of the same modulation logic used in our production memory system, where it influences retrieval breadth. On ARC-3, if the prompt hint is doing useful work, we expect to see fewer wasted exploration actions on simple levels (state biases toward exploit when hypothesis confidence is high) and more breadth on stuck levels (cortisol decays back to homeostasis, exploration resumes). We'll measure this by comparing per-action novelty distributions across runs with the modulation enabled and disabled.

3. Open-weights 72B reasoning is competitive with frontier hosted models on this task

Qwen2.5-72B at 4-bit quantization on a single H100 produces around 50 tokens per second of decode. With our 9 hour budget and the action efficiency our memory layer should buy, we have room for around 50 actions per game across 25 games with a few hundred tokens of reasoning per turn. That's more headroom than we'd get with a frontier reasoning model that emits 5 to 10x more chain-of-thought tokens per turn. If a 72B open model armed with persistent memory matches a 200B+ frontier model running fresh on each game, that's a structurally interesting result independent of the leaderboard rank.

Where we expect to land

Realistically, somewhere between FORGE v19's 0.5 percent (currently rank 20 on Kaggle, achieved with local game-class instantiation and BFS search) and an undetermined ceiling that depends on how many ARC-3 levels actually have learnable shared mechanics across games. FORGE has a structural advantage we choose not to exploit: the competition ships the game source code as part of the dataset, and FORGE deepcopies the game class to run offline BFS. That works and is legal under the rules, but it's not the same problem as "play the game from observations." We want the agent to act like a player, not a solver, because the part we're testing is the memory layer.

If we land in the same neighborhood as FORGE without the source-code shortcut, we'll consider that a strong result. If we land higher, even better.

Why we're publishing this

Three reasons. First, the open-source requirement: the entire stack will be on GitHub before the milestone deadline, so we'd rather explain it upfront than have people wonder. Second, ARC-AGI-3 is genuinely interesting as a benchmark for memory-equipped agents, and the community has been collaborative about sharing approaches (FORGE published mid-competition, and that helped us calibrate). Third, Bitterbot is a memory system that exists outside of this benchmark, so we're using the leaderboard as a public test of whether the design holds up against an external scoring function, not the other way around.

How to follow along

The code lives under benchmarks/arc-agi-3/kaggle/ in this repo. The Python memory port is pip install-able as a standalone library if anyone wants to use it for other projects.

Comments, criticism, and "your novelty score formula is wrong because..." are all welcome.

Acknowledgments

Thanks to Ashvin Singh for publishing FORGE v15 mid-competition. Reading that source gave us a much faster start on understanding the Kaggle scoring environment than we'd have gotten from the docs alone. Thanks also to the ARC Prize Foundation for designing a benchmark that explicitly rewards what memory can buy you instead of what raw model size can.

[VGIL77]

Quick update on this thread.

Bitterbot is on the official Kaggle ARC-AGI-3 leaderboard as of 2026-05-23: rank 121, score 0.34%. Surrounding cluster (leaderboard):

Rank  Submitter            Score  Entries  Last
119   HARIDOSS24           0.34   1        2d
120   Matthew Blake Ward   0.34   20       17h
121   Victor M. Gil        0.34   1        4h   ← Bitterbot
122   MuChat007            0.33   20       15d

Important context: this was a baseline submission, not the full PLAN-19b stack the post above describes. The Kaggle scoring slot we got this round ran on an RTX 6000 rather than the H100 the architecture targets, and the agent ran without the Qwen2.5-72B reasoning model and the full biological-memory layer wired up.

So 0.34% from a trimmed setup on the smaller GPU is our floor, not our ceiling. The architecture target adds the 72B reasoning model and the persistent rule-transfer memory the post above describes, running on the H100 tier of compute. There is meaningful headroom from here.

What's next:

• Wire up the full Bitterbot stack (Qwen2.5-72B AWQ + the Python biological-memory port) for the next submission
• Target the H100 scoring tier
• Push the score higher before the June 30 milestone

Open source code at benchmarks/arc-agi-3/kaggle/ in this repo.

[enzo485389]

Great open-source AI agent with a unique local-first design and persistent memory system. The project is ambitious, well-designed, and has a lot of potential for AI automation and experimentation.

[ferhimedamine]

The hormonal modulation of exploration breadth is a creative approach — mapping biological motivation systems to retrieval parameters is something most memory architectures skip entirely.

The hybrid keyword + vector retrieval on the knowledge graph is the architecture we independently converged on. A few data points from production that might help calibrate expectations:

1. Keyword vs vector contribution: On temporal reasoning tasks (which ARC pattern recognition shares structural similarity with), BM25 keyword matching contributes ~20% of retrievals that pure vector similarity misses. Specific pattern names, grid dimensions, and transformation labels have keyword signatures that vector embeddings dilute.

2. Decay as curiosity: Your count-based curiosity maps to what we implement as importance-weighted decay — rules that the agent uses frequently get reinforced, while rules that were learned but never successfully applied decay. This naturally pushes exploration toward less-explored solution strategies without an explicit curiosity module.

3. Knowledge graph structure for rules: We found that entity-relationship representation of learned rules (rule → applies_to → pattern_type, rule → requires → condition) gives better retrieval than flat vector lookup because you can traverse from a recognized pattern to ALL rules that apply to that pattern type.

Knowledge graph traversal + hybrid retrieval: https://github.com/Dakera-AI/dakera-py/blob/main/examples/knowledge_graph.py

[VGIL77]

Thanks @ferhimedamine, this is one of the more useful comments this thread has gotten, and the convergence is striking.

On keyword vs vector: we landed in the same place and run RRF over a hybrid (roughly 70% vector, 30% BM25), for exactly the reasons you give. Specific identifiers like grid dimensions, transformation labels, and pattern names have sharp keyword signatures that embeddings smear out, and on anything symbolically or temporally precise that long tail is where a lot of the real recall lives. Your ~20% figure tracks what we see.

On decay as curiosity: agreed, and we split it into two layers that reinforce each other. Memories and rules carry an importance-weighted Ebbinghaus decay, so a rule that gets used gets reinforced while one that was learned but never successfully applied fades, which is the implicit exploration pressure you describe. On top of that we run an explicit intrinsic-reward layer (count and novelty based). The decay alone gets you most of the way; the explicit layer mainly helps when the agent needs to deliberately revisit somewhere it has been avoiding.

On graph structure over flat lookup: this one is timely. We just shipped a graph-abstraction pass that detects communities in the entity-relationship graph and synthesizes summary nodes over them, so the reader can start at a pattern-level summary and descend to all the rules that hang off it instead of scanning a flat frontier. Your "traverse from a recognized pattern to ALL rules that apply to that pattern type" is almost word for word the motivation, and the gap you point at only widens as the rule set grows, which is exactly when you need it not to degrade.

Thanks for the pointer to the Dakera example, I will take a look. And fair on calibration generally: our current leaderboard number is a trimmed baseline rather than the full stack, so we are reading our own early results with the same caution.

Appreciate you taking the time to write this up.