recordstore — User Guide

recordstore is a versioned key→record store layered over any content-addressed bytes store. This guide covers the concepts, the full public API, the canonicity contract, running against a real Swarm/Bee node, common versioning patterns, and current limitations.

It is a tutorial: it explains and walks through. When you already know what you're doing and just need to look something up — a signature, an error, an extra, a default — use the compact REFERENCE.md instead (definition-first, no narrative, pinned against the code by the test suite; also the right document to hand to an AI agent).

Everything documented here is importable from the top-level package:

from recordstore import (
    RecordStore,
    MemoryBytesStore, DirBytesStore, FsspecBytesStore, BeeBytesStore,
    MemoryPointer, FilePointer, SwarmFeedPointer,
    swarm_store,
    canonical_bytes,
)

0. Background: is this reinventing the wheel?

Short answer: the core idea isn't new, but this particular instantiation fills a real gap. Worth reading before you decide whether to build on this vs. something else.

The idea has prior art, and a well-documented failure mode

"Content-addressed trie → canonical root, independent of edit history" is the same idea behind Ethereum's Merkle Patricia Trie, and behind Noms (Attic Labs, Go, now dead/archived) and its living successor Dolt ("Git for data"), whose "prolly trees" achieve the same property via content-defined chunking rather than pure key-branching. Academically, Auvolat & Taïani's Merkle Search Trees (2019) formalize exactly this requirement and prove ordinary B-trees don't have it — insertion order can change their shape even with identical final content — tracing back to Naor & Teague's 2001 work on history-independent data structures.

Getting the canonical-form invariants wrong has real consequences, not just theoretical ones:

Ethereum's MPT needs hex-prefix encoding to disambiguate leaf vs. extension nodes and odd/even nibble parity — precisely the "empty node with one child" ambiguity this trie's canonicalization rules close. Early implementations (geth vs ethereumj) diverged on root hash after just two inserts into an empty trie.
In November 2016, geth and Parity handled an edge case (empty-account deletion under EIP-161) differently under out-of-gas conditions, producing different state roots for identical transaction histories — an actual chain fork at block 2,686,351.
IPFS's own UnixFS HAMT is not canonical — its CID depends on chunk size and DAG balancing, a limitation still being addressed years later (IPIP-499).

So: people keep needing this, keep reinventing it, and keep getting subtle parts of it wrong. This library's two canonicalization rules (§1, "The canonicity guarantee") are the minimal fix for the same failure class MPT took years of bugs to close — closer in spirit to MPT's pure radix-branching approach than to prolly trees' content-defined chunking, which trades a probabilistic depth-balancing guarantee (useful under adversarial or long-shared-prefix key sets) for a mechanism this codebase doesn't need at its current scale.

Related but distinct ideas, for context: Merkle-CRDTs (Protocol Labs) and Git's own object model don't have this property, for different reasons — Merkle-CRDTs are a causal DAG of deltas with no single "current state" root, and Git trees, while genuinely canonical, are hierarchical paths rather than a flat key-value map with point lookup.

Where the actual value is

Not the idea — the fit:

Simpler encoding than MPT, in exactly the place MPT kept breaking. Plain byte prefixes and canonical JSON instead of nibble-packed RLP: same canonicity guarantee, far less surface area for the encoding bugs above.
Much smaller than its closest relatives. Dolt is a full SQL database; Irmin is a general Git-like store with branching and merge built in. This library deliberately has neither — merge is left to the application layer (see docs/SWARM_DESIGN.md §5 in the OntoDAG repo) — because the consuming use case only ever needed put/get/commit/snapshot with a canonical root, not a database engine.
First Python + Swarm instance of this pattern, as far as we found. Noms is dead, Dolt and Irmin are Go/OCaml with much larger footprints, and IPLD's own "prolly tree" ADL spec isn't finalized. Nothing turned up filling "small, dependency-light, Python, pluggable content-addressed backend including Bee."

If you need cross-language interop

If roots ever need to be produced identically from Go, JS, or another language, the relevant prior art is IPLD's approach: a CID tags both codec and hash function, and dag-cbor/dag-json are deterministic codec specs (sorted map keys, no duplicate keys, shortest-form numbers) — not "canonical JSON" left to each language's own encoder. IPLD backs this with published cross-language test fixtures (hex blocks + expected CIDs) that Go/JS/Python implementations must all reproduce. That fixture-suite approach — not just a prose spec — is what would have caught the geth/ethereumj divergence before it shipped, and is the template to follow if this store's wire format ever needs a second-language implementation.

1. Concepts

Records

A record is any JSON-compatible value — dict, list, str, number, bool, None — stored under a non-empty string key. Keys are plain strings with no imposed structure; because iteration is sorted and prefix-filtered, /-separated prefixes ("users/alice") give you cheap namespacing for free.

Roots and versions

All records live in a persistent (copy-on-write) compacted radix trie whose nodes are stored as blobs in the bytes store. The trie's root reference — a hex string — identifies one immutable, self-consistent snapshot of the entire dataset. A root is to a dataset what a commit hash is to a git tree:

hold a root ⇒ you can read that exact version forever (as long as the blobs exist),
compare two roots with == ⇒ you know whether two datasets are identical,
share a root ⇒ someone else can open the same version.

Staging and commit

A RecordStore accumulates put/delete calls in memory. Nothing touches the bytes store until commit(), which writes the changed records and trie paths and returns the new root. Reads are read-your-writes: staged changes shadow the committed state, so get/keys/contains always reflect what you would see after committing.

Because versions share structure, a commit writes only the blobs along the paths that changed; everything else is reused from the previous version.

The canonicity guarantee

Equal content ⇒ equal root. Two datasets containing the same key→value pairs have byte-identical roots, no matter what sequence of puts, deletes, and commits produced them.

This holds because every encoding in the stack is deterministic:

Values are encoded with canonical_bytes — JSON with sorted object keys, minimal separators, UTF-8, NaN/Infinity rejected (they have no canonical JSON form). Two structurally equal values always produce the same bytes, hence the same blob reference.
Trie nodes are canonically encoded, and the trie maintains two structural invariants so its shape is a pure function of its contents: a node with no value and no children does not exist, and a node with no value and exactly one child is merged into that child.

Consequences you can rely on:

deduplication — identical values are stored once;
O(1) version comparison — root_a == root_b;
history independence — replaying operations in any order that reaches the same final content reaches the same root (this is the substrate for building CRDT-style merges above this layer);
idempotent commits — committing with nothing staged (or with staged writes equal to what's already stored) returns the same root.

One subtlety: JSON does not distinguish 1 from 1.0, and Python types that JSON normalizes (tuples become lists) come back in their JSON form. Store what JSON can represent faithfully.

canonical_bytes(value) is exported for anything that needs byte-identical encodings of its own (hashing application-level objects, testing).

2. `RecordStore` API

Constructing

RecordStore(bytes_store, root=None, pointer=None)

bytes_store — any object satisfying the BytesStore protocol (§3).
root — open at an existing version. None starts from the empty dataset (or from the pointer's value, see next).
pointer — any object satisfying the Pointer protocol (§4). If given and root is None, the store opens at pointer.get(); every successful commit() then advances the pointer to the new root.

RecordStore.at(root, bytes_store)   # classmethod

Opens a read-only snapshot at root. Reads work as usual; put/delete/commit raise TypeError. Use this whenever you want to read a version without any risk of writing.

store.root   # property

The root of the last committed state (staged changes are not included — it changes only on commit()). None for a brand-new empty store.

Reading

store.get(key)         # → value (deep copy); raises KeyError if absent
store.contains(key)    # → bool
store.keys(prefix="")  # → iterator of keys, sorted, staged overlay included
store.items(prefix="") # → iterator of (key, value), sorted, staged included

Returned values are deep copies — mutating them never mutates the store. Write changes back with put.
keys("users/") iterates only keys starting with that prefix, in sorted order. The prefix is a plain string prefix, not a path component — keys("users") also matches "users2/x".
items() is the way to read many records at once: it fetches the value blobs in a single batch, so over BeeBytesStore the reads run concurrently instead of one serial round trip per record. Prefer it over keys() + get() in a loop when hydrating a whole store or prefix (see §6).

What changed between two versions? store.diff(other_root) yields (key, mine, theirs) for every key whose value differs between the store's committed root and other_root — a side missing the key gets the ABSENT sentinel (a stored value can legitimately be None, so None can't mean "missing"). It walks the same structural trie diff merge uses, pruning every shared subtree, so the cost is proportional to the difference, not the dataset — diffing a root against itself reads nothing. Staged changes are part of no root, so commit first; to compare two arbitrary published roots, RecordStore.at(a, blobs).diff(b).

Writing

store.put(key, value)  # stage an insert/overwrite
store.delete(key)      # stage a removal; raises KeyError if absent

put validates immediately: the key must be a non-empty string (ValueError otherwise) and the value must be canonically encodable (TypeError/ValueError from the JSON encoder otherwise — this fails fast at put time, never at commit time).
put stores a detached copy of the value: mutating your object after put does not change what was staged.
Staged changes are read-your-writes and can overwrite each other freely; only the final staged state matters at commit.

Committing

root = store.commit()  # → new root reference (hex string), or None if the
                       #   dataset is empty

Flushes staged changes in deterministic (sorted-key) order, updates store.root, advances the pointer if one is attached, and returns the new root. The pointer moves only after every blob write has succeeded, so a reader following the pointer sees all of a commit or none of it.

Deleting a key that was staged-but-never-committed simply drops it; committing a delete of a key that never existed in the trie is a no-op.

Converging with concurrent writers. commit(reconcile=True, resolver=None, retries=5) (with a pointer attached) merges instead of overwriting when the pointer moved under you — see §5 for the mechanism, the resolver, and the in-process-vs-networked guarantees. Plain commit() is last-write-wins.

Merging versions directly. RecordStore.merge(bytes_store, base, ours, theirs, resolver=None) → root three-way merges two divergent roots and is the primitive reconcile builds on; full semantics and examples in §5.

Proving (0.16.0+)

from recordstore import verify_proof, ProofError, ABSENT

proof = store.prove(key)               # against the committed root
value = verify_proof(proof, root)      # pure: no store access at all

prove(key) returns a small, JSON-ready dict — {format, version, addressing, root, key, present, nodes, value} — carrying the raw trie-node blobs along the key's one possible path (plus the value blob when present). verify_proof(proof, root) replays the walk over those exact bytes, recomputing every reference with the addressing scheme the envelope names: it returns the record for an inclusion proof, the ABSENT sentinel for an absence proof, and raises ProofError on any mismatch. The verifier needs the root reference and nothing else — no bytes store, no network, no trust in whoever produced the proof.

Absence is provable, not just inclusion: the canonical encoding gives a key exactly one possible location under a given root, so exhibiting the path where the walk dies is authoritative. That is a property most Merkle structures do not have, and it falls out of the canonicity contract (§1).

Details worth knowing:

Proofs are statements about the committed root; prove of a key with staged, uncommitted changes raises ValueError (commit first). Other staged keys don't interfere.
addressing is detected from the bytes store (sha256 for MemoryBytesStore and the default DirBytesStore/FsspecBytesStore; swarm for BeeBytesStore and swarm-addressed local stores); pass prove(key, addressing="sha256") for duck-typed stores.
Every proof is self-verified before being returned, so a mismatch — e.g. a Bee node whose erasure coding makes server references diverge from the plain content address — fails loudly at prove time, never silently at the verifier.
Proofs are O(depth): a handful of nodes even for one key among thousands, and they survive json.dumps/loads (they are designed to travel).
Proofs are the trust-free path — the only one. Ordinary reads trust the bytes store's endpoint completely (see §7): verify_proof checks every byte it returns, store.get checks nothing.

Error summary

Situation	Raised
`get`/`delete` of a missing key	`KeyError`
empty or non-string key	`ValueError`
non-JSON-encodable value in `put`	`TypeError` / `ValueError`
`NaN` / `Infinity` in a value	`ValueError`
write on a read-only snapshot	`TypeError`
key exists but value bytes unreachable (evicted + offline; backend blob missing)	`RecordUnavailable` — deliberately not a `KeyError`
unresolved merge conflict (no `resolver`)	`MergeConflict` (`.conflicts`)
`reconcile` cannot land after `retries`	`RuntimeError`
`prove` of a key with staged changes	`ValueError`
`prove` on a store with unknown addressing	`ValueError` (pass `addressing=`)
proof fails verification (any tampering)	`ProofError`

3. Bytes store backends

The BytesStore protocol is two methods:

class BytesStore(Protocol):
    def put(self, data: bytes) -> str: ...   # → reference
    def get(self, ref: str) -> bytes: ...    # KeyError if missing

Any content-addressed store satisfying it works — the reference just has to be a stable hex string determined by the content.

`MemoryBytesStore()`

A dict keyed by SHA-256. Use it for tests and ephemeral work; len(store) gives the blob count. Data lives only as long as the object.

`BeeBytesStore(api_url, postage_batch_id="auto", deferred_upload=True, max_concurrent_reads=16, min_batch_ttl=86400)`

A real Swarm Bee node over its HTTP API (POST/GET /bytes) — named for that endpoint specifically: /bytes is Bee's blob-level API, distinct from the raw /chunks/{address} single-chunk primitive that this class does not use. References are Swarm BMT references. Requirements and behavior:

requests is imported lazily inside the constructor — install the [bee] extra.
A usable postage batch is required for writes. The default postage_batch_id="auto" picks the node's usable batch with the longest remaining validity, via swarmfs >= 0.4.0 (imported lazily; pip install 'recordstore[stamps]' for "auto", or pass an explicit batch id and recordstore stays swarmfs-free). Selection only, never purchase — a library must not spend the node wallet's xBZZ on its own. To buy programmatically, use swarmfs's StampManager.plan(size, ttl)/buy(amount, depth) and pass the returned id; batches below ~1 day of validity are rejected by the network, and a fresh purchase takes on the order of a minute to become usable.
A batch that is about to expire is not selected. "auto" requires a day of remaining validity (AUTO_MIN_BATCH_TTL, overridable per store with min_batch_ttl=). swarmfs's own floor is 60 seconds, which is right for a one-shot upload and wrong for a store meant to outlive the process: a batch with a minute left would be chosen, and everything written under it would stop being paid for a minute later.
Two warnings you should act on, both raised when a batch is selected because both are otherwise silent until they bite:
- under a week of validity left — renew it. Your records live exactly as long as the batch, and an expired batch cannot be revived: the node drops it, a top-up against it fails, and the chunks it paid for become the first candidates for eviction. Renewal is additive, so topping up early costs nothing extra.
- the fullest bucket is ≥ 80% full on an immutable batch — dilute it. Chunks land in 65 536 buckets by their address; when one fills, further chunks hashing there are refused with HTTP 402 batch is overissued, even though the batch has capacity elsewhere. A record store keeps appending, so it walks into this gradually.

Watching and renewing the batch

Renewal is the caller's move, but recordstore will tell you when it is needed:

blobs = BeeBytesStore("http://localhost:1633")

info, _ = blobs.batch_status()                 # cheap: the /stamps summary
print(info.ttl / 86400, "days left",
      info.utilization, "of", info.bucket_capacity, "in the fullest bucket")

info, buckets = blobs.batch_status(buckets=True)   # exact, ~2 MB response
print(buckets.max_load, "of", buckets.capacity, "|", buckets.headroom, "free")

From a whole store, reach it through blobs — RecordStore(...).blobs.batch_status().

Cron that, and renew before it matters. Any of these does it:

swarmlite stamps --check --min-ttl 7d          # if you have swarmlite
swarmlite stamps topup <batchID> --for 4w
swarm-cli stamp topup <batchID> --amount ...   # the ecosystem CLI

from swarmfs.stamps import StampManager          # or directly, in Python
plan = await mgr.plan_topup(batch_id, ttl_secs=4 * 7 * 86400)
await mgr.topup(batch_id, plan.added_amount)     # spends xBZZ; you decide

If a bucket has filled, the cure is capacity rather than time: StampManager.dilute(batch_id, depth + 1) doubles every bucket and preserves the counters, so the write that was refused then succeeds. Dilution costs only gas but roughly halves the remaining validity per depth step, so dilute first and top up afterwards. A write refused for this reason raises with that recipe in the message, and nothing already stored is lost — the batch keeps paying for what it has stamped.

deferred_upload=True (default) returns as soon as the node has the data locally, with push-sync to the network happening in the background; False waits. Check network retrievability with Bee's GET /stewardship/{ref} if you need the guarantee.
Values larger than one 4 KB chunk are handled transparently by Bee's splitter — any payload yields exactly one reference.
Concurrent I/O. It keeps one pooled, keep-alive HTTP session (no handshake per op) and implements get_many/put_many, which recordstore uses to parallelise reads (items(), prefix scans) and a commit's value writes. max_concurrent_reads (default 16) caps in-flight requests and sizes the connection pool; raise it on a high-latency link, lower it to be gentle on a shared node. It bounds concurrency — a huge get_many never opens more than this many sockets at once.
HTTP timeouts are 120 s; a 404 surfaces as KeyError, other HTTP errors as requests.HTTPError.

A quick smoke against a local node:

from recordstore import RecordStore, BeeBytesStore

blobs = BeeBytesStore("http://localhost:1633")  # "auto": picks a usable batch
store = RecordStore(blobs)
store.put("hello", {"world": True})
root = store.commit()
print(RecordStore.at(root, blobs).get("hello"))

Because chunks are immutable and content-addressed, mixing backends is safe in one direction: anything written through one BeeBytesStore is readable through any other Bee node that can retrieve the chunks.

4. Pointers

A root reference identifies a version forever, but something has to name “the latest version.” That is a Pointer:

class Pointer(Protocol):
    def get(self) -> Optional[str]: ...
    def set(self, root: str) -> None: ...

A pointer may also implement compare_and_set(expected, new) -> bool; commit(reconcile=True) (§5) uses it for race-free updates and falls back to a best-effort read-then-set when it is absent.

Attach one at construction and it is read at open and advanced on every commit:

from recordstore import RecordStore, MemoryBytesStore, FilePointer

pointer = FilePointer("/var/lib/myapp/ROOT")
store = RecordStore(blobs, pointer=pointer)   # opens at the pointed root
store.put("k", 1)
store.commit()                                  # pointer now names the new root

The whole store on Swarm, in one call

Assembling the two Swarm pieces by hand is easy to get subtly wrong — the obvious wiring, RecordStore(BeeBytesStore(...), FilePointer(...)), puts the blobs on Swarm but leaves the latest-root pointer on local disk, so nothing is publishable. swarm_store does both halves:

from recordstore import swarm_store

store = swarm_store("my-notes", signer=key)          # read + write
store = swarm_store("my-notes", owner=address)       # read someone else's
store = swarm_store("my-notes", signer=key,
                    api_url="http://node:1633",
                    stamp="<batch id>")              # defaults: localhost, "auto"

Blobs go to BeeBytesStore, the latest root to a SwarmFeedPointer, and the postage batch is resolved once (possibly from "auto") and shared, so the feed's SOC writes and the blob writes are paid from the same batch — so one batch's expiry takes down both, and store.blobs.batch_status() reports for both. Needs the direct-on-Swarm bundle: pip install "recordstore[swarm-only]" (= [bee,feeds,stamps]) (stamps only for "auto" and batch health). Everything above RecordStore stays backend-neutral — this factory is the single answer to "where is Swarm specified?".

Local-first: disk now, Swarm in the background (0.17.0+)

The backends above make you choose one home for your blobs — memory forgets, disk doesn't publish, Bee makes every commit a network (and postage) liability. local_first_store stops the choosing:

from recordstore import local_first_store

store = local_first_store("~/.myapp/store", "http://localhost:1633")
store.put("k", {"v": 1})
store.commit()        # local disk, instant, works offline
store.sync()          # optional barrier: confirmed ON the Swarm network
print(store.sync_status())  # pinned vs evictable bytes, per-commit rungs

Commits always land on local disk and are recorded in the directory's journal; a background worker pushes them to Swarm and confirms arrival (Bee's stewardship check retrieves through the network peer-to-peer, so "confirmed" means the network has it, not just your node). Offline is the normal mode — commits never block on the network; certainty is on demand via sync(). With max_bytes= the directory becomes a budgeted working set: unpushed data is pinned (the limit is soft for it — you can always save work), only Swarm-confirmed blobs are evicted under pressure, and reading an evicted record transparently re-fetches and re-verifies it. This also closes the §7 read-trust gap for evicted reads: healed bytes are hashed against their reference. Needs swarmfs ≥ 0.9 (pip install "recordstore[local-first-swarm]"; that floor also brings keccak in swarmfs's base install, which the default addressing="swarm" requires); the design lives in swarmfs's docs/localstore-design.md.

The working set is yours to shape (0.18.0+): store.pin("hot", "users/") keeps everything under a prefix on disk no matter the eviction pressure (unpin releases); store.fetch("users/") warms a subtree back from Swarm before you go offline. When a record's bytes are unreachable — the key exists, its value is on Swarm, you have no network — reads raise RecordUnavailable, deliberately not a KeyError, so absence and unreachability can never be confused. Publishing to a Swarm feed follows confirmation: store.publish(feed_pointer) (or local_first_store(..., publish_pointer=...) to auto-publish) points readers only at roots the network provably serves. And history is a choice, not a tax: every commit is a snapshot, confirmed history costs almost nothing locally (it evicts), but if you accumulate unpushed history offline, store.squash_history() collapses the lineage to the current root and frees the disk.

Durable blobs without Swarm

MemoryBytesStore is ephemeral and BeeBytesStore needs a node, so two backends cover the middle ground:

from recordstore import RecordStore, DirBytesStore, FilePointer, FsspecBytesStore

# a versioned store entirely on local disk
store = RecordStore(DirBytesStore("~/.myapp/blobs"),
                    pointer=FilePointer("~/.myapp/root"))

# or on any fsspec filesystem
store = RecordStore(FsspecBytesStore("s3://bucket/blobs"))

DirBytesStore(path, addressing="sha256") — the file name is the reference. Writes go to a temp file and are os.replaced into place, so a crash never leaves a torn blob; re-putting existing content is a no-op; names fan out two hex characters deep (ab/cdef…) so a million blobs do not land in one directory.
FsspecBytesStore(url, addressing="sha256", **storage_options) — same contract over local paths, S3, GCS, Azure, HTTP, SFTP or memory://. Needs the [fsspec] extra. It refuses bzz://: fsspec is path-addressed, but a Swarm reference is produced by the write, so aiming it at Swarm would throw that addressing away — use BeeBytesStore or swarm_store for Swarm.

Addressing. Both default to "sha256", which matches MemoryBytesStore, so a dataset has the same root in memory and on disk — that is what makes the backends interchangeable. Pass addressing="swarm" to name blobs by their Swarm reference instead (computed locally by swarmfs.splitter, no node needed): the directory then shares Swarm's address space, so you can build offline and publish later with nothing re-addressed. Note that roots are not comparable across the two schemes, and that a Bee node adding erasure coding returns a different root for the same bytes, so an offline mirror stays address-compatible only if you upload with redundancy disabled. Any bytes -> str callable also works.

MemoryPointer(root=None) — in-process only; implements an atomic compare_and_set, so commit(reconcile=True) is race-free in-process.
FilePointer(path) — one root in a local file; set writes a temp file and os.replaces it, which is atomic on POSIX, so a crash never leaves a torn pointer. A missing file reads as None.
SwarmFeedPointer(api_url, topic, *, signer=None, owner=None, postage_batch_id=None, ...) — the "latest root" as an owner-signed Swarm feed. Each set publishes a signed single-owner chunk (SOC); get resolves the latest via a feed lookup. Needs the swarm-bee package (pip install "recordstore[feeds]") for the BMT/secp256k1 signing, imported lazily so the core stays stdlib-only.

Pass a signer (32-byte secp256k1 private key, hex) to read and write — the owner address is derived from it; or an owner address (hex) for a read-only pointer. Writing also needs a postage_batch_id. topic is a namespace string hashed to the feed topic.

Because Swarm feed lookups are unreliable per call on a light node (transient 404s, or a stale-early index — see §7), SwarmFeedPointer does not trust a single lookup: it serves your own writes from a read-your-writes cache (feed_ttl seconds), floors the write index monotonically so back-to-back commits never collide, and retries cold reads with exponential backoff, ignoring a result that regresses below the newest index it has seen. A never-written feed resolves to None after the retries exhaust (which RecordStore treats as the empty dataset). The retry/backoff/TTL knobs are constructor arguments.

5. Versioning patterns

Time travel. Keep the roots your application cares about (commit ids, checkpoints); open any of them later:

v1 = store.commit()
store.put("config", {"mode": "fast"})
v2 = store.commit()

old = RecordStore.at(v1, blobs)   # v1 is untouched by later commits

Undo and redo (0.20.0+). You do not have to keep those roots yourself: a pointer that keeps a timeline remembers where it has been, which is the only thing time travel was ever missing. A root is the state, so going back is pointing back — nothing is recovered and nothing is rewritten:

store = RecordStore(DirBytesStore("~/.myapp/blobs"),
                    pointer=FilePointer("~/.myapp/root"))
store.put("config", {"mode": "fast"})
store.commit(message="go fast")

for version in store.history():          # newest first, like `git log`
    print(version.root[:12], version.at, version.message, version.current)

store.undo()      # the previous state; None if there is nothing before it
store.redo()      # forward again; None at the tip
store.checkout(some_root)                # jump to a state the timeline holds
store.status()    # {root, staged, undoable, redoable, ...} — like `git status`

The semantics are an editor's, not a journal's: a line of states and a position in it. A commit made after an undo abandons the redo tail, exactly as typing after undo does. Nothing is destroyed by that — the abandoned root is still readable by ref, and a local-first store's journal keeps every root it ever committed (git's reflog to this timeline's branch). Staged-but-uncommitted changes are dropped by an undo: the state you asked for is the state you get.

This works for a plain disk store and for a local-first one with no extra wiring, since the latter's HEAD is a FilePointer. Two caveats worth knowing: an undo moves the pointer, so followers see the older state (and on a published feed, so do they); and an undo does not travel through a merge — a peer that merges this replica afterwards re-adds what was undone, because merge only ever adds. Undo is local time travel, not a retraction others honour.

Commit messages are labels, not content. commit(message=...) records the message in the timeline, and deliberately not in the root. This is the one place the git analogy breaks, and it is load-bearing: a git commit hashes its message, so the same change described differently is a different commit, while a root here hashes state alone — which is what makes equal content converge to one root, dedup structurally, and merge without conflict. Two people who make the same change with different words still agree on the state. If attribution has to travel, sign a record about the claim; a message only ever explains what this replica did.

Long consistent reads. Snapshots are immutable, so a reporting job can iterate keys() and get() for hours against one root while writers commit new versions concurrently — no locks, no torn reads.

Cheap change detection. Two stores (or two moments of one store) are equal iff their roots are equal. To sync, compare roots first and walk keys only on mismatch.

Branching and merge. Open two stores at the same root, let them diverge, and commit each — two versions sharing most of their structure (equal-content branches converge to the same root by canonicity). Reconcile them with a three-way merge against their common ancestor:

from recordstore import RecordStore, MergeConflict, ABSENT, DELETE

merged = RecordStore.merge(blobs, base_root, our_root, their_root)

A change made on only one side is taken; the same change on both sides is taken once; a change made on both sides to different values is a conflict. By default a conflict raises MergeConflict (its .conflicts lists the keys) — nothing is silently dropped. Supply a resolver to settle them:

def resolver(key, base, ours, theirs):   # each is the value or ABSENT
    if ours is ABSENT or theirs is ABSENT:
        return DELETE                    # e.g. delete wins over modify
    return max(ours, theirs)             # your policy

merged = RecordStore.merge(blobs, base, ours, theirs, resolver=resolver)

The merge is efficient by canonicity: both the read (a structural diff that prunes subtrees equal on both sides) and the write (only the merged diff, applied to base) are proportional to the divergence, not the dataset. It is commutative when the resolver is symmetric in its ours/theirs arguments (the built-in raise-on-conflict is).

Automatic reconciliation. With a pointer attached you rarely call merge by hand — pass reconcile=True to commit, and it converges with concurrent writers instead of overwriting them:

store = RecordStore(blobs, pointer=pointer)   # opens at the pointer's root
store.put("users/alice", {...})
store.commit(reconcile=True, resolver=resolver)   # merges if the pointer moved

If the pointer still points where this store branched from, the commit lands directly; if another writer advanced it, commit three-way merges the two and retries (up to retries=) until it lands. A pointer exposing compare_and_set (e.g. MemoryPointer) gets race-free updates; FilePointer / SwarmFeedPointer fall back to a best-effort read-then-set.

Idempotent writers. A writer that recomputes and re-puts the same records produces the same root — safe to re-run, and downstream consumers comparing roots see “no change.”

6. Performance notes

One record = one value blob, plus one blob per trie node along the key's path. Trie nodes are compacted (radix), so path length tracks key distinctiveness, not key length.
Commits write O(changed paths) blobs, not O(dataset).
Trie nodes are cached in memory after first load (they are immutable, so the cache never invalidates). Re-reading the same region of a store or snapshot is cheap; a cold open pays one blob fetch per trie level per distinct path.
Reads through BeeBytesStore are one HTTP roundtrip per (uncached) blob. To hydrate an entire dataset (or a prefix) use items() rather than keys() + get() per key: it batches the value-blob reads (and the trie walk fetches each node's children as a batch too), so BeeBytesStore fetches them concurrently instead of one serial round trip at a time — a large win on a high-latency link. Tune the parallelism with BeeBytesStore(..., max_concurrent_reads=N) (default 16).
BeeBytesStore keeps a pooled, keep-alive HTTP session, so no blob op pays a fresh TCP/TLS handshake — the single biggest per-op saving on a slow link.
On write, commit() writes bottom-up in concurrent batches: all value blobs first, then the trie nodes one level at a time (children before parents, since a parent's reference is the Bee-assigned hash of its children). Only the nodes surviving in the final root are written — orphaned intermediates from key-by-key insertion are pruned. A commit therefore costs roughly O(trie depth) concurrent round-trip rounds, not one serial round trip per node. Still prefer fewer, larger commits on a high-latency link (fewer feed-pointer updates and fewer depth rounds overall).

7. Limitations and roadmap

The near-term limitations and their incremental fixes are below. Larger, multi-release bets (e.g. the canonical-POT convergence track) live in the repo roadmap.

Ordinary reads are unverified — the Bee endpoint is trusted. BeeBytesStore.get returns what the node sends without re-hashing it against the reference, so a compromised or lying endpoint could serve wrong bytes undetected. Content-addressing makes every read verifiable; recordstore just doesn't spend the CPU by default. This is the reasonable default against your own Bee node — which is the intended deployment — and the wrong trust model for a public gateway; don't point BeeBytesStore at one. When you need trust-free answers today, that is exactly what prove/verify_proof (§2) are for — and, since 0.17.0, what the local-first path (§3) does for every blob it fetches back from the network: an evicted record heals only if the fetched bytes hash to their reference. The plain BeeBytesStore.get path remains unverified by design (swarmfs's docs/localstore-design.md, Verification and trust, has the full trust story).
The version timeline is per-replica, and best-effort under racing processes. history()/undo()/redo() (§5) read the timeline a pointer keeps beside itself, so they describe this replica: another machine following the same feed has its own line, and an undo does not travel through a merge (merge only ever adds, so a peer re-adds what you undid). Two bare processes committing at the same instant can lose a timeline entry the same way they can lose a pointer update — no data is lost with it, since every root stays readable by reference, and a local-first store serializes its writers with a lock. keep_history=False on FilePointer opts out entirely.
Concurrency control is opt-in, and best-effort across a network. commit(reconcile=True) (§5) makes concurrent writers converge — three-way merge and retry when the pointer moved under it — while plain commit() still last-write-wins. It is race-free against a pointer with an atomic compare_and_set: MemoryPointer has one, so in-process multi-writer is correct. SwarmFeedPointer provides a best-effort compare_and_set — it reads the feed head fresh (so it reliably catches a feed that already advanced, the common case) and verifies its own write read-back — but a Swarm feed has no atomic index claim (Bee accepts and overwrites a second update at the same index), so two writers committing at the exact same index simultaneously can still race. This narrows the window to near-simultaneous collisions rather than any concurrent write. FilePointer has no CAS (plain read-then-set).
Swarm feed lookups are unreliable per call. SwarmFeedPointer (implemented in v0.4.0, see §4) works around this — read-your-writes cache, monotonic write-index floor, and retry-until-stable reads with a stale-early guard, following swarmfs's bzzf:// layer. As of v0.4.1 it also passes Bee's after index hint once it has a confirmed index to resume from, so lookups start near the tip; because swarm-bee's typed API does not expose after (see bee-py#2) this goes through the client transport. As of v0.4.1 index discovery no longer depends on the flaky lookup at all — it probes the feed's SOC chunks directly (individually retrievable even when the lookup 404s), so cold reads resolve in one attempt and an empty feed returns None at once. The one remaining rough edge is that the after hint reaches Bee through a private swarm-bee transport surface until bee-py#2 exposes it publicly. Full rationale is in the SwarmFeedPointer docstring in recordstore.py.
Concurrency tuning across a real link. The read/write parallelism cap (BeeBytesStore(max_concurrent_reads=…), default 16) is a single per-store value; its optimum depends on the client↔node link and is best found with a two-node benchmark — writer and reader on separate nodes (ideally separate locations) so reads force real Swarm retrieval rather than local-store hits. That measurement may also motivate splitting the cap into separate read/write limits.
Trie depth is bounded by recursion. insert, delete, and merge recurse to the trie's depth, so a key set that nests very deeply — thousands of keys each a prefix of the next, or keys sharing a multi-thousand-byte prefix — can exceed Python's recursion limit (~1000) and raise RecursionError. Realistic key sets (path-like, moderate length) produce shallow compacted tries and are unaffected; the practical ceiling is ~1000 levels of distinct nesting. Converting the trie walks to explicit stacks would remove the limit if a use case ever needs it.
No garbage collection. Old versions' blobs are never deleted by this library. On Swarm, chunk lifetime is governed by postage: when a batch expires its chunks stop being paid for and become the first candidates for eviction from the network — not deleted at that instant, but the window is unpredictable, and an expired batch cannot be topped up, so treat expiry as loss. Keep the batch alive instead (see Watching and renewing the batch). Local pinning keeps your node's copy regardless of postage, but it does not keep the content retrievable from the network. For MemoryBytesStore everything lives until the process exits.
Record schema version. Every value blob is wrapped in {"rsv": 1, "val": ...}; a future format bump will change rsv and readers reject unknown versions rather than misread them. Trie nodes carry an analogous "tn": 1.

8. Testing your own usage

The test suite doubles as executable documentation:

tests/test_recordstore.py — the API contract: canonical roots, snapshot isolation, structural sharing, no aliasing, pointer atomicity.
tests/test_recordstore_fuzz.py — randomized put/delete histories against a dict oracle, asserting the canonical-root property throughout.
tests/test_recordstore_bee.py — the same store over a live Bee node (BEE_API/BEE_BATCH env vars; skips otherwise).
tests/test_recordstore_feed.py — SwarmFeedPointer over a live Bee node (read-your-writes, network resolution, read-only pointer, end-to-end RecordStore reopen); skips unless BEE_API is set and swarm-bee is installed.
tests/test_boundaries.py — enforces that module-level imports stay stdlib-only.

When building on the BytesStore or Pointer protocols, the MemoryBytesStore/MemoryPointer pairing plus the fuzz test's oracle pattern is a good template for validating an implementation.

FilesExpand file tree

USER_GUIDE.md

Latest commit

History