Skip to content

Commit b3b2e1b

Browse files
committed
docs(crdt): document CRDT history offload to the blob plane
CRDTSpec.ArchiveHistory + Config.Documents.ArchiveHistory; the seal-on-Compact flow into fabriq_crdt_segments; HistoryReader/SegmentLister/HistoryPurger capabilities + SegmentInfo; storage-required fail-fast.
1 parent 2a87a88 commit b3b2e1b

1 file changed

Lines changed: 81 additions & 2 deletions

File tree

docs/content/docs/(data-planes)/documents.mdx

Lines changed: 81 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Documents
3-
description: The CRDT document plane — append-only update log, seq-vector sync, compaction, and quiet-window materialization back into ordinary entities.
3+
description: The CRDT document plane — append-only update log, seq-vector sync, compaction with optional history offload to the blob plane, and quiet-window materialization back into ordinary entities.
44
---
55

66
The document plane backs `KindDocument` entities: collaborative documents — page-builder pages, annotations — where concurrent editing is normal and last-write-wins rows would destroy work. These entities are **not** written through the command plane (`Exec` rejects them). They converge through CRDT merges in an append-only log and only periodically *materialize* into ordinary rows. The plane is implemented in `adapters/postgres` (`DocStore`), folding the log through grove's `crdt.MergeEngine` (HLC-stamped field merge) — referenced, never reimplemented.
@@ -14,6 +14,12 @@ type CRDTSpec struct {
1414
Engine string // engine reference, e.g. "grove-crdt"
1515
SnapshotEvery int // compact after this many updates
1616
QuietWindow time.Duration // idle window before materialization
17+
18+
// ArchiveHistory offloads sealed update history to the blob plane on
19+
// Compact (latest state stays in the DB). nil = inherit the global
20+
// Config.Documents.ArchiveHistory default; a non-nil pointer overrides
21+
// it per entity. See "History offload" below.
22+
ArchiveHistory *bool
1723
}
1824
```
1925

@@ -95,6 +101,78 @@ type Materialized struct {
95101

96102
`Compact` folds the update log into a snapshot row at the current high-water seq and trims log rows with `seq <= last_seq`, in one transaction. It changes storage shape only — never merge results — and bounds reconnect cost for long-lived documents. Cadence is governed by `CRDTSpec.SnapshotEvery`; in production it runs as the worker's leader-elected compactor job, not from request handlers.
97103

104+
By default the trimmed log tail is simply dropped — the compacted snapshot already carries its merged effect. With **history offload** enabled (below), that tail is instead *sealed into a blob segment* before it leaves the log, so the full raw edit history stays reconstructable without keeping it in Postgres.
105+
106+
## History offload
107+
108+
Long-lived documents accumulate a large raw update log. Compaction keeps the *current* state cheap, but the trimmed updates are gone. **History offload** keeps them — moved off Postgres into the [blob plane](/docs/file-plane) as immutable, content-addressed segments, so full edit history remains available at object-store cost while the hot path stays small.
109+
110+
On `Compact` with offload enabled, the tail being trimmed (`seq <= last_seq`) is first **sealed** into one immutable segment: its updates are serialized and written to the blob store, and a single index row is recorded in `fabriq_crdt_segments` mapping the contiguous `[seqLo, seqHi]` range to the segment's blob key — then the log rows are deleted, all in the compaction transaction. Snapshot and Sync are unaffected: they read the compacted snapshot plus the live tail exactly as before, byte-for-byte identical whether offload is on or off.
111+
112+
### Reading offloaded history
113+
114+
Offloading adds three **optional, type-asserted capabilities** on the store (consumers assert for them, mirroring `blob.Presigner`/`Ranger`). Stores that do not offload need not implement them:
115+
116+
```go
117+
// Reconstruct a raw update range, transparently spanning sealed blob
118+
// segments and the still-in-DB tail.
119+
type HistoryReader interface {
120+
ReadHistory(ctx context.Context, docID string, seqLo, seqHi int64) ([]HistoryUpdate, error)
121+
}
122+
123+
// List a document's sealed segments (storage-shape metadata; blob keys
124+
// are an internal detail and intentionally not exposed).
125+
type SegmentLister interface {
126+
ListSegments(ctx context.Context, docID string) ([]SegmentInfo, error)
127+
}
128+
129+
// Delete a document's offloaded history — segment blobs + index rows.
130+
// The admin delete path purges history when a document entity is removed.
131+
type HistoryPurger interface {
132+
DeleteHistory(ctx context.Context, docID string) error
133+
}
134+
```
135+
136+
```go
137+
type SegmentInfo struct {
138+
SegSeq int64 // segment ordinal
139+
SeqLo int64 // inclusive seq range start
140+
SeqHi int64 // inclusive seq range end
141+
UpdateCount int64 // updates sealed in this segment
142+
ByteSize int64 // segment payload size
143+
At time.Time // seal time
144+
}
145+
```
146+
147+
`ReadHistory` returns every update with `seqLo <= seq <= seqHi` in seq order, unioning the sealed segments (fetched from the blob store and served through an in-process LRU cache) with any updates still in the live log. `fabriq_crdt_segments` is a tenant-scoped content table with the same [scope-aware RLS](/docs/tenancy) as the CRDT update/snapshot tables.
148+
149+
### Enabling it
150+
151+
Offload is off by default and opt-in at two levels:
152+
153+
```go
154+
f, _, err := fabriq.Open(ctx, reg, fabriq.Config{
155+
// ...
156+
Documents: fabriq.DocumentsConfig{ArchiveHistory: true}, // global default
157+
})
158+
```
159+
160+
Per entity, `CRDTSpec.ArchiveHistory *bool` overrides the global default (`nil` inherits it):
161+
162+
```go
163+
CRDT: &registry.CRDTSpec{
164+
Engine: "grove-crdt",
165+
SnapshotEvery: 200,
166+
ArchiveHistory: ptr(true), // this entity offloads even if the global default is off
167+
},
168+
```
169+
170+
<Callout type="warn">
171+
Offload requires a configured [blob store](/docs/file-plane) (`Storage`) — the sealed
172+
segments live there. `Open` **fails fast** if `Documents.ArchiveHistory` (or any entity's
173+
`CRDTSpec.ArchiveHistory`) is set while no storage driver is configured.
174+
</Callout>
175+
98176
## Live sync transport
99177

100178
Bidirectional sync rides the subscription hub's connection layer with **no conflation and no coalescing** — CRDT frames must arrive complete and in order. The conflating delta path ([Subscriptions](/docs/subscriptions)) and the document sync path share connections, never semantics. The seam is the hub's raw channel pair, `Hub.SubscribeRaw` / `Hub.PublishRaw`, distinct from the conflated `Subscribe` / `Publish`.
@@ -130,6 +208,7 @@ Until a document goes quiet, its edits exist only in the CRDT log — projection
130208
</Callout>
131209

132210
<Cards>
133-
<Card title="Registry" href="/docs/registry">CRDTSpec — declaring Engine, SnapshotEvery, and QuietWindow on a document entity.</Card>
211+
<Card title="Registry" href="/docs/registry">CRDTSpec — declaring Engine, SnapshotEvery, QuietWindow, and ArchiveHistory on a document entity.</Card>
212+
<Card title="File Plane" href="/docs/file-plane">The blob store that holds sealed history segments — content-addressed, reference-counted, reconciler-GC'd.</Card>
134213
<Card title="Deployment" href="/docs/deployment">The leader-elected worker that runs the materializer and compactor jobs.</Card>
135214
</Cards>

0 commit comments

Comments
 (0)