You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/content/docs/(data-planes)/documents.mdx
+81-2Lines changed: 81 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
---
2
2
title: Documents
3
-
description: The CRDT document plane — append-only update log, seq-vector sync, compaction, and quiet-window materialization back into ordinary entities.
3
+
description: The CRDT document plane — append-only update log, seq-vector sync, compaction with optional history offload to the blob plane, and quiet-window materialization back into ordinary entities.
4
4
---
5
5
6
6
The document plane backs `KindDocument` entities: collaborative documents — page-builder pages, annotations — where concurrent editing is normal and last-write-wins rows would destroy work. These entities are **not** written through the command plane (`Exec` rejects them). They converge through CRDT merges in an append-only log and only periodically *materialize* into ordinary rows. The plane is implemented in `adapters/postgres` (`DocStore`), folding the log through grove's `crdt.MergeEngine` (HLC-stamped field merge) — referenced, never reimplemented.
@@ -14,6 +14,12 @@ type CRDTSpec struct {
14
14
Enginestring// engine reference, e.g. "grove-crdt"
15
15
SnapshotEveryint// compact after this many updates
16
16
QuietWindow time.Duration// idle window before materialization
17
+
18
+
// ArchiveHistory offloads sealed update history to the blob plane on
19
+
// Compact (latest state stays in the DB). nil = inherit the global
20
+
// Config.Documents.ArchiveHistory default; a non-nil pointer overrides
21
+
// it per entity. See "History offload" below.
22
+
ArchiveHistory *bool
17
23
}
18
24
```
19
25
@@ -95,6 +101,78 @@ type Materialized struct {
95
101
96
102
`Compact` folds the update log into a snapshot row at the current high-water seq and trims log rows with `seq <= last_seq`, in one transaction. It changes storage shape only — never merge results — and bounds reconnect cost for long-lived documents. Cadence is governed by `CRDTSpec.SnapshotEvery`; in production it runs as the worker's leader-elected compactor job, not from request handlers.
97
103
104
+
By default the trimmed log tail is simply dropped — the compacted snapshot already carries its merged effect. With **history offload** enabled (below), that tail is instead *sealed into a blob segment* before it leaves the log, so the full raw edit history stays reconstructable without keeping it in Postgres.
105
+
106
+
## History offload
107
+
108
+
Long-lived documents accumulate a large raw update log. Compaction keeps the *current* state cheap, but the trimmed updates are gone. **History offload** keeps them — moved off Postgres into the [blob plane](/docs/file-plane) as immutable, content-addressed segments, so full edit history remains available at object-store cost while the hot path stays small.
109
+
110
+
On `Compact` with offload enabled, the tail being trimmed (`seq <= last_seq`) is first **sealed** into one immutable segment: its updates are serialized and written to the blob store, and a single index row is recorded in `fabriq_crdt_segments` mapping the contiguous `[seqLo, seqHi]` range to the segment's blob key — then the log rows are deleted, all in the compaction transaction. Snapshot and Sync are unaffected: they read the compacted snapshot plus the live tail exactly as before, byte-for-byte identical whether offload is on or off.
111
+
112
+
### Reading offloaded history
113
+
114
+
Offloading adds three **optional, type-asserted capabilities** on the store (consumers assert for them, mirroring `blob.Presigner`/`Ranger`). Stores that do not offload need not implement them:
115
+
116
+
```go
117
+
// Reconstruct a raw update range, transparently spanning sealed blob
`ReadHistory` returns every update with `seqLo <= seq <= seqHi` in seq order, unioning the sealed segments (fetched from the blob store and served through an in-process LRU cache) with any updates still in the live log. `fabriq_crdt_segments` is a tenant-scoped content table with the same [scope-aware RLS](/docs/tenancy) as the CRDT update/snapshot tables.
148
+
149
+
### Enabling it
150
+
151
+
Offload is off by default and opt-in at two levels:
152
+
153
+
```go
154
+
f, _, err:= fabriq.Open(ctx, reg, fabriq.Config{
155
+
// ...
156
+
Documents: fabriq.DocumentsConfig{ArchiveHistory: true}, // global default
157
+
})
158
+
```
159
+
160
+
Per entity, `CRDTSpec.ArchiveHistory *bool` overrides the global default (`nil` inherits it):
161
+
162
+
```go
163
+
CRDT: ®istry.CRDTSpec{
164
+
Engine: "grove-crdt",
165
+
SnapshotEvery: 200,
166
+
ArchiveHistory: ptr(true), // this entity offloads even if the global default is off
167
+
},
168
+
```
169
+
170
+
<Callouttype="warn">
171
+
Offload requires a configured [blob store](/docs/file-plane) (`Storage`) — the sealed
172
+
segments live there. `Open`**fails fast** if `Documents.ArchiveHistory` (or any entity's
173
+
`CRDTSpec.ArchiveHistory`) is set while no storage driver is configured.
174
+
</Callout>
175
+
98
176
## Live sync transport
99
177
100
178
Bidirectional sync rides the subscription hub's connection layer with **no conflation and no coalescing** — CRDT frames must arrive complete and in order. The conflating delta path ([Subscriptions](/docs/subscriptions)) and the document sync path share connections, never semantics. The seam is the hub's raw channel pair, `Hub.SubscribeRaw` / `Hub.PublishRaw`, distinct from the conflated `Subscribe` / `Publish`.
@@ -130,6 +208,7 @@ Until a document goes quiet, its edits exist only in the CRDT log — projection
130
208
</Callout>
131
209
132
210
<Cards>
133
-
<Cardtitle="Registry"href="/docs/registry">CRDTSpec — declaring Engine, SnapshotEvery, and QuietWindow on a document entity.</Card>
211
+
<Cardtitle="Registry"href="/docs/registry">CRDTSpec — declaring Engine, SnapshotEvery, QuietWindow, and ArchiveHistory on a document entity.</Card>
212
+
<Cardtitle="File Plane"href="/docs/file-plane">The blob store that holds sealed history segments — content-addressed, reference-counted, reconciler-GC'd.</Card>
134
213
<Cardtitle="Deployment"href="/docs/deployment">The leader-elected worker that runs the materializer and compactor jobs.</Card>
0 commit comments