[DRAFT] Baser→Suber Migration - Phase 1-6 Prep (Issue #1163)

[jaelliot]

Migrate Baser subdbs to all use Suber

Part of #1163 - Remove technical debt. Upgrade all databases in Baser to use Suber subclasses.

Status: Draft / Work in Progress (7 of 18 databases complete)

This is a phased migration to reduce risk. This PR completes phases 1-6 (foundation + low-risk databases). Remaining phases planned with detailed prep work.

Progress

• Databases migrated: 7 of 18 (39%)
• Tests passing: 288/289 (99.65%)
• Breaking changes: 0 (backward compatible)

---

What's in this PR

Phase 1: LMDB Study & Documentation

• Created comprehensive LMDB best practices guide (600+ lines)
• Studied Database Internals (Chapters 1-6: B-trees, CoW, MVCC, transactions)
• Updated 7 instruction files with LMDB patterns

Phase 2: POC Migration

• dtss → CesrSuber(klas=Dater) - DateTime storage with ISO-8601 validation
• fels → OnSuber() - First-seen event log indices

Phase 3: KEL & Escrow Databases

• kels → OnIoDupSuber() - Key event log indices (post-quantum safe via IoSet)
• pses → OnIoDupSuber() - Partial signed escrow
• pwes → OnIoDupSuber() - Partial witnessed escrow
• dels → OnIoDupSuber() - Duplicitous events
• ldes → OnIoDupSuber() - Likely duplicitous events

Phase 4: Integration Testing

• Ran full test suite (core, db, app layers)
• Caught and fixed getDts() breaking change
• Added backward compatibility layer (internal Dater, external bytes)
• Result: 288/289 tests passing

Phase 5: Database Analysis

• Audited 11 remaining databases
• Created risk matrix and migration roadmap
• Estimated 19-25h for remaining work

Phase 6 Prep: Signature Database Baseline

• Audited 75 consumer call sites (sigs: 35, wigs: 40)
• Created performance benchmark script
• Established baseline: putSigs 2.47ms, getSigs 0.01ms, ~582 events/sec bulk
• Defined acceptance: <5% regression

---

What's remaining

Next: Signature Databases (Phase 6)

• sigs → CesrDupSuber(klas=Siger)
• wigs → CesrDupSuber(klas=Siger)
• Risk: CRITICAL (hot path - signature verification)
• Prep complete: consumer audit + performance baseline done
• Estimated: 3-4h

Phase 7: Core Storage

• evts → SerderSuber(klas=SerderKERI) - main event storage
• aess → CatCesrSuber(klas=(Prefixer, Cipher)) - seal couples
• Risk: HIGH (core KEL operations)
• Estimated: 4-5h

Phase 8: Receipt Databases

• rcts, ures, vrcs, vres → various CatCesrIoSetSuber (post-quantum safe)
• Risk: MEDIUM (receipt processing, TEL coordination)
• Estimated: 8-10h

Phase 9: Investigation

• imgs, iimgs - determine if active features or deprecated
• Risk: LOW
• Estimated: 2-3h

Note: ooes deferred (Keanu working separately)

---

Testing

Test results:

• Core: 157/157 passing
• Database: 48/48 passing
• App: 83/84 passing
• Total: 288/289 passing (99.65%)

Validated:

• Event processing workflows
• KEL operations (forward/backward replay, last-only)
• Escrow operations (pses, pwes, dels, ldes)
• Witness coordination
• Delegated identifiers
• Key state validation

Backward compatibility approach:

All migrations maintain external API compatibility while using Suber internally. Example:

def getDts(self, key):
    dater = self.dtss.get(keys=key)  # Internal: Dater object
    if dater is None:
        return None
    return dater.dtsb  # External: bytes (backward compatible)

This pattern was validated when Phase 4 testing caught the getDts() breaking change - fixed immediately by adding this compatibility layer.

---

Database migration map

Database	Status	Suber Class	Risk
dtss	Done	CesrSuber(Dater)	Low
fels	Done	OnSuber()	Low
kels	Done	OnIoDupSuber()	Low
pses	Done	OnIoDupSuber()	Low
pwes	Done	OnIoDupSuber()	Low
dels	Done	OnIoDupSuber()	Low
ldes	Done	OnIoDupSuber()	Low
sigs	Prep complete	CesrDupSuber(Siger)	Critical
wigs	Prep complete	CesrDupSuber(Siger)	Critical
evts	Planned	SerderSuber(SerderKERI)	High
aess	Planned	CatCesrSuber(...)	High
rcts	Planned	CatCesrDupSuber(...)	Medium
ures	Planned	CatCesrIoSetSuber(...)	Medium
vrcs	Planned	CatCesrIoSetSuber(...)	Medium
vres	Planned	CatCesrIoSetSuber(...)	Medium
imgs	TBD	TBD	Low
iimgs	TBD	TBD	Low
ooes	Deferred	(Keanu's work)	-

---

Performance

No performance-critical databases in this PR (only datetime storage, KEL indices, escrow).

Baseline established for Phase 6 signature databases:

• putSigs (3 sigs): 2.47ms
• getSigs (3 sigs): 0.01ms
• Bulk write: ~582 events/sec
• Acceptance threshold: <5% regression

---

Documentation

Created:

• lmdb-best-practices.md (600+ lines covering CoW B+ trees, MVCC, transaction management, KERI patterns)
• Phase-specific prep docs (Phases 4-9)
• Performance benchmark script
• PR defense document

Updated:

• 7 instruction files with LMDB patterns
• ADR-008 with Database Internals learnings

---

Why draft?

This PR is 39% complete (7 of 18 databases). Remaining work estimated at 19-25h across 4 more phases.

Opening as draft to:

1. Get early feedback on approach before continuing
2. Show progress on long-running migration
3. Validate backward compatibility strategy
4. Coordinate with Keanu (working on ooes separately)

Next steps depend on review feedback:

• Option A: Complete remaining phases in this PR
• Option B: Merge foundation work, continue in separate PRs per phase

---

Questions for reviewers

1. Is the phased migration approach acceptable?
2. Is the backward compatibility pattern (internal Suber, external bytes) appropriate?
3. Should remaining phases be completed in this PR or separate PRs?
4. Is <5% performance regression an appropriate threshold for Phase 6 (signatures)?

---

Related: #1163

External documentation: keri-notes task folder

[m00sey]

Seems like this conflicts with #1159 as both address aess.

Perhaps update the plan without it, #1159 seems manageable to merge on its own.

[iFergal]

@jaelliot imgs is used to store images for remote contacts, which is also an API on KERIA. There is no corresponding Signify-TS interface that uses it right now (unsure besides Signify-TS/Java), but we will use it and iimgs in the coming months for our wallet.

@SmithSamuelM iimgs is for local identifiers instead of remote contacts - from the recent merge #1084 - we copied the TODO clean that was there from imgs to be safe. So I would say definitely not used yet.

[jaelliot]

Closing Draft PR

Closing per @SmithSamuelM's directive to submit one database per PR for easier review and iterative feedback.

This draft validated migration patterns across 7 databases (48/49 tests passing). Starting fresh with individual PRs for assigned databases: dtss, dels, ldes.

Status: Closing (not merging)
Next: Individual PRs incorporating feedback on IoSet changes for signatures

Related: #1163