feat: SQLite storage backend (in-tree) with developer mode by LeeroyHannigan · Pull Request #182 · ExtendDB/extenddb

LeeroyHannigan · 2026-06-26T16:24:18Z

Bring the SQLite storage backend into the workspace as crates/storage-sqlite, per RFC-0002 (backends as in-tree crates). PostgreSQL remains the default; SQLite is selected by Cargo feature. Targets local development, CI, and embedded/air-gapped use.

Consolidation:

crates/storage-sqlite as a workspace member using workspace dependencies.
bin features: sqlite and sqlite-memory (zero-config ephemeral in-memory).
Uniform backend linking: each backend exposes link() and the bin's backends::link_all() references every enabled backend, so inventory registrations are never dead-stripped.

Developer mode (build-time dev-mode feature):

Plain HTTP on loopback and open IAM authorization, with SigV4 still verified.
Credential adopted from the standard AWS_* env, else AWS's documented example.
A compile_error guard makes dev-mode + the Postgres backend fail to build, so an insecure production binary cannot exist.

Fixes found bringing the backend to parity:

Resolve a relative SQLite path to absolute before daemonizing (chdir to / otherwise breaks file-backed serve in the default daemon mode).
Remove a stale PID file before daemonizing (spurious "failed to start").
Generate extenddb-prefixed access keys/secrets; reject non-ASCII identifiers (matches PostgreSQL).
Hash-only GSI scan pagination on a composite-key base table no longer skips rows (include the base sort key in ORDER BY and the pagination predicate).
Seed a self-service IAM policy on user creation.
Restrict the database file (and WAL/SHM sidecars) to 0600.
Parse session tags in array and object form; treat expired sessions as absent.
Mark a restored table ACTIVE immediately.
Bind every placeholder in delete_index_row_multi.

Testing:

SQLite integration job added to CI, mirroring the Postgres job.
run-tests works with non-Postgres backends; import/export allowed paths use $TMPDIR.
Conformance green against SQLite: 721 pytest + 326 comprehensive, plus fmt, clippy -D warnings, and cargo test --workspace.

Checklist

I have read CONTRIBUTING.md
All tests pass (cargo test --workspace)
Code is formatted (cargo fmt --check)
Clippy is clean (cargo clippy -- -W clippy::pedantic)
I have added or updated tests for new functionality
I have updated documentation if behavior changed
Breaking changes are noted below (if any)
If this changes the wire protocol, Storage trait, auth model, on-disk
format, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).

ADR / RFC:

Breaking changes

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

jcshepherd · 2026-06-26T17:56:48Z

+// binary that also contains a production backend. Postgres is the production
+// backend, so the combination is rejected at compile time — there is no runtime
+// path by which a Postgres deployment can serve in dev mode.
+#[cfg(all(feature = "dev-mode", feature = "postgres"))]


What would you think of making this require sqlite or sqlite-memory with dev mode, rather than prevent postgres and dev mode specifically? Most backends we add will be considered production backends, I think, but very few will be considered as suitable for dev only.

jcshepherd · 2026-06-26T17:58:24Z

@@ -44,6 +44,76 @@ pub struct ServeArgs {
 /// Bind the listening socket, daemonize, then start the tokio runtime.


This comment looks detached from the function it was associated with (run()).

jcshepherd · 2026-06-26T18:03:39Z

+        .unwrap_or_else(|| DEFAULT_SECRET.to_owned());
+
+    // The credential store dispatches on the access-key prefix; keep that
+    // invariant intact rather than special-casing it for dev mode.


I don't understand this. I mean, I understand the words but I don't understand why we need to do anything special for dev mode.

Dev mode only relaxes authorization (the IAM policy decision) SigV4 authentication stays identical to production, so the server must hold a credential whose secret matches what the SDK signs with, or every request fails signature verification. seed_dev_credential supplies that: it adopts AWS_ACCESS_KEY_ID/SECRET if present, else seeds AWS's well-known example key, keeping us a drop-in for DynamoDB Local without forking (and under-testing) the auth path.

That seems orthogonal to the key type. What bad thing happens if someone uses an ASIA-type key in dev mode?

jcshepherd

Part 1 of N. Few minor comments/questions around dev mode, etc.

Addresses PR #182 review feedback (jcshepherd): the run() doc comment was left detached from run() when seed_dev_credential was inserted between them. Signed-off-by: Lee Hannigan <lhnng@amazon.com>

…ng postgres Addresses PR #182 review feedback (jcshepherd): invert the dev-mode compatibility guard from a deny-list (dev-mode + postgres) to an allow-list (dev-mode requires sqlite). Every backend is a production backend unless proven otherwise, so a deny-list would have to grow with each new backend; requiring a known dev backend keeps any current or future production backend incompatible with dev-mode automatically. sqlite-memory enables sqlite, so one check covers both modes. Signed-off-by: Lee Hannigan <lhnng@amazon.com>

jcshepherd · 2026-07-01T18:21:46Z

+        );
+    }
+
+    // The default account is created during bootstrap (by `init`, or by


Do we have an explicit concept of a default account or admin account? If not, what is the assurance that the first account returned by list_all_accounts() is a/the admin account? Or is it okay to create the dev user under any account?

Fixed. There's now an explicit default-account concept: bootstrap records default_account_id in settings (exposed via CatalogStore::default_account_id()), and seed_dev_credential resolves the account through it rather than list_all_accounts().first(). Recorded on both bootstrap paths (file-backed init + in-memory serve-time), idempotent so it also backfills existing catalogs.

jcshepherd · 2026-07-01T18:34:43Z

    let metrics = Arc::new(extenddb_core::metrics::MetricsCollector::new());

-    let tls_enabled = app_config.server.tls.enabled;
+    let tls_enabled = app_config.server.tls.enabled && !cfg!(feature = "dev-mode");


Minor: I'd feel a shade more comfortable if there was a nice is_tls_enabled() utility function that wrapped this && . I haven't made it through the rest of the review but I'm seeing several checks in manage_http.rs on tls.enabled that don't appear to be updated.

jcshepherd · 2026-07-01T20:40:20Z

+    );
+    // Monotonic clamp within the partition (RFC 3339 strings sort lexically).
+    let part_max: Option<String> =
+        sqlx::query_scalar("SELECT MAX(ready_at) FROM gsi_pending WHERE worker_partition = ?")


Is ready_at indexed?

Yes idx_gsi_pending_claim ON gsi_pending (worker_partition, ready_at, id), which matches the drain query exactly (filter partition, order by ready_at, tiebreak id)

jcshepherd · 2026-07-01T21:07:40Z

+async fn next_stream_seq(
+    tx: &mut sqlx::Transaction<'_, sqlx::Sqlite>,
+) -> Result<i64, StorageError> {
+    sqlx::query("UPDATE seq_counters SET value = value + 1 WHERE name = 'stream'")


Does this imply a global ordering for all stream records across all shards? Is that what we want?

The counter is global/monotonic, but records are stored and consumed per shard (stream_records PK (shard_id, sequence_number)), so consumers only ever see per-shard order, which is all DynamoDB guarantees. Global monotonicity is just a cheap superset on single-writer SQLite; it doesn't imply any cross-shard ordering contract.

jcshepherd

Left a few more comments/questions.

General questions: are there "migration scripts" to go with this PR, for setting up system tables, etc., needed by ExtendDB in MySQL?

LeeroyHannigan · 2026-07-02T10:34:45Z

Left a few more comments/questions.

General questions: are there "migration scripts" to go with this PR, for setting up system tables, etc., needed by ExtendDB in MySQL?

I assume you mean SQLLite? Schema setup is one authoritative migration, crate::schema::apply (mirrors Postgres 001_schema.sql), applied at extenddb init (file-backed) and at serve-time bootstrap (in-memory), with extenddb migrate for existing catalogs.

eldondevat · 2026-07-06T14:29:42Z

Just want to say I pulled this down to build and started testing with it. Very excited here. 😋

LeeroyHannigan · 2026-07-06T15:51:40Z

Thanks @eldondevat 🚀

We'll get things finalized this week and hopefully get it into your hands very soon 💪🏻

yesyayen · 2026-07-08T16:57:41Z

+                .await?;
+            }
+        }
+
+        tx.commit()
+            .await
+            .map_err(|e| StorageError::Internal(e.to_string()))?;
+
+        // Data DDL after catalog commit.
+        if let Some(updates) = &input.global_secondary_index_updates {


couple of things I'd like to discuss regarding GSI creation and backfill process,

[HIGH] Backfill runs under the global write lock, so it blocks writes to all base tables in the instance for the full duration

GSI is advertised ACTIVE (after Catalog table COMMIT) before base table data is backfilled, since these are non-atomic multi step update. (DescribeTable respond with ACTIVE for the index, but Get or Query will result in resource not found since data table does not yet exist)

Create + backfill is non-atomic, and a process crash between them leaves an ACTIVE index row with no data table, and nothing reconciles it on restart

yesyayen · 2026-07-08T17:16:59Z

+        maps: &ExpressionMaps,
+        stream: Option<&StreamCapture>,
+    ) -> Result<Option<Item>, StorageError> {
+        let indexes = fetch_indexes_for_table(&key_info.table_id, &self.pool).await?;


[not a blocker]
table's index metadata is fetched from the catalog on the write hot-path for every request (put, update, delete, and TWI). I believe we should cache this on the already cached TableKeyInfo and skip the read. Same pattern exists in the Postgres backend, and can be resolved by making this change on the shared layer.

yesyayen · 2026-07-08T17:24:58Z

+        let mut errored = false;
+        loop {
+            match process_gsi_batch(&engine).await {
+                Ok(n) if n > 0 => continue,


[not a blocker] GSI async propagation, divergent failure modes across backends.

Postgres and SQLite handle async GSI propagation differently on failure. Postgres runs 4 workers partitioned by pk-hash and, on a bad update, logs-and-drops it so the queue keeps draining (liveness preserved, that one index write silently lost).

SQLite uses a single worker across all tables/accounts and, on a bad row, rolls back the claimed batch and re-claims the same rows forever (no drop). So one poison row permanently stalls all GSI propagation instance-wide and the queue grows unbounded, not just head-of-line delay.

Neither of these matches DynamoDB; flagging the divergence as a discussion point.

yesyayen · 2026-07-08T17:53:51Z

+-- Idempotency tokens for TransactWriteItems.
+CREATE TABLE IF NOT EXISTS idempotency_tokens (
+    token       TEXT PRIMARY KEY,
+    fingerprint TEXT NOT NULL,
+    created_at  TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now'))
+);


this table is keyed on token alone, but TWI idempotency tokens are unique per account, not globally. Key it on (account_id, token) so two accounts reusing the same token don't clash (currently they do: silent replay or false IdempotentParameterMismatch across accounts).

yesyayen · 2026-07-08T18:44:21Z

+        // Pre-fetch index metadata per distinct table (outside the write lock).
+        let mut table_indexes: HashMap<String, Vec<IndexMeta>> = HashMap::new();
+        for op in ops {
+            let name = op_table_name(op);
+            if !table_indexes.contains_key(name) {
+                let indexes = fetch_indexes_for_table(op_table_id(op), &self.pool).await?;
+                table_indexes.insert(name.to_owned(), indexes);
+            }
+        }
+
+        let _writer = self.write_lock.lock().await;


there is a TOCTOU race condition here, index metadata is read before taking write_lock, so a GSI added by a concurrent UpdateTable in this gap can be missed, and this write won't maintain it, leaving those items permanently missing from the new GSI. I believe we should read the index set after acquiring the lock.

this also exists on other write APIs.

Adds the SQLite storage backend (crates/storage-sqlite) as an alternative to PostgreSQL for local/dev use, with GSI async-propagation parity to the Postgres queue design and the review-driven correctness fixes: - idempotency tokens keyed per (account_id, token) - index metadata read inside the write lock on all write paths (TOCTOU) - GSI queue drops unprocessable rows instead of stalling propagation - GSIs build as CREATING and are reconciled to ACTIVE on startup after backfill; DescribeTable reports CREATING until the backfill completes - data-plane ops on a CREATING/DELETING table return ResourceNotFound Signed-off-by: Lee Hannigan <lhnng@amazon.com>

LeeroyHannigan · 2026-07-09T13:03:57Z

Thanks @yesyayen — really appreciate the thorough pass. All five are addressed; mapping each to the change:

1. GSI create/backfill (update_table.rs) [HIGH] — split into the three sub-issues you raised:

ACTIVE advertised before data exists — fixed. The index is now inserted as CREATING and flipped to ACTIVE only after the data table is created and backfilled, so DescribeTable reports CREATING for the whole build window and a Query/GetItem never hits a not-yet-populated index.
Non-atomic create+backfill (crash leaves an orphan) — fixed. Added a startup reconciler (reconcile_incomplete_gsis) that finds any GSI left in CREATING, drops any partial data table, rebuilds + backfills it, then flips it ACTIVE. Idempotent.
Backfill runs under the global write lock — acknowledged and deliberately deferred to a follow-up. Making it non-blocking (chunked backfill + online catch-up) is a real redesign against SQLite's single-writer model; I didn't want to bundle that into this backend-landing PR. Called out explicitly in the commit message.

2. TOCTOU on the index read (transactions.rs + others) — fixed. The index-metadata read is now taken after acquiring write_lock on all four write paths (Put/Update/Delete/TransactWriteItems), so a GSI added by a concurrent UpdateTable (which holds the same lock) can't be missed.

3. Idempotency token keyed globally (schema.rs) — fixed. idempotency_tokens is now keyed on (account_id, token), and the account is threaded through the check. Note: the PostgreSQL backend has the identical bug — tracked separately, out of scope for this SQLite PR.

4. GSI worker poison-row stall (workers.rs) [not a blocker] — fixed anyway, since it's a real liveness bug. Each pending row is now applied under a SAVEPOINT; an unprocessable row is logged-and-dropped (matching the PostgreSQL backend) instead of rolling back the batch and re-claiming it forever.

5. Hot-path metadata read (put_item.rs) [not a blocker] — agreed it belongs in the shared layer (same pattern in Postgres), so I've left it as a follow-up rather than a SQLite-local patch here.

Also added a data-plane parity fix: ops on a CREATING/DELETING table now return ResourceNotFoundException (was ResourceInUseException), matching DynamoDB and the Postgres backend.

Tests: idempotency (replay/mismatch/cross-account), poison-row drop, and reconciler crash-recovery added as in-crate tests. Full verification on the current tip — Python comprehensive suite 1106 passed / 0 failed, Rust integration 372 passed / 0 failed, unit 16 passed, fmt + clippy -D warnings clean.

The branch was also linearized (it carried internal merge commits that the merge queue rejects) and rebased onto current main.

The migration registry added pending_data_migrations to the Bootstrapper trait; the SQLite bootstrapper was missing it. SQLite applies its full schema in run_catalog_migrations (single file) with no separately-tracked data migrations, so it reports none pending. Signed-off-by: Lee Hannigan <lhnng@amazon.com>

clippy useless_borrows_in_formatting on the sqlx log-filter format arg. Signed-off-by: Lee Hannigan <lhnng@amazon.com>

LeeroyHannigan requested review from amrith, c33howard, jcshepherd, pdf-amzn and yesyayen as code owners June 26, 2026 16:24

LeeroyHannigan force-pushed the feat/storage-sqlite-in-tree branch from 3317349 to ba7fdee Compare June 26, 2026 16:30

jcshepherd reviewed Jun 26, 2026

View reviewed changes

jcshepherd reviewed Jul 1, 2026

View reviewed changes

yesyayen reviewed Jul 8, 2026

View reviewed changes

LeeroyHannigan force-pushed the feat/storage-sqlite-in-tree branch from 4e704f8 to 968e746 Compare July 9, 2026 12:54

LeeroyHannigan added 2 commits July 9, 2026 13:55

fix(sqlite): drop redundant borrow in serve log-filter format

4613dea

clippy useless_borrows_in_formatting on the sqlx log-filter format arg. Signed-off-by: Lee Hannigan <lhnng@amazon.com>

		@@ -44,6 +44,76 @@ pub struct ServeArgs {
		/// Bind the listening socket, daemonize, then start the tokio runtime.

Uh oh!

Conversation

LeeroyHannigan commented Jun 26, 2026

Checklist

Breaking changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcshepherd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jcshepherd left a comment

Choose a reason for hiding this comment

Uh oh!

LeeroyHannigan commented Jul 2, 2026

Uh oh!

eldondevat commented Jul 6, 2026

Uh oh!

LeeroyHannigan commented Jul 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yesyayen Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LeeroyHannigan commented Jul 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yesyayen Jul 8, 2026 •

edited

Loading