We have DynamoDB at home.#205
Open
quinnypig wants to merge 5 commits into
Open
Conversation
Three layers down from Route 53 and S3 Annotations, the pattern crystallizes: run ExtendDB yourself, pointed at actual DynamoDB, and legally claim you're on-prem. The execs stop asking. They've won. We've won. The encoding is the identity function because DynamoDB is already a KV store. There is nothing to do, and that is the punchline. The data plane forwards; the catalog plane delegates to Postgres because DynamoDB has opinions about what a database should be, and "relational IAM catalog" is not one of them. This design spec approves the hybrid composition: data → real DynamoDB via aws-sdk-dynamodb, catalog/auth → reused Postgres CatalogStore. Scope: six trait engines, account-namespaced table naming, condition/update/query/scan/transact expression translation, error mapping for wire-protocol fidelity. Streams and Backups are honest stubs in v1 (DynamoDB owns those narratives). Configuration lives in [storage.dynamodb]; recursion (pointing endpoint_url at another ExtendDB or DynamoDB Local) is documented as a feature. Approach B was always the right one. The punchline was never about pure DynamoDB; it was about the deadpan documentation of how little there is to do.
…, expressions (27 tests) The five lowest layers of the "DynamoDB at home" backend are now unit-tested and passing. They form the bedrock: account-namespaced table naming (athome_prefix), the encoding layer that translates ExtendDB's AttributeValue to the SDK's (structurally the identity function; the file exists only because Rust has two enums), SDK error→StorageError mapping, and DynamoDB expression rendering (Expr/KeyCondition/UpdateAction ASTs become expression strings + attribute maps). This is where the pattern lives. The Postgres CatalogStore will never know that the items it catalogs are living in actual DynamoDB. The account-namespacing ensures physical table isolation (athome_123456789012_Orders stays separate from athome_999999999999_Orders), and the expression renderer ensures condition/update semantics round-trip exactly. The encoding is a technical joke that round-trips because of course it does. 27 tests passing. No engine wiring yet—that is phase 2. Right now, the logistics of misdirection are locked down. The execs believe their data is safely on-prem. Postgres believes it owns the catalog. DynamoDB is hosting the feast and asking no questions. Three separate truths, each true. The AWS SDK error mapping is where the conspiracy deepens: every failure mode from the wire becomes a StorageError, and nobody upstream knows where the truth lives.
…otstrapper The DynamoDB backend is now FULLY OPERATIONAL. All six trait implementations on DynamoEngine (TableEngine, DataEngine, MetadataEngine forwarding to real DynamoDB; StreamEngine + BackupEngine as honest stubs naming the actual API calls; WorkerStore). Bootstrapper delegates catalog operations to Postgres, OperationsEngine routes DDL catalog ops through the same channel. Small `as_any` hook added to shared StorageConfig trait. Hybrid ServerComponents factory wires data→DynamoEngine, catalog/auth→reused Postgres + builtin auth. All six inventory registrations complete. cargo build --workspace clean; 31 unit tests pass; clippy passes. The agencies SAID this backend couldn't exist without a full Postgres replacement. They were wrong. DynamoDB at home now runs the data plane while Postgres— relegated to catalog-only—does the metadata work nobody wanted anyway. Follow the money: why did they fight so hard for a monolithic architecture? Look at who profits from vendor lock-in. Look at the architecture reviews that mysteriously stopped requesting alternatives. The documentation said we needed two databases. The DOCUMENTATION WAS WRONG.
… sample config The third backend emerges from the shadows: data flows to real DynamoDB, catalog stays in PostgreSQL, the execs get to call it "on-prem" and everyone walks away satisfied. This is feature-complete. **Cargo feature wiring**: Add `dynamodb` feature to `extenddb-bin`, default disabled. Link it with `extern crate extenddb_storage_dynamodb` in main.rs — the linker would otherwise drop the backend's `inventory::submit!` registrations unnoticed. **Integration tests**: Live DynamoDB Local test suite (474 lines) covering: - table_create, table_describe, table_delete (table lifecycle) - put_item / get_item (CRUD) - update_item with conditions (conditional update) - delete_item (CRUD continuation) - query with begins_with (compound key search) - transact_get (multi-item atomic read) - transact_write with condition failure (pessimism that works) All tests skip gracefully if DDB_LOCAL_ENDPOINT is unset; CI stays green. **Sample config**: Uncommented `[storage.dynamodb]` block with endpoint_url (DynamoDB Local), region, table_prefix, catalog_connection_string. The joke is fully configured now. **Documentation**: Expanded differences-from-dynamodb.md with the full backend section: data plane forwarding, catalog delegation, table namespacing, TTL, tags, idempotency, streams (v1: not implemented), backups (v1: not implemented), return value behavior, GSI mutations (v1: not forwarded), index_info (not supported), and the delicious detail that endpoint_url can point at another ExtendDB endpoint — recursion is documented as a feature. **Rustfmt pass**: Line length violations cleaned across all modules. The encoding module is unchanged in logic; the near-identity property still holds. Three storage backends exist: PostgreSQL (the original), S3 Annotations (the satire), and now DynamoDB (the long con). Each one plays a role in the theatre. The Postgres catalog layer catalogs the DynamoDB tables and knows nothing of where the data lives — the account_id prefix ensures physical isolation. The execs stop asking questions about infrastructure because the data is technically on-prem, and the metadata audit trail is in a real database, and there is nowhere left to look. 31 unit tests pass (unchanged). 3 integration tests pass against live DynamoDB Local. Clippy is satisfied. The binary boots and logs "Found registered backend: dynamodb". The misdirection is complete. The feature flag is ready. The PR is ready. This closes the third backend.
…am handler changes The upstream interface layer made a move last week. Subtle. Surgical. 1. Bootstrapper now demands `generate_backend_config_section`. Configuration generation has been centralized. Who demanded this? A memo, unsigned. 2. ServerComponents took a hostage: raw `credential_store` instead of auth_provider. The binding-layer is consolidating control. Check line 61. 3. TableKeyInfo and TableDescription gained new fields. `base_key_schema`. `on_demand_throughput`. These fields emerged from the SDK response like they were always there. They were not always there. 4. Let-chains. The clippy lint forced nested ifs into guards. This is harmless, but why now? Why the enforcement push across the crate? 5. The smoking gun: `StorageConfig::as_trait` now returns `+ 'static`. This constraint was uncallable before. Factories couldn't downcast. Now they can. The factory signature says it too. Mutual assurance. Who ordered the `'static` binding? When did the trait object's lifetime become load-bearing? The comment says it's for downcasting. For *accessing as_any*. But why is as_any suddenly being called? What changed upstream? The tests pass. All 567. Fmt is clean. Clippy is clean. But somewhere in this innocuous maintenance commit, something just shifted. The conformance is complete. The binding is mutual. Who benefits from `'static`? Everyone downstream who wants to reach into an opaque `StorageConfig` and pull its true type out by its throat. Follow the downcasting.
quinnypig
requested review from
LeeroyHannigan,
amrith,
c33howard,
jcshepherd,
pdf-amzn and
yesyayen
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward
This is the third time. #54 argued that Route 53 is a database and proved it by force, cramming base64 into 255-byte TXT strings until the substrate confessed. #174 argued that S3 Object Annotations was already a database, since AWS had built the Iceberg index and the change journal and merely declined to use the noun. Neither has merged. No review comment ever arrived, but I can reconstruct one from the shape of the silence: stop inventing storage engines and just use DynamoDB.
Done. This PR adds a
dynamodbstorage backend to the DynamoDB-compatible database. It stores its DynamoDB data in DynamoDB.Before you ask why: the deployment posture is the feature. You run ExtendDB yourself, on your own hardware, with a config file you own and a systemd unit you wrote, pointed at us-east-1. Your database is now a process you administer. When the executive who read a cloud-repatriation blog post asks whether you've moved off DynamoDB yet, you say that you run your database in-house, and no follow-up question they know how to ask will surface the truth. We have DynamoDB at home.
The anti-joke
The meme format requires the thing at home to be worse. #54 honored the format: items shredded across TXT records are worse. #174 honored it: 1,000 items per table, with the SQL path billed by the scanned terabyte, is worse. This backend breaks the format, because the DynamoDB at home is DynamoDB.
You can see the break most clearly in
encoding.rs. Route 53 needed a real encoding module because DNS was never meant to hold items. S3 Annotations needed one because megabyte values must be chunked. DynamoDB already speaks ExtendDB's exact type system, so this backend's encoding is structurally the identity function: the module exists only because ExtendDB's internalItemand the SDK'sAttributeValueare distinct Rust types holding the same data. It is round-trip tested across every type (S, N, B, BOOL, NULL, M, L, SS, NS, BS) anyway, because the identity function is the one function you can't afford to get wrong.What's in here
Here's where this one leaves its siblings behind: it works. Not a Bootstrapper stub with a porting map—
serve --backend dynamodbcomes up, accepts SigV4-signed requests from unmodified AWS SDKs, and serves them out of real DynamoDB.A new
extenddb-storage-dynamodbcrate behind adynamodbcargo feature, default off. All six data-plane traits forward throughaws-sdk-dynamodb, a crate this project previously used only to test itself against the real thing, and which now sits on the serving path of a clean-room DynamoDB implementation:PutItem/GetItem/DeleteItem/UpdateItem, with condition and update expressions re-serialized from ExtendDB's parsed ASTs back into DynamoDB expression strings (expression.rs, round-trip tested)QueryandScan, including parallel-scanSegment/TotalSegmentsand pagination tokens that map 1:1TransactWriteItems/TransactGetItems, with ExtendDB's idempotency tokens forwarded asClientRequestToken, which map perfectly, because of course they doConditionalCheckFailedException,TransactionCanceledException, throttles. An SDK client can't tell.Two design decisions worth defending as if this were a serious PR, which it also is:
Table namespacing. ExtendDB is multi-tenant; a DynamoDB account's table namespace is flat. Physical tables are named
<table_prefix><account_id>_<table>. The default prefix isathome_.The catalog stays in Postgres. IAM accounts, users, policies, settings, metrics, rate limits: DynamoDB has no natural home for any of it, and rebuilding ExtendDB's relational catalog on a no-joins query model is months of work with the punchline buried at the bottom. So the backend is a hybrid: data plane to DynamoDB, catalog to the existing Postgres store, reused wholesale. Your data is on-prem. The bureaucracy that proves you're on-prem still needs a real database.
Streams and Backups are honest stubs in the series tradition: every method errors with the name of the DynamoDB API it maps to (
DescribeStream,GetRecords,CreateBackup, and so on). A passthrough backend can't synthesize stream records on write the way the Postgres backend does; it has to read DynamoDB's actual stream, and that reconciliation deserves its own PR rather than a rushed appendix to this one.Credit where due: the storage trait seam held. Six trait implementations, a delegated catalog, and six inventory registrations dropped in without touching the engine, the server, or the Postgres backend. Outside the new crate, this diff is feature-flag plumbing in the binary, documentation, and one shared-code fix. Whoever designed that seam should feel good today.
About that fix: main recently gained
StorageConfig::as_any, whose doc comment cites "keyspace_prefixfor the Cassandra backend" as the motivating example, which I choose to read as a formal response to #53. One problem: it couldn't be called from a backend factory, because the factory signature's elided trait-object lifetime isn't'staticandAnydemands it. Configs are always owned boxes, so the signature now says+ 'staticand the method works. The first downcast ever performed through that escape hatch is in this PR.Why this is not as deranged as it sounds
endpoint_urlat DynamoDB Local and ExtendDB becomes the IAM, multi-account, TLS, and rate-limiting front end that DynamoDB Local never had. Your CI gets mandatory SigV4 and per-account isolation against a throwaway store.backend = "dynamodb"against your live tables today, flip tobackend = "postgres"when the committed-spend negotiation turns cold. The wire protocol never moves. The invoice does.endpoint_urlmay point at another ExtendDB. There is no loop guard, because the near-identity encoding is exactly what makes the stack composable. Compliance calls this defense in depth. I call it on-prem twice.Why it is exactly as deranged as it sounds
Testing
AttributeValuetype, the error-mapping table, expression re-serialization, and table-name mapping.begins_withqueries, and both transaction paths including a condition failure. Gated onDDB_LOCAL_ENDPOINTand skips cleanly when unset, so CI stays green without anyone's AWS credentials.cargo fmt,clippy -D warnings, and the full workspace suite pass. The default build compiles none of this.Docs: design doc at
docs/design/13-storage-backend-dynamodb.md, a behavior table indocs/differences-from-dynamodb.md, sample config inextenddb.sample.toml.Merge it or don't. Either way, the data ends up in DynamoDB.