db-apply and db-assert walk large database fleets in parallel (--parallel N) - #442
Merged
Conversation
…tabases (weasel#431) db-apply walked its databases strictly sequentially, which at fleet scale is the deployment cost itself: 1,037 target databases at ~0.5s apiece is 8m41s of dead wall time for a deploy where every single one reports "No changes detected". db-assert has the identical loop and the identical pain. Both commands now run through a shared batch runner with a --parallel flag on WeaselInput (default 1 -- strictly sequential, exactly the old behavior, so nothing changes without opting in). The shape follows the direction agreed on the issue: - The unit of parallelism is the *physical* database: targets are grouped by descriptor.DatabaseUri(), the parallelism applies across groups, and a group always runs sequentially within itself. Parallel DDL against one physical database only contends on its locks, so "--parallel 8" means 8 physical databases in flight -- also the right unit to reason about against a server's max_connections ceiling. - Failure semantics are keep-going-and-aggregate at every parallelism, not just above 1 -- "fail fast at 1, aggregate at 8" reads as a bug later. Every database is attempted, each failure prints as it happens, and db-apply terminates with an AggregateException of per-database DatabaseApplyExceptions (identity in the message, original stack trace in the inner exception), which is what turns the exit code non-zero. Note Parallel.ForEachAsync cancels its remaining iterations when a body throws, so the aggregation is built by catching inside the body -- once, in the runner, instead of in every caller. - AnsiConsole is not safe for concurrent writers, so all output from inside the batch goes through a lock. The migration DDL is the other half of that problem: under parallelism each database's DDL is routed to its own buffer via the redirectable logger seam (#437) and flushed as one unit directly above that database's completion line. At --parallel 1 the logger is left alone so a genuinely long migration still streams its SQL live -- on a >90 minute restore pass that stream is how the operator knows the run is alive, and nothing is interleaving with it anyway. - The positional (i+1)/total progress stops meaning anything once completions happen out of order, so it is now a completion counter, and the run closes with a summary block -- N unchanged, M migrated, K failed, failures listed -- which is the cheapest possible answer to "so how did the deploy actually go?". Cancellation propagates through both the scheduler and the per-database operation; connection pools are still released per finished database (deliberately not with the batch token -- a cancelled run is exactly when abandoning idle pools hurts most). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #431 —
db-applywalked 1,037 databases strictly sequentially: 8m41s of dead deploy time for a no-op. Implements the direction from the issue thread.Design (per the thread)
--parallel/-plives onWeaselInput, default 1 — nothing changes for anyone who doesn't set it, anddb-assertgets the same knob (identical loop, identical pain).descriptor.DatabaseUri(): N physical databases in flight, strict sequencing within a group.Parallel.ForEachAsyncbody (a body throw would otherwise cancel remaining iterations), printed as they happen, and the command terminates with anAggregateExceptionof newDatabaseApplyExceptionwrappers (database identity in message, original as inner). Non-zero exit if any failed.(i+1)/totalis meaningless in parallel), console writes under a lock, and per-database migration DDL is buffered and flushed as a unit above its completion line when--parallel > 1— at 1 the logger streams live, preserving liveness for the >90-minute single-database case. Rides the redirectable-logger seam from98aecb1(refactor(core): make the migration logger redirectable per database (weasel#431) #437); a host's deliberately redirected logger is left alone.N unchanged / M migrated / K failed, failures listed.Behavior changes (called out per the thread)
db-assertnow reports unexpected (non-validation) exceptions as failed assertions instead of crashing the command.Tests
Weasel.CommandLine.Tests56/56 on net9.0 and net10.0. New coverage: runner unit tests (every DB exactly once; bounded concurrency with no overlap within a physical DB, using the field's two-targets-per-physical-DB topology in miniature; failures collected, not fatal; cancellation propagates;<= 1runs sequentially) and integration tests against real Postgres (cross-physical-DB parallel apply; aggregate-and-continue at--parallel 4and at 1; DDL attribution/counter/summary via captured console; no-op reapply;db-assertparallel pass and keep-going-on-failure). Docs updated for both commands.Merging this also flushes the 11 commits pending on master since 9.23.2 (discovery progress, SQLite generated-columns fix, per-fingerprint stamp keying, SQL Server CREATE DATABASE postcondition, redirectable migration logger) into the next release.
🤖 Generated with Claude Code