Skip to content

fix: finish topition inserts before create_topic writes watermarks - #929

Open
solace-aross wants to merge 1 commit into
mainfrom
fix/sol-155830-pg-race-test-flake
Open

solace-aross wants to merge 1 commit into
mainfrom
fix/sol-155830-pg-race-test-flake

Conversation

@solace-aross

Copy link
Copy Markdown
Collaborator

Postgres create_topic could return success for a topic that was missing or had no watermark row. This makes it wait for each insert and check the result.

Why

  • pg::tests::race_concurrent_commit_and_abort failed once in a merge queue run with UnknownTopicOrPartition, 0.22 s in, before the race it checks. Produce returns that error when the topition or the watermark row is missing.
  • create_topic sent its topition and watermark inserts together and dropped the result streams unread. tokio-postgres returns from query_raw after the bind, so an error from either insert was never seen.
  • COMMIT on a failed transaction answers success. create_topic then returned the topic id with nothing committed.
  • The watermark insert selects the topition row the other insert creates. If it reached the server first it matched no rows and reported no error, which left a topic with no watermark.

Changes

  • Run the topition inserts to completion, then the watermark inserts. Each phase still runs its partitions together.
  • Fail if an insert does not affect exactly one row.
  • Remove tx_prepare_query_raw, which had no other caller.
  • Add a test that warms only the watermark statement on the pool connection. It fails before the change and passes after.

Not confirmed

  • I could not reproduce the CI failure, so I cannot say this is its cause. About 760 runs of the pg tests on CI runners, 260 local runs, 50,000 topic setups and 100,000 race iterations all passed, with and without CPU load.
  • Nothing that touches topics or clusters is shared between tests; all of it is scoped by a random cluster name.
  • The cache ordering above cannot happen with the current callers. The test sets it up directly. The dropped insert errors are the likelier match for the symptom, but I did not see one fire.

Verified

  • just fmt and cargo clippy --workspace --all-features --all-targets -- -D warnings pass.
  • cargo nextest run -p nisshi-storage-sql --all-features: 49 passed against a local Postgres 17.
  • 200 runs of the pg tests with this change: 200 passed.

🤖 Generated with Claude Code

The Postgres create_topic sent its topition and watermark inserts
together and dropped the result streams unread. The watermark insert
selects the topition row the other insert creates, so if it reached the
server first it matched nothing and reported no error. Any error from
either insert was dropped the same way, and COMMIT on an aborted
transaction answers success, so create_topic could return Ok for a topic
that was missing or had no watermark.

Run the topition inserts to completion, then the watermark inserts, and
fail if any insert does not affect exactly one row.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Signed-off-by: Andrea Ross <168456375+solace-aross@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant