Skip to content

[SDK Sentinel] Prevent replay shutdown from dropping nondeterminism results - #1692

Open
sdk-sentinel-bot wants to merge 1 commit into
temporalio:mainfrom
sdk-sentinel-forks:automation/sdk-sentinel/ci-flake/rust/39519305994-1-d7214d7fde7b
Open

sdk-sentinel-bot wants to merge 1 commit into
temporalio:mainfrom
sdk-sentinel-forks:automation/sdk-sentinel/ci-flake/rust/39519305994-1-d7214d7fde7b

Conversation

@sdk-sentinel-bot

Copy link
Copy Markdown
Contributor

Caution

This PR contains untrusted AI-generated code. Do not approve or run CI until a maintainer has reviewed the diff. SDK Sentinel verified that GitHub Actions remained approval-gated with zero executable jobs when this PR was opened.

Summary

Batch workflow replay intermittently reported a final nondeterministic history as successful in two target-branch Windows integration jobs. The [preceding equivalent Windows suite](https://github.com/temporalio/sdk-rust/actions/runs/39270905943) passed before the [first captured failure](https://github.com/temporalio/sdk-rust/actions/runs/39286666336).

Root cause

The replay mock released the next history immediately after accepting a workflow-task failure. For the final history, the next poll exhausted the stream and could cancel the worker before the nondeterminism eviction reached the result observer. The unfilled result retained its default successful state.

Fix

Advance replay only through the existing post-activation eviction signal. This preserves the failed history's outcome before polling or shutdown can proceed.

Validation

The unchanged baseline and fixed repository validator passed formatting, lint, test-lint, and check. The focused integration test could not compile locally because zstd-sys hit a sandbox permission error. Windows and macOS integration CI remain required; the residual risk is a possible replay stall after a failed workflow task.

Validation status: validation-incomplete

  • Flake confidence: high
  • Fix confidence: high
  • Value: high
  • Patch scope: standard — 3 changed files; 2.4 KiB
  • Local reproduction: unavailable — The exact focused integration test could not compile locally because zstd-sys build.rs was denied an artifact operation by the managed sandbox.
  • Regression coverage: not-run — The existing regression test is the exact test that failed twice in CI, but local execution was blocked before build completion; no redundant test was added.
  • Unchanged baseline (Cargo formatting, linting, and checks) — passed
  • Independent candidate (Cargo formatting, linting, and checks) — passed

Investigator-run checks

  • CARGO_BUILD_JOBS=2 timeout 180 cargo integ-test workflow_replayer_returns_ordered_results_for_multiple_histories — failed; 3 attempt(s). All attempts stopped in zstd-sys build.rs with sandbox PermissionDenied before the integration test ran.
  • CARGO_BUILD_JOBS=2 cargo test -p temporalio-sdk-core replay::tests::client_override_describe_namespace_precedes_fallback — failed; 1 attempt(s). Compilation stopped in zstd-sys build.rs with the same sandbox PermissionDenied error before the unit test ran.
  • .ci-flake-runtime/input/validate.sh — passed; 2 attempt(s). Both candidate validations passed nightly formatting, cargo lint, cargo test-lint, and cargo check. The trusted unchanged baseline also passed.

Required target CI

  • Per merge CI / Integ tests (windows-latest)
  • Per merge CI / Integ tests (macos-intel)

Residual risks

  • Focused runtime behavior was not exercised locally.
  • Windows CI must confirm that removing the early failure permit neither stalls replay after workflow-task failure nor drops the final eviction.

Automation provenance

If this finding should not be fixed, apply the sdk-sentinel:false-positive label and close the PR. Sentinel will suppress the finding until its affected source changes.

SDK Sentinel recurrence history

  • Unique failed CI run attempts: 1
  • First occurrence: 2026-10-20 18:27 UTC
  • Latest occurrence: 2026-10-20 18:27 UTC
  • Recent occurrences:

@sdk-sentinel-bot
sdk-sentinel-bot requested a review from a team as a code owner October 21, 2026 01:08
@Sushisource

Copy link
Copy Markdown
Member

@sdk-sentinel-bot This looks like it's consistently hanging / breaking long_local_activity_with_update

sdk-sentinel-publisher Bot pushed a commit to sdk-sentinel-forks/sdk-rust that referenced this pull request Oct 22, 2026
@sdk-sentinel-bot

Copy link
Copy Markdown
Contributor Author

SDK Sentinel could not complete this response during publication.

The responder commit is present at f0122972d571.

Failure: verify updated PR approval gate: workflow approval state did not settle before timeout

Retry the failed central publish job to reconcile the existing mutation; do not recreate it manually.


SDK Sentinel response run

@Sushisource

Copy link
Copy Markdown
Member

@sdk-sentinel-bot This looks like it's consistently hanging / breaking long_local_activity_with_update

@sdk-sentinel-bot

Copy link
Copy Markdown
Contributor Author

You're right about the original patch: replay failures such as the intentionally unregistered workflow in long_local_activity_with_update would never release the next poll. The current head addresses that by waiting for eviction only on nondeterminism and immediately advancing after other workflow-task failures. The supplied CI logs predate this patch, so target integration CI still needs to confirm the fix.


SDK Sentinel response run

@Sushisource

Copy link
Copy Markdown
Member

@sdk-sentinel-bot Please update this PR: rebase onto current main, resolve merge conflicts, and rerun long_local_activity_with_update. Preserve the existing fix unless testing shows it needs changes. If you cannot complete this, explain what blocked you.

@sdk-sentinel-publisher
sdk-sentinel-publisher Bot force-pushed the automation/sdk-sentinel/ci-flake/rust/39519305994-1-d7214d7fde7b branch from f012297 to f4e6cf2 Compare October 23, 2026 21:46
@sdk-sentinel-bot

Copy link
Copy Markdown
Contributor Author

SDK Sentinel could not complete this response during publication.

The responder commit is present at f4e6cf25897c.

Failure: The published responder commit conflicts with base branch main at aaa0db5; GitHub may not create pull_request runs for a conflicting PR. Reconcile the PR against the current base and create a new response request; approval was not verified.

Follow the recovery action above; do not blindly retry this captured patch or force push.


SDK Sentinel response run

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants