Fix duplicate executions of the same action that caused scheduler state to get out of sync [1.6-patch-6]

[cormacrelf]

Description

The scheduler keeps a map from action key to the operation running that action. That map is how it notices two identical actions and makes them share one run.

When a client disconnected, the cleanup code deleted that action's map entry every time. It never checked whether the entry still pointed at the operation being cleaned up. So if an operation had already finished but its client entry was still hanging around, cleaning it up deleted the entry belonging to a newer operation for the same action digest. This caused "...are out of sync" / "should have had the unique_key" errors, hung CI builds with concurrent identical actions.

The cause is a race that orphans an action-key deduplication entry. The race starts when all clients of an operation disconnect while a worker still executes that operation. For example, after client_action_timeout_s with no client keepalive, the state manager marks the in-flight operation Completed(DeadlineExceeded) while the worker still executes it.

In memory_awaited_action_db, the cleanup of a completed operation removed its action_info_hash_key_to_awaited_action entry in all cases. A late client drop can occur after a newer operation claims the same action key. In this case, the cleanup deleted the entry of the newer operation. Later requests for that action key did not deduplicate onto the live operation. Instead, the scheduler started a third operation for the same action. Each completion that occurs after a replacement claims the key repeats the fault. Now the cleanup removes the entry only when the entry still points at the operation that it cleans up.

This change adds late_client_drop_does_not_orphan_replacement_operation as a regression test.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

This change adds one regression test in nativelink-scheduler/tests/simple_scheduler_test.rs:

• late_client_drop_does_not_orphan_replacement_operation — this test creates a race between a late client-drop cleanup and a replacement operation for the same action key. The test makes sure that the deduplication entry of the replacement operation survives.

Checklist

• Updated documentation if needed
• Tests added/amended
• bazel test //... passes locally
• PR is contained in a single commit, using git amend see some docs

---

This change is 

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nativelink	Ready	Preview	Aug 4, 2026 1:19am
nativelink-aidm	Ready	Preview	Aug 4, 2026 1:19am

[MarcusSorealheis]

@palfrey could you take a look as well?