[BUG] Controllers sharing a queue repeatedly contend for the same job reservations

[alexhraber]

Describe the bug

Two independent controllers with distinct controller IDs targeting the same Buildkite queue repeatedly received overlapping job sets and attempted to reserve the same jobs.

During a 14-minute capture, 294 reservation results were not_reserved. Of those 294 results, 290 corresponded to job IDs that were accepted and passed downstream by the peer controller. Successful reservations were split 443 versus 375.

The logs directly demonstrate cross-controller reservation contention. They do not establish a fairness defect, phase-lock, restart dependence, or a connection to reported queue-wait latency.

To Reproduce

The behavior is timing-sensitive:

1. Deploy two independent controllers with distinct IDs targeting the same Buildkite queue.

2. Give them identical or similar polling configuration.

3. Feed a steady stream of jobs into the queue.

4. Compare:

   • buildkite_monitor_jobs_returned_total
   • Reservation results in both controller logs
   • buildkite_deduper_jobs_marked_running_total

5. Correlate job IDs and look for both controllers receiving the same jobs, followed by one reserving them successfully while the peer receives not_reserved.

Expected behavior

Occasional reservation races may be unavoidable, and an exact 50/50 distribution is not expected.

Occasional reservation races may be expected when multiple controllers share a queue, and an exact 50/50 distribution is not expected. It is unclear whether the repeated overlap observed here is expected behavior at this frequency.

Environment

• agent-stack-k8s version: v0.47.0
• Kubernetes version: GKE; exact version not captured
• Deployment method: Two independent Helm installations in separate clusters
• max-in-flight: 150 per controller
• job-creation-concurrency: 50 per controller

Logs

# Controller B
09:31:54.264 job processing completed agent-api-jobs-processed=2
09:31:54.264 reserving jobs count=2
09:31:54.610 job reservation completed reserved-job-count=2 not-reserved-job-count=0

# Controller A
09:31:54.482 job processing completed agent-api-jobs-processed=2
09:31:54.482 reserving jobs count=2
09:31:54.650 job reservation completed reserved-job-count=0 not-reserved-job-count=2

Aggregate results from the 14-minute capture:

Controller A: 443 reserved, 170 not_reserved
Controller B: 375 reserved, 124 not_reserved

Total not_reserved: 294
Job IDs matched to downstream handling by peer: 290

Affiliation (optional)

Buildkite

Additional context

Inspection of v0.47.0 shows:

• internal/controller/monitor/monitor.go drives polling with a fixed time.NewTicker. There is no per-poll jitter that continually varies relative timing between controllers.
• internal/controller/controller.go places reservation before the downstream max-in-flight limiter.
• internal/controller/reserver/reserver.go attempts to reserve the fetched jobs, splitting the received set into chunks of up to 1,000.

Reservation of the fetched set is therefore not bounded by the limiter’s current downstream capacity. These implementation details allow repeated cross-controller contention, but the available evidence does not establish why it occurred at the observed frequency or whether that frequency is expected.

One hypothesis worth testing is that the fixed polling cadence allows the controllers’ relative polling phase to remain stable, producing recurring overlap and a timing advantage during reservation. Per-poll jitter could be tested as an experiment or mitigation, but phase-lock has not been established as the root cause.

The logs were captured in a different window from the original queue-wait reports, so this issue makes no claim that reservation contention caused those waits.

If repeated reservation contention is intentional as part of the multi-controller coordination model, clarification of the expected behavior and operating characteristics would be helpful.