[all] Add slot based booking#2486
Draft
DiegoTavares wants to merge 9 commits into
Draft
Conversation
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Collaborator
Author
|
This branch is a rework of #2115 |
8 tasks
DiegoTavares
force-pushed
the
slot_rework
branch
from
July 16, 2026 16:58
b78ac57 to
38e0b3d
Compare
Slot-based scheduling adds two orthogonal axes: a per-host concurrency cap
and a per subscription/folder/job max, parallel to cores/gpus.
- layer.int_slots_required (0 = not slot-based)
- host.int_concurrent_slots_limit (-1 = not slot-based)
- proc.int_slots_reserved (single source of truth for slot accounting)
- {subscription,folder_resource,job_resource}.int_max_slots
(-1 unlimited, 0 reject-all, N cap)
host_stat.int_running_slots is intentionally NOT added: per-host slot usage
derives from SUM(proc.int_slots_reserved), so RQD need not report slots.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the per-host concurrency-slot axis to the scheduler. Slot hosts (host.concurrent_slots_limit set) and slot layers (layer.int_slots_required > 0) are strictly paired: a slot host runs only slot layers and vice-versa. - Host: concurrent_slots_limit + running_slots_count (seeded from SUM(proc.int_slots_reserved) per host, incremented in-memory on booking). - DispatchLayer/DispatchFrame/VirtualProc carry slots_required; slot layers are forced non-threadable. - placement gate (epvm_gate/saturation_gate): match_slots enforces strict pairing and the per-host cap, replacing the cores/mem floor for slot work. - host_cache index: slot hosts share a fixed sentinel key so they never slide out of the search range while free slots remain. - consume_host_virtual_resources: slot procs reserve 0 cores + 0 memory and only consume slots; proc_dao writes int_slots_reserved. 177 lib tests pass; clippy clean (no new warnings). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Slot frames reserve 0 cores and 0 memory, so RQD must run them without CPU pinning. reserve_cores(0) is already a safe no-op returning an empty affinity list, so the only change RQD needs is to accept num_cores == 0 when slots_required > 0. The scheduler populates RunFrame.slots_required from the proc. RQD deliberately does NOT track or report slot usage: the scheduler owns slot accounting via proc.int_slots_reserved (design decision 7c), so none of the PR's original RQD-side slot-state tracking is carried over. - proto: RunFrame.slots_required (field 30) - rqd: relax num_cores validation for slot frames - scheduler: send slots_required in prepare_rqd_run_frame Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a slots limit axis to the accounting Store, parallel to and fully independent of cores/gpus, enforced at subscription/folder/job level. A slot frame books slots with 0 cores/gpus and vice-versa. - BookingDelta.slot_delta; Counter slot double-buffer (inflight/settled) mirrors cores/gpus through book/confirm/rollback/apply_release/overwrite/seed. - Slot caps per vertex with a distinct rule (over_slot_cap): -1 unlimited, 0/missing = reject-all (fail-closed before the seed), N caps at N. 0 is a valid admin value, so it cannot read as "unset/unlimited" like the cores path. - dao: SUM(int_slots_reserved) in the booked snapshot; int_max_slots in the three limit queries. recompute/limit_reseed carry slots (whole counts, no centicore conversion). listener: slots in acct_release, slot variants in acct_limit_change. - dispatcher sets slot_delta = slots_required on the booking delta. 182 lib tests pass (5 new slot-axis tests); clippy clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Complete the slot accounting round-trip on the Cuebot side: - VirtualProc.slotsReserved, hydrated from proc.int_slots_reserved by the shared VIRTUAL_PROC_MAPPER (GET_VIRTUAL_PROC gains the column; GET_VIRTUAL_PROC_LIST already selects proc.*). - AccountingNotifier.notifyRelease includes negated slots in the acct_release payload (0 for regular procs, so a no-op on the slot axis for them). - notifySubscriptionMaxSlots / notifyFolderMaxSlots / notifyJobMaxSlots emit acct_limit_change on the new slot caps (whole counts, -1 unlimited, 0 reject-all), called by the admin setters added in the management-surface change. compileJava passes (JDK 11). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t surface Expose slot configuration across the stack, mirroring the max_cores/burst pattern. Proto: max_slots on Group/Job/Subscription + SetMaxSlots RPCs; slots_required on Layer; concurrent_slots_limit on Host + SetConcurrentSlotsLimit RPC. Cuebot: setMaxSlots down servant -> service -> DAO for group/job/subscription (each DAO fires AccountingNotifier.notify*MaxSlots when scheduler-managed); host setConcurrentSlotsLimit (per-host cap, no notifier); whiteboard mappers + SELECTs read the new columns for display (incl. the reused Nested job mapper). Layer submission path: cjsl-1.16.dtd slots_required element, JobSpec parse, LayerDetail.slotsRequired, LayerDaoJdbc insert. pycue: setMaxSlots (group/job/subscription), setConcurrentSlotsLimit (host). pyoutline: slots_required layer arg + serialization (spec_version 1.16). cuegui: HostMonitorTree "Concurrent Slots" column + setConcurrentSlotsLimit action (-1 disable / 0 reject-all / N cap). Slots are whole counts (no centicore conversion). compileJava + spotlessJavaCheck pass (JDK 11); pycue/cuegui/pyoutline py_compile clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- placement.rs: gate tests covering the full pairing matrix (slot host x slot
layer within/over cap, both pairing-mismatch rejections, core/mem floor ignored
on slot hosts, constant E-PVM score for slot placements).
- docs: slot-based-booking.md covering the two axes, strict pairing, the
-1/0/N limit convention, and how slot accounting derives from proc.
Slot limit-exceeded observability comes free via the existing
accounting_limit_exceeded_total{table} counter (subscription_slots/folder_slots/
job_slots labels). 188 scheduler tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
DiegoTavares
force-pushed
the
slot_rework
branch
from
July 16, 2026 17:16
38e0b3d to
5188f45
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a new booking mode that doesn't take cores and memory into consideration, but a predefined limit on how many concurrent frames a host is allowed to run.
Rationale: Booking by slot is useful for pipelines where frames are small and limited not by their
cpu/memory consumption but by other resources like storage bandwidth or network availability. In
these scenarios, limiting the concurrency is more important than the resource consumption.
Attention:* This branch is stacked on top of #2002
Tasks:
Cuegui Changes
Add a new column to HostMonitorTree to show the number of concurrent slots:
Add a new menu option on HostMonitorTree to change the concurrent slots limit value:
Attention
The new feature is initially being developed on top of the new scheduler module. At this time Cuebot will not take slot booking into consideration. A future PR will handle implementing the Cuebot changes to avoid touching too many files on a single PR.