Summary
A pending pod with a required inter-pod anti-affinity against the pods that reclaim/preempt would evict can never be placed by those actions, even when eviction would fully clear a node. Every scenario fails the k8s InterPodAffinity filter with node(s) didn't match pod anti-affinity rules.
Mechanism
Statement.Evict moves the victim to Releasing and calls node.UpdateTask, which is RemoveTask followed by addTask. addTask unconditionally calls PodAffinityInfo.AddPod, so the evicted pod goes straight back into the node's k8s NodeInfo that the predicates plugin hands to InterPodAffinity. The scheduler ends up with two inconsistent views of a Releasing pod:
- resources: future-free (
ReleasingVector), so a pending pod can be Pipelined onto the node;
- inter-pod affinity index: present, so the pending pod's required anti-affinity rejects the node.
A Pipelined task is never bound in the same cycle (commitPipeline → Cache.TaskPipelined, a status update; only commitAllocate reaches BindPod), so the two pods would never actually coexist. Keeping the Releasing pod in the affinity index buys no safety and defeats reclaim/preempt for any workload that uses required anti-affinity. The symmetric check makes it worse: the victims' own required anti-affinity terms also reject the incoming pod while they are Releasing.
Impact observed (v0.17.0)
An 8-node, 64-GPU gang and a 48-GPU job sat Pending while 157 reclaimable GPUs were held by lower-priority pods on 22 nodes that eviction would have fully cleared. Replaying the captured scheduler snapshot offline: 777 reclaim scenarios tried, 675 explicit anti-affinity rejections on nodes whose only pods were the evicted victims, Didn't find a reclaim strategy in 2.2s. Stripping the required anti-affinity terms from the snapshot and replaying (nothing else changed) → Scenario solved for 8 tasks.
Versions
Reproduced on v0.17.0; the code path is unchanged on main (pkg/scheduler/api/node_info/node_info.go, addTask → PodAffinityInfo.AddPod).
Proposed fix
Do not index Releasing tasks for inter-pod affinity, and un-index symmetrically in RemoveTask keyed on the stored copy's status. PR to follow with unit tests, an action-level reclaim test, and the production snapshot replay as evidence.
Summary
A pending pod with a required inter-pod anti-affinity against the pods that
reclaim/preemptwould evict can never be placed by those actions, even when eviction would fully clear a node. Every scenario fails the k8s InterPodAffinity filter withnode(s) didn't match pod anti-affinity rules.Mechanism
Statement.Evictmoves the victim toReleasingand callsnode.UpdateTask, which isRemoveTaskfollowed byaddTask.addTaskunconditionally callsPodAffinityInfo.AddPod, so the evicted pod goes straight back into the node's k8sNodeInfothat the predicates plugin hands to InterPodAffinity. The scheduler ends up with two inconsistent views of a Releasing pod:ReleasingVector), so a pending pod can bePipelinedonto the node;A
Pipelinedtask is never bound in the same cycle (commitPipeline→Cache.TaskPipelined, a status update; onlycommitAllocatereachesBindPod), so the two pods would never actually coexist. Keeping the Releasing pod in the affinity index buys no safety and defeats reclaim/preempt for any workload that uses required anti-affinity. The symmetric check makes it worse: the victims' own required anti-affinity terms also reject the incoming pod while they are Releasing.Impact observed (v0.17.0)
An 8-node, 64-GPU gang and a 48-GPU job sat
Pendingwhile 157 reclaimable GPUs were held by lower-priority pods on 22 nodes that eviction would have fully cleared. Replaying the captured scheduler snapshot offline: 777 reclaim scenarios tried, 675 explicit anti-affinity rejections on nodes whose only pods were the evicted victims,Didn't find a reclaim strategyin 2.2s. Stripping the required anti-affinity terms from the snapshot and replaying (nothing else changed) →Scenario solved for 8 tasks.Versions
Reproduced on v0.17.0; the code path is unchanged on
main(pkg/scheduler/api/node_info/node_info.go,addTask→PodAffinityInfo.AddPod).Proposed fix
Do not index
Releasingtasks for inter-pod affinity, and un-index symmetrically inRemoveTaskkeyed on the stored copy's status. PR to follow with unit tests, an action-level reclaim test, and the production snapshot replay as evidence.