Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
152 changes: 151 additions & 1 deletion docs/KeyFeatures/HierarchicalQueue.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,42 @@ data:
- name: nodeorder
```

### Configure Ancestor Reclaim Scope

`ancestorReclaimLevel` is configured as an argument of the `capacity` plugin. It is only meaningful when hierarchical queue mode is enabled with `enableHierarchy: true` and the scheduler has the `reclaim` action enabled.

```yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: volcano-scheduler-configmap
namespace: volcano-system
data:
volcano-scheduler.conf: |
actions: "allocate, preempt, reclaim"
tiers:
- plugins:
- name: priority
- name: gang
enablePreemptable: false
- plugins:
- name: drf
enablePreemptable: false
- name: predicates
- name: capacity
enableHierarchy: true
arguments:
ancestorReclaimLevel: 1
- name: nodeorder
```

The value controls how many ancestor levels are checked when Volcano decides whether a running workload from another queue can be reclaimed:

- `0`: keep the default behavior. Volcano does not add ancestor-level reclaim restrictions.
- `1`: add parent-level deserved-resource checks for cross-parent reclaim.
- `2`: add grandparent-level checks when the queue topology reaches that level.
- `N`: continue the same pattern for deeper queue trees.

### Building Hierarchical Queues
A new `parent` field has been added to the Queue spec to specify the parent queue:

Expand Down Expand Up @@ -134,4 +170,118 @@ spec:

When cluster resources are insufficient for pod requirement, pod's resources can be reclaimed. For pods in different queues, they will first reclaim pods in sibling queues (if the allocated resources of the sibling queue exceed the `deserved` value). If the resources in sibling queues are still insufficient to meet the pod's requirements, the hierarchical structure of the queues (i.e., ancestor queues) will be traversed upward to find sufficient resources. For example, if job-a and job-c are submitted first and the cluster resources are insufficient for job-b, job-b will first reclaim job-a. If reclaiming job-a does not meet the resource requirements, job-c will then be considered for reclaiming.

Note that in the current version, users can only submit jobs to **leaf queues**. If tasks have already been submitted to a parent queue, child queues cannot be created under that queue. This ensures effective management of resources and tasks across different levels in the queue hierarchy. Additionally, the sum of the `deserved`/`guarantee` values of child queues cannot exceed the `deserved`/`guarantee` values configured for the parent queue. Each child queue's `capability` values cannot exceed the `capability` limits of the parent queue. If a queue does not specify the `capability` value for a certain resource dimension, it will inherit the `capability` from its parent queue. If the parent queue and all ancestor queues do not specify it, the value will finally inherit from the root queue. By default, the root queue's `capability` is set to the total available resources of that dimension in the cluster.
Note that in the current version, users can only submit jobs to **leaf queues**. If tasks have already been submitted to a parent queue, child queues cannot be created under that queue. This ensures effective management of resources and tasks across different levels in the queue hierarchy. Additionally, the sum of the `deserved`/`guarantee` values of child queues cannot exceed the `deserved`/`guarantee` values configured for the parent queue. Each child queue's `capability` values cannot exceed the `capability` limits of the parent queue. If a queue does not specify the `capability` value for a certain resource dimension, it will inherit the `capability` from its parent queue. If the parent queue and all ancestor queues do not specify it, the value will finally inherit from the root queue. By default, the root queue's `capability` is set to the total available resources of that dimension in the cluster.

### Understanding `ancestorReclaimLevel`

Hierarchical queues allow one part of the queue tree to use unused `deserved` resources from another part of the queue tree. When a queue later has pending work, reclaim can give those borrowed resources back from a reclaimable queue that is over its own `deserved` resources.

`ancestorReclaimLevel` controls how much of the hierarchy must be considered before a queue is treated as a valid reclaim victim. This is useful for any queue tree that encodes ownership, quota boundaries, workload classes, accelerator pools, environments, cost centers, tenants, or any other resource-sharing model.

#### Scenario 1: parent-level reclaim is allowed

In this example, `queue-b` is over its own `deserved` resources and its parent `parent-b` is also over its `deserved` resources. `queue-a` has a pending pod that cannot be scheduled without reclaim.

```mermaid
graph TD

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems that mermaid didn't be rendered in the netlify preview page, could you check it mate?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah thanks, I forgot to add the dependency.
I think I will wait for vanishka with this PR, so we don't conflict.
#533

@hajnalmt hajnalmt Jul 10, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebased on Vanishka's change and centered the diagrams. Everything works now properly.

R[root]
PA[parent-a<br/>deserved: a100=1<br/>allocated: a100=0]
QA[queue-a<br/>deserved: a100=1<br/>pending: p-reclaimer a100=1]
PB[parent-b<br/>deserved: a100=3<br/>allocated: a100=4]
QB[queue-b<br/>deserved: a100=1<br/>allocated: a100=4<br/>running: p1 a100=1, p2 a100=1, p3 a100=1, p4 a100=1]
R --> PA
PA --> QA
R --> PB
PB --> QB
```

With `ancestorReclaimLevel: 1`, Volcano checks the leaf queue and then the parent level. Because `queue-b` and `parent-b` are both over their relevant `deserved` resources, one running pod from `queue-b` can be reclaimed and the pending pod in `queue-a` can be pipelined.

#### Scenario 2: parent-level reclaim is blocked

Here the leaf queue `queue-b` is over its own `deserved` resources, but its parent is still within its share. That means the parent branch is not borrowing more than it should at the level being protected.

```mermaid
graph TD
R[root]
PA[parent-a<br/>deserved: a100=1<br/>allocated: a100=0]
QA[queue-a<br/>deserved: a100=1<br/>pending: p-reclaimer a100=1]
PB[parent-b<br/>deserved: a100=4<br/>allocated: a100=2]
QB[queue-b<br/>deserved: a100=1<br/>allocated: a100=2<br/>running: p1 a100=1, p2 a100=1]
R --> PA
PA --> QA
R --> PB
PB --> QB
```

With `ancestorReclaimLevel: 1`, reclaim from `queue-b` is blocked. The leaf queue is over deserved, but the protected parent-level check fails because `parent-b` is not over deserved.

#### Scenario 3: level 0 keeps the previous permissive behavior

Using the same topology as Scenario 2, `ancestorReclaimLevel: 0` does not add the parent-level gate. Volcano keeps the previous behavior and relies on the existing reclaim checks. If `queue-b` is reclaimable and over its leaf `deserved` resources, the pending pod in `queue-a` can reclaim from `queue-b` even though `parent-b` is not over deserved.

This is the compatibility setting for clusters that already rely on the existing hierarchical reclaim behavior.

#### Scenario 4: grandparent-level checks protect deeper trees

For deeper queue trees, `ancestorReclaimLevel: 2` also evaluates the grandparent level. This is useful when reclaim should respect a boundary above the direct parent.

```mermaid
graph TD
R[root]
GA[group-a<br/>deserved: cpu=2, memory=2Gi<br/>allocated: cpu=0, memory=0]
PA[parent-a<br/>deserved: cpu=2, memory=2Gi<br/>allocated: cpu=0, memory=0]
QA[queue-a<br/>deserved: cpu=2, memory=2Gi<br/>pending: p-reclaimer cpu=2, memory=2Gi]
GB[group-b<br/>deserved: cpu=3, memory=3Gi<br/>allocated: cpu=2, memory=2Gi]
PB[parent-b<br/>deserved: cpu=1, memory=1Gi<br/>allocated: cpu=2, memory=2Gi]
QB[queue-b<br/>deserved: cpu=1, memory=1Gi<br/>allocated: cpu=2, memory=2Gi<br/>running: p-victim cpu=2, memory=2Gi]
R --> GA
GA --> PA
PA --> QA
R --> GB
GB --> PB
PB --> QB
```

With `ancestorReclaimLevel: 1`, reclaim can pass because the parent-level check for `parent-b` passes. With `ancestorReclaimLevel: 2`, reclaim is blocked because the grandparent-level check for `group-b` fails: `group-b` is not over its `deserved` resources. The running pod in `queue-b` is therefore not a valid reclaim victim for this pending pod.

#### Scenario 5: shared ancestors do not create an extra boundary without a reclaimer signal

For sibling queues, both queues already share the direct parent. The parent is not a boundary between different branches.

```mermaid
graph TD
R[root]
P[parent<br/>deserved: cpu=4, memory=4Gi]
QA[queue-a<br/>deserved: unset<br/>pending: p-reclaimer cpu=2, memory=2Gi]
QB[queue-b<br/>deserved: unset<br/>running: p-victim cpu=2, memory=2Gi]
R --> P
P --> QA
P --> QB
```

In this case, `ancestorReclaimLevel: 1` does not make the shared parent an additional reclaim gate. However, `queue-a` also has no relevant `deserved` value for the pending pod's requested resources, so the reclaimer queue has no leaf-level deserved signal for reclaim. Volcano can block reclaim to avoid evicting a sibling workload when the pending workload is not tied to a deserved share.

#### Scenario 6: sibling reclaim is allowed when the reclaimer has a deserved signal

This example uses the same shared-parent topology as Scenario 5, but `queue-a` now has a `deserved` value that matches the pending pod's requested resources.

```mermaid
graph TD
R[root]
P[parent<br/>deserved: cpu=4, memory=4Gi]
QA[queue-a<br/>deserved: cpu=2, memory=2Gi<br/>pending: p-reclaimer cpu=2, memory=2Gi]
QB[queue-b<br/>deserved: unset<br/>running: p-victim cpu=2, memory=2Gi]
R --> P
P --> QA
P --> QB
```

With `ancestorReclaimLevel: 1`, the shared parent still does not add another boundary between the sibling queues. Because the reclaimer queue has a relevant `deserved` signal for the pending pod, Volcano can reclaim the running pod from `queue-b` and pipeline the pending pod in `queue-a`.

#### Choosing a value

- Use `0` when you want compatibility with the previous reclaim behavior and do not need ancestor-level restrictions.
- Use `1` when the direct parent should be the first protected boundary for cross-parent reclaim.
- Use `2` or higher when the queue hierarchy has deeper quota boundaries and reclaim should respect those higher levels too.
- Configure `deserved` on every ancestor level that should participate in reclaim decisions for the relevant resource. Without a relevant `deserved` value, an ancestor cannot provide a meaningful over-quota signal for that resource.
16 changes: 12 additions & 4 deletions docusaurus.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ const config = {
title: "Volcano",
tagline: "Cloud native batch scheduling system",


future: {
v4: true,
faster: false,
},

url: "https://volcano.sh",
Expand Down Expand Up @@ -45,7 +45,15 @@ const config = {
projectName: "website",

onBrokenLinks: "throw",
onBrokenMarkdownLinks: "throw",
markdown: {
format: "detect",
mermaid: true,
hooks: {
onBrokenMarkdownLinks: "throw",
},
},

themes: ["@docusaurus/theme-mermaid"],

i18n: {
defaultLocale: "en",
Expand Down Expand Up @@ -85,7 +93,7 @@ const config = {
"classic",
{
docs: {
sidebarPath: require.resolve('./sidebar.js'),
sidebarPath: require.resolve("./sidebar.js"),
showLastUpdateTime: true,
showLastUpdateAuthor: false,
editUrl: "https://github.com/volcano-sh/website/tree/master/",
Expand Down Expand Up @@ -191,4 +199,4 @@ const config = {
},
};

export default config;
export default config;
Loading
Loading