Skip to content

Commit 4f41e4d

Browse files
committed
docs: sync missing plugin documentation from volcano core repository
Signed-off-by: hemantch01 <hemantchaudhary905@gmail.com>
1 parent ec7c352 commit 4f41e4d

8 files changed

Lines changed: 532 additions & 0 deletions

content/en/docs/plugins.md

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -173,4 +173,50 @@ The Numa-Aware Plugin aims to address these limitations.
173173

174174
Common scenarios for NUMA-Aware are computation-intensive jobs that are sensitive to CPU parameters, scheduling delays. Such as scientific calculation, video decoding, animation rendering, big data offline processing and other specific scenes.
175175

176+
### Usage
176177

178+
#### Overview
179+
The Usage-based scheduling plugin evaluates actual real-time resource utilization (e.g., CPU, Memory) collected from monitoring systems like Prometheus instead of only depending on requested resources. It prevents new pods from being scheduled onto overloaded nodes and actively balances the cluster workload.
180+
181+
#### Scenario
182+
Useful in clusters experiencing unbalanced node resource consumption where some nodes are overburdened while others remain idle despite having similar requested resources.
183+
184+
### Rescheduling
185+
186+
#### Overview
187+
The Rescheduling plugin periodically rebalances the cluster by evaluating real resource utilization. It actively evicts pods from heavily utilized nodes and shuffles them to under-utilized nodes based on configured target thresholds and strategies like LowNodeUtilization or OfflineOnly.
188+
189+
#### Scenario
190+
Perfect for long-running clusters where dynamic workload lifecycles lead to fragmentation and resource imbalances over time.
191+
192+
### ResourceQuota
193+
194+
#### Overview
195+
The ResourceQuota plugin interfaces with Kubernetes' native `ResourceQuota` objects to ensure that a PodGroup is only enqueued if there is sufficient resource capacity in its namespace.
196+
197+
#### Scenario
198+
Highly beneficial in multi-tenant environments to prevent jobs from entering the scheduling pipeline and clogging the queue when they have no chance of running due to namespace quota restrictions.
199+
200+
### Pod Disruption Budget (PDB)
201+
202+
#### Overview
203+
The PDB Plugin ensures that Volcano respects user-defined Kubernetes PodDisruptionBudget (PDB) constraints during any eviction-based scheduling actions, such as `reclaim`, `preempt`, and `shuffle`.
204+
205+
#### Scenario
206+
Crucial for highly available workloads where simultaneous eviction of multiple replicas could result in service disruption.
207+
208+
### Overcommit
209+
210+
#### Overview
211+
The Overcommit Plugin allows the scheduler to artificially inflate the apparent "idle resources" of the cluster by a configurable factor (e.g., 1.2), permitting more jobs to enqueue in the scheduling pipeline than the physical capacity.
212+
213+
#### Scenario
214+
Useful when administrators want the scheduler to tolerate a larger backlog of `pending` pods waiting for resources without rejecting them outright during peak loads.
215+
216+
### DeviceShare
217+
218+
#### Overview
219+
The DeviceShare Plugin provides a unified framework for sharing specialized hardware devices such as GPUs, NPUs, and FPGAs across multiple pods.
220+
221+
#### Scenario
222+
Ideal for advanced AI/ML environments needing granular hardware sharing, like vGPU, vNPU, and GPU exclusive deployments.
Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
+++
2+
title = "DeviceShare Plugin"
3+
4+
date = 2026-05-11
5+
lastmod = 2026-05-11
6+
7+
draft = false
8+
toc = true
9+
type = "docs"
10+
11+
linktitle = "DeviceShare"
12+
[menu.docs]
13+
parent = "user-guide"
14+
weight = 4
15+
+++
16+
17+
## Introduction
18+
19+
The **DeviceShare Plugin** is an advanced resource scheduling plugin in Volcano that provides a common framework for sharing specialized hardware devices (like GPUs, NPUs, FPGAs) across multiple pods.
20+
21+
Rather than implementing fragmented logic for each new hardware accelerator, Volcano exposes a unified `Devices` interface. The `deviceshare` plugin leverages this interface to perform robust allocation, node filtering, and resource tracking for shared devices.
22+
23+
## Mechanism
24+
25+
The DeviceShare plugin works in conjunction with device-specific implementations. It exposes standard scheduling operations such as `Predicate` (filtering nodes based on available device capacity) and `Allocate`/`Release` (assigning portions of a device to specific pods).
26+
27+
Currently, the `deviceshare` plugin serves as the underlying engine powering features like:
28+
- **GPU Sharing**: Allowing multiple pods to request fractions of a single physical GPU's memory.
29+
- **vGPU and vNPU**: Virtualizing accelerator slices.
30+
- **GPU Exclusive**: Restricting a pod to exclusively own a GPU to avoid contention.
31+
32+
## Configuration and Usage
33+
34+
The `deviceshare` plugin is typically enabled implicitly when you enable device sharing predicates in the Volcano scheduler config map. However, if you are developing custom device sharing logic or need to explicitly declare it, it can be configured in your `volcano-scheduler-configmap`:
35+
36+
```yaml
37+
actions: "enqueue, allocate, backfill"
38+
tiers:
39+
- plugins:
40+
- name: priority
41+
- name: gang
42+
- name: conformance
43+
- name: deviceshare # Enable the device share framework plugin
44+
- plugins:
45+
- name: overcommit
46+
- name: drf
47+
- name: predicates
48+
- name: proportion
49+
- name: nodeorder
50+
- name: binpack
51+
```
52+
53+
> **Note:** For specific guides on how to configure your workloads to request shared GPUs or NPUs, please refer to the dedicated guides for [GPU Sharing](../user_guide_how_to_use_gpu_sharing) and [vNPU](../user_guide_how_to_use_vnpu).
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
+++
2+
title = "HCCLRank Plugin"
3+
4+
date = 2026-05-11
5+
lastmod = 2026-05-11
6+
7+
draft = false
8+
toc = true
9+
type = "docs"
10+
11+
linktitle = "HCCLRank"
12+
[menu.docs]
13+
parent = "user-guide"
14+
weight = 4
15+
+++
16+
17+
## Introduction
18+
19+
In distributed AI training, particularly when using Ascend NPUs (Neural Processing Units) or MindSpore frameworks, the compute nodes need a deterministic rank or index to communicate over HCCL (Huawei Collective Communication Library).
20+
21+
The **HCCLRank Plugin** is a Volcano Job plugin that automatically injects a `hccl/rankIndex` annotation into the Pods of a Volcano Job. It calculates a unique rank for each pod based on its task type (`master` or `worker`) and its replica index.
22+
23+
## Mechanism
24+
25+
During the Pod creation phase (`OnPodCreate`), the HCCLRank Plugin intercepts the pod and adds the `hccl/rankIndex` annotation to it.
26+
27+
The calculation is as follows:
28+
- **Master Role**: Rank = Pod Index
29+
- **Worker Role**: Rank = (Total Master Replicas) + Pod Index
30+
31+
If the Pod already has a `RANK` environment variable defined in its container specifications, the plugin will use that value instead and simply map it to the `hccl/rankIndex` annotation.
32+
33+
## Configuration
34+
35+
To enable the HCCLRank plugin, configure it within the Volcano job controller's configuration or add it to the `plugins` field of your `VolcanoJob` spec.
36+
37+
```yaml
38+
apiVersion: batch.volcano.sh/v1alpha1
39+
kind: Job
40+
metadata:
41+
name: ascend-distributed-training
42+
spec:
43+
minAvailable: 3
44+
schedulerName: volcano
45+
plugins:
46+
hcclrank:
47+
- --master=master
48+
- --worker=worker
49+
tasks:
50+
- replicas: 1
51+
name: master
52+
template:
53+
spec:
54+
containers:
55+
- name: master
56+
image: my-ascend-image
57+
- replicas: 2
58+
name: worker
59+
template:
60+
spec:
61+
containers:
62+
- name: worker
63+
image: my-ascend-image
64+
```
65+
66+
### Arguments
67+
68+
The HCCLRank plugin supports overriding the default task names used to identify master and worker roles:
69+
70+
- **`--master`**: The name of the master role task in your Job spec. Default is `master`.
71+
- **`--worker`**: The name of the worker role task in your Job spec. Default is `worker`.
Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
+++
2+
title = "Overcommit Plugin"
3+
4+
date = 2026-05-11
5+
lastmod = 2026-05-11
6+
7+
draft = false
8+
toc = true
9+
type = "docs"
10+
11+
linktitle = "Overcommit"
12+
[menu.docs]
13+
parent = "user-guide"
14+
weight = 4
15+
+++
16+
17+
## Introduction
18+
19+
In typical cluster environments, the scheduler calculates available idle resources strictly based on physical node capacity minus allocated resources. However, when cluster resources are nearly fully utilized, many PodGroups are rejected from entering the scheduling pipeline and are left completely un-enqueued, which might not be desirable for scenarios where you want the scheduler to tolerate a larger backlog of `pending` pods.
20+
21+
The **Overcommit Plugin** allows the scheduler to artificially inflate the apparent "idle resources" of the cluster by applying an `overcommit-factor`. This permits more jobs to be enqueued and wait in the scheduling pipeline than the physical resources might typically allow.
22+
23+
## Mechanism
24+
25+
The Overcommit plugin evaluates whether a job can be enqueued based on the requested `MinResources` of the PodGroup and the expanded idle resources.
26+
27+
Expanded idle resource is calculated as:
28+
`Idle Resource = (Total Resource * overcommit-factor) - Used Resource`
29+
30+
If the job's minimal requested resources can fit into this expanded idle resource pool, the job is permitted to be enqueued.
31+
32+
## Configuration
33+
34+
To use the Overcommit Plugin, add it to your `volcano-scheduler-configmap` under the `enqueue` tier, and provide an `overcommit-factor`.
35+
36+
```yaml
37+
actions: "enqueue, allocate, backfill"
38+
tiers:
39+
- plugins:
40+
- name: overcommit # Enable the overcommit plugin
41+
arguments:
42+
overcommit-factor: 1.2 # The overcommit factor. Default is 1.2
43+
- name: priority
44+
- name: gang
45+
- name: conformance
46+
- plugins:
47+
- name: drf
48+
- name: predicates
49+
- name: proportion
50+
- name: nodeorder
51+
- name: binpack
52+
```
53+
54+
### Arguments
55+
56+
- **`overcommit-factor`**: A float value greater than or equal to `1.0`. For example, `1.2` means the scheduler will pretend the cluster has 20% more total resources when deciding whether to enqueue jobs into the pipeline. If a value less than `1.0` is provided, the plugin will automatically fallback to the default value of `1.2`.
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
+++
2+
title = "Pod Disruption Budget (PDB) Plugin"
3+
4+
date = 2026-05-11
5+
lastmod = 2026-05-11
6+
7+
draft = false
8+
toc = true
9+
type = "docs"
10+
11+
linktitle = "Pod Disruption Budget"
12+
[menu.docs]
13+
parent = "user-guide"
14+
weight = 4
15+
+++
16+
17+
## Introduction
18+
19+
When users deploy highly available jobs or applications on Volcano, they often need to limit the number of pod replicas that can be evicted or destroyed simultaneously to avoid downtime. This constraint is managed via Kubernetes **PodDisruptionBudget (PDB)** resources.
20+
21+
The **PDB Plugin** ensures that Volcano respects user-defined PDB constraints during the scheduling process, specifically during eviction actions like `reclaim`, `preempt`, and `shuffle`.
22+
23+
## Prerequisites
24+
25+
- Your Kubernetes version must be 1.21 or later.
26+
- You must have created valid `PodDisruptionBudget` resources for your workloads.
27+
28+
## Mechanism
29+
30+
The PDB Plugin registers several functions (`ReclaimableFn`, `PreemptableFn`, and `VictimTasksFn`) under the `reclaim`, `preempt`, and `shuffle` actions. It maintains a cache of PDBs using `v1.PodDisruptionBudgetLister`.
31+
32+
During eviction scenarios, the plugin filters out tasks whose eviction would violate the configured PDB constraints. It calculates the `DisruptedPods` (pods whose eviction was processed but not yet observed by the PDB controller) and ensures the remaining available replicas satisfy the budget.
33+
34+
## Configuration
35+
36+
To enable the PDB Plugin, update the `volcano-scheduler-configmap` to include the `pdb` plugin in your configuration tiers.
37+
38+
```yaml
39+
actions: "reclaim, preempt, shuffle"
40+
tiers:
41+
- plugins:
42+
- name: pdb # Enable the PDB plugin
43+
- name: priority
44+
- name: gang
45+
- name: conformance
46+
- plugins:
47+
- name: overcommit
48+
- name: drf
49+
- name: predicates
50+
- name: proportion
51+
- name: nodeorder
52+
- name: binpack
53+
```
54+
55+
*Note: The PDB plugin will be actively invoked when actions like `reclaim`, `preempt`, or `shuffle` are executed in the scheduler workflow.*
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
+++
2+
title = "Rescheduling Plugin"
3+
4+
date = 2026-05-11
5+
lastmod = 2026-05-11
6+
7+
draft = false
8+
toc = true
9+
type = "docs"
10+
11+
linktitle = "Rescheduling"
12+
[menu.docs]
13+
parent = "user-guide"
14+
weight = 4
15+
+++
16+
17+
## Introduction
18+
19+
Unbalanced resource utilization across a Kubernetes cluster often occurs due to unreasonable scheduling strategies, dynamic changes in job lifecycles, and node status changes (such as added/removed nodes or taint/affinity modifications).
20+
21+
The **Rescheduling** plugin addresses these issues by actively rebalancing the cluster's resource utilization among nodes. It accomplishes this by evaluating real resource utilization (via Prometheus metrics) instead of merely the requested resource amounts, and it periodically evicts pods based on custom configured rescheduling strategies.
22+
23+
## Rescheduling Workflow
24+
25+
1. **Resource Filter**: Filters workloads which are eligible to be evicted based on queues or labels.
26+
2. **Strategy Evaluation**: Evaluates filtered workloads against the configured rescheduling strategies to determine which ones should be evicted.
27+
3. **Eviction**: Evicts the pods attached to the identified workloads.
28+
4. **Periodical Execution**: Executes the above process periodically.
29+
30+
## Rescheduling Strategies
31+
32+
Volcano's rescheduling plugin supports multiple strategies to select potential evictees:
33+
34+
- **LowNodeUtilization**: Targets unbalanced nodes by evicting pods from highly utilized nodes and shuffling them to low utilized nodes based on configured target thresholds.
35+
- **OfflineOnly (OLO)**: Only selects offline workloads (annotated with `preemptable: true`) for rescheduling.
36+
- **LowPriorityFirst (LPF)**: Sorts workloads by priority and evicts lower priority pods first.
37+
- **ShortLifeTimeFirst (SLTF)**: Sorts workloads by running time. Pods with the shortest life time will be rescheduled first to ensure long-running workloads are not interrupted.
38+
- **BigObjectFirst (BOF)**: Selects workloads which request the most dominant resource and reschedules them first to improve system throughput and avoid small workloads starvation.
39+
- **MoreReplicasFirst (MRF)**: Sorts workloads by replica number. Workloads with the most replicas are rescheduled first, making it friendly to `gang` scheduling by considering `minAvailable`.
40+
41+
## Configuration
42+
43+
To enable the Rescheduling plugin, you must configure the `volcano-scheduler-configmap` by adding the `shuffle` action and configuring the `rescheduling` plugin within the tiers.
44+
45+
```yaml
46+
actions: "enqueue, allocate, backfill, shuffle" ## Add 'shuffle' action
47+
tiers:
48+
- plugins:
49+
- name: priority
50+
- name: gang
51+
- name: conformance
52+
- name: rescheduling ## Rescheduling plugin
53+
arguments:
54+
interval: 5m ## Optional. Frequency at which the strategies are called. Default is 5m.
55+
metricsPeriod: 5m ## Optional. The duration of metrics to consider. Default is 5m.
56+
strategies: ## Required. Strategies to execute in order.
57+
- name: offlineOnly
58+
- name: lowPriorityFirst
59+
- name: lowNodeUtilization
60+
params:
61+
thresholds:
62+
"cpu" : 20 ## Threshold below which a node is considered under-utilized
63+
"memory": 20
64+
"pods": 20
65+
targetThresholds:
66+
"cpu" : 50 ## Target utilization to reach for balance
67+
"memory": 50
68+
"pods": 50
69+
queueSelector: ## Optional. Select workloads in specified queues as potential evictees. All queues by default.
70+
- default
71+
- test-queue
72+
labelSelector: ## Optional. Select workloads with specified labels as potential evictees. All labels by default.
73+
business: offline
74+
team: test
75+
- plugins:
76+
- name: overcommit
77+
- name: drf
78+
- name: predicates
79+
- name: proportion
80+
- name: nodeorder
81+
- name: binpack
82+
```
83+
84+
> **Note:** The rescheduling decisions consider metrics collected from Prometheus. Ensure your metrics configuration is correctly set up as it evaluates real node resource utilization instead of requested resource amounts.

0 commit comments

Comments
 (0)