[QST]: Support NVIDIA MIG and dynamic MIG allocation for Kubernetes workloads

[JiaHanDong]

Summary

Add support for NVIDIA Multi-Instance GPU (MIG) resources in slurm-bridge, including:

1. Scheduling workloads onto pre-configured MIG instances.
2. Supporting dynamically created and removed MIG instances through Kubernetes Dynamic Resource Allocation (DRA).

This issue follows up on the roadmap discussion where the maintainers suggested opening a separate issue to track and prioritize MIG and GPU partitioning support.

Motivation

MIG-capable NVIDIA GPUs can be partitioned into multiple hardware-isolated GPU instances. Each MIG instance has dedicated compute and memory resources and can be allocated independently.

Without MIG awareness, slurm-bridge can only account for the physical GPU as a whole. This prevents Kubernetes and native Slurm workloads from safely sharing a MIG-enabled GPU while retaining consistent resource accounting between the two schedulers.

A typical use case is a node with one physical GPU partitioned into several MIG instances, where:

• Kubernetes workloads request individual MIG profiles through DRA.
• Native Slurm jobs may also request MIG resources.
• slurm-bridge must ensure that both scheduling domains use a consistent view of the available MIG capacity.
• Two workloads may use different MIG instances on the same physical GPU without being assigned the same instance or exceeding the parent GPU's capacity.

Requested functionality

1. Pre-configured MIG instances

Support nodes where MIG mode and MIG instances have already been configured outside slurm-bridge.
slurm-bridge should be able to:

• Discover MIG devices advertised through Kubernetes DRA ResourceSlice objects.
• Associate each MIG instance with its parent physical GPU.
• Preserve the MIG profile and device identity during scheduling.
• Translate a Kubernetes MIG allocation into the corresponding Slurm resource request or reservation.
• Prevent the same MIG instance from being allocated to both a Kubernetes workload and a native Slurm job.
• Allow different MIG instances on the same physical GPU to be allocated independently when the Slurm configuration permits it.

2. Dynamic MIG

Support DRA drivers that advertise possible MIG partitions before the corresponding MIG instances physically exist.

For example, the NVIDIA DRA driver can advertise abstract MIG devices using partitionable-device capacity. After scheduling, its kubelet plugin creates the selected GPU Instance and Compute Instance during NodePrepareResources, and removes them during NodeUnprepareResources.

For this mode, slurm-bridge should be able to:

• Understand partitionable GPU devices and their shared-capacity relationships.
• Preserve the selected parent GPU, MIG profile, placement, and capacity in the scheduling decision.
• Reserve the corresponding parent-GPU capacity in the Slurm external job.
• Avoid allocations that select overlapping or incompatible MIG placements.
• Coordinate correctly with the DRA prepare lifecycle, where the concrete MIG UUID may not exist until after scheduling.
• Release the Slurm-side reservation when the workload finishes or device preparation fails.
• Recover consistently after scheduler, controller, kubelet plugin, or node restarts.

The goal is not necessarily for slurm-bridge itself to configure MIG hardware. Hardware creation and removal may remain the responsibility of the NVIDIA DRA driver. The requirement is that slurm-bridge can schedule and account for those dynamic partitions correctly.

Expected scheduling behavior

Given one physical GPU with multiple compatible MIG partitions:

1. A Kubernetes workload requests a specific MIG profile or capacity.
2. slurm-bridge represents the request in the external Slurm job.
3. Slurm selects a node without conflicting with native Slurm allocations.
4. Kubernetes binds the DRA allocation to a compatible MIG partition on that node.
5. For dynamic MIG, the DRA driver creates the concrete MIG instance during resource preparation.
6. The workload receives only the allocated MIG device.
7. When the workload terminates, the Slurm reservation is released and a dynamically created MIG instance may be removed by the DRA driver.

Multiple workloads should be able to use non-overlapping MIG instances on the same physical GPU without oversubscribing its compute, memory, or placement capacity.

[killianmuldoon]

Thanks for opening the issue - I'll see how we can prioritise it internally. I have a couple of questions to help understand the use case - what kind of workloads are you running where you want MIG sharing scheduled with slurm bridge? Does the cluster have a wide range of job sizes? Would some nodes have MIG enabled and others not?

[JiaHanDong]

Thanks for following up. The workloads for which we want to use GPU partitioning are interactive workloads, such as Jupyter notebook sessions, that need only a fraction of a GPU rather than an entire device.

We use slurm-bridge so that Kubernetes and Slurm can share a subset of compute nodes and improve overall utilization. The cluster is heterogeneous in this respect: some nodes will have MIG enabled, while others will continue to expose full GPUs. Administrators will explicitly control which nodes are MIG-enabled. Consequently, GPU request sizes vary between smaller interactive workloads using MIG instances and workloads requiring full GPUs.

Our initial idea was to represent partitioned GPU capacity in Slurm using the Shard mechanism. Although Shard does not provide resource or fault isolation, that may be sufficient here because slurm-bridge external jobs do not execute the actual workloads; they only reserve and account for resources. The DRA GPU driver would remain responsible for creating and exposing MIG instances and enforcing isolation, while slurm-bridge would coordinate the amount of partitioned GPU capacity reserved in Slurm.

One issue we are still considering is how to prevent conflicts between GPU resources used by native Slurm jobs and those reserved by external jobs for Kubernetes pods. Once part of a physical GPU has been reserved by an external placeholder job, we want the remaining partitions on that GPU to be available only to other external jobs, not to regular Slurm jobs. We are investigating how best to enforce this policy on the Slurm side.

I also noticed your response in the dra-driver-cpu issue regarding device-index consistency. The GPU index used by the DRA driver may not match the index used by Slurm. Our current idea is to introduce an administrator-managed ConfigMap as the authoritative mapping between DRA GPU identities and Slurm GPU indices, with slurm-bridge performing the necessary translation.

This is only an initial proposal, and we would appreciate your thoughts on a more suitable integration model.

[killianmuldoon]

I did some playing around with how we could support this. The bottom line is that static MIG support is tractable, but dynamic MIG would take significant work and may not fit with slurm-bridge at all.

One issue we are still considering is how to prevent conflicts between GPU resources used by native Slurm jobs and those reserved by external jobs for Kubernetes pods. Once part of a physical GPU has been reserved by an external placeholder job, we want the remaining partitions on that GPU to be available only to other external jobs, not to regular Slurm jobs. We are investigating how best to enforce this policy on the Slurm side.

We have a change coming in the upcoming release which should prevent slurm and kubernetes from scheduling to the same node. It sounds like that could solve your issue.

The GPU index used by the DRA driver may not match the index used by Slurm. Our current idea is to introduce an administrator-managed ConfigMap as the authoritative mapping between DRA GPU identities and Slurm GPU indices, with slurm-bridge performing the necessary translation

We're making a lot of changes to DRA in the upcoming release - the basics are already merged and available on main for the example GPU driver. We're explicitly tracking which GPUs are used in DRA by setting this to the Slurm Node Extra field. You can try it out and see if it fits your model, but there's still a lot of changes pending which will change how the DRA flow works, so it could change day-to-day.

[killianmuldoon]

By the way - are you active in the Kubernetes slack? It's a good place to talk about these issues more broadly.