[FEA]: Make managed-node tainting optional

[rogowski-piotr]

Is this a new feature, an improvement, or a change to existing functionality?

Improvement

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

Slurm Bridge currently taints every bridged Kubernetes node with slinky.slurm.net/managed-node=<schedulerName>:NoExecute. This is useful for fully isolated Slurm-managed nodes, but it is too broad for shared-node deployments where Slurm Bridge only manages selected namespaces through admission.managedNamespaces.

In this setup, normal cluster infrastructure with housekeeping components should continue to run through the default scheduler. The global NoExecute taint forces all of these unrelated components to carry Slurm Bridge tolerations, even though they are not Slurm-managed workloads. This becomes especially brittle for hook Jobs or operator-generated Pods whose templates might be immutable after creation.

Feature Description

Make managed-node tainting optional. Slurm Bridge should be able to run with node registration and node state reconciliation enabled, but without automatically adding taint.

The default behavior should remain unchanged.

Describe your ideal solution

Add an optional Helm value (default true), for example:

controllers:
  nodeTaint:
    enabled: true

Describe any alternatives you have considered

No response

Additional context

No response

Code of Conduct

• I agree to follow Slurm Bridge's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[killianmuldoon]

Thanks @rogowski-piotr - I can definitely see the friction with the node taint. Unfortunately I'm not sure I understand how to safely expose an option like this - even if it could work in some clusters.

If we have two schedulers running there's going to be race conditions. Even avoiding those races, there's a split brain problem with resource accounting in Slurm. CPUs / memory consumed by non-slurm scheduled pods won't be mirrored between Slurm and Kubernetes in a way that makes sense.

If the pain point is mainly around adding the toleration we could add a mutation at admission time to add the toleration if the pod is selected by slurm-bridge. Would that be a worthwhile improvement?

We already do this 😅