Set priorityClassName for Spark tasks

[madiyar-wayve]

Tracking issue

Related to #6348.

Why are the changes needed?

These changes will allow setting priorityClassName for Spark tasks.

What changes were proposed in this pull request?

The Spark K8s plugin now propagates the priorityClassName from the resolved pod spec onto both the driver and executor specs of the generated SparkApplication:

• In createDriverSpec, set spec.sparkSpec.PriorityClassName from podSpec.PriorityClassName.
• In createExecutorSpec, set spec.sparkSpec.PriorityClassName from podSpec.PriorityClassName.

The value is set via the existing strPtr helper, which returns nil for an empty string. This means that when no priority class is configured, the field is left unset rather than being populated with an empty string, preserving the previous behavior in that case.

The podSpec.PriorityClassName value honors anything coming from the task's pod template / overlay pod spec, since it is read after MergeOverlayPodSpecOntoBase is applied.

How was this patch tested?

Added unit tests in spark_test.go:

• TestBuildResourcePriorityClassName: sets priorityClassName on the pod spec and asserts that both the driver and executor specs of the resulting SparkApplication have the expected PriorityClassName.
• TestBuildResourceNoPriorityClassName: verifies that when no priority class is set, both the driver and executor PriorityClassName fields are left nil (unset) rather than an empty string.

All new and existing Spark plugin unit tests pass.

Labels

• added

Setup process

N/A

Screenshots

N/A

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

N/A

Stack

Docs link

N/A

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[madiyar-wayve]

I don't know what happened, but I didn't want to close this PR

[Tom-Newton]

LGTM. Maybe in future we should switch to using the driver and exectutor pod templates https://github.com/kubeflow/spark-operator/blob/master/docs/api-docs.md#sparkpodspec. I think that would enable a simpler implementation where we don't need to manually add support for new properties.