Add OCS and GCS Backend Support to clustermq

[ernst-bablick]

Hi Michael,

I would like to contribute support for Open Cluster Scheduler (OCS) and Gridware Cluster Scheduler (GCS) to clustermq. Both are successors of Sun Grid Engine (SGE) and Univa Grid Engine (UGE).

• OCS (Open Source):
  https://github.com/hpc-gridware/clusterscheduler

• GCS (Commercial / Enterprise):
  https://hpc-gridware.com/

Background

While testing the existing SGE backend, I observed that it does not work out of the box with OCS/GCS and requires adjustments.

We currently have a prospect who intends to use clustermq on both OCS and GCS clusters concurrently. For this reason, we require distinct scheduler templates for each system. I have implemented the necessary changes in a forked repository and created product-specific templates for both OCS and GCS.

Before opening a pull request, I would like to clarify a few implementation questions and discuss design considerations.

Questions

1. Array Job Behavior and Worker Cancellation

In SGE-like systems, workloads are typically scheduled as array jobs. This means that workers belonging to a single Q() call may not start simultaneously. As a result, there can be situations where some workers remain queued even though the workload has already completed.

It appears that remaining queued workers are not automatically canceled when the master process finishes.

• Is there currently a mechanism to cancel remaining queued workers?
• If not, where would be the most appropriate place in the architecture to implement such cancellation logic?

2. The cores Parameter in SGE and PBS Backends

The SGE and PBS backends use a cores parameter to request multiple cores per worker job (possibly via a parallel environment). I would like clarification on:

• When and how is this parameter used?
• Does this relate to MPI-based R workloads?
• Or is it primarily intended to allocate multiple scheduler slots/cores per clustermq worker process?

From my observation, Q() does not expose a cores argument directly. How and where is this parameter expected to be configured?

As I am relatively new to R, I would also appreciate insight into whether there are common R workloads requiring MPI-style parallelism within a clustermq worker, or whether this parameter primarily controls resource allocation.

3. CPU and Memory Binding Features (GCS)

GCS provides enterprise binding features that allow binding jobs to specific hardware units such as:

• Hardware threads
• CPU cores
• Dies
• Sockets
• CPU caches
• NUMA nodes

This enables the scheduler to place jobs on hardware resources sharing cache or memory locality, which can significantly improve performance for certain workloads.

Does clustermq currently provide a way to expose or control scheduler-level binding features?

If not, would it be acceptable to extend Q() with an optional parameter that allows users to specify binding requirements for their jobs?

In some scenarios, assigning one CPU core per worker is not optimal. Binding to a NUMA node or cache domain may provide better performance characteristics, but I am unsure if this also applies to R workloads.

Next Steps

I would be happy to share my fork and proposed changes for review. Before opening a pull request, I would appreciate some guidance.

[mschubert]

Hi,

Thanks for the heads-up! I'm very happy to accept meaningful contributions.

1. Array Job Behavior and Worker Cancellation

Is there currently a mechanism to cancel remaining queued workers?

If any workers are remaining (that includes pending) when we call cleanup() or the gc collects the QSys object, we will call the shell command to kill all remaining workers; example:

clustermq/R/qsys_sge.r

Lines 48 to 54 in bd85c48

	finalize = function(quiet = TRUE) { # self$workers_running == 0
	if (!private$is_cleaned_up) {
	system(paste("qdel", private$job_id),
	ignore.stdout=quiet, ignore.stderr=quiet, wait=FALSE)
	}
	private$is_cleaned_up = TRUE
	}

Each QSys implementation (e.g. SGE, or the two you propose) should provide or inherit a working finalize() function that does that.

2. The cores Parameter in SGE and PBS Backends

The SGE and PBS backends use a cores parameter to request multiple cores per worker job (possibly via a parallel environment). I would like clarification on:

• When and how is this parameter used?
• Does this relate to MPI-based R workloads?
• Or is it primarily intended to allocate multiple scheduler slots/cores per clustermq worker process?

From my observation, Q() does not expose a cores argument directly. How and where is this parameter expected to be configured?

You can supply this via Q(..., template=list(var=value)), as any other possible template value. Here, the cores parameter is not a first-class citizen like memory that can be passed directly to Q, which was a (rather arbitrary) design choice at the time.

As I am relatively new to R, I would also appreciate insight into whether there are common R workloads requiring MPI-style parallelism within a clustermq worker, or whether this parameter primarily controls resource allocation.

You could run an R function call that uses multiple cores/threads via Q, also without MPI. If we set cores=2, we provide the resource allocation, and the user is responsible for actually using those cores.

3. CPU and Memory Binding Features (GCS)

GCS provides enterprise binding features that allow binding jobs to specific hardware units such as [...]
Does clustermq currently provide a way to expose or control scheduler-level binding features?

These can all be passed via template(var=value), as long as there is a valid template field with that name.

In some scenarios, assigning one CPU core per worker is not optimal. Binding to a NUMA node or cache domain may provide better performance characteristics, but I am unsure if this also applies to R workloads.

This is user responsibility, we just provide the allocation and execute whatever R code is provided.

Next Steps

I would be happy to share my fork and proposed changes for review. Before opening a pull request, I would appreciate some guidance.

I would expect that inheriting from SGE like PBS/Torque do here would be a good option:

clustermq/R/qsys_sge.r

Lines 60 to 72 in bd85c48

	PBS = R6::R6Class("PBS",
	inherit = SGE,
	public = list(
	initialize = function(..., template=getOption("clustermq.template", "PBS")) {
	super$initialize(..., template=template)
	private$array_idx = "$PBS_ARRAY_INDEX"
	private$job_id = private$qsub_stdout[1]
	}
	),
	cloneable = FALSE
	)

I see that you already went this way with your fork and this probably already works!

Please do not change the documentation order of the schedulers. When Slurm is mentioned first this is because it's the most widely used (in my experience). OCS/GCS should probably be mentioned after SGE, because they're related.

Are OCS/GCS using "$PBS_ARRAY_INDEX" as index variable, or is this a copy/paste mistake?