new subcommand `canary flux run` to manage special flux-y behavior

[mdmosby]

Flux is a hierarchical scheduler. For maximum throughput it is best to create an enclosing allocation and then run all jobs within so that you don't have to sit in the queue. Furthermore, it naturally supports inter-job dependencies. We would like to have an execution mode for canary that determines (or has specified on CLI) the appropriate enclosing job size allocation, then submits the test DAG to Flux to delegate all scheduling, resource allocation/mapping, etc.

Proposal:

canary flux run [--nodes,-N NODES] ...

  --nodes,-N NODES
          Run tests in an enclosing allocation across NODES nodes.
          NODES is determined using internal heuristics if unspecified.

Broad strokes, should create the enclosing allocation from alloc_handle = Flux().alloc(...) and forward to the HPC-connect backend for Flux for all further job submission. Coupled with other work (done?) supporting submission of jobs with dependencies, each test's handle should be held by Canary and console reporting should look similar to canary run rather than canary hpc run.

Other considerations:

• failed jobs should auto-cancel dependent jobs
• when all jobs finish, enclosing allocation should be finalized
• ctrl-C should cancel all running jobs, then enclosing allocation, then exit
• If NODES specified, use appropriately sized resource pool to determine which tests can run
• GPU device masking? Does flux run handle this for us?

[mdmosby]

Had a brainwave that this could be implemented by submitting a job that calls canary with appropriate flags and forwards output rather than trying to do everything in-core. This has the added benefit of being potentially able to bootstrap a flux instance into a host scheduler.