Skip to content

Add support for scheduled array task #14

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 49 commits into
base: master
Choose a base branch
from
Draft

Add support for scheduled array task #14

wants to merge 49 commits into from

Conversation

@arteymix
Copy link
Member

@arteymix arteymix commented Oct 25, 2025 •
edited
Loading

A scheduled array task

TODO:

  • integrate this with batch parameter to collapse multiple scheduled tasks into a job array
  • integrate with sacct
  • add a submission mecahnism based on Slurm REST API

Depends on #12 being merged.

arteymix and others added 30 commits October 21, 2025 18:48
@arteymix
Improve sratools support
92f82ba
@arteymix
Add support for Cell Ranger
ccc61a5
@arteymix
Add an option to DynamicTaskWithOutputMixin to not unpack singleton o…
01439e2 
…utput
@arteymix
Add a task namespace for CellRangerCount
04f9e76
@arteymix
Disable singleton unpacking by default for DynamicTaskWithOutputMixin
e8521a1
@arteymix
Revert "Disable singleton unpacking by default for DynamicTaskWithOut…
8e9d493 
…putMixin"

This reverts commit 176cb04.
@arteymix
Add Python 3.14 to the CI
adcb7b7
@arteymix
Add cellranger.BamToFastq
ea59d20
@arteymix
Write CellRanger to a temporary output directory
4a5b71b
@arteymix
Cap the length of Slurm job names to 100 characters
b7d3ef1
@arteymix
Fix typo in walltime description
f799b81
@arteymix
Write cutadapt output atomatically
da39ee4 
Eliminate RemoveTaskOutputOnFailureMixin.
@arteymix
Make multiqc task atomic
b683caa
@arteymix
Add a resources for running Cell Ranger count and bamtofastq
e5ae23c
@arteymix
Add a resource for cutadapt jobs
fcf2795
@arteymix
Add an io_jobs resource for I/O intensive jobs
f4005f3
@arteymix
Cleanups cutadapt optional parameter defaults
22e75d8
@arteymix
Use Luigi's builtin temporary output for fastq-dump
41649db
@arteymix
More work...
785441a
@arteymix
Improvements for scheduled external programs
0363b8d 
Move schedulers in a new module and add an interface for schedulable
tasks.

Include Luigi metadata in the Slurm --comment field.
@arteymix
Make fastqc.GenerateReport atomic
2db27ca
@arteymix
Add a .fastq.gz to the temporary output files of cutadapt
bad7b8e
@arteymix
cutadapt: Use the patched temporary_path() implementation to preserve…
d63187f 
… file extension
@arteymix
Add io_jobs to the default resources
fd975bb
@arteymix
Add fastqc_jobs and mark prefetch consume an io_jobs resource
248d38a
@arteymix
bcftools: Make all output atomic
6c59b7f
@arteymix
Drop empty gatk module
8f69622
@arteymix
vep: Make vep.Annotate atomic
0b4bfc8
@arteymix
sratoolkit: Fix formatting
ef7699f
@arteymix
Formatting and minor cleanups
568455f
arteymix and others added 19 commits October 22, 2025 10:36
@arteymix
Defer checks for task parameters
f524a11 
Task parameters can be overwritten by the subclass, so it's best to
defer their validation when they are actually used.
@arteymix
Add a local and ssh scheduler definitions
ca5886c 
Make srun binary configurable

Add individual configuration sections for each scheduler.

Add a test configuration to use for the CI.
@arteymix
Add testspaths to pytest.ini
370af87
@arteymix
fixup! Add a local and ssh scheduler definitions
66b60ee
@arteymix
Fix various typing issues
cd94527
@arteymix
Migrate to pyproject.toml
00dcaac
@arteymix
Allow schedulers to provide task status and progress updates
17531bd
@arteymix
Add development dependencies
efed7c7
@arteymix
schedulers: Few improvements to task status tracking with Slurm
f5a05ec 
Simply use the task ID in the comment instead of nesting a JSON payload.

Cache squeue output to minimize the number of calls to squeue,
especially when multiple workers are overseeing Slurm jobs.

Peform more validation with graceful logging when the squeue payload
does not match expectations.
@arteymix
Add missing bioluigi.tasks package
101e8fa
@arteymix
schedulers: Do not set cwd for srun, --chdir already takes care of it
0243d77
@arteymix
Add dev dependencies for building docs
6573466
@arteymix
Ignore build directories
6d1dda4
@arteymix
slurm: Synchronize squeue calls and minimize task status updates
d272937
@arteymix
slurm: Add missing stdout/stderr when not capturing output
eb60068
@arteymix
sratoolkit: Simplify output of FastqDump to simply be the output dire…
009357a 
…ctory
@arteymix
slurm: Filter keys included in the status message
0fc9915
@arteymix
fixup! slurm: Filter keys included in the status message
1558199
@arteymix
Add support for array jobs
86d4d17
@arteymix arteymix self-assigned this Oct 25, 2025
@arteymix arteymix mentioned this pull request Oct 27, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@arteymix arteymix

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@arteymix