Streamlining/ optimization/ cleanup tasks for future + new features related to nextflow

[ppreshant]

Noting things to improve the nextflow code to make things more maintainable in the future~!
This comment is for assembly, and other stuff is in notes below

assembly_mag subworkflow

• Default to nf-core modules wherever they were modified and use modules.config to add modifications instead
  • (future): Austin will identify modules from nf-core that have been modified/moved to local eventually and add comments about changes. (Slack, 5/Sep/25) -- Could start from somatem_mags.nf's diff in latest commit
  • SingleM, TaxBurst: directly in local ; Bakta moved to local ; checkm2_parse: custom made likely in local.
• Harmonize process-specific resource specifications RAM, CPUs provided in modules.config to instead be closer to the process_low/medium/high labels for smoother harmonization

Specific modules

• flye: Is there any benefit in the logs being renamed with the file prefix in flye/main.nf:L51?

gzip -c assembly.fasta > ${prefix}.assembly.fasta.gz
gzip -c assembly_graph.gfa > ${prefix}.assembly_graph.gfa.gz
gzip -c assembly_graph.gv > ${prefix}.assembly_graph.gv.gz
mv assembly_info.txt ${prefix}.assembly_info.txt
mv flye.log ${prefix}.flye.log
mv params.json ${prefix}.params.json

• Unnecessary/confusing channel manipulations in assembly_mags.nf subwk. Don't need to merge channels (check nf-core's standard version; don't add environment that can be passed as a parameter easier.

    // Binning with SemiBin2
    ch_asm_bam = FLYE.out.fasta.join(SAMTOOLS_SORT.out.bam, by: [0])
    
    // Set SemiBin2 environment parameter
    ch_asm_bam_with_env = ch_asm_bam.map { meta, fasta, bam ->
        def new_meta = meta.clone()
        new_meta.semibin_env = params.sample_environment
        [new_meta, fasta, bam]
    }
    
    SEMIBIN_SINGLEEASYBIN(ch_asm_bam_with_env)

Technical features

• flye database download: should include the light database using option --type light in ext.args. what should the user call this?

As this taxonomic-untargeted database is fairly demanding in terms of storage consumption, we also provide a lightweight DB type providing all non-coding feature information but only PSCC information from UniRef50 clusters for CDS. If download bandwiths or storage requirements become an issue or if shorter runtimes are favored over more-specific annotation, the light DB will do the job.

• Include a co-assembly mode to generate more bins? (needs more thought for data preperation here)

Multi-sample binning requires more complex steps to prepare input data as well as more computation but can also result in more bins (particularly in complex habitats). See Figure 3b in the SemiBin1 manuscript for a comparison of multi sample vs. single sample. source: semibin docs

• Check if we give the .gfa files to semibin2 and read documentation for improvements and downsides expected?

Claude says: In metaFlye's case, the graph is explicitly output (as a .gfa file), and SemiBin2 and similar binners can optionally use graph connectivity — not just sequence similarity — to decide which contigs belong together

[ppreshant]

Notes

• modules that seem to be customized ; use this list to move to nf-core versions

~/Somatem$ grep -r "custom somatem module" modules/*
modules/local/taxburst_convert/main.nf:// custom somatem module
modules/local/taxburst/main.nf:// custom somatem module
modules/local/checkm2/parse/main.nf:// custom somatem module

modules/nf-core/nanoplot/main.nf:// custom somatem module
modules/nf-core/flye/main.nf:// custom somatem module
modules/nf-core/semibin/singleeasybin/main.nf:// custom somatem module
modules/nf-core/checkm2/predict/main.nf:// custom somatem module
modules/nf-core/samtools/sort/main.nf:// custom somatem module
modules/nf-core/samtools/coverage/main.nf:// custom somatem module

Relocated to nf-core now:

• samtools/sort : d2efbc1

• semibin/singleeasybin : 17024a0)

• fly: work in progress. need the workflow level publish sorted before doing this. pop this stash when needed: (after git stash list) : stash@{0}: On main: flye_nf-core shift

• ...

• Need to move the publishDirs to these relocated processes to workflow level publish (in the current branch ~ better than the old branch for merge/current reasons~?)

Brief work on magsv3 branch

• Side project: testing magsv3 with cleaned up modules.config on latest nextflow 25.10.

  • big_mag assembly fails now (31/Oct/25) ; fixed in mag: stalling at semibin/singleeasybin with update 25.10 and nf-core' samtools/sort #65
  • Using smaller test dataset ; Create small test datasets for full assembly_mags workflow #66

• Moving to standards of nextflow 25.10

  • Change Channel. to channel.

  The use of Channel to access channel factories is deprecated -- use channel insteadnextflow
  look for examples like this or use seqera AI to fix? if the agentic stuff works!?

$ grep -r "Channel\." subworkflows/*
subworkflows/local/assembly_mags.nf:    ch_versions = Channel.empty()
subworkflows/local/utils/nf-core-compatibility.nf:        tuple_out = Channel.fromPath(reads)
subworkflows/local/utils/nf-core-compatibility.nf:        tuple_out = Channel.fromPath("${reads}/*.fastq.gz")
subworkflows/local/download_databases.nf:    ch_hostile_db = Channel.empty()
subworkflows/local/download_databases.nf:    ch_emu_db = Channel.empty()
subworkflows/local/download_databases.nf:    ch_lemur_db = Channel.empty()
subworkflows/local/download_databases.nf:    ch_checkm2_db = Channel.empty()
subworkflows/local/download_databases.nf:    ch_bakta_db = Channel.empty()
subworkflows/local/download_databases.nf:    ch_singlem_db = Channel.empty()
subworkflows/local/taxonomic-profiling.nf:    ch_versions = Channel.empty() // collect versions from all modules
subworkflows/local/genome-dynamics.nf:    ch_versions = Channel.empty() // collect versions from all modules
subworkflows/local/utils_nfcore_somatem_pipeline/main.nf:    ch_versions = Channel.empty()
subworkflows/local/pre-processing.nf:    ch_versions = Channel.empty() // collect versions from all modules
subworkflows/nf-core/utils_nfcore_pipeline/main.nf:    return ch_versions.unique().map { version -> processVersionsFromYAML(version) }.unique().mix(Channel.of(workflowVersionToYAML()))

[ppreshant]

General nextflow

• need to organize the outputs within the results/pipeline_info/.. into execution_report/ ; timeline/ and trace/ for better access
• Apply strict syntax: Whici is default in v26.04.0 (disable it by setting the environment variable NXF_SYNTAX_PARSER=v1)
  • What does this entail? and what are the benefits?
  • Need to use the def for variable declaration (first time): example: def ch_versions = channel.empty()
• Need to implement a quick way to check for file presence/absence before crashing the workflow with cryptic error log! see how nf-core handles this!

Other subworkflows/modules

Hostile:

• hostile log location: move the log.info entry from download_databases.nf into the hostile/fetch module? this should activate/print log only when cache doesn't go through and DB is actually downloaded ; check of modules can have logging commands

 log.info "Fetching hostile index/database: ${db_name_without_extension} for minimap2. Will take > 5 min"

• Support mouse index for mouse microbiome samples! (notes in pre-processing.nf)
  // TODO: future update: enable hostile db change between human default and mouse based on params.sample_environment
  // Then change the matcher to =~ /human|mouse/ to pick up both!

Rhea

• Add module for minigraph and pass in alignment file along with flye output to rhea to make the nextflow process more efficient and avoid flye being called as an executable within rhea
• Here's how you would call rhea (tested this one in repo when the code errored out after the minigraph part)

#!/bin/bash
rhea rhea_results/t0.gaf rhea_results/t1.gaf \
    --input-graph rhea_results/metaflye/assembly_graph.gfa \
    --bp-table rhea_results/bp_counts.tsv \
    --output-dir rhea_results/post_test \
    --threads 50 \