Migrate BUSCO to nf-core module #730

dialvarezs · 2024-12-12T05:14:23Z

This PR migrates from the local BUSCO module to the nf-core one.

Main changes

Updates to latest BUSCO version (v5.8.2)
Uses csvtk/concat to merge BUSCO and UNTAR to prepare database instead of the custom modules
BUSCO now runs in batch mode, so it runs one process per sample instead of one per bin
In the current version, BUSCO only generates a single file, not one per domain and one specifiic, so the GTDBTk part that uses QC metrics to filter bins got simplified.
Removed the logic related to collect "failed bins", they were not being used.
Handle --save_busco_db with publishDir directly.
Replace COMBINE_TSV local module by nf-core csvtk/concat

Breaking changes

--busco_clean is not supported in this new setup, so the param should be deprecated

PR checklist

Closes #484.

nf-core-bot · 2024-12-12T13:52:47Z

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.1.0.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

dialvarezs · 2024-12-12T17:01:45Z

@jfy133 It's green now 🎉

Some questions:

How should I deprecate a param (busco_clean)?
About --save_busco_db, the current module doesn't have the db downloads as output, so I'm adding it here: Add busco downloads as output modules#7210. I will try with publishDir. I'll try using publishDir. Previously, the issue was that multiple BUSCO processes were publishing to the same directory, but now BUSCO runs per sample rather than per bin, which should result in far fewer simultaneous publish operations.

EDIT:

Regarding second point, I managed to get a pretty simple and elegant solution to store all the downloaded lineages across all the samples, basically puting each of the lineage directories as ouputs instead of the whole busco_downloads directory. So, the custom module for that is no longer necessary.

The PR should be ready merging nf-core/modules#7210.

prototaxites · 2024-12-13T13:22:59Z

How should I deprecate a param (busco_clean)?

Is there scope to add this as an optional input to the BUSCO module itself? For some context, the problem I encountered with BUSCO was that the temporary working files take up a lot of storage space on disk, so running BUSCO on (in my case) 1000s of bins meant that my available scratch quota was quickly filling up and leading to mag runs that couldn't finish. So it would be good to try and keep this as an option if possible as I imagine this is a plausible issue for other mag users also!

dialvarezs · 2024-12-13T13:42:45Z

Hi @prototaxites,
I don’t think there should be any issue with porting the "clean" part of the script to the nf-core module. I’ll go ahead and open a PR for it.
By the way, the temporary storage usage should be significantly reduced now, as it only runs one BUSCO instance per sample.

prototaxites · 2024-12-13T14:23:24Z

By the way, the temporary storage usage should be significantly reduced now, as it only runs one BUSCO instance per sample.

If that's the case, maybe we don't need it?

dialvarezs · 2024-12-13T14:57:40Z

The tests I ran for this PR look like this:

auto

auto_prok

How do these numbers look to you?
For 20 samples, I estimate we could trim ~150 GB in auto (and about half of that in auto_prok).

dialvarezs · 2025-01-02T07:30:54Z

I made a last minor addition to this PR, I replaced the module used to combine bin qc summaries by a nf-core one (csvtk/concat).

conf/test.config

jfy133

Thank you for your patience @dialvarezs !

Some initial comments, however when comparing -profile test on your branch and current dev, I get the following results:

(nf-core) james@bionb103:~/git/nf-core/mag/testing (dev)$ cat results_dev/GenomeBinning/QC/busco_summary.tsv 
GenomeBin       Specific lineage dataset        %Complete (specific)    %Complete and single-copy (specific)    %Complete and duplicated (specific)     %Fragmented (specific)  %Missing (specific)        Total number (specific)
MEGAHIT-MetaBAT2-test_minigut_sample2.unbinned.2.fa     bacteria_odb10  0.0     0.0     0.0     0.0     100.0   124
SPAdes-MaxBin2-test_minigut_sample2.noclass.1.fa        bacteria_odb10  0.0     0.0     0.0     0.0     100.0   124
SPAdes-MetaBAT2-test_minigut.1.fa       bacteria_odb10  13.7    13.7    0.0     2.4     83.9    124
MEGAHIT-MetaBAT2-test_minigut.unbinned.2.fa     bacteria_odb10  0.0     0.0     0.0     0.8     99.2    124
MEGAHIT-MaxBin2-test_minigut.001.fa     bacteria_odb10  19.4    19.4    0.0     0.8     79.8    124
SPAdes-MetaBAT2-test_minigut.unbinned.2.fa      bacteria_odb10  0.0     0.0     0.0     0.0     100.0   124
MEGAHIT-MetaBAT2-test_minigut_sample2.unbinned.1.fa     bacteria_odb10  0.0     0.0     0.0     0.0     100.0   124
MEGAHIT-MetaBAT2-test_minigut.1.fa      bacteria_odb10  12.1    12.1    0.0     3.2     84.7    124
SPAdes-MetaBAT2-test_minigut_sample2.unbinned.1.fa      bacteria_odb10  0.0     0.0     0.0     0.0     100.0   124
SPAdes-MetaBAT2-test_minigut.2.fa       bacteria_odb10  20.2    20.2    0.0     1.6     78.2    124
SPAdes-MaxBin2-test_minigut_sample2.noclass.2.fa        bacteria_odb10  0.0     0.0     0.0     0.0     100.0   124
MEGAHIT-MetaBAT2-test_minigut.2.fa      bacteria_odb10  19.4    19.4    0.0     0.0     80.6    124
MEGAHIT-MetaBAT2-test_minigut.unbinned.1.fa     bacteria_odb10  0.0     0.0     0.0     0.0     100.0   124
MEGAHIT-MaxBin2-test_minigut.002.fa     bacteria_odb10  13.7    13.7    0.0     4.8     81.5    124
SPAdes-MetaBAT2-test_minigut_sample2.unbinned.2.fa      bacteria_odb10  0.0     0.0     0.0     0.0     100.0   124
SPAdes-MetaBAT2-test_minigut.unbinned.1.fa      bacteria_odb10  0.0     0.0     0.0     0.8     99.2    124
SPAdes-MaxBin2-test_minigut.001.fa      bacteria_odb10  20.2    20.2    0.0     1.6     78.2    124
SPAdes-MaxBin2-test_minigut_sample2.001.fa      bacteria_odb10  2.4     2.4     0.0     4.8     92.8    124
SPAdes-MaxBin2-test_minigut.002.fa      bacteria_odb10  13.7    13.7    0.0     3.2     83.1    124
(nf-core) james@bionb103:~/git/nf-core/mag/testing (dev)$ cat results_dialvarezs-busco/GenomeBinning/QC/BUSCO/test_minigut/test_minigut-bacteria_odb12-busco.batch_summary.txt 
Input_file      Dataset Complete        Single  Duplicated      Fragmented      Missing n_markers       Scaffold N50    Contigs N50     Percent gaps    Number of scaffolds
SPAdes-MetaBAT2-test_minigut.unbinned.2.fa      Run failed; check logs
SPAdes-MetaBAT2-test_minigut.unbinned.1.fa      Run failed; check logs

i.e, the runs on your BUSCO implementation are failing for some reason

We need to investigate if this is valid output or not given the new version of BUSCO and the database.

It would also be helpful if @skrakau (if she has time) could look at this to make sure we aren't removing anything important - I note that the custom busco_run.sh script of Sabrina's original module has A LOT more LoC than the new module has... (which maybe due to the new version of BUSCO) but I have 0 experience with BUSCO myself.

jfy133 · 2025-01-19T15:50:34Z

conf/modules.config

+                path: { "${params.outdir}/GenomeBinning/QC/BUSCO" },
+                mode: params.publish_dir_mode,
+                overwrite: false,
+                pattern: "busco_downloads/lineages/*",


Does this result in the same path structure in the --outdir as the old BUSCO_SVE_DOWNLOAD module (if it is, all good - if not might need to update output.md)

jfy133 · 2025-01-19T15:51:25Z

CHANGELOG.md

@@ -7,13 +7,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

 ### `Added`

+- [#730](https://github.com/nf-core/mag/pull/730) - Migrated from local BUSCO module to nf-core one (added by @dialvarezs)
+
 ### `Changed`


I woul mention the default database change too

jfy133 · 2025-01-19T15:52:26Z

modules/nf-core/checkm2/predict/main.nf

@@ -13,7 +13,7 @@ process CHECKM2_PREDICT {

    output:
    tuple val(meta), path("${prefix}")                   , emit: checkm2_output
-    tuple val(meta), path("${prefix}/quality_report.tsv"), emit: checkm2_tsv
+    tuple val(meta), path("${prefix}_checkm2_report.tsv"), emit: checkm2_tsv


This needs to be reflected in the CHANGELOG and the OUTPUT.md

jfy133 · 2025-01-19T15:54:09Z

nextflow_schema.json

@@ -768,11 +768,6 @@
                    "description": "Save the used BUSCO lineage datasets provided via `--busco_db`.",
                    "help_text": "Useful to allow reproducibility, as BUSCO datasets are frequently updated and old versions do not always remain accessible."
                },
-                "busco_clean": {


hrm... my instinct here is not to remove this but just add that functionality to the nf-core module.

I've been doing this in some of the metagenomic database modules, where I have an input channel (boolean val() called 'keep intermediates') which if given don't do anything, but if remains false injects an rm command into the script block.

jfy133 · 2025-01-19T15:55:14Z

subworkflows/local/bin_qc.nf

-                BUSCO_DB_PREPARATION(ch_busco_db)
-                ch_db_for_busco = BUSCO_DB_PREPARATION.out.db.map { meta, db ->
-                    [[id: meta, lineage: 'Y'], db]
+                BUSCO_UNTAR([[id: 'busco_db'], ch_busco_db])


Suggested change

BUSCO_UNTAR([[id: 'busco_db'], ch_busco_db])

BUSCO_UNTAR(ch_busco_db.map{db -> [[id: 'busco_db'], db]})

jfy133 · 2025-01-19T15:57:50Z

subworkflows/local/bin_qc.nf


+        qc_summaries = BUSCO_BUSCO.out.batch_summary


Are these ``batch_summaryfiles different from the old BUSCO output files? If so we should check if the descriptions inoutput.md` need to be updated

jfy133 · 2025-01-19T16:24:44Z

subworkflows/local/gtdbtk.nf

-            .map { row ->
-                        def completeness  = -1
-                        def contamination = -1
-                        def missing, duplicated


This deletion means that the pipeline is no longer backwards compatible with older versions of the database, correct? This is important to list in te changelog

jfy133 · 2025-01-19T16:33:45Z

reason for failures I think:

********************************************************************************
Running BUSCO on file: /home/james/git/nf-core/mag/testing/work/24/90cfdc2ff2e2e1083f6c1f6e74b3ff/input_seqs/SPAdes-MetaBAT2-test_minigut.unbinned.2.fa
********************************************************************************


2025-01-19 15:55:58 INFO:busco.BuscoRunner      Input file is /home/james/git/nf-core/mag/testing/work/24/90cfdc2ff2e2e1083f6c1f6e74b3ff/input_seqs/SPAdes-MetaBAT2-test_minigut.unbinned.2.fa
2025-01-19 15:55:58 ERROR:busco.BuscoRunner     Unable to run BUSCO in offline mode. Dataset /home/james/git/nf-core/mag/testing/work/24/90cfdc2ff2e2e1083f6c1f6e74b3ff/busco_db/lineages/bacteria_odb12 does not exist.
2025-01-19 15:55:58 INFO:busco.BuscoRunner

********************************************************************************
Running BUSCO on file: /home/james/git/nf-core/mag/testing/work/24/90cfdc2ff2e2e1083f6c1f6e74b3ff/input_seqs/SPAdes-MetaBAT2-test_minigut.unbinned.1.fa
********************************************************************************


2025-01-19 15:55:58 INFO:busco.BuscoRunner      Input file is /home/james/git/nf-core/mag/testing/work/24/90cfdc2ff2e2e1083f6c1f6e74b3ff/input_seqs/SPAdes-MetaBAT2-test_minigut.unbinned.1.fa
2025-01-19 15:55:58 ERROR:busco.BuscoRunner     Unable to run BUSCO in offline mode. Dataset /home/james/git/nf-core/mag/testing/work/24/90cfdc2ff2e2e1083f6c1f6e74b3ff/busco_db/lineages/bacteria_odb12 does not exist.
2025-01-19 15:55:58 DEBUG:urllib3.connectionpool        Starting new HTTPS connection (1): busco-data.ezlab.org:443
2025-01-19 15:55:58 DEBUG:urllib3.connectionpool        Starting new HTTPS connection (1): busco-data.ezlab.org:443
2025-01-19 15:55:59 DEBUG:urllib3.connectionpool        https://busco-data.ezlab.org:443 "PUT /upload/rundata202501191555589915025.txt HTTP/11" 201 0
2025-01-19 15:55:59 DEBUG:urllib3.connectionpool        https://busco-data.ezlab.org:443 "PUT /upload/rundata202501191555589915025.txt HTTP/11" 201 0
2025-01-19 15:55:59 DEBUG:busco.run_BUSCO       File uploaded successfully.

Migrate to nf-core BUSCO module (WIP)

f85d7b0

dialvarezs changed the title ~~Migrate to nf-core BUSCO module (WIP)~~ Migrate BUSCO to nf-core module Dec 12, 2024

Several busco-related fixes

25b72f1

dialvarezs force-pushed the dev-busco branch from cab7dbd to 25b72f1 Compare December 12, 2024 06:10

dialvarezs added 3 commits December 12, 2024 10:54

More fixes

db88a3c

Fixes in combine_tables script

091a54a

Add more cpus to busco, and fix comparison in combine_tables

4a22358

Exclude failed BUSCO bins from GTDB-Tk, update changelog

bd22078

dialvarezs marked this pull request as ready for review December 12, 2024 19:55

dialvarezs and others added 2 commits December 12, 2024 18:15

Save busco downloaded db

9512e95

Remove 'BUSCO_SAVE_DOWNLOAD' from conf/modules

9cc1d74

dialvarezs added 6 commits December 15, 2024 17:56

Merge remote-tracking branch 'upstream/dev' into dev-busco

46d19b1

Merge cleanup and fixes

edf4192

Define busco_lineage

6f200ef

Updade nf-core busco module

9702a87

Remove unneded config

abc1760

Fix busco channels and namespace in summary script

a44eb56

dialvarezs force-pushed the dev-busco branch from e5255ed to a44eb56 Compare December 16, 2024 12:39

dialvarezs added 3 commits January 2, 2025 04:10

Merge remote-tracking branch 'origin/dev' into dev-busco

37dbeed

Merge remote-tracking branch 'upstream/dev' into dev-busco

53dc7d3

Replace local combine_tsv module by nf-core csvtk/concat

eed82db

dialvarezs added 2 commits January 2, 2025 04:35

Fix changelog

299ff59

Update nf-core checkm2/databasedownload

89c7455

dialvarezs and others added 8 commits January 2, 2025 04:42

Fix tiara

c1d0552

Linting fixes

0b81462

Add ids to channel used by CONCAT_BINQC_TSV

f93e007

Update csvtk/concat module

07905a8

Fix gtdb usage of qc summary

1671aca

Update tests to bacteria odb12

3d4ba8b

Last fixes to get gtdb and bin summary working

d60ae40

Fix comment

135e703

dialvarezs added the ready to review label Jan 8, 2025

prototaxites reviewed Jan 8, 2025

View reviewed changes

conf/test.config Outdated Show resolved Hide resolved

dialvarezs added 2 commits January 8, 2025 11:14

busco_clean deprecation cleanup

44306da

Minor busco tweaks

f71249b

jfy133 requested changes Jan 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate BUSCO to nf-core module #730

Migrate BUSCO to nf-core module #730

dialvarezs commented Dec 12, 2024 •

edited

Loading

nf-core-bot commented Dec 12, 2024 •

edited

Loading

dialvarezs commented Dec 12, 2024 •

edited

Loading

prototaxites commented Dec 13, 2024

dialvarezs commented Dec 13, 2024

prototaxites commented Dec 13, 2024

dialvarezs commented Dec 13, 2024

dialvarezs commented Jan 2, 2025 •

edited

Loading

jfy133 left a comment

jfy133 Jan 19, 2025

jfy133 Jan 19, 2025

jfy133 Jan 19, 2025

jfy133 Jan 19, 2025

jfy133 Jan 19, 2025

jfy133 Jan 19, 2025

jfy133 Jan 19, 2025

jfy133 commented Jan 19, 2025

	BUSCO_UNTAR([[id: 'busco_db'], ch_busco_db])
	BUSCO_UNTAR(ch_busco_db.map{db -> [[id: 'busco_db'], db]})

Migrate BUSCO to nf-core module #730

Are you sure you want to change the base?

Migrate BUSCO to nf-core module #730

Conversation

dialvarezs commented Dec 12, 2024 • edited Loading

Main changes

Breaking changes

PR checklist

nf-core-bot commented Dec 12, 2024 • edited Loading

dialvarezs commented Dec 12, 2024 • edited Loading

prototaxites commented Dec 13, 2024

dialvarezs commented Dec 13, 2024

prototaxites commented Dec 13, 2024

dialvarezs commented Dec 13, 2024

dialvarezs commented Jan 2, 2025 • edited Loading

jfy133 left a comment

Choose a reason for hiding this comment

jfy133 Jan 19, 2025

Choose a reason for hiding this comment

jfy133 Jan 19, 2025

Choose a reason for hiding this comment

jfy133 Jan 19, 2025

Choose a reason for hiding this comment

jfy133 Jan 19, 2025

Choose a reason for hiding this comment

jfy133 Jan 19, 2025

Choose a reason for hiding this comment

jfy133 Jan 19, 2025

Choose a reason for hiding this comment

jfy133 Jan 19, 2025

Choose a reason for hiding this comment

jfy133 commented Jan 19, 2025

dialvarezs commented Dec 12, 2024 •

edited

Loading

nf-core-bot commented Dec 12, 2024 •

edited

Loading

dialvarezs commented Dec 12, 2024 •

edited

Loading

dialvarezs commented Jan 2, 2025 •

edited

Loading