Longread only functionality #718

muabnezor · 2024-11-28T12:28:45Z

This PR adds long-read only functionality to mag.

closes #662, #659, #275

PR checklist

…esheets

… 97 to 90 when dealing with longreads

…and reads

github-actions · 2024-11-28T12:28:58Z

Warning

Newer version of the nf-core template is available.

Your pipeline is using an old version of the nf-core template: 3.1.0.
Please update your pipeline to the latest version.

For more documentation on how to update your pipeline, please see the nf-core documentation and Synchronisation documentation.

d4straub

I only had a short look, sorry, but I just wanted to express my gratitude that you tackle long read assembly!

nextflow.config

conf/modules.config

muabnezor · 2024-11-28T14:18:45Z

I only had a short look, sorry, but I just wanted to express my gratitude that you tackle long read assembly!

My pleasure hehe. Still WIP. I have to tidy up the code and run some validation on real data, but we're getting there!

… in config

…rtread assemblers, when assembly input is given

…empty if no files are given

…d also return remainder in case the ch_short_reads channel is empty. Change config for FILTLONG to only use '--trim' option if shortreads are passed

…orkflow

muabnezor · 2024-12-12T10:19:58Z

So far in this PR I have

changed the validation schema to allow for long read only input
Added a host removal track for long reads using minimap2 as aligner
Added Flye and metaMDBG as longread assemblers
refactoring of code so that all the assembly code is moved to a subworkflow

Downstream from the assembly, except for the binning preparation, the long read and short read assemblies are treated the same.

muabnezor · 2024-12-12T14:23:10Z

Me and some colleagues are running some tests on ont data, and I think this works as expected. We will continue testing and play around with the parameters, but if anyone wants to have a look through the code and suggest improvements, that would be awesome!

…nnel

jfy133

OK I'm not seeing anything obvious! But I would still like to run a test (can't do that today).

Once commetns (mostly questions) addressed, I will start the manual tests :)

jfy133 · 2025-01-20T11:34:15Z

CHANGELOG.md

@@ -24,10 +24,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - [#707](https://github.com/nf-core/mag/pull/707) - Make Bin QC a subworkflow (added by @dialvarezs)
 - [#707](https://github.com/nf-core/mag/pull/707) - Added CheckM2 as an alternative bin completeness and QC tool (added by @dialvarezs)
 - [#708](https://github.com/nf-core/mag/pull/708) - Added `--exclude_unbins_from_postbinning` parameter to exclude unbinned contigs from post-binning processes, speeding up Prokka in some cases (added by @dialvarezs)
+- [#718](https://github.com/nf-core/mag/pull/718) - Added metaMDBG and Flye as longread assemblers (added by @muabnezor)


Suggested change

- [#718](https://github.com/nf-core/mag/pull/718) - Added metaMDBG and Flye as longread assemblers (added by @muabnezor)

- [#718](https://github.com/nf-core/mag/pull/718) - Added metaMDBG and (meta)Flye as long read assemblers (suggested by ljmesi [and many others] added by @muabnezor)

jfy133 · 2025-01-20T11:35:31Z

CHANGELOG.md

 - [#732](https://github.com/nf-core/mag/pull/732) - Added support for Prokka's compliance mode with `--prokka_with_compliance --prokka_compliance_centre <xyz>` (reported by @audy and @Thomieh73, added by @jfy133)

 ### `Changed`

+- [#718](https://github.com/nf-core/mag/pull/718) - Longread only input (added by @muabnezor)


Suggested change

- [#718](https://github.com/nf-core/mag/pull/718) - Longread only input (added by @muabnezor)

- [#718](https://github.com/nf-core/mag/pull/718) - Longread only input is now an option (added by @muabnezor)

jfy133 · 2025-01-20T11:35:42Z

CHANGELOG.md

@@ -37,16 +40,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - [#716](https://github.com/nf-core/mag/pull/692) - Make short read processing a subworkflow (added by @muabnezor)
 - [#708](https://github.com/nf-core/mag/pull/708) - Fixed channel passed as GUNC input (added by @dialvarezs)
 - [#729](https://github.com/nf-core/mag/pull/729) - Fixed misspecified multi-FASTQ input for single-end data in MEGAHIT (reported by John Richards, fix by @jfy133)
+- [#718](https://github.com/nf-core/mag/pull/718) - refactoring assembly into subworkflow (added by @muabnezor)


Suggested change

- [#718](https://github.com/nf-core/mag/pull/718) - refactoring assembly into subworkflow (added by @muabnezor)

- [#718](https://github.com/nf-core/mag/pull/718) - refactoring assembly steps into subworkflow (added by @muabnezor)

jfy133 · 2025-01-20T11:37:34Z

CITATIONS.md

@@ -66,6 +66,10 @@

 - [Filtlong](https://github.com/rrwick/Filtlong)

+- [Flye](https://www.nature.com/articles/s41592-020-00971-x)
+
+  > Kolmogorov, M., Bickhart, D.M., Behsaz, B. et al. metaFlye: scalable long-read metagenome assembly using repeat graphs. Nat Methods 17, 1103–1110 (2020). https://doi.org/10.1038/s41592-020-00971-x


Suggested change

> Kolmogorov, M., Bickhart, D.M., Behsaz, B. et al. metaFlye: scalable long-read metagenome assembly using repeat graphs. Nat Methods 17, 1103–1110 (2020). https://doi.org/10.1038/s41592-020-00971-x

> Kolmogorov, M., Bickhart, D.M., Behsaz, B. et al. metaFlye: scalable long-read metagenome assembly using repeat graphs. Nat Methods 17, 1103–1110 (2020). doi: 10.1038/s41592-020-00971-x

jfy133 · 2025-01-20T11:38:13Z

CITATIONS.md

@@ -106,6 +110,14 @@

  > Levy Karin, E., Mirdita, M. & Söding, J. MetaEuk—sensitive, high-throughput gene discovery, and annotation for large-scale eukaryotic metagenomics. Microbiome 8, 48 (2020). 10.1186/s40168-020-00808-x

+- [metaMDBG](https://www.nature.com/articles/s41587-023-01983-6)
+
+  > Benoit, G., Raguideau, S., James, R. et al. High-quality metagenome assembly from long accurate reads with metaMDBG. Nat Biotechnol 42, 1378–1383 (2024). https://doi.org/10.1038/s41587-023-01983-6


Suggested change

> Benoit, G., Raguideau, S., James, R. et al. High-quality metagenome assembly from long accurate reads with metaMDBG. Nat Biotechnol 42, 1378–1383 (2024). https://doi.org/10.1038/s41587-023-01983-6

> Benoit, G., Raguideau, S., James, R. et al. High-quality metagenome assembly from long accurate reads with metaMDBG. Nat Biotechnol 42, 1378–1383 (2024). doi:10.1038/s41587-023-01983-6

jfy133 · 2025-01-20T11:58:58Z

subworkflows/local/binning_preparation.nf

-    BOWTIE2_ASSEMBLY_BUILD ( assemblies )
+    ch_versions       = Channel.empty()
+    ch_multiqc_files  = Channel.empty()
+        // multiple symlinks to the same assembly -> use first of sorted list


Suggested change

// multiple symlinks to the same assembly -> use first of sorted list

// multiple symlinks to the same assembly -> use first of sorted list

What is this comment referring to, it sounds a bit scary?

jfy133 · 2025-01-20T12:08:02Z

subworkflows/local/longread_binning_preparation.nf

+    ch_minimap2_input_idx = ch_minimap2_input
+        .map { meta_idx, index, meta, reads -> [ meta_idx, index ] }
+
+    MINIMAP2_ASSEMBLY_ALIGN ( ch_minimap2_input_reads, ch_minimap2_input_idx, true, 'bai', false, false )


What are the true/false/falses? something should be parameterasable by the user?

jfy133 · 2025-01-20T12:10:33Z

subworkflows/local/shortread_assembly.nf

+    ch_short_reads_spades
+
+    main:
+


Suggested change

jfy133 · 2025-01-20T12:13:45Z

subworkflows/local/utils_nfcore_mag_pipeline/main.nf

@@ -244,12 +247,6 @@ def validateInputParameters(hybrid) {
    if (params.host_fasta && params.host_genome) {
        error('[nf-core/mag] ERROR: Both host fasta reference and iGenomes genome are specified to remove host contamination! Invalid combination, please specify either --host_fasta or --host_genome.')
    }
-    if (hybrid && (params.host_fasta || params.host_genome)) {
-        log.warn('[nf-core/mag]: Host read removal is only applied to short reads. Long reads might be filtered indirectly by Filtlong, which is set to use read qualities estimated based on k-mer matches to the short, already filtered reads.')
-        if (params.longreads_length_weight > 1) {


Can you verify this warning is no onger necessary?

(the one before is fine obviously)

jfy133 · 2025-01-20T12:15:14Z

workflows/mag.nf

@@ -112,6 +107,8 @@ workflow MAG {

    if (!params.keep_phix) {
        ch_phix_db_file = Channel.value(file("${params.phix_reference}"))
+    } else {


Is this the fix for the phix bug? If so would appreciate to split this out into a separate PR:

muabnezor · 2025-01-20T14:07:05Z

thank you @jfy133. I'm away this week, but I'll try to find time to go through your comments asap.

muabnezor added 12 commits November 25, 2024 11:34

Add longread only test config

5ffb327

Change samplesheet validation schema to allow for longread-only sampl…

695da2b

…esheets

Check if both short, and long reads are given

f67b6b3

fix validation schema for samplesheet

bb78aab

Merge remote-tracking branch 'upstream/dev' into longread_only

992b01e

Add separate wubworkflow for longread host removal

bfd404c

Add longread meta-assemblers metaflye and metamdbg

387284d

Add longread assemble config

6948316

Prepare binning for longread assemblies

762f12b

Fix config and how long reads are prepared for binning

c7a5cdf

change jgi_summarize_bam_contig_depths --percentidentity default from…

18ae00d

… 97 to 90 when dealing with longreads

Fix longread binning preparation from all combination of assemblies, …

8a22939

…and reads

muabnezor added the WIP Work in progress label Nov 28, 2024

muabnezor requested a review from jfy133 November 28, 2024 12:28

muabnezor changed the base branch from master to dev November 28, 2024 12:29

d4straub reviewed Nov 28, 2024

View reviewed changes

nextflow.config Show resolved Hide resolved

prototaxites reviewed Nov 28, 2024

View reviewed changes

conf/modules.config Outdated Show resolved Hide resolved

muabnezor added 11 commits November 29, 2024 07:16

format

038a87b

Fix longread hostremoval, and fix --longread_percentidentiy parameter…

3ff8de2

… in config

Add test_longread to profiles, and ci testing

e277290

Fix validation schema, and fix assembly channels for longread and sho…

75a36f2

…rtread assemblers, when assembly input is given

Change logic in samplesheet validation, ch_raw_short_reads should be …

67d44d0

…empty if no files are given

fix custom samtools view module

ca4d101

Merge branch 'dev' into longread_only

cfd43e7

Make sure filtlong works without short reads. the join operator shoul…

6765ab1

…d also return remainder in case the ch_short_reads channel is empty. Change config for FILTLONG to only use '--trim' option if shortreads are passed

Make sure FILTLONG is not run when there are no long reads

e73975c

Fix grouping logic for channels in longreads_binning_preparation subw…

686fafe

…orkflow

make assembly into subworkflow

a0279fe

muabnezor added 2 commits December 12, 2024 10:24

Fix long read assembly input

28f72af

Update docs

1c4b364

Update citations for tools

34067db

muabnezor removed the WIP Work in progress label Dec 12, 2024

Fix versions channel in subworkflows

26a0fe7

muabnezor added the ready to review label Dec 13, 2024

muabnezor added 2 commits January 7, 2025 08:49

Fix bug when running with --keep_phix, make ch_phix_db_file empty Cha…

d9564da

…nnel

Merge branch 'dev' into longread_only

9087f2f

muabnezor mentioned this pull request Jan 7, 2025

When running the pipeline with --skip_clipping and --keep_phix: ERROR ~ No such variable: ch_phix_db_file #740

Open

muabnezor added 6 commits January 7, 2025 10:59

fix modules.config

ba0f831

Fix linting

d46aa6d

Use nf-core official module for samtools fastq

f90e5f5

Change modules config

85c2e82

change hybrid logic

7398ba3

fix samplesheet validation

ae1953d

jfy133 linked an issue Jan 20, 2025 that may be closed by this pull request

Add separate Nanopore input option #275

Open

jfy133 reviewed Jan 20, 2025

View reviewed changes

jfy133 mentioned this pull request Jan 21, 2025

Use channel empty when no phix reference supplied because skipping #749

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Longread only functionality #718

Longread only functionality #718

muabnezor commented Nov 28, 2024 •

edited

Loading

github-actions bot commented Nov 28, 2024 •

edited by nf-core-bot

Loading

d4straub left a comment

muabnezor commented Nov 28, 2024

muabnezor commented Dec 12, 2024

muabnezor commented Dec 12, 2024

jfy133 left a comment

jfy133 Jan 20, 2025

jfy133 Jan 20, 2025

jfy133 Jan 20, 2025

jfy133 Jan 20, 2025

jfy133 Jan 20, 2025

jfy133 Jan 20, 2025

jfy133 Jan 20, 2025

jfy133 Jan 20, 2025

jfy133 Jan 20, 2025

jfy133 Jan 20, 2025

jfy133 Jan 20, 2025

muabnezor commented Jan 20, 2025

	- [#718](https://github.com/nf-core/mag/pull/718) - Added metaMDBG and Flye as longread assemblers (added by @muabnezor)
	- [#718](https://github.com/nf-core/mag/pull/718) - Added metaMDBG and (meta)Flye as long read assemblers (suggested by ljmesi [and many others] added by @muabnezor)

	> Kolmogorov, M., Bickhart, D.M., Behsaz, B. et al. metaFlye: scalable long-read metagenome assembly using repeat graphs. Nat Methods 17, 1103–1110 (2020). https://doi.org/10.1038/s41592-020-00971-x
	> Kolmogorov, M., Bickhart, D.M., Behsaz, B. et al. metaFlye: scalable long-read metagenome assembly using repeat graphs. Nat Methods 17, 1103–1110 (2020). doi: 10.1038/s41592-020-00971-x

	> Benoit, G., Raguideau, S., James, R. et al. High-quality metagenome assembly from long accurate reads with metaMDBG. Nat Biotechnol 42, 1378–1383 (2024). https://doi.org/10.1038/s41587-023-01983-6
	> Benoit, G., Raguideau, S., James, R. et al. High-quality metagenome assembly from long accurate reads with metaMDBG. Nat Biotechnol 42, 1378–1383 (2024). doi:10.1038/s41587-023-01983-6

	// multiple symlinks to the same assembly -> use first of sorted list
	// multiple symlinks to the same assembly -> use first of sorted list

Longread only functionality #718

Are you sure you want to change the base?

Longread only functionality #718

Conversation

muabnezor commented Nov 28, 2024 • edited Loading

PR checklist

github-actions bot commented Nov 28, 2024 • edited by nf-core-bot Loading

d4straub left a comment

Choose a reason for hiding this comment

muabnezor commented Nov 28, 2024

muabnezor commented Dec 12, 2024

muabnezor commented Dec 12, 2024

jfy133 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

muabnezor commented Jan 20, 2025

muabnezor commented Nov 28, 2024 •

edited

Loading

github-actions bot commented Nov 28, 2024 •

edited by nf-core-bot

Loading