Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrap-up version 3.2.0 #73

Merged
merged 169 commits into from
Mar 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
169 commits
Select commit Hold shift + click to select a range
18f8895
add illumina docker test (#44)
fmalmeida May 10, 2023
07ca6ac
Update github actions for testing profiles (#46)
fmalmeida May 10, 2023
8ea57fc
30 change to unicycler v050 (#45)
fmalmeida May 18, 2023
a000d3f
31 new directory called final output (#48)
fmalmeida May 18, 2023
d4b8977
22 add polypolish tool (#49)
fmalmeida May 19, 2023
ee363fe
include citation information (#51)
fmalmeida Oct 7, 2023
623993a
Update README.md
fmalmeida Oct 7, 2023
60d8781
raven does not have --weaken parameter anymore
fmalmeida Oct 29, 2023
0963cda
allow users to change -k and -w
fmalmeida Oct 29, 2023
e680501
added high_quality_longreads parameter
fmalmeida Oct 29, 2023
f54d61a
add defaults for high quality reads
fmalmeida Oct 29, 2023
dd5cd56
added high-quality definition for assemblers that allow it
fmalmeida Oct 29, 2023
4e2a428
update medaka version
fmalmeida Oct 30, 2023
0ee44eb
update docker image version
fmalmeida Oct 30, 2023
b4cc5ae
update max_memory default
fmalmeida Oct 31, 2023
a4aeae3
fixed if else statemens
fmalmeida Nov 4, 2023
a89362f
update docker image
fmalmeida Nov 4, 2023
19b743a
fix flye module
fmalmeida Nov 4, 2023
4dcdb90
Add Citation
fmalmeida Nov 10, 2023
2ad842b
Add Citation
fmalmeida Nov 10, 2023
3ffbf87
Update .readthedocs.yml
fmalmeida Nov 11, 2023
747d41d
Update .readthedocs.yml
fmalmeida Nov 11, 2023
df86d76
Update requirements.txt
fmalmeida Nov 11, 2023
f2c1187
Merge branch 'master' of https://github.com/fmalmeida/MpGAP
fmalmeida Nov 20, 2023
9f1b1f8
Merge branch 'master' of https://github.com/fmalmeida/MpGAP
fmalmeida Nov 20, 2023
154c05f
Saving code that was wrongly merged to master instead of dev (#56)
fmalmeida Dec 24, 2023
3951a41
fix input definition
fmalmeida Dec 28, 2023
786c1f1
update readthedocs yml
fmalmeida Jan 25, 2024
79d9e22
add mkdocs yml
fmalmeida Jan 25, 2024
f65f7cf
add examples from ngs-preprocess pipeline
fmalmeida Jan 25, 2024
cdf4fb9
fix image link
fmalmeida Jan 25, 2024
9cf778d
updated index page
fmalmeida Jan 25, 2024
93f07de
update colors
fmalmeida Jan 25, 2024
90b142d
update installation page
fmalmeida Jan 25, 2024
71373b7
update quickstart page
fmalmeida Jan 25, 2024
dbdf2a4
fix indentations
fmalmeida Jan 25, 2024
1e90b44
fix block indentation
fmalmeida Jan 25, 2024
404a21f
update configs page
fmalmeida Jan 25, 2024
abbece3
reorganize docs assets
fmalmeida Jan 25, 2024
ff70bf3
start converting manual page
fmalmeida Jan 25, 2024
adf937c
update manual page
fmalmeida Jan 25, 2024
0df200c
increase size of table cell
fmalmeida Jan 25, 2024
03e0632
remove unused examples.md
fmalmeida Jan 25, 2024
3a43be3
update samplesheet page
fmalmeida Jan 25, 2024
aa3b36e
fix code block
fmalmeida Jan 27, 2024
4b2e77d
update non-bacterial dataset example
fmalmeida Jan 27, 2024
5925b65
update outputs.md page
fmalmeida Jan 27, 2024
f55b936
trying out including example reports
fmalmeida Jan 27, 2024
175b1c8
first update on page (needs data rerun)
fmalmeida Jan 27, 2024
26458ca
update non bacteria dataset command
fmalmeida Jan 27, 2024
f143e3f
fix indentation and typos
fmalmeida Feb 1, 2024
93f5995
Merge branch 'dev' of https://github.com/fmalmeida/MpGAP into 58-add-…
fmalmeida Feb 12, 2024
e07db03
Update changelog
fmalmeida Feb 12, 2024
898a59b
update docs output overview and generated files
fmalmeida Feb 13, 2024
9748f89
update page
fmalmeida Feb 13, 2024
d1c5ed2
add mention of polypolish in documentation
fmalmeida Feb 13, 2024
a76a57d
also run hybrid tests on dev PRs
fmalmeida Feb 13, 2024
9306245
hybid test uses pacbio
fmalmeida Feb 13, 2024
b72c040
add missing parameter to schema
fmalmeida Feb 13, 2024
f304c2a
fix IDs of hybrid workflow
fmalmeida Feb 13, 2024
ef01232
Merge pull request #62 from fmalmeida/58-add-example-of-non-bacterial…
fmalmeida Feb 13, 2024
c667cb1
Merge branch 'dev' of https://github.com/fmalmeida/MpGAP into 52-inco…
fmalmeida Feb 13, 2024
19bc5a9
fix error message
fmalmeida Feb 13, 2024
0b8e130
delete empty line
fmalmeida Feb 13, 2024
098dd9a
adjust tuple when hybrid strategy is both
fmalmeida Feb 13, 2024
c4ea390
skip haslr and singularity runs
fmalmeida Feb 14, 2024
e866b0e
Update test_pr_hybrid_docker.yml
fmalmeida Feb 14, 2024
546051e
add a checker to YAML samplesheet to prohibit use of corrected_longre…
fmalmeida Feb 15, 2024
8c23bdc
add def statements
fmalmeida Feb 15, 2024
0334038
add explanation of parameter in docs
fmalmeida Feb 16, 2024
5694080
Merge pull request #63 from fmalmeida/52-incomplete-pipeline-and-diff…
fmalmeida Feb 16, 2024
a8f2466
change multiqc resource label
fmalmeida Feb 16, 2024
4bd6242
add new start_asm resources parameter and update defaults
fmalmeida Feb 16, 2024
ad32a19
update default resources for modules without labels, modify assembly …
fmalmeida Feb 16, 2024
14ae77f
fix order of inclusion
fmalmeida Feb 16, 2024
eb0a71c
fix quast resources function
fmalmeida Feb 16, 2024
150d0f0
add explanation of parameter in documentation
fmalmeida Feb 16, 2024
0b9d484
add parameter in nextflow schema and update defaults
fmalmeida Feb 16, 2024
db878fb
update changelog
fmalmeida Feb 16, 2024
a9c4358
also wrap-up 1st attempt in check_max function to avoid requesting mo…
fmalmeida Feb 16, 2024
2ec8280
update docs/assets
fmalmeida Feb 16, 2024
82840bb
Merge pull request #64 from fmalmeida/61-add-a-simple-parameter-to-ha…
fmalmeida Feb 16, 2024
684b7c3
update docker to download latest busco datasets
fmalmeida Feb 16, 2024
99c63aa
add ignore on SAM/BAM files
fmalmeida Feb 16, 2024
22b4fb4
update shasum
fmalmeida Feb 16, 2024
2c55341
add busco instead of quast-busco
fmalmeida Feb 17, 2024
1b00a46
fix quast mem/cpus variable
fmalmeida Feb 17, 2024
d62c994
update param description
fmalmeida Feb 17, 2024
cb86547
update how busco params default is handled
fmalmeida Feb 17, 2024
d18826e
pre-download busco db
fmalmeida Feb 17, 2024
90cae14
update docker image
fmalmeida Feb 17, 2024
f007b41
define automatic busco lineage control
fmalmeida Feb 17, 2024
3ece903
add param to nextflow schema
fmalmeida Feb 17, 2024
898543f
add parameter in documentation
fmalmeida Feb 17, 2024
3b5fadb
add BUSCO in tools listing
fmalmeida Feb 17, 2024
1ec4eaf
fix mqc module
fmalmeida Feb 17, 2024
9f9d548
add new parameter for skipping pilon or polypolish
fmalmeida Feb 18, 2024
a12596a
add a checker to have at least one sreads polisher if running hb stra…
fmalmeida Feb 18, 2024
9de583a
quast values are not params
fmalmeida Feb 23, 2024
5afa9c1
added code to generate template samplesheet for bacannot
fmalmeida Feb 23, 2024
9b44cd0
fix BUSCO samplenames
fmalmeida Feb 23, 2024
5e65cbc
Update environment.yml
fmalmeida Feb 23, 2024
b2d6cd9
generate a versions.yml for all modules
fmalmeida Feb 23, 2024
96c176f
fix indentation
fmalmeida Feb 23, 2024
73ba7eb
include generation of versions yml
fmalmeida Feb 23, 2024
5a036f5
add lines to avoid saving versions.yml files
fmalmeida Feb 23, 2024
522f010
all modules use my image
fmalmeida Feb 23, 2024
a108dc2
Update test.config
fmalmeida Feb 23, 2024
de97097
fixed execution of versions file generation
fmalmeida Feb 24, 2024
dce365a
added a proper configuration for new MultiQC
fmalmeida Feb 24, 2024
af4b88a
Update manual.md
fmalmeida Feb 25, 2024
61d0e27
Update nextflow.config
fmalmeida Feb 25, 2024
9962ae7
Update nextflow.config
fmalmeida Feb 25, 2024
ec6f540
Update CHANGELOG.md
fmalmeida Feb 25, 2024
19400bc
Merge pull request #65 from fmalmeida/50-add-skip-parameter-for-sread…
fmalmeida Feb 25, 2024
98bb620
Merge branch 'dev' of https://github.com/fmalmeida/MpGAP into 66-add-…
fmalmeida Feb 26, 2024
bd35164
re-organize configuration
fmalmeida Feb 26, 2024
a9e4946
fix container definitions
fmalmeida Feb 26, 2024
3cd7bd2
all use the same container
fmalmeida Feb 26, 2024
35d4d5e
fix docker image
fmalmeida Feb 26, 2024
225b419
standardize quotation
fmalmeida Feb 26, 2024
4c7149f
update changelog
fmalmeida Feb 26, 2024
b5e523c
<add lines for incorporating hifiasm>
scintilla9 Feb 27, 2024
62e390f
Merge branch 'dev' of https://github.com/fmalmeida/MpGAP into add-hif…
fmalmeida Feb 27, 2024
44a88d0
update docker image
fmalmeida Feb 27, 2024
44220ac
fix module path
fmalmeida Feb 27, 2024
500a0e1
update output examples in documentation
fmalmeida Feb 27, 2024
4c681b5
add hifi testing case
fmalmeida Feb 27, 2024
70247fb
fix container for all modules
fmalmeida Feb 27, 2024
4d21f7a
skip hifiasm for non-hifi tests
fmalmeida Feb 27, 2024
a45c38d
add new testing workflow
fmalmeida Feb 27, 2024
da03d54
min for canu
fmalmeida Feb 27, 2024
142b4ac
add def statement
fmalmeida Feb 27, 2024
d27c7d6
Merge pull request #69 from fmalmeida/66-add-automatic-samplesheet-fo…
fmalmeida Feb 27, 2024
ed0719c
Merge branch 'dev' of https://github.com/fmalmeida/MpGAP into add-hif…
fmalmeida Feb 27, 2024
1e4ea6b
update changelog
fmalmeida Feb 27, 2024
ceaef66
github actions cannot tun hifi (not enough mem)
fmalmeida Feb 27, 2024
a48dfd3
add information of new tool in docs
fmalmeida Feb 28, 2024
11004e1
Merge pull request #70 from scintilla9/add-hifiasm-assembler
fmalmeida Feb 28, 2024
7b07f7e
add checker for haslr outputs
fmalmeida Feb 28, 2024
08667de
add checker for hifiasm outputs
fmalmeida Feb 28, 2024
711f462
manually build haslr
fmalmeida Feb 28, 2024
cd42745
fix haslr installation
fmalmeida Feb 28, 2024
bbea428
fixed docker image building order
fmalmeida Feb 29, 2024
6974405
update dockerfile and correct tool path
fmalmeida Feb 29, 2024
0fade0e
update custom error messages indentations
fmalmeida Feb 29, 2024
e454ef7
only execute hifiasm when reads are corrected or high quality
fmalmeida Feb 29, 2024
9a0c6fd
fix if else checker
fmalmeida Feb 29, 2024
1ba7d65
fix if else
fmalmeida Feb 29, 2024
545db8d
update module to have a single main script for any profile
fmalmeida Mar 2, 2024
f646cf6
update image download info
fmalmeida Mar 2, 2024
82c7e8c
update changelog
fmalmeida Mar 2, 2024
24fd51b
Merge branch 'master' of https://github.com/fmalmeida/MpGAP into dev
fmalmeida Mar 2, 2024
8a6ab88
fix quast resources parser
fmalmeida Mar 2, 2024
c31941b
fix indentation
fmalmeida Mar 2, 2024
4169227
include new parameter to manage quast&busco databases
fmalmeida Mar 3, 2024
e9c7d6a
fix function so objects have always the correct class
fmalmeida Mar 3, 2024
cbef6d8
add singularity runOptions to make it proper solve SIF images from do…
fmalmeida Mar 4, 2024
c7992c2
whitelist SINGULARITY_TMPDIR tmpdir
fmalmeida Mar 5, 2024
e23ec6d
Merge branch 'dev' of https://github.com/fmalmeida/MpGAP into dev
fmalmeida Mar 5, 2024
bf9934c
add TODO
fmalmeida Mar 5, 2024
9958f5e
use correct image
fmalmeida Mar 6, 2024
48ea776
fix permissions in medaka
fmalmeida Mar 6, 2024
2698a3a
use it locally to avoid problems with singularity
fmalmeida Mar 6, 2024
2cd3fdf
update docs assets aspergillus example results
fmalmeida Mar 6, 2024
696e7c2
update command used in example
fmalmeida Mar 6, 2024
bc95c65
is singularity git actions working?
fmalmeida Mar 6, 2024
38d86c1
update docs on singularity
fmalmeida Mar 6, 2024
df74632
Update CHANGELOG.md
fmalmeida Mar 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
37 changes: 37 additions & 0 deletions .github/workflows/test_pr_hybrid_docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Testing hybrid / docker from PR
on:
pull_request:
branches: [master, dev]
types: [opened, synchronize, reopened]

jobs:
run_nextflow:
name: Run pipeline for the upcoming PR
runs-on: ubuntu-latest

steps:

- name: Check out pipeline code
uses: actions/checkout@v2

- name: Install Nextflow
env:
CAPSULE_LOG: none
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/

- name: Clean environment
run: |
sudo rm -rf /usr/local/lib/android # will release about 10 GB if you don't need Android
sudo rm -rf /usr/share/dotnet # will release about 20GB if you don't need .NET

- name: Run tests for hybrid
run: |
nextflow run main.nf -profile docker,test,hybrid,pacbio --skip_haslr
rm -r work .nextflow*

- name: View results
run: |
sudo apt-get install -y tree
tree hybrid_test_pacbio
42 changes: 42 additions & 0 deletions .github/workflows/test_pr_hybrid_singularity.bkp
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: Testing hybrid / singularity from PR
on:
pull_request:
branches: [master, dev]
types: [opened, synchronize, reopened]

jobs:
run_nextflow:
name: Run pipeline for the upcoming PR
runs-on: ubuntu-latest

steps:

- name: Check out pipeline code
uses: actions/checkout@v2

- name: Install Singularity
uses: eWaterCycle/setup-singularity@v7
with:
singularity-version: 3.8.3

- name: Install Nextflow
env:
CAPSULE_LOG: none
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/

- name: Clean environment
run: |
sudo rm -rf /usr/local/lib/android # will release about 10 GB if you don't need Android
sudo rm -rf /usr/share/dotnet # will release about 20GB if you don't need .NET

- name: Run tests for hybrid
run: |
nextflow run main.nf -profile singularity,test,hybrid,pacbio
rm -r work .nextflow*

- name: View results
run: |
sudo apt-get install -y tree
tree hybrid_test
37 changes: 37 additions & 0 deletions .github/workflows/test_pr_illumina_docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Testing illumina / docker from PR
on:
pull_request:
branches: [ master, dev ]
types: [ opened, synchronize, reopened ]

jobs:
run_nextflow:
name: Run pipeline for the upcoming PR
runs-on: ubuntu-latest

steps:

- name: Check out pipeline code
uses: actions/checkout@v2

- name: Install Nextflow
env:
CAPSULE_LOG: none
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/

- name: Clean environment
run: |
sudo rm -rf /usr/local/lib/android # will release about 10 GB if you don't need Android
sudo rm -rf /usr/share/dotnet # will release about 20GB if you don't need .NET

- name: Run tests for illumina
run: |
nextflow run main.nf -profile docker,test,sreads
rm -r work .nextflow*

- name: View results
run: |
sudo apt-get install -y tree
tree sreads_test
42 changes: 42 additions & 0 deletions .github/workflows/test_pr_illumina_singularity.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
name: Testing illumina / singularity from PR
on:
pull_request:
branches: [ master, dev ]
types: [ opened, synchronize, reopened ]

jobs:
run_nextflow:
name: Run pipeline for the upcoming PR
runs-on: ubuntu-latest

steps:

- name: Check out pipeline code
uses: actions/checkout@v2

- name: Install Singularity
uses: eWaterCycle/setup-singularity@v7
with:
singularity-version: 3.8.3

- name: Install Nextflow
env:
CAPSULE_LOG: none
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/

- name: Clean environment
run: |
sudo rm -rf /usr/local/lib/android # will release about 10 GB if you don't need Android
sudo rm -rf /usr/share/dotnet # will release about 20GB if you don't need .NET

- name: Run tests for illumina
run: |
nextflow run main.nf -profile singularity,test,sreads
rm -r work .nextflow*

- name: View results
run: |
sudo apt-get install -y tree
tree sreads_test
37 changes: 37 additions & 0 deletions .github/workflows/test_pr_lreads_docker_ont.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Testing long-reads / docker (ONT) from PR
on:
pull_request:
branches: [ master, dev ]
types: [ opened, synchronize, reopened ]

jobs:
run_nextflow:
name: Run pipeline for the upcoming PR
runs-on: ubuntu-latest

steps:

- name: Check out pipeline code
uses: actions/checkout@v2

- name: Install Nextflow
env:
CAPSULE_LOG: none
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/

- name: Clean environment
run: |
sudo rm -rf /usr/local/lib/android # will release about 10 GB if you don't need Android
sudo rm -rf /usr/share/dotnet # will release about 20GB if you don't need .NET

- name: Run tests for long-reads (ont)
run: |
nextflow run main.nf -profile docker,test,lreads,ont --max_memory '6.GB' --max_cpus 2 --skip_flye --skip_canu
rm -r work .nextflow*

- name: View results
run: |
sudo apt-get install -y tree
tree lreads_test_ont
37 changes: 37 additions & 0 deletions .github/workflows/test_pr_lreads_docker_ont_hifi.bkp
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
name: Testing long-reads HIFI / docker (ONT) from PR
on:
pull_request:
branches: [ master, dev ]
types: [ opened, synchronize, reopened ]

jobs:
run_nextflow:
name: Run pipeline for the upcoming PR
runs-on: ubuntu-latest

steps:

- name: Check out pipeline code
uses: actions/checkout@v2

- name: Install Nextflow
env:
CAPSULE_LOG: none
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/

- name: Clean environment
run: |
sudo rm -rf /usr/local/lib/android # will release about 10 GB if you don't need Android
sudo rm -rf /usr/share/dotnet # will release about 20GB if you don't need .NET

- name: Run tests for long-reads (ont)
run: |
nextflow run main.nf -profile docker,test,lreads,ont_hifi --max_memory '6.GB' --max_cpus 4
rm -r work .nextflow*

- name: View results
run: |
sudo apt-get install -y tree
tree lreads_test_ont_hifi
48 changes: 48 additions & 0 deletions .github/workflows/test_pr_lreads_singularity_ont.bkp
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
name: Testing long-reads / singularity (ONT) from PR
on:
pull_request:
branches: [ master, dev ]
types: [ opened, synchronize, reopened ]

jobs:
run_nextflow:
name: Run pipeline for the upcoming PR
runs-on: macos-latest

steps:

- name: 'Set up latest Oracle JDK 17'
uses: oracle-actions/setup-java@v1
with:
website: oracle.com
release: 17

- name: Check out pipeline code
uses: actions/checkout@v2

- name: Install Singularity
uses: eWaterCycle/setup-singularity@v7
with:
singularity-version: 3.8.3

- name: Install Nextflow
env:
CAPSULE_LOG: none
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/

- name: Clean environment
run: |
sudo rm -rf /usr/local/lib/android # will release about 10 GB if you don't need Android
sudo rm -rf /usr/share/dotnet # will release about 20GB if you don't need .NET

- name: Run tests for long-reads (ont)
run: |
nextflow run main.nf -profile singularity,test,lreads,ont
rm -r work .nextflow*

- name: View results
run: |
sudo apt-get install -y tree
tree lreads_test_ont
1 change: 1 addition & 0 deletions .nf-core.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
repository_type: pipeline
18 changes: 5 additions & 13 deletions .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,12 @@ version: 2
build:
os: ubuntu-22.04
tools:
python: "3.11"
python: "3.9"

# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/conf.py
mkdocs:
configuration: mkdocs.yml

# Build documentation with MkDocs
# mkdocs:
# configuration: mkdocs.yml

# Optionally build your docs in additional formats such as PDF and ePub
formats: all

# Optionally set the version of Python and requirements required to build your docs
# Optionally declare the Python requirements required to build your docs
python:
install:
- requirements: docs/requirements.txt
- requirements: docs/requirements.txt
2 changes: 1 addition & 1 deletion .zenodo.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"description": "<p>MpGAP is built using Nextflow, a workflow tool to run tasks across multiple compute infrastructures in a very portable manner. It uses Docker/Singularity containers making installation trivial and results highly reproducible. It is an easy to use pipeline that adopts well known software for _de novo_ genome assembly of Illumina, Pacbio and Oxford Nanopore sequencing data through illumina only, long reads only or hybrid modes.</p>",
"license": "other-open",
"title": "fmalmeida/MpGAP: A generic multi-platform genome assembly pipeline",
"version": "v3.1.4",
"version": "v3.2",
"upload_type": "software",
"creators": [
{
Expand Down
Loading
Loading