SPAdes is running abnormally, with some errors.

[TAXICANI]

Description of bug

Hello！
I tried to use Spades to assemble metagenomes on HPC by submitting Slurm scripts. After running the program for a period of time, the log file showed the following error, but Slurm indicated that the job had not been terminated. After waiting for 20 hours, there was still no update to the log file. I suspect that there is an abnormality in the program.

spades.log

Command line: /public/home/sk_wrj/anaconda3/envs/spades/bin/spades.py --meta -1 /public/home/sk_wrj/DG_1_5_2018.07_1.fq.gz -2 /public/home/sk_wrj/DG_1_5_2018.07_2.fq.gz -o /public/home/sk_wrj/BASALT/spades -m 800 -t 80

System information:
SPAdes version: 4.0.0
Python version: 3.12.5
OS: Linux-3.10.0-693.el7.x86_64-x86_64-with-glibc2.17

Output dir: /public/home/sk_wrj/BASALT/spades
Mode: read error correction and assembling
Debug mode is turned OFF

Dataset parameters:
Metagenomic mode
Reads:
Library number: 1, library type: paired-end
orientation: fr
left reads: ['/public/home/sk_wrj/DG_1_5_2018.07_1.fq.gz']
right reads: ['/public/home/sk_wrj/DG_1_5_2018.07_2.fq.gz']
interlaced reads: not specified
single reads: not specified
merged reads: not specified
Read error correction parameters:
Iterations: 1
PHRED offset will be auto-detected
Corrected reads will be compressed
Assembly parameters:
k: [21, 33, 55]
Repeat resolution is enabled
Mismatch careful mode is turned OFF
MismatchCorrector will be SKIPPED
Coverage cutoff is turned OFF
Assembly graph output will use GFA v1.2 format
Other parameters:
Dir for temp files: /public/home/sk_wrj/BASALT/spades/tmp
Threads: 80
Memory limit (in Gb): 800

======= SPAdes pipeline started. Log can be found here: /public/home/sk_wrj/BASALT/spades/spades.log

/public/home/sk_wrj/DG_1_5_2018.07_1.fq.gz: max reads length: 150
/public/home/sk_wrj/DG_1_5_2018.07_2.fq.gz: max reads length: 150

Reads length: 150

===== Before start started.

===== Read error correction started.

===== Read error correction started.

== Running: /public/home/sk_wrj/anaconda3/envs/spades/bin/spades-hammer /public/home/sk_wrj/BASALT/spades/corrected/configs/config.info

0:00:00.000 1M / 24M INFO General (main.cpp : 76) Starting BayesHammer, built from N/A, git revision N/A
0:00:00.000 1M / 24M INFO General (main.cpp : 77) Loading config from "/public/home/sk_wrj/BASALT/spades/corrected/configs/config.info"
0:00:00.002 1M / 24M INFO General (main.cpp : 79) Maximum # of threads to use (adjusted due to OMP capabilities): 1
0:00:00.002 1M / 24M INFO General (memory_limit.cpp : 55) Memory limit set to 800 Gb
0:00:00.002 1M / 24M INFO General (main.cpp : 87) Trying to determine PHRED offset
0:00:00.005 1M / 24M INFO General (main.cpp : 93) Determined value is 33
0:00:00.005 1M / 24M INFO General (hammer_tools.cpp : 40) Hamming graph threshold tau=1, k=21, subkmer positions = [ 0 10 ]
0:00:00.006 1M / 24M INFO General (main.cpp : 114) Size of aux. kmer data 24 bytes
=== ITERATION 0 begins ===
0:00:00.006 1M / 24M INFO K-mer Counting (kmer_data.cpp : 284) Estimating k-mer count
0:00:00.014 17M / 24M INFO K-mer Counting (kmer_data.cpp : 289) Processing "/public/home/sk_wrj/DG_1_5_2018.07_1.fq.gz"
0:29:41.886 17M / 24M INFO K-mer Counting (kmer_data.cpp : 298) Processed 68853519 reads
0:29:41.929 17M / 24M INFO K-mer Counting (kmer_data.cpp : 289) Processing "/public/home/sk_wrj/DG_1_5_2018.07_2.fq.gz"
0:59:38.318 17M / 24M INFO K-mer Counting (kmer_data.cpp : 298) Processed 137707038 reads
0:59:38.322 17M / 24M INFO K-mer Counting (kmer_data.cpp : 303) Total 137707038 reads processed
0:59:38.593 17M / 24M INFO K-mer Counting (kmer_data.cpp : 306) Estimated 4373713076 distinct kmers
0:59:38.593 1M / 24M INFO K-mer Counting (kmer_data.cpp : 310) Filtering singleton k-mers
mimalloc: warning: unable to allocate aligned OS memory directly, fall back to over-allocation (21751660544 bytes, address: 0x2b102eed6000, alignment: 67108864, commit: 1)
0:59:47.654 21G / 21G INFO K-mer Counting (kmer_data.cpp : 316) Processing "/public/home/sk_wrj/DG_1_5_2018.07_1.fq.gz"
2:31:33.242 21G / 21G INFO K-mer Counting (kmer_data.cpp : 325) Processed 68853519 reads
2:31:33.274 21G / 21G INFO K-mer Counting (kmer_data.cpp : 316) Processing "/public/home/sk_wrj/DG_1_5_2018.07_2.fq.gz"
4:05:16.302 21G / 21G INFO K-mer Counting (kmer_data.cpp : 325) Processed 137707038 reads
4:05:16.338 21G / 21G INFO K-mer Counting (kmer_data.cpp : 330) Total 137707038 reads processed
4:05:16.346 21G / 21G INFO General (kmer_index_builder.hpp : 258) Splitting kmer instances into 16 files using 1 threads. This might take a while.
4:05:16.348 21G / 21G INFO General (file_limit.hpp : 43) Open file limit set to 51200
4:05:16.348 21G / 21G INFO General (kmer_splitter.hpp : 94) Memory available for splitting buffers: 259.91 Gb
4:05:16.348 21G / 21G INFO General (kmer_splitter.hpp : 102) Using cell size of 4194304
4:05:16.359 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 98) Processing "/public/home/sk_wrj/DG_1_5_2018.07_1.fq.gz"
4:05:36.087 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 278842 reads
4:05:55.448 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 549642 reads
4:06:14.815 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 823835 reads
4:06:33.258 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 1094912 reads
4:06:50.805 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 1365502 reads
4:07:08.815 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 1637243 reads
4:07:47.336 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 2179767 reads
4:10:21.780 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 4354339 reads
4:15:11.433 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 8429429 reads
4:25:04.488 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 16877076 reads
4:44:38.696 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 33559358 reads
5:23:30.499 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 67127654 reads
5:25:32.521 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 98) Processing "/public/home/sk_wrj/DG_1_5_2018.07_2.fq.gz"
6:36:11.970 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 108) Processed 134478466 reads
6:39:33.260 21G / 21G INFO K-mer Splitting (kmer_data.cpp : 113) Total 137707038 reads processed
6:39:33.263 21G / 21G INFO General (kmer_index_builder.hpp : 264) Starting k-mer counting.
wrj/BASALT/spades/configs/debruijn/rna_mode.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/rna_mode.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/careful_mda_mode.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/careful_mda_mode.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/careful_mda_mode.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/config.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/config.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/config.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/detail_info_printer.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/detail_info_printer.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/detail_info_printer.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/distance_estimation.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/distance_estimation.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/distance_estimation.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/metaplasmid_mode.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/metaplasmid_mode.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/metaplasmid_mode.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/metaviral_mode.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/metaviral_mode.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/metaviral_mode.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/pe_params.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/pe_params.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/pe_params.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/rnaviral_mode.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/rnaviral_mode.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/rnaviral_mode.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/sewage_mode.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/sewage_mode.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/sewage_mode.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/simplification.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/simplification.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/simplification.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/toy.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/toy.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/toy.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn', '/public/home/sk_wrj/BASALT/spades/configs/debruijn', '[Errno 2] No such file or directory'), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs', '/public/home/sk_wrj/BASALT/spades/configs', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs'")]

== Error == exception caught: <class 'shutil.Error'>

In case you have troubles running SPAdes, you can report an issue on our GitHub repository github.com/ablab/spades
Please provide us with params.txt and spades.log files from the output directory.

SPAdes log can be found here: /public/home/sk_wrj/BASALT/spades/spades.log

Thank you for using metaSPAdes! If you use it in your research, please cite:

Nurk, S., Meleshko, D., Korobeynikov, A. and Pevzner, P.A., 2017. metaSPAdes: a new versatile metagenomic assembler. Genome research, 27(5), pp.824-834.
doi.org/10.1101/gr.213959.116

params.txt

Command line: /public/home/sk_wrj/anaconda3/envs/spades/bin/spades.py --meta -1 /public/home/sk_wrj/DG_1_5_2018.07_1.fq.gz -2 /public/home/sk_wrj/DG_1_5_2018.07_2.fq.gz -o /public/home/sk_wrj/BASALT/spades -m 800 -t 80

System information:
SPAdes version: 4.0.0
Python version: 3.12.5
OS: Linux-3.10.0-693.el7.x86_64-x86_64-with-glibc2.17

Output dir: /public/home/sk_wrj/BASALT/spades
Mode: read error correction and assembling
Debug mode is turned OFF

Dataset parameters:
Metagenomic mode
Reads:
Library number: 1, library type: paired-end
orientation: fr
left reads: ['/public/home/sk_wrj/DG_1_5_2018.07_1.fq.gz']
right reads: ['/public/home/sk_wrj/DG_1_5_2018.07_2.fq.gz']
interlaced reads: not specified
single reads: not specified
merged reads: not specified
Read error correction parameters:
Iterations: 1
PHRED offset will be auto-detected
Corrected reads will be compressed
Assembly parameters:
k: [21, 33, 55]
Repeat resolution is enabled
Mismatch careful mode is turned OFF
MismatchCorrector will be SKIPPED
Coverage cutoff is turned OFF
Assembly graph output will use GFA v1.2 format
Other parameters:
Dir for temp files: /public/home/sk_wrj/BASALT/spades/tmp
Threads: 80
Memory limit (in Gb): 800

SPAdes version

4.0.0

Operating System

Linux-3.10.0-693.el7.x86_64-x86_64-with-glibc2.17

Python Version

3.12.5

Method of SPAdes installation

conda

No errors reported in spades.log

• Yes

[asl]

The log reads:

wrj/BASALT/spades/configs/debruijn/rna_mode.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/rna_mode.info'"), ('/public/home/sk_wrj/anaconda3/envs/spades/share/spades/configs/debruijn/careful_mda_mode.info', '/public/home/sk_wrj/BASALT/spades/configs/debruijn/careful_mda_mode.info', "[Errno 2] No such file or directory: '/public/home/sk_wrj/BASALT/spades/configs/debruijn/careful_mda_mode.info'"),
...

I would suspect some kind of I/O issue on the storage system you're using. I would suggest you to ask your system administrator.