Performance regression in v1.3.1 vs v1.1.0 on large PE gz FASTQ with --detect_adapter_for_pe

[Abieskawa]

Hi, I found a major performance regression in fastp v1.3.1 compared with v1.1.0. It seems like v1.3.1 is slow and stuck in some unknown process.

Environment:

• Linux
• Docker
• paired-end gz FASTQ
• large samples (~2.3G–3.1G per FASTQ)

Command:

fastp -w 8 \
  -i S13_1.fastq.gz -I S13_2.fastq.gz \
  -o S13_1.cleaned.fastq.gz -O S13_2.cleaned.fastq.gz \
  -l 30 -q 20 -e 20 \
  --detect_adapter_for_pe \
  -f 10 -F 10 \
  -h S13.fastp.html -j S13.fastp.json

Observed in v1.3.1:
fastp starts and prints adapter-detection messages
output FASTQ files stay at 0 bytes
html/json reports are not produced
process stays alive for many hours
CPU usage remains active, but no measurable output progress
only some large samples are affected

Observed in v1.1.0:
same workflow is much faster and completes normally

[sfchen]

Thanks for reporting.

I couldn't reproduce this issue with my own data, can you please upload your data here, so that I can test with your data.

[Abieskawa]

Thanks you for the quick reply. The data may not be appropriate for released here, and it is quite large. But I think you can still test it on https://www.ncbi.nlm.nih.gov/sra/SRX030390. The speed should be quite different between v1.1.0 and v1.3.1.

[sfchen]

Besides the performance drop, did you observe any output inconsistencies? The unzipped FASTQ files should have same md5.

[Abieskawa]

No, but I have to say that I did not wait v1.3.1 to finish its work and output report. I applied fastp (v1.3.1) on three samples with 5X (-w) CPU assigned to each samples, and after four days, and it was still running, so I killed it. It only used one CPU when I checked it.

[KimYannn]

how many cpu threads do u have and assign to your container? r u using -w greater than you have in container?

[Abieskawa]

I did not limit it, and I got 160+ cpus available.
I test it with new container with specifying fastp=1.1.0 and lag solved in the end.
The following is the actual I run container:
nohup docker run --rm --gpus device=0 -v /home/abieskawa/analysis/2020_Tilapia_skimSeq_Disease_Resistance:/data -v /home/abieskawa/analysis/kraken2_standard_20251015:/kraken2_db -v /home/abieskawa/output:/home/abieskawa/output -w /data dgbreeding-gpu:test --config_file /data/configure.txt > pipeline_test.log 2>&1 &

[KimYannn]

@Abieskawa yep, this is a issue between 1.2.0 and 1.3.0 which aims to do performance optimization in io, are u sharing this machine with someone and the cpu load is too high all the time ?

[Abieskawa]

Yeah, I am sharing this machine with others, but it seems like they did not run with any process requiring lots of computing resources in recent days. BTW, the PI gave us root authority, which is why I can run docker container on it.

[KimYannn]

@Abieskawa could u have a try #679, u could find the static bin in the ci page https://github.com/OpenGene/fastp/actions/runs/24123835261?pr=679