nextflow 24.04.4 never exit due to incomplete file transfer #5363
Description
Bug report
Expected behavior and actual behavior
I have used nextflow 22.10.6.5843, which runs smoothly. After I updated my nextflow to v24.0.4.4, the same script hangs with some files not finished for transferring. The files to be transferred are totally around 50Gb.
Steps to reproduce the problem
Program output
Oct-03 12:42:12.157 [main] DEBUG nextflow.Session - Session await > all processes finished
Oct-03 12:42:17.082 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: slurm) - terminating tasks monitor poll loop
Oct-03 12:42:17.082 [main] DEBUG nextflow.Session - Session await > all barriers passed
Oct-03 12:42:17.093 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'TaskFinalizer' shutdown completed (hard=false)
Oct-03 12:42:22.095 [main] INFO nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (7 files)
Oct-03 12:43:22.102 [main] DEBUG nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (7 files)
Oct-03 12:44:22.104 [main] DEBUG nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (6 files)
Oct-03 12:45:22.106 [main] DEBUG nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (6 files)
Oct-03 12:46:22.108 [main] DEBUG nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (6 files)
Oct-03 12:47:22.110 [main] DEBUG nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (6 files)
Oct-03 12:48:22.112 [main] DEBUG nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (4 files)
Oct-03 12:49:22.114 [main] DEBUG nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (3 files)
Oct-03 12:50:22.116 [main] DEBUG nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (3 files)
.......
.......
.......
Oct-04 00:41:23.430 [main] DEBUG nextflow.util.ThreadPoolHelper - Waiting for file transfers to complete (3 files)
Oct-04 00:42:18.432 [main] WARN nextflow.util.ThreadPoolHelper - Exiting before file transfers were completed -- Some files may be lost
Oct-04 00:42:18.432 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false)
Oct-04 00:42:18.463 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=32; failedCount=0; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=226d 11h 31m 9s; failedDuration=0ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=5; peakCpus=125; peakMemory=0; ]
Oct-04 00:42:18.733 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done
Oct-04 00:42:18.820 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false)
Oct-04 00:42:18.820 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
Environment
- Nextflow version: 24.0.4.4
- Java version: openjdk version "20.0.2-internal" 2023-07-18
- Operating system: linux
- Bash version: GNU bash, version 5.1.16(1)-release