GTDB won't stop running and produce results

[lyisrae1]

Hello,

May you please help me figure out what is going wrong? I have been trying to use GTDB for the past month, and it does not stop running. It just goes on forever.

I ran check_install to see if it was something on the installation end, but everything looks good:
[2025-03-28 13:00:19] INFO: |-- FastTree OK
[2025-03-28 13:00:19] INFO: |-- FastTreeMP OK
[2025-03-28 13:00:19] INFO: |-- guppy OK
[2025-03-28 13:00:19] INFO: |-- hmmalign OK
[2025-03-28 13:00:19] INFO: |-- hmmsearch OK
[2025-03-28 13:00:19] INFO: |-- mash OK
[2025-03-28 13:00:19] INFO: |-- pplacer OK
[2025-03-28 13:00:19] INFO: |-- prodigal OK
[2025-03-28 13:00:19] INFO: |-- skani OK
[2025-03-28 13:00:19] INFO: Checking integrity of reference package: /scratch/genomics/yisraell/GTDB/release220
[2025-03-28 13:00:26] INFO: |-- pplacer OK
[2025-03-28 13:00:26] INFO: |-- masks OK
[2025-03-28 13:00:30] INFO: |-- markers OK
[2025-03-28 13:00:30] INFO: |-- radii OK
[2025-03-28 13:00:42] INFO: |-- msa OK
[2025-03-28 13:00:42] INFO: |-- metadata OK
[2025-03-28 13:00:42] INFO: |-- taxonomy OK
[2025-03-28 13:42:23] INFO: |-- skani OK
[2025-03-28 13:42:23] INFO: |-- mrca_red OK
[2025-03-28 13:42:23] INFO: Done.

The program also know where my database is, which is great! But it does not give me any results and seems to run forever.

Here is the code I used for my script:
start-mamba
mamba activate gtdbtk-2.1.1
gtdbtk classify_wf --genome_dir /scratch/public/genomics/yisraell/50_percent_complete_and_10_percent_contamination/All_bins --out_dir /scratch/public/genomics/yisraell/GTDB/GTDB_Classify --mash_db -f --pplacer_cpus 40 --cpus 40

(I included mash_db since it is the first time)

And it will just run without stopping, and without producing results. It has been running since Feb. 26th.

Can you please help me figure this out?

Environment

• Installed via pip (include the output of pip list)
• [ X] Using a conda environment (include the output of conda list && conda list --revisions)
• Using a Docker container (include the IMAGE ID of the container)

(gtdbtk-2.1.1) [yisraell@login01 yisraell]$ conda list && conda list --revisions

packages in environment at /home/yisraell/.conda/envs/gtdbtk-2.1.1:

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
boost 1.70.0 py38h9de70de_1 conda-forge
boost-cpp 1.70.0 h7b93d67_3 conda-forge
bzip2 1.0.8 h4bc722e_7 conda-forge
ca-certificates 2025.1.31 hbcca054_0 conda-forge
capnproto 1.0.2 h2b92303_1 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
dendropy 5.0.6 pyhdfd78af_0 bioconda
fastani 1.34 h4dfc31f_4 bioconda
fasttree 2.1.11 h7b50bb2_5 bioconda
gsl 2.7 he838d99_0 conda-forge
gtdbtk 2.4.0 pyhdfd78af_2 bioconda
hmmer 3.1b2 3 bioconda
icu 67.1 he1b5a44_0 conda-forge
ld_impl_linux-64 2.43 h712a8e2_2 conda-forge
libblas 3.9.0 28_h59b9bed_openblas conda-forge
libcblas 3.9.0 28_he106b2a_openblas conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc 14.2.0 h77fa898_1 conda-forge
libgcc-ng 14.2.0 h69a702a_1 conda-forge
libgfortran 14.2.0 h69a702a_1 conda-forge
libgfortran5 14.2.0 hd5240d6_1 conda-forge
libgomp 14.2.0 h77fa898_1 conda-forge
liblapack 3.9.0 28_h7ac8fdf_openblas conda-forge
liblzma 5.6.4 hb9d3cd8_0 conda-forge
liblzma-devel 5.6.4 hb9d3cd8_0 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libopenblas 0.3.28 pthreads_h94d23a6_1 conda-forge
libsqlite 3.46.0 hde9e2c9_0 conda-forge
libstdcxx 14.2.0 hc0a3c3a_1 conda-forge
libstdcxx-ng 14.2.0 h4852527_1 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.2.13 h4ab18f5_6 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
mash 2.3 hc74b729_7 bioconda
ncurses 6.5 h2d0b736_3 conda-forge
numpy 1.24.4 py38h59b608b_0 conda-forge
openssl 3.4.0 h7b32b05_1 conda-forge
pip 24.3.1 pyh8b19718_0 conda-forge
pplacer 1.1.alpha19 h9ee0642_2 bioconda
prodigal 2.6.3 h7b50bb2_10 bioconda
pydantic 1.10.12 py38h5eee18b_1
python 3.8.19 hd12c33a_0_cpython conda-forge
python_abi 3.8 5_cp38 conda-forge
readline 8.2 h8228510_1 conda-forge
setuptools 75.3.0 pyhd8ed1ab_0 conda-forge
skani 0.2.2 ha6fb395_2 bioconda
tk 8.6.13 noxft_h4845f30_101 conda-forge
tqdm 4.67.1 pyhd8ed1ab_0 conda-forge
typing-extensions 4.11.0 py38h06a4308_0
typing_extensions 4.11.0 py38h06a4308_0
wheel 0.45.1 pyhd8ed1ab_0 conda-forge
xz 5.6.4 hbcc6ac9_0 conda-forge
xz-gpl-tools 5.6.4 hbcc6ac9_0 conda-forge
xz-tools 5.6.4 hb9d3cd8_0 conda-forge
zlib 1.2.13 h4ab18f5_6 conda-forge
zstd 1.4.9 ha95c52a_0 conda-forge
2025-02-10 21:53:01 (rev 0)
+_libgcc_mutex-0.1 (conda-forge/linux-64)
+_openmp_mutex-4.5 (conda-forge/linux-64)
+boost-1.70.0 (conda-forge/linux-64)
+boost-cpp-1.70.0 (conda-forge/linux-64)
+bzip2-1.0.8 (conda-forge/linux-64)
+ca-certificates-2025.1.31 (conda-forge/linux-64)
+capnproto-1.0.2 (conda-forge/linux-64)
+colorama-0.4.6 (conda-forge/noarch)
+dendropy-5.0.6 (bioconda/noarch)
+fastani-1.3 (bioconda/linux-64)
+fasttree-2.1.11 (bioconda/linux-64)
+gsl-2.7 (conda-forge/linux-64)
+gtdbtk-2.1.1 (bioconda/noarch)
+hmmer-3.1b2 (bioconda/linux-64)
+icu-67.1 (conda-forge/linux-64)
+ld_impl_linux-64-2.43 (conda-forge/linux-64)
+libblas-3.9.0 (conda-forge/linux-64)
+libcblas-3.9.0 (conda-forge/linux-64)
+libffi-3.4.2 (conda-forge/linux-64)
+libgcc-14.2.0 (conda-forge/linux-64)
+libgcc-ng-14.2.0 (conda-forge/linux-64)
+libgfortran-14.2.0 (conda-forge/linux-64)
+libgfortran5-14.2.0 (conda-forge/linux-64)
+libgomp-14.2.0 (conda-forge/linux-64)
+liblapack-3.9.0 (conda-forge/linux-64)
+liblzma-5.6.4 (conda-forge/linux-64)
+liblzma-devel-5.6.4 (conda-forge/linux-64)
+libnsl-2.0.1 (conda-forge/linux-64)
+libopenblas-0.3.28 (conda-forge/linux-64)
+libsqlite-3.46.0 (conda-forge/linux-64)
+libstdcxx-14.2.0 (conda-forge/linux-64)
+libstdcxx-ng-14.2.0 (conda-forge/linux-64)
+libuuid-2.38.1 (conda-forge/linux-64)
+libxcrypt-4.4.36 (conda-forge/linux-64)
+libzlib-1.2.13 (conda-forge/linux-64)
+lz4-c-1.9.3 (conda-forge/linux-64)
+mash-2.3 (bioconda/linux-64)
+ncurses-6.5 (conda-forge/linux-64)
+numpy-1.24.4 (conda-forge/linux-64)
+openssl-3.4.0 (conda-forge/linux-64)
+pip-24.3.1 (conda-forge/noarch)
+pplacer-1.1.alpha19 (bioconda/linux-64)
+prodigal-2.6.3 (bioconda/linux-64)
+python-3.8.19 (conda-forge/linux-64)
+python_abi-3.8 (conda-forge/linux-64)
+readline-8.2 (conda-forge/linux-64)
+setuptools-75.3.0 (conda-forge/noarch)
+tk-8.6.13 (conda-forge/linux-64)
+tqdm-4.67.1 (conda-forge/noarch)
+wheel-0.45.1 (conda-forge/noarch)
+xz-5.6.4 (conda-forge/linux-64)
+xz-gpl-tools-5.6.4 (conda-forge/linux-64)
+xz-tools-5.6.4 (conda-forge/linux-64)
+zlib-1.2.13 (conda-forge/linux-64)
+zstd-1.4.9 (conda-forge/linux-64)

2025-02-19 14:52:57 (rev 1)
fastani {1.3 (bioconda/linux-64) -> 1.34 (bioconda/linux-64)}
gtdbtk {2.1.1 (bioconda/noarch) -> 2.4.0 (bioconda/noarch)}
+pydantic-1.10.12 (defaults/linux-64)
+skani-0.2.2 (bioconda/linux-64)
+typing-extensions-4.11.0 (defaults/linux-64)
+typing_extensions-4.11.0 (defaults/linux-64)

Server information

• CPU: grep -m 1 "^model name" /proc/cpuinfo && grep -c "^processor" /proc/cpuinfo
• RAM: grep "^MemTotal" /proc/meminfo
• OS: cat /etc/os-release

model name : Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz 48
MemTotal: 131887596 kB
NAME="Rocky Linux"
VERSION="8.9 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.9"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.9 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2029-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.9"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.9"

Debugging information

• [X ] gtdbtk.log has been included (drag and drop the file to upload).
• Genomes have been included (if possible, and there are few).

mamba (miniforge3) v24.1.2 loaded, to start type 'start-mamba'

• Wed Feb 26 16:00:21 EST 2025 job GTDB_Megahit3 started in uThC.q with jobID=5992246 on compute-75-04

Additional comments

[pchaumeil]

Hi,
Could you please try to update GTDB-Tk to the latest version ( 2.4.0)? GTDB-Tk 2.1 does not support release 220. FYI we are planning to release a new version of GTDB-Tk in the coming days.

[lyisrae1]

This is the latest version. The environment is named gtdbtk-2.1.1, but it is version 2.4.0. It is listed in my environment updates: gtdbtk 2.4.0 pyhdfd78af_2 bioconda

[lyisrae1]

Do you think this is a download issue?

[pchaumeil]

Do you have any gtdbtk.log file in the /scratch/public/genomics/yisraell/GTDB/GTDB_Classify folder?

[lyisrae1]

No I do not. All I have is the log file generated by the cluster.
It says: mamba (miniforge3) v24.1.2 loaded, to start type 'start-mamba'

• Wed Feb 26 14:07:47 EST 2025 job GTDB_Megahit1 started in uThC.q with jobID=5992203 on compute-76-13

The job is still running ever since February 26th.

[pchaumeil]

Thanks!

I just have a few more questions to help troubleshoot the issue:

What is the walltime of your job? 20 days seems quite long; are you sure the job is actually running and not just sitting in the queue?
Have you successfully run a job on this cluster before?
Have you tried running an interactive job to test things? If so, did it complete successfully?

[lyisrae1]

Hello again,

I have successfully run jobs on this cluster many times. I have done interactive jobs before, and have never had an issue like this.
I have checked the job, and it says that it is running has been running for that long. I even made a second run at a cut off time of 6 days, and the job was killed after 6 days with no data produced