clam and -t (--threads)

[brantfaircloth]

Hi y'all,

As noted in #333, I've been testing clam w/ loci having highly variable coverage. One thing that I've noticed in these runs (and during snparcher runs, more generally) is that clam seems to mostly run in single-threaded mode. As a result, test runs inputting different parameters for clam are [really] slow.

After activating the correct conda environment, i've been running clam manually:

clam loci -o bcf-callable_loci.INF.zarr -t 24 --per-sample -m 10 depths.zarr/ &> bcf-callable_loci.zarr.INF.log

At first, there seems to be some activity involving multiple cores/threads, but after a while, CPU activity drops from >100% (usually around 400-600%) to 95-100%, where it stays. I haven't investigate the clam code - so this may be expected behavior (although it seems suspect).

This is clam 1.1.4 which was packaged up in snparcher v2.1. Also happy to move this over to the clam page, but I figured I would report here given #333.

Thanks!
-b

[cademirch]

Hi Brant,

Thanks for the issue, I will look into this asap!

Relevant to this, I've been working on https://github.com/pbzarr/pbzarr to replace the Zarr internals of clam and make it easier to do exploratory analyses on coverage data like you have. Let me know if this is interesting to you, and if you have availability it would be great to hop on a call to better understand your data and how pbzarr could be helpful (or what features might be missing).

Cade

[brantfaircloth]

Cool - thanks 👍. I can package up the depths.zarr and get that to you if it would help debug. And, sure - let me take a look at pbzarr. I also need to play around with clam a little bit more to see how it seems to work for this particular use-case.

[cademirch]

Cool - thanks 👍. I can package up the depths.zarr and get that to you if it would help debug. And, sure - let me take a look at pbzarr. I also need to play around with clam a little bit more to see how it seems to work for this particular use-case.

That would be helpful! Take a look here for how to inspect your depth zarr https://github.com/cademirch/clam/tree/main/notebooks

[brantfaircloth]

Here's the depths.zarr. Its not super huge... ~1.3 G zipped up.