Skip to content

Fixed Conda dependency issues in environment.yml #211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 26 commits into
base: dev
Choose a base branch
from
Open

Fixed Conda dependency issues in environment.yml #211

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
c1875d2
changed base.config process_medium settings for local runs
KurayiChawatama Jul 31, 2025
4a0e1ca
fixed conda profile highly_variable column not boolean error
KurayiChawatama Jul 31, 2025
0fa1ee6
added missing singlecellexperiment and r2py libs to environment.yml
KurayiChawatama Jul 31, 2025
eda483e
fixed rpy2 version fetch crash
KurayiChawatama Jul 31, 2025
8f1f0ca
fixed upset plot script python fetch fail crash
KurayiChawatama Jul 31, 2025
e74e1ad
added missing libs to doublet removal env yml
KurayiChawatama Jul 31, 2025
2a02909
fixed doublet removal script python fetch fail crash
KurayiChawatama Jul 31, 2025
b22c1a0
added evironment yml to scds to allow conda based running
KurayiChawatama Jul 31, 2025
f05a981
updated liana env yml to include missing conda packadges
KurayiChawatama Jul 31, 2025
e596e27
added missing scrublet lib to env yml
KurayiChawatama Jul 31, 2025
95f8c8e
added missing libs to doublet detection env yml
KurayiChawatama Jul 31, 2025
93a62e9
added missing libs to solo env yml
KurayiChawatama Jul 31, 2025
0e711ff
untested attempted addition of SingleR support to the pipeline
KurayiChawatama Aug 1, 2025
4edf537
Revert "untested attempted addition of SingleR support to the pipeline"
KurayiChawatama Aug 1, 2025
9ab7a8b
reset base.config to it's original state
KurayiChawatama Aug 1, 2025
8e9ce72
reset the scds module to it's original condaless state
KurayiChawatama Aug 1, 2025
8363758
added correct conda dependencies for liana rankaggregate to run
KurayiChawatama Aug 1, 2025
784d8e6
reset liana rankagreggate env to it's original state
KurayiChawatama Aug 1, 2025
5b01930
reverted nf-core solo env.yml to original state
KurayiChawatama Aug 2, 2025
8dfca4f
restored nf-core doubletdetection env.yml to original version
KurayiChawatama Aug 2, 2025
fc9c88c
Fix: Sync logo to match nf-core template (lint fix)
KurayiChawatama Aug 2, 2025
270f77b
resolved the issues raised in the PR
KurayiChawatama Aug 19, 2025
bee50ca
added the container images to match env yaml changes, removed changes…
KurayiChawatama Aug 19, 2025
089c0b0
matched liana py to upstream, removed scrublet environment yaml
KurayiChawatama Aug 19, 2025
72dfd31
fix(doublet_removal): update environment to NumPy 2–compatible stack
KurayiChawatama Aug 20, 2025
119efb0
updated doublet removal contianers to match changes to env yml
KurayiChawatama Aug 20, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions modules/local/adata/extend/templates/extend.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@ def load_pickle_or_csv(path):
df = load_pickle_or_csv(path).reindex(adata.var_names)
adata.var = pd.concat([adata.var, df], axis=1)

# Ensure 'highly_variable' is boolean if present
if "highly_variable" in adata.var:
adata.var["highly_variable"] = adata.var["highly_variable"].astype(bool)

Comment on lines +41 to +44
Copy link
Collaborator

@nictru nictru Aug 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Ensure 'highly_variable' is boolean if present
if "highly_variable" in adata.var:
adata.var["highly_variable"] = adata.var["highly_variable"].astype(bool)

I explained the issue that you are trying to fix in #215 and implemented a different fix in the meantime. That fix is not optimal, but for me preferable over this. So please revert this for now

I am not happy with the situation in this aspect, but I have not yet come up with a really clean way of handling this

for path in obsm_paths:
df = pd.read_pickle(path).reindex(adata.obs_names)
adata.obsm[simple_name(path)] = np.float32(df.to_numpy())
Expand Down
2 changes: 2 additions & 0 deletions modules/local/adata/readrds/environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,3 +6,5 @@ dependencies:
- bioconda::anndata2ri=1.3.2
- conda-forge::anndata=0.11.1
- conda-forge::r-seurat=5.1.0
- bioconda::bioconductor-singlecellexperiment=1.24.0
- conda-forge::rpy2=3.5.11
4 changes: 2 additions & 2 deletions modules/local/adata/readrds/main.nf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ process ADATA_READRDS {

conda "${moduleDir}/environment.yml"
container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
'oras://community.wave.seqera.io/library/anndata2ri_bioconductor-singlecellexperiment_anndata_r-seurat:c4b75a61a89ec006':
'community.wave.seqera.io/library/anndata2ri_bioconductor-singlecellexperiment_anndata_r-seurat:5fae42aabf7a1c5f' }"
'oras://community.wave.seqera.io/library/anndata2ri_bioconductor-singlecellexperiment_anndata_r-seurat_rpy2:5e5acc8a27b2f90a':
'community.wave.seqera.io/library/anndata2ri_bioconductor-singlecellexperiment_anndata_r-seurat_rpy2:5e785d9c16504ed6' }"

input:
tuple val(meta), path(rds)
Expand Down
6 changes: 4 additions & 2 deletions modules/local/adata/readrds/templates/readrds.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

os.environ["TMPDIR"] = "."

import importlib.metadata

import anndata as ad
import anndata2ri
import rpy2
Expand Down Expand Up @@ -53,10 +55,10 @@ def format_yaml_like(data: dict, indent: int = 0) -> str:
adata.write_h5ad("${prefix}.h5ad")

versions = {
"${task.process}": {
"NFCORE_SCDOWNSTREAM:SCDOWNSTREAM:LOAD_H5AD:ADATA_READRDS": {
Comment on lines -56 to +58
Copy link
Collaborator

@nictru nictru Aug 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert this please, unless you have a very good explanation for why this is needed

"anndata": ad.__version__,
"anndata2ri": anndata2ri.__version__,
"rpy2": rpy2.__version__,
"rpy2": importlib.metadata.version("rpy2"),
Copy link
Collaborator

@nictru nictru Aug 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the issue description you state:

Used importlib.metadata to dynamically retrieve the rpy2 version, preventing crashes during rds reading.

I don't understand what exactly the issue with the original implementation is

Copy link
Author

@KurayiChawatama KurayiChawatama Aug 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey, the original implementation was crashing and raising,
AttributeError: module 'rpy2' has no attribute 'version'

Copy link
Collaborator

@nictru nictru Aug 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did the following to validate your concern:

  1. docker run -it community.wave.seqera.io/library/anndata2ri_bioconductor-singlecellexperiment_anndata_r-seurat_rpy2:5e785d9c16504ed6
  2. python3
  3. import rpy2
  4. rpy2.__version__

Which correctly printed a version string. So if you get an error and I do not, I think the following are possible explanations:

  • You have a different version of python in the conda env (in docker it's 3.12.11)
  • You have a different version of rpy2 in the conda env (in docker it's 3.5.11) - but since the rpy2 version is encoded in the environment.yml, this should not be possible
  • The problem is related to the host computer, outside of conda

The reason I come to these explanations is that the docker is basically a minimal ubuntu, with micromamba installed and all the specified packages installed in the base environment. So if it works in this setting, but not in yours, there is not many places one needs to check.

Anyways, I don't want to accept this change until we have a better understanding of the problem. The __version__ access is used in many places throughout the pipeline (and all of nf-core) and if there is a fundamental problem with it, we need to make a lot more changes than this single one

"pandas": pd.__version__,
"seurat": seurat.__version__
}
Expand Down
2 changes: 1 addition & 1 deletion modules/local/adata/upsetgenes/templates/upsetplot.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/opt/conda/bin/python
#!/usr/bin/env python

import os
import platform
Expand Down
9 changes: 7 additions & 2 deletions modules/local/doublet_detection/doublet_removal/environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,10 @@ channels:
- conda-forge
- bioconda
dependencies:
- conda-forge::anndata=0.11.1
- conda-forge::upsetplot=0.9.0
- python=3.11.9
- numpy=2.0.2
- scipy=1.14.1
- pandas=2.2.2
- matplotlib=3.8.4
- anndata=0.11.1
- upsetplot=0.9.0
4 changes: 2 additions & 2 deletions modules/local/doublet_detection/doublet_removal/main.nf
View file
Open in desktop
Copy link
Author

@KurayiChawatama KurayiChawatama Aug 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixes the error that happened because the doublet_removal module tried to load a pickle (sample_scrublet.pkl) that was created with NumPy ≥2.0, which uses the internal module numpy._core. The environment for doublet_removal was pinned to NumPy 1.23.5, where numpy._core does not exist, so unpickling failed with ModuleNotFoundError. Updating the environment to a NumPy-2–compatible stack resolved the mismatch.

Copy link
Collaborator

@nictru nictru Aug 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, updating to newer versions is generally fine, just make sure to use the full accessions (e.g. conda-forge::anndata=0.11.1 here as well, not just anndata=0.11.1)

Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ process DOUBLET_REMOVAL {

conda "${moduleDir}/environment.yml"
container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
'oras://community.wave.seqera.io/library/anndata_upsetplot:922f36ea0326a629':
'community.wave.seqera.io/library/anndata_upsetplot:784e0f450da10178' }"
'oras://community.wave.seqera.io/library/anndata_matplotlib_numpy_pandas_pruned:a603920d55ea7f32':
'community.wave.seqera.io/library/anndata_matplotlib_numpy_pandas_pruned:9674070abcd72c6f' }"

input:
tuple val(meta), path(h5ad), path(predictions)
Expand Down
2 changes: 1 addition & 1 deletion modules/local/doublet_detection/doublet_removal/templates/doublet_removal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/opt/conda/bin/python
#!/usr/bin/env python3

import os
import platform
Expand Down
Loading