Wrong version of ESMF used in ESMFPy when running in a Jupyter notebook that's uses a non-base conda environment

[colinsauze]

I am trying to run ESMPy on a Jupyter Hub system with ESMPy 8.8.1 (and ESMF 8.8.1) installed in the base Conda environment. My notebook uses a second conda environment with ESMPy/ESMF 8.9.1 installed. If I run Python code in the terminal in the second environment with 8.9.1 installed everything is ok, but if I try to run a Jupyter notebook which imports ESMPy I get a Version Mismatch error saying:

ESMF installation version 8.8.1 differs from ESMPy version 8.9.1

despite having 8.9.1 installed in this environment.

The root cause of this appears to be that line 25 of loadESMF.py reads the environment variable ESMFMKFILE. When I run in a Jupyter notebook the ESMFMKFILE environment variable is set to /opt/conda/lib/esmf.mk regardless of what conda environment the notebook is run in. When run from the terminal it picks up the path to esmf.mk from the conda environment.

As a work around I can manually set the os.environ['ESMFMKFILE'] in the Jupyter notebook or this code checks if the path of the Python interpreter differs from the path of the ESMFMKFILE.

import os, sys
# get the path of our python interpreter, e.g. /opt/conda/bin/python or /home/user/my_conda_env/bin/python
# this should return something like /opt/conda or /home/user/my_conda_env/
python_path = os.path.dirname(os.path.dirname(sys.executable))

# get the path of the ESMF mk file, e.g. /opt/conda/lib/esmf.mk or /home/user/my_conda_env/lib/esmf.mk
# this should also return something like /opt/conda or /home/user/my_conda_env/
esmf_path = os.path.dirname(os.path.dirname(os.environ["ESMFMKFILE"]))

# the python interpreter and ESMF mk file should share the same parent directory, if they don't then use the python interpreter path to set the ESMFMKFILE path.
if python_path != esmf_path:
    os.environ["ESMFMKFILE"] = python_path + "/lib/esmf.mk"

# the import os esmpy should now succeed in all cases
import esmpy

If this was inserted around line 25 of src/addon/src/addon/src/loadESMPF.py (https://github.com/esmf-org/esmf/blob/develop/src/addon/esmpy/src/esmpy/interface/loadESMF.py) then it would solve the problem.

[anntsay]

@billsacks just curious if you have already fixed this for us in 8.9.0? we have vague memory of dealing with this before.. thanks!

[billsacks]

I did fix some ESMPy version-related issues in the past, but I think this is something different.

As noted in the documentation, the general recommendation is to set ESMFMKFILE explicitly. However, there's some code to try to do this automatically for conda installations, and it sounds like the guessing that's done there isn't as robust as it should be.

At first I thought you might want to change this:

esmf/src/addon/esmpy/src/esmpy/interface/loadESMF.py

Line 29 in bae8e92

os.path.join(sys.prefix, 'lib', 'esmf.mk'), # conda build of esmf

but then I realized that this may actually be coming from the esmpy conda package scripts, not from ESMF/ESMPy itself:

https://github.com/conda-forge/esmpy-feedstock/blob/c4b2f63ca92c4a7ab2e0f5476bd0fcd7ff5aefd6/recipe/scripts/activate.sh#L11-L15

If you're finding that the ESMFMKFILE env var is actually automatically set, then I think it's probably coming from that conda package logic. I don't understand this well enough myself to know what logic is robust here, but if you agree that this seems to be coming from the conda package logic, then I think the right thing to do is to open an issue with the esmpy-feedstock conda-forge package - or a PR if you have a sense of how to fix it there.

[anntsay]

followed and created a ticket in conda-forge as recommended by Bill above:

conda-forge/esmpy-feedstock#94

[xylar]

Thanks for the detailed report — this looks like an environment inheritance issue rather than a problem with the ESMPy or conda packages themselves.

Below is the results from a discussion I had with ChatGPT about this issue. I think these suggestions seem sensible.

In ESMPy, the ESMFMKFILE environment variable takes precedence when determining which ESMF installation to use. If it is already set, ESMPy will use that path even if the Python interpreter is running from a different conda environment.

In many Jupyter/JupyterHub setups, the notebook server is started in a "base" environment and kernels inherit its environment variables. In that case the ESMFMKFILE from the base environment can leak into kernels launched from other environments. Since kernels are often started without a full conda activate, the activation scripts that normally reset ESMFMKFILE for the target environment may never run.

That would explain the behavior you’re seeing: the kernel is using Python from the new environment but still seeing ESMFMKFILE pointing to the base environment.

A few things that may help confirm or resolve this:

1. Check the variable inside the notebook

   import os, sys
   print(sys.prefix)
   print(os.environ.get("ESMFMKFILE"))

2. Try unsetting it before importing ESMPy

   import os
   os.environ.pop("ESMFMKFILE", None)
   import esmpy

   If that fixes the import, it confirms the inherited environment variable is the issue.

3. Ensure the kernel environment activates correctly Depending on the Jupyter setup, this might mean:

   • launching kernels via conda run -n <env> ...
   • reinstalling the kernel with python -m ipykernel install --user --name <env>
   • or setting ESMFMKFILE explicitly in the kernelspec.

4. Alternatively, clearing ESMFMKFILE in the notebook server environment (or JupyterHub spawner config) can prevent the base environment value from leaking into kernels.

If you can share the outputs of the check in (1), that would help confirm whether this is what’s happening in your setup.