feat: supporting pixi environment managing

[MoAly98]

Summary

• Adds pixi configuration for environments
• Includes pre-configured environments mirroring coffea's optional dependencies (dask, dev, parsl, rucio, triton, xrootd)

Motivation

Why

In my experience developing with coffea:

• xrootd from pip has given me runtime errors that didn't occur with the conda-forge build
• Managing multiple conda environments got slow and ate up disk space
• Pure venvs weren't flexible enough

Pixi has worked well for me, fast resolution, shared cache, and I can mix conda-forge packages (like xrootd) with an editable install of coffea.

[lgray]

Can you create a new ci task that demonstrates that the pixi configuration works?

[ikrommyd]

I'll cc @matthewfeickert here as the expert. Should we be shipping the lock file here or just that toml config? Is that what people do when providing pixi setups?

[NJManganelli]

I don’t know what conventions are, but my personal take is that the central recipe is kept, and lock files for known-good setups are also useful (I had also proposed using pixi for the in-container builds, as those could be nicely translated for direct installation too) Nick

…

On Sat, Dec 13, 2025 at 14:30 Iason Krommydas ***@***.***> wrote: *ikrommyd* left a comment (scikit-hep/coffea#1503) <#1503 (comment)> I'll cc @matthewfeickert <https://github.com/matthewfeickert> here as the expert. Should we be shipping the lock file here or just that toml config? Is that what people do when providing pixi setups? — Reply to this email directly, view it on GitHub <#1503 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJDSMOSYUR3BLNCIICN37X34BOP6HAVCNFSM6AAAAACO3G6JKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNBYHE2TCOBXGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[matthewfeickert]

Should we be shipping the lock file here or just that toml config? Is that what people do when providing pixi setups?

This is a good question that I don't think I actually have a good rule on yet for libraries. I think this comes down to a project maintainer question of "how often do we want to update the lock file"? Once a week? Once a month? Once a release?

For an application project the idea of adding a few kB or even a few MB on a regular basis from the lock file is a no-brainer "yes"

$ ls -lh ./pixi.lock
-rw-rw-r-- 1 feickert feickert 302K Dec 16 15:24 ./pixi.lock

I don't know if I feel the same way about the lock file for libraries though. While I very much understand the appeal of "we know that at every commit of this project that we can get a dev environment that works and isn't broken" is a huge boon, there's also the question of "how much do you need that if you're developing?" and "do you want to have a stable environment if your users would be getting a broken one without that lock file?".

I will need to think on this more, but I would be genuinely curious to get all of your thoughts as well.

I had also proposed using pixi for the in-container builds, as those could be nicely translated for direct installation too

@NJManganelli for sure they are useful here. I think it also makes the use of Linux containers as software deployment technologies much more transparent and helps to strip away as thinking of them as packaging technologies (which I would now argue they aren't). c.f. https://proceedings.scipy.org/articles/nwuf8465#deploying-environments-to-remote-compute

[matthewfeickert]

So this works, but this is also going to get the Python package version of all of these. If you're using Pixi and want to also leverage the conda package versions of your tools for different environment setups then you need to do some duplication work like I started to do in scikit-hep/pyhf#2638.

[matthewfeickert]

Also relevant to the discussion: scipy/scipy#23999

[lgray]

My 2 cents: Since we pin from below and track it consistently, and also do not take any rc versions or specific versions as dependencies: I don't think a lock file is really necessary.

It makes more sense in a package like pocketcoffea or other derived framework which may desire a specific version of coffea.

[matthewfeickert]

My 2 cents: Since we pin from below and track it consistently, and also do not take any rc versions or specific versions as dependencies: I don't think a lock file is really necessary.

SGTM. 👍

[lgray]

@MoAly98 Please go ahead and resolve conflicts, remove the lock file, and get things to work with conda. Please add some sort of test. :-)

[ikrommyd]

So should we have a pixi setup that is identical to the requirements from the conda forge feedstock?

[matthewfeickert]

So should we have a pixi setup that is identical to the requirements from the conda forge feedstock?

The conda-forge feedstock recipe is only providing the base library requirements. For library development you're going to want to have the equivalent of extras and dependency groups working via multiple environments. I'll try to get scikit-hep/pyhf#2638 finished soon so that there's maybe a reference working example in the Scikit-HEP ecosystem, but I think that is more what the target should be like than the conda-forge recipe.

[andrzejnovak]

I think you'd want it to just fetch the requirements you have in pyproject. I have been testing locally for mplhep with sth like

[tool.pixi.workspace]
name = "mplhep"
authors = ["andrzejnovak <novak5andrzej@gmail.com>"]
description = "Matplotlib styles for HEP"
channels = ["conda-forge"]
platforms = ["linux-64", "osx-64", "osx-arm64", "win-64"]

[tool.pixi.pypi-dependencies]
mplhep = { path = ".", editable = true, extras = ["dev", "test", "docs"] }

You could probably scope it for CI usage into different envs, but as a main dev entrypoint that pulls everything this is very easy.

[matthewfeickert]

I think you'd want it to just fetch the requirements you have in pyproject. I have been testing locally for mplhep with sth like

[tool.pixi.workspace]
name = "mplhep"
authors = ["andrzejnovak <novak5andrzej@gmail.com>"]
description = "Matplotlib styles for HEP"
channels = ["conda-forge"]
platforms = ["linux-64", "osx-64", "osx-arm64", "win-64"]

[tool.pixi.pypi-dependencies]
mplhep = { path = ".", editable = true, extras = ["dev", "test", "docs"] }

You can for sure do that, but I guess the question to me is if you have the ability to develop with conda packages compared to Python packages, is there any reason not to (outside of a conda-forge feedstock that is known to lag behind PyPI, but even then you could just have that be a Python package and the rest conda based)?

[NJManganelli]

Is it time to come back to this?

[ikrommyd]

I would like this