Dependencies: Relax version pinning on transitive dependencies

[amotl]

About

Locking down transitive dependencies to patch release versions and applying them as main package runtime dependency constraints isn't always the best option when aiming to cooperate with other packages.

Problem

This became apparent with dependencies of marshmallow and requests, making it difficult to use ingestr in a wider ecosystem together with other packages in the same Python environment, either as a library, or just installed side-by-side, due to package version conflicts. ¹

[...] because croud==1.13.0 depends on marshmallow==3.22.0 and ingestr==0.13.52 depends on marshmallow==3.26.1 [...], installing both together doesn't work.

Thoughts

Application builds of ingestr might take a different route, e.g. by using separate requirements.txt files that include the whole shebang, using a uv pip compile incantation that omits the --no-deps option flag.

References

• Dependencies: Relax version pinning on marshmallow #230
• Dependencies: Relax version pinning on requests #231

Footnotes

1. • See also https://github.com/crate/croud/issues/575, that it's also a note to self, i.e. we also need to address the same details on a package that took version pinning too serious. Edit: See also https://github.com/crate/croud/pull/576.
   ↩

[karakanb]

Thanks for the PR.

ingestr is primarily built and distributed as a CLI tool rather than a library. In order to merge this, we would need to figure out a way of releasing the CLI with strict dependencies while allowing a more relaxed version for library builds, for which I don't have an idea at the moment. I am happy to hear if you have any proposals, but at the moment I don't think we can move forward with this as is.

[amotl]

Hi @karakanb. Thanks for your swift reply.

ingestr is primarily built and distributed as a CLI tool [...]
we would need to figure out a way of releasing the CLI with strict dependencies while allowing a more relaxed version for library builds [...]
I am happy to hear if you have any proposals [...]

We hear you. I am happy to provide proposals to bring both details together optimally. As I am just learning ingestr, can I humbly ask you to provide pointers to where the CLI tool building and distribution takes place [if there are others than below]?

Is it mostly about building the OCI image, driven by workflows/release.yml, or are there also other distribution variants to be considered, like standalone binaries, or other variants we are currently not thinking of?

[karakanb]

@amotl thanks for helping with this. The two ways we distribute ingestr currently are:

• the docker image, as you have identified
• the Makefile commands for build and upload-release which allow us to release ingestr as a CLI tool in PyPI