Package size - we are exceeding the total PyPI size limit of 10Gb

[marcus-oscarsson]

We are starting to hit the size limit, total for all packages, on PyPi. I had to remove the really old packages, that I assume that no one is using (or should not be).

Our release strategy is indeed a bit unconventional which creates quite alot of packages. However I have a feeling that our packages are quite big.

One issue is that the .hkli files are quite big. The entire /mxcubecore/configuration/ directory can probably be moved out of the package which would make the package smaller.

[rhfogh]

Specifically for the *.hkli files I shall put it on my TODO list to remove them. I guess no one else is using them, and it does indeed not make sense that two thirds of the space for the mxcubecore package is taken up by the gphl/test_samples directory. On the other hand that will still only save you 180MB, which is a far cry from the 10GB we have.

On the general issue of configuration/ files I am not sure. It is good to have the various configuration files available when you are developing (it would be even better if they were up to date, of course) even if those actually used at the beamlines live somewhere else. Would it help if configuration files were moved to mxcubeweb / mxcubeqt? Or are they needed for testing with mxcubecore only?

[marcus-oscarsson]

@rhfogh thanks for the cleanup :). Its the size of the files times the number of releases so it quickly grows large. I agree that the configuration files are useful and necessary for the tests. However they don't need to be part of the mxcubecore package, they can be kept in the root like for mxcubeweb. I also agree that a cleanup and assessment of which configuration files that are actually needed would be very welcome. I think we mentioned it before but that could perhaps come as a step in the YAML work, as we need to update all the configuration files at some point. right ?

[fabcor-maxiv]

180MB

Oh wow! Indeed, I did not know.

Well it is 180 MB uncompressed. The .tar.gz and the .whl files that we upload to PyPI are luckily compressed so it is somewhat smaller, but still 30+ MB per file, so 60+ MB per release.

It should be possible and easy to exclude those files from the distribution archives that we upload on PyPI. This is the documentation on how to exclude files with Poetry.

But of course, if it makes sense to remove those file from the git repository itself, then let's do that instead.

[rhfogh]

I shall put it no my TODO list and get to it as soon as I have time - after the Trieste meeting.

[fabcor-maxiv]

Potential fix: #1106.