Replace astropy.utils.data.download_file with httpx

[hugobuddel]

download_file() from astropy.utils.data can lead to various Exceptions with the univie webserver, e.g.

urllib.error.ContentTooShortError: <urlopen error File was supposed to be 10713600 bytes but we only got 10713359 bytes. Download failed.

This bothers end users, and makes our tests fail unreliably.

Replacing download_file() with equivalent functionality from httpx with proper retry functions seems to 'fix' this problem. So we should use that everywhere.

Places where we use download_file():

• ScopeSim/scopesim/server/example_data_utils.py
  13:from astropy.utils.data import download_file

• ScopeSim/scopesim/server/OLD_database.py
  12:from astropy.utils.data import download_file

• ScopeSim/scopesim/effects/ter_curves_utils.py
  9:from astropy.utils.data import download_file

• Pyckles/pyckles/utils.py
  3:from astropy.utils.data import download_file

• ScopeSim_Templates/scopesim_templates/extragalactic/galaxies.py
  8:from astropy.utils.data import download_file

• irdb/MICADO/docs/example_notebooks/3_scopesim_SCAO_4mas_fv-psf.ipynb
  275: "from astropy.utils.data import download_file\n",

So this issue transcends just ScopeSim.

[teutoburg]

Maybe some of those might also be better off using pooch, especially that one notebook. See also AstarVienna/speXtra#10

[hugobuddel]

Thanks, pooch seems a nice way to fetch things. Assuming it doesn't have the same problem..

[teutoburg]

Thanks, pooch seems a nice way to fetch things. Assuming it doesn't have the same problem..

Done with #446 I guess. I forgot we already had the pooch conversation half a year ago 😂

[teutoburg]

The one in ScopeSim_Templates was removed in AstarVienna/ScopeSim_Templates#88.

[teutoburg]

The one in pyckles seems to have caused this: https://github.com/AstarVienna/ScopeSim/actions/runs/10233695289/job/28316097461

[teutoburg]

The one in the MICADO notebook was removed in AstarVienna/irdb#185

4 down, 2 to go 🚀