conda environment

[svw26]

Hello there!

Unfortunately, I am stuck on the first step, when creating the conda environment. I get the following message

packagesNotFoundError: The following packages are not available from current channels:
  - fitsio
  - healpy
Current channels:
  - https://repo.anaconda.com/pkgs/main/osx-arm64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/osx-arm64
  - https://repo.anaconda.com/pkgs/r/noarch

Are there alternative packages that you'd recommend?

Kind regards,
Sarah

[sbailey]

Thanks for reporting! Try adding the conda-forge channel to your search path, like

conda create -c conda-forge --name desi 'numpy<2' scipy 'astropy<7' ipython jupyter \
        matplotlib numba pytest fitsio h5py healpy requests

In my personal setup I used miniforge rather than anaconda. Miniforge includes the conda-forge channel by default and it found the packages there, so hopefully that will also work for anaconda.

Please report back if that works for you and we'll either update the instructions or continue to iterate.

[svw26]

Yep, that works for me. Many thanks!

[svw26]

Hmm, me again. I can install the GitHub repositories fine, but then with install_desimodel_data I get an error that ends with:

  File "/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/desimodel/install.py", line 31, in assert_svn_exists
    raise AssertionError("svn command is not executable. Install svn to use the install script. Original Error is: '{0}'.".format(e.strerror))
AssertionError: svn command is not executable. Install svn to use the install script. Original Error is: 'No such file or directory'.

When I try conda install svn it then says

Solving environment: - 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.

and completes with

  - zipp -> python[version='>=3.12,<3.13.0a0'] -> __osx[version='>=11.0']
  - zstandard -> __osx[version='>=11.0']
  - zstd -> __osx[version='>=11.0']

Your installed version is: 14.3.1

But then, after typing svn --version, it says zsh: command not found: svn...

[sbailey]

@svw26 It looks like you are on a Mac, which doesn't come with svn pre-installed, and that's not a python package that can be installed with conda either. See https://subversion.apache.org/packages.html#osx for a few options on how to install svn / subversion.

@weaverba137 do you have any ideas for how we could support the equivalent of the svn exports with URL downloads instead of requiring svn for the operations in desimodel.install?

[weaverba137]

do you have any ideas for how we could support the equivalent of the svn exports with URL downloads instead of requiring svn

I do not. In principle it is possible to view a svn repository with plain http, but actually downloading without using svn is something I've never heard of.

[svw26]

Thanks again, @sbailey

I confirm that I have installed svn (e.g. running svn help works), and install_deismodel_data appears to have completed successfully. However, I then run into a problem with the install_redrock_templates step. Please see below for the full Traceback.

DEBUG: Reading templates from /Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates
Reading templates from ['/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates/rrtemplate-GALAXY-None-v2.6.fits', '/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates/rrtemplate-QSO-LOZ-v1.0.fits', '/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates/rrtemplate-QSO-HIZ-v1.1.fits', '/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates/rrtemplate-STAR-A-v0.1.fits', '/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates/rrtemplate-STAR-B-v0.1.fits', '/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates/rrtemplate-STAR-CV-v0.1.fits', '/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates/rrtemplate-STAR-F-v0.1.fits', '/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates/rrtemplate-STAR-G-v0.1.fits', '/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates/rrtemplate-STAR-K-v0.1.fits', '/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates/rrtemplate-STAR-M-v0.1.fits', '/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/data/templates/rrtemplate-STAR-WD-v0.1.fits']
Traceback (most recent call last):
  File "/Users/sarahwhite/anaconda3/envs/desi/bin/install_redrock_templates", line 74, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/sarahwhite/anaconda3/envs/desi/bin/install_redrock_templates", line 64, in main
    templates = redrock.templates.load_templates()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/templates.py", line 464, in load_templates
    tx = Template(filename, zscan_galaxy=zscan_galaxy,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/templates.py", line 69, in __init__
    self.flux = np.asarray(native_endian(fx['BASIS_VECTORS'].data),
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sarahwhite/anaconda3/envs/desi/lib/python3.12/site-packages/redrock/utils.py", line 39, in native_endian
    return data.byteswap().newbyteorder()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: `newbyteorder` was removed from the ndarray class in NumPy 2.0. Use `arr.view(arr.dtype.newbyteorder(order))` instead.

[sbailey]

Thanks for your patience, @svw26 . Unfortunately we are not yet compatible with numpy 2.x, which is the reason that the conda create ... command has 'numpy<2' to restrict the numpy version installed. Please double check numpy.__version__ and the conda create command to make sure that you have an environment with numpy 1.x instead of 2.x.

A similar issue applies for astropy: we are not yet compatibile with astropy 7.x, thus the restriction 'astropy<7' in the conda create command.

We are actively working on numpy 2 support (e.g. desiutil #212, #219; desispec #2471, #2474), but we haven't completely converged on that yet.

That particular "newbyteorder" incompatibility was fixed in desispec #2471, so if you really really need numpy 2.x for some other reason, the tutorials might work by installing desiutil/main and desispec/main instead, but we have not yet tested that ourselves. In the meantime, the recommendation is to use numpy 1.x.

[svw26]

Ah OK, then you may want to remove (from the instructions) the note that " you can replace those tag X.Y.Z numbers with "main" ". I had installed "main" in case the problem that I ran into with fitsio and healpy could be overcome with the most-recent versions.

I started the conda-environment creation from scratch and, happily, can say that I was able to complete the install_redrock_templates step.

However, unhappily, downloading the 'tiny_dr1' has gotten stuck at the Downloading files for healpix 23040... stage for over 24 hours. I think that it would be good if more testing by complete novices could be done, and to maybe move the note about the tutorial requiring ~40 GB of data(!) to the top of the page? (My bad for not reading all the way through, but thankfully I have a new laptop with plenty of memory left.)

[sbailey]

Thanks for your real-world user testing and suggestions.

We have tested the tutorials with main + numpy 1.x; what we haven't tested is tutorials with main + numpy 2.x because we are still fixing non-tutorial issues with numpy 2.x and our packages. Caveat: although we do our best with automated testing prior to merging updates, sometimes incompatibilities creep in, so usually "main" works but not always.

Download time: longer than 24 hours is unusually slow. In our pre-release testing, a collaborator in Columbia was able to download this dataset in 30 minutes. The desi_get_dr_subset script will skip any files that have already been downloaded, so if you are still stuck, try ^C to exit the script and re-run it. Below are the complete set of healpix 23040 files that it should eventually download (1.5 GB):

tiny_dr1/spectro/redux/iron/healpix/main/dark/230/23040/coadd-main-dark-23040.fits
tiny_dr1/spectro/redux/iron/healpix/main/dark/230/23040/hpixexp-main-dark-23040.csv
tiny_dr1/spectro/redux/iron/healpix/main/dark/230/23040/emline-main-dark-23040.fits
tiny_dr1/spectro/redux/iron/healpix/main/dark/230/23040/spectra-main-dark-23040.fits.gz
tiny_dr1/spectro/redux/iron/healpix/main/dark/230/23040/qso_mgii-main-dark-23040.fits
tiny_dr1/spectro/redux/iron/healpix/main/dark/230/23040/redux_iron_healpix_main_dark_230_23040.sha256sum
tiny_dr1/spectro/redux/iron/healpix/main/dark/230/23040/qso_qn-main-dark-23040.fits
tiny_dr1/spectro/redux/iron/healpix/main/dark/230/23040/redrock-main-dark-23040.fits
tiny_dr1/spectro/redux/iron/healpix/main/dark/230/23040/rrdetails-main-dark-23040.h5

The data under tiny_dr1/spectro/redux/iron/tiles is 17 GB and the tiny_dr1/spectro/redux/iron/zcatalog/v1/zall-pix-iron.fits is the biggest piece at 21 GB.

[svw26]

Hurray! I have reached All downloads completed successfully!
(And the total space taken up is 43 GB. )

I had to keep monitoring the download as it may have kept running into a timeout problem. Something connected to the security of the WiFi connection, I'm told, and the download speed being slow for us living in South Africa :)

[sbailey]

Glad it finally worked! We'll update the documentation based upon your experience here. Thanks for your interest in the DESI data.

I'll wait to close this ticket until we've updated the documentation:

• conda create -c conda-forge ... needed when using anaconda instead of miniconda
• comment on 'numpy<2' and 'astropy<7' restriction
• mention that svn is also required and provide link to installation instructions
• highlight that downloads are big, and that the script can be run multiple times in a row if it gets stuck