Set up the icepyx gallery

[salvis2]

This repo is meant to hold the usage example notebooks for icepyx. The original effort was to integrate this into icepyx, but there were some troubles, see the PR icesat2py/icepyx#125. I am currently working on setting up the functionality on my fork.

The Pangeo Gallery is here. It has a contributor guide that I have been following to set up this repo.

Currently, I am stuck on image management. The image that we used for the IceSat-2 Hackweek is here, but has old, API-breaking changes that are not compatible with the current examples. @JessicaS11, @scottyhq, and I have been tossing around various solutions to this problem. We need a different image than previously used and I think there are two main solutions:

1. Leave the image on the IceSat-2 Hackweek image repository. We can get a new conda-lock file for the icesat2 hackweek image and release a new tag. This doesn't interfere with the image for the Zenodo release and gets a new image that we know will work quickly, but it's not in the same place that icepyx people are working.
2. Make a repo for this image in the icesat2py org. This allows for minimizing the packages involved so the image is smaller / faster, is technically the right place for this image to live, where icepyx is being developed, and is independent of the hackweek, but will be more work.

[scottyhq]

Make a repo for this image in the icesat2py org. This allows for minimizing the packages involved so the image is smaller / faster, is technically the right place for this image to live, where icepyx is being developed, and is independent of the hackweek, but will be more work.

This seems like a good idea. I recommend using this to setup a repository that builds an image with the minimal conda dependencies for icepyx https://github.com/jupyterhub/repo2docker-action

[JessicaS11]

@salvis2, I say we go with @scottyhq's suggestion to set up a repo2docker-action. Let me know if you need anything from me!

[salvis2]

@JessicaS11 , I can set up the repo. I'm assuming the only things that it will need are whatever I find in the example notebooks import statements. Let me know if there's anything else it needs.

We will need to decide whose DockerHub credentials will be used for pushing the image (probably not mine). Also would be good to create an icepyx organization on DockerHub (or we can put it somewhere else). Just want somewhere for the image to live. It could just be under your username for now. I'll set up the image repo in the next few days and ping you there but that's what I think will need doing.

[JessicaS11]

@salvis2 I think that should be good (what's in the import statements), unless we need to be more proactive for installing icepyx itself (currently that will need to be done with pip, as we're not on conda yet, so it could make sense to create the environment with needed dependencies. One of them is gdal, which tends to mess things up if you try to add it later).

I'll look into creating an icepyx organization on DockerHub, but I'm fine for it to be under my username for now.

[salvis2]

Per a suggestion by @scottyhq , I put in my own credentials and a place for an image to live under my username, and the build worked! https://github.com/salvis2/icepyx-examples-image/actions/runs/316625746

@JessicaS11 if you have credentials to put into the icepyx-gallery repo tomorrow, I'll use them to build the image and get to testing the gallery. I can test with my own image otherwise and we will just switch things later.

[salvis2]

So I tried building the notebooks and found out it doesn't have icepyx installed. It doesn't have anything that I specified in requirements.txt. I can find packages from environment.yml though. @scottyhq , do you know why it wouldn't pick up the requirements.txt file? I figured everything in the root folder would be fine: https://repo2docker.readthedocs.io/en/latest/usage.html#where-to-put-configuration-files. The configuration files documentation says it supports the file, but their example doesn't have git links.

[salvis2]

Fixed the requirements.txt issue in icesat2py/icepyx-examples-image@bd6cbd8

[salvis2]

Okay! We are getting to a good spot. The notebooks can launch with binder, start building, and import icepyx correctly at least. They are currently failing with different errors:

ICESat-2_DAAC_DataAccess2_Subsetting.ipynb

CellExecutionError: An error occurred while executing the following cell:
------------------
region_a.earthdata_login('username','email')
------------------
...
AssertionError: Enter a properly formatted email address

ICESat-2_DAAC_DataAccess_Example.ipynb

CellExecutionError: An error occurred while executing the following cell:
------------------
region_a = ipx.Query(short_name, spatial_extent, date_range)
------------------
...
AssertionError: Check that the path and filename of your geometry file are correct

ICESat-2_DEM_comparison_Colombia_working.ipynb

CellExecutionError: An error occurred while executing the following cell:
------------------
%%bash
cd ~
# git clone https://github.com/icesat2py/icepyx.git
# git clone https://github.com/ICESAT-2HackWeek/topohack.git
# git clone https://github.com/darribas/contextily.git

cd contextily
pip install -e .
cd ../topohack
pip install -e .
cd ../icepyx
pip install -e .
------------------
...
CalledProcessError: Command 'b'cd ~\n# git clone https://github.com/icesat2py/icepyx.git\n# git clone https://github.com/ICESAT-2HackWeek/topohack.git\n# git clone https://github.com/darribas/contextily.git\n\ncd contextily\npip install -e .\ncd ../topohack\npip install -e .\ncd ../icepyx\npip install -e .\n'' returned non-zero exit status 1.

As far as that last one is concerned, the three modules here should be installed via pip, so we shouldn't need to install them here.

[JessicaS11]

This is looking great @salvis2! For the last issue, I think you're right that we can take those out since we've included those resources by default in the environment. That example was originally developed to be run on an older image that didn't have those modules.

As for the other two, I'll admit I'm not entirely sure why the subsetting example is returning them but the data access example did not. Any thoughts on that?

[salvis2]

Ah well the first problem is that I pasted the wrong notebook name in second position. I've edited the comment to reflect this.

So the data subsetting notebook fails because the email provided is literally 'email'. The data access notebook fails on the query because of something about the geometry file.

The geometry file may not actually get to binder. The default setup of binderbot is to only clone files in the base repo and I while changed that for the GitHub Action, it may be a limitation for binder as well. There was another example where I had to put the data file into a notebook cell to get it to run.

[salvis2]

I'm trying a fix for the last notebook with the most obvious error but I don't think I will get all the notebooks building by myself. I'll set up a PR for all the files today and y'all can take it from here. I've opened up another issue #3 with brief instructions on finalizing the link between this repo and the gallery once everything else is ready.

[weiji14]

Just to follow up on this (after almost half a year). Am I correct in saying that the main issue is now having an email/password combination in the notebook to run the gallery example code? Was wondering if there's a way around the authentication issue, e.g. a way of caching the data or using an alternative authentication method (does NSIDC have API tokens)?