Streamline parcels-benchmarks

[VeckoTheGecko]

This PR reworks parcels-benchmarks in a way (I hope) is much easier to work with. Follow the README and let me know what you think.

Changes:

• Replaces the parcels_benchmarks internal package (which provided the CLI tool for adding dataset hashes etc.). Now instead:

  • An intake-xarray catalog is defined in catalogs/parcels-benchmarks/catalog.yml. The top of the file has a comment which contains the link to the ZIP to be downloaded.
    • This streamlines our approach, making it easier for the benchmarking scripts to go straight from data on disk to xarray dataset.
    • We can use other options available via intake
    • This approach allows us to get familiar with intake which will likely be used for our HPC systems after v4 is released.
  • A script (scripts/download-catalog.py) downloads the data for a catalog and takes a output_dir (both via CLI args). This uses curl to download the dataset, and then unzips all nested zip files (deleting the original zips). This script also copies the catalog file into the output_dir (which is good since the datasets in the catalog are defined relative to this catalog file).
    • If a catalog is already downloaded (i.e., if the folder already exists) its skipped
    • Pro: The use of curl here means this approach is quite transparent - one can easily see download speeds and decide to cancel
    • Con: There is no longer the concept of "known hashes" - this is something we can get back if we want in future¹
  • Pixi is used, via the setup-data task, to download all the datasets.
    • This makes our data approach much more flexible should we want to change it in future

• Requires a PARCELS_BENCHMARKS_DATA_FOLDER environment variable to be explicitly set which is then acts as the working space for the data. This environment variable is used in the download and benchmarking code.

We needed the following things to ease development:

• Download all datasets before running benchmarks

• Make it transparent the download progress of datasets

Footnotes

1. given we are the sole owners of our data sources I don't think this is a concern ↩

[VeckoTheGecko]

Not all the benchmarks are running. Once this is merged I'll fix the rest in #40 .

Let me know what you think of this @fluidnumerics-joe

[VeckoTheGecko]

Oh, and since Parcels is now a submodule I think you'll need to do git submodule update --init --recursive (if you aren't doing a fresh clone from the README)

[fluidnumerics-joe]

Is it intentional to not have the FESOM and ICON datasets in the catalog ? I'm confused about where that went.

[VeckoTheGecko]

I should have mentioned in the PR description, I was planning on having it in a future PR (wanted to avoid conflicts with the other reworking of the ingestion code)

[VeckoTheGecko]

Also I need to figure out exactly how intake integrates with Uxarray. The fact Uxarray doesn't initialise from an xarray dataset (ie has uxr.open_mfdataset as the main entry) slightly complicated things

[erikvansebille]

What's the difference between the catalogues in parcels-benchmarks and the parcels-examples? They seem to be the same now?

[VeckoTheGecko]

Yes, to be updated in a future PR (mainly focussing on the actual downloading of the datasets - will fix the catalogs and ingestion at the same time)