Add zenodo mirroring - #297
Merged
Merged
Conversation
The relative spectral responses and the atmospheric correction LUTs are downloaded from zenodo.org. When many processing chains download them at about the same time the upstream host may start limiting the traffic, and the downloads fail. Add the possibility to mirror the data on a web location of one's own: * The 'PSP_DATA_BASE_URL' environment variable, or the 'download_base_url' configuration option that it overrides, gives the base URL to download from instead of zenodo.org. * The new 'mirror_pyspectral_data.py' script, and the 'mirror_data' function behind it, download the tarballs into the directory layout such a mirror is expected to serve. The data version is part of the path, so one mirror can serve several versions of the data at the same time, and mirroring a new version does not disturb installations still using an older one. Only missing files are downloaded, so the script can be re-run from cron. The URLs are now built by 'get_rsr_url' and 'get_rayleigh_lut_url' rather than taken from the module level constants, which keep the zenodo.org locations used when no mirror is configured. There is deliberately no fallback to zenodo.org when a mirror fails, as that would defeat the purpose and hide a broken mirror. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #297 +/- ##
==========================================
+ Coverage 91.17% 91.77% +0.60%
==========================================
Files 25 25
Lines 2673 2868 +195
==========================================
+ Hits 2437 2632 +195
Misses 236 236
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The URL the relative spectral responses and the atmospheric correction LUTs are downloaded from was logged at debug level only, so a chain running at info level showed where the data were stored but not where they came from. With a mirror configurable through 'PSP_DATA_BASE_URL' this matters: an operator could not tell from normal logs whether the mirror or zenodo.org was actually used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Member
Author
|
I have also verified this to work with Trollflow2 chains. I've used the new download script to fetch the Rayleigh LUTs and RSRs to localhost, copied them over to a S3 bucket and referenced the location via HTTPS in |
adybbroe
approved these changes
Sep 21, 2026
adybbroe
left a comment
adybbroe
left a comment
Collaborator
There was a problem hiding this comment.
Many thanks for adding this feature. And thanks for testing it also in a real-time setup. Much appreciated.
LGTM.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I've had problems with Zenodo throttling/blocking download of the lookup tables and spectral responses when multiple of my containers are updating at the same time. So I asked Claude for a way to mirror the files somewhere else. I haven't yet reviewed the changes.
Oh, and also preliminary
AGENTS.mdis here. It just desribes the repository structure.pytest pyspectralflake8 pyspectralAUTHORS.mdif not there already