Update develop-ref after dtcenter/METplotpy#576#3288
Merged
georgemccabe merged 1445 commits intodevelop-reffrom Apr 21, 2026
Merged
Update develop-ref after dtcenter/METplotpy#576#3288georgemccabe merged 1445 commits intodevelop-reffrom
georgemccabe merged 1445 commits intodevelop-reffrom
Conversation
* new branch testing * marine and cryosphere datasets and workflow * medium_range datasets and workflow * pbl and precep files for datasets and workflow * s2s_mjo and s2s files datasets and workflow * Per #2741, updated METplus Workflow section * first 5 short_range files * Per #2741, rearranged and modified information for consistency. * short_range directories starting with m or p * short_range/UserScript files * space_weather files * tc_and_extra_tc and unstructured_grids files * removing extra # * fixes Julie suggested * more fixes and new updates * second pass of short_range files * last directories clean up * Tidy up UGRID use case doc file. * Minor changes to tc_and_extra_tc docs. * Minor changes to space_weather docs. * Minor changes to short_range docs. * Minor changes to s2s_mjo docs. * Minor changes to s2s docs. * Minor changes to precipitation docs. * Minor changes to pbl docs. * Minor changes to medium_range docs. * Minor changes to marine_and_cryo docs. * Clarifying on climo dataset. * Adds INIT_END and INIT_INCREMENT to workflow. * Removes unused conf item. * Adds other timing control elements for completeness. * Removes unused conf item. * Removes unused conf item and switches min/max for readability. * Adds additional timing control conf items for completeness. * INIT_END is not set so changed to None. * Removes 2014 for VALID_BEG because it actually uses data from 2016, and updated the information to reflect that. * Adds other timing control for consistency. * Adds other timing control for consistency. * Adds other timing control for consistency. * Clarifies that despite LEAD_SEQ not being set, it does use multiple forecast leads because it uses output from another use case. * Clean up CyclonePlotter METplus Workflow. * Adds clarity about the 60h lead time. * Adds the best description of timing control I could think of, since it's a bit confusing for the TC- tools. * Adds timing info for more clarity and consistency with template. * Per #2741, correcting formatting --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> Co-authored-by: Daniel Adriaansen <dadriaan@ucar.edu>
This reverts commit d762570.
* Per #2697, update conda environment creation scripts to create v6.1 environments that use Python 3.12.0. Specified python package versions to match what was used to create METbaseimage, but left the rest of the packages versionless to allow mamba/conda to install the desired versions. Once envs have successfully been created, I will need to go back and add the version numbers that were used to make sure future envs will be created consistently * change how cartopy feature download script is called -- see SciTools/cartopy#2263 * get older set of tar_files that includes jasper version needed to install gfdl tracker * revert back to using py3.10.4 for spacetime environment because pyngl is not supported in py3.12.0 * remove args now that SSL cert for dtcenter site is renewed * Per #2697, update version of conda environments to use envs created using python 3.12.0 * added scipy to python embedding environment because it is now a requirement of MET Python Embedding * remove deprecated import * update swpc env script * add option to specify branch to use for METplotpy/METcalcpy/METdataio * add workflow dispatch inputs to override METplotpy/METcalcpy/METdataio branch for use case testing, changed default values to empty string to determine develop or main_vX.Y * simplify METplus Analysis branch override * do not pull METplus image from DockerHub before building to see if it will properly pull the latest changes from MET that are not being incorporated in the METplus image * fix warnings in python script * fix error using numpy 2.X * added multi-variate mode quick search keyword so use case is more discoverable * updated config settings with new variables * add back pull and use cache of previously generated docker image since I determined it was not the cause of the MET content not being updated properly * revert back override of MET image * changes to appease SonarQube * removed duplicate entries * rearrange args to appease SonarQube
* fix sonarqube findings * update test to ensure it fails as expected even if the OUTPUT_BASE is set in the defaults * force https protocol to satisfy SQ * copy explicit files/dirs into Docker image instead of using a wildcard to satisfy SonarQube -- tested that METplus unit tests can still run successfully instead a Docker container that was created using the copy method
* Use case config file from @briannen. * Documentation template for this use case. * Changes GRIBv2 record numbers to match subsetted forecast files. * Updates to add new use case to automation. * Updates input locations to the correct place where sample data are stored. * Adds use case gallery image. * Removes apostrophe from Point2GridConfig_wrapped MET config file and adds literalinclude of this file to the Point2Grid met_tool_wrapper use case documentation. * Use case documentation file. * Updates to remove using grib record numbers because they could vary, which necessitated using three FCST/OBS vars for each cloud level. Also used CUSTOM_LOOP_LIST to process both goes16 ans goes18 for each run, and switched to only a single forecast for demonstration purposes. * Changes docker attempts to 20 instead of 10. * Minor tweaks to docs from PR review.
* fix unit test to work with python 3.12 and 3.10 * per #2697, update python version to 3.12 for GHA workflows, create python requirements files for 3.12 while preserving 3.10 versions to allow for testing of multiple versions * update recommended python version to 3.12.0 * per #2697, run unit tests using both py3.10 and py3.12 * add missing comma * add creation of conda env used for development for METplus wrappers (includes METplus wrappers requirements, unit test reqs, and documentation reqs) * use scripts for python ersions to ensure 3.10 is not truncated to 3.1 * update version of python to use for RTD to 3.12 * added version numbers for the python package requirements that were used when they were created without specifying versions to ensure that the same environment will be created if the env is regenerated * added instructions to create the metplus_dev environment so it can easily be created locally * update required python for install * update conda environments for use case tests to use versions that were created with python 3.12 * disable spacetime use case because pyngl is not supported in python 3.12 * change MJO use cases to use mp_analysis conda env instead of spacetime * update python package versions * install METplotpy and METcalcpy automatically if mp_analysis environment is used * use conda to clone an existing conda env because the --clone argument doesn't appear to work using a newer version of mamba (v2.0)
* Add NRL GFS-ASOS use case * Moves use case from met_tool_wrapper to model_applications and renames use case conf file and directoy. * Latest versions to use. * Changes variable names to have stn_ prefix since latitude and longitude were being overwritten with zeros, and also multiples cloud fraction by 100 to make it a percentage to match the forecast. * Adds checking to see if it's a fraction variable or not before converting to percent (assuming the other variable is altitude), and simplifies the height/level used so that observation is a single value from the obs file. * Scripts to read GFS data with Python embedding and interpolate vertically from cloud base pressure to cloud base height. * Latest changes for Python embedding of forecast data to compare cloud base height with ASOS, add MPR output, and shift around various settings to the config subgroups. * Removes INIT_INCREMENT because only one INIT time is being processed. * Turns off MPR output. * Adds reading of topography from the model to convert the geopotential height field from meters MSL to meters AGL to compare with observations. * Adds valid time to obs filename so it dynamically selects the correct date for the input file. * Adds use case documentation gallery image. * Renames gallery image filename and updates path in docs file. * Changes to run for a single lead time, and removes python embedding for the forecast for low_cloud_base_altitude in favor of the ceiling field from GFS. Based on comments in NOAA's UPP code, ceiling is the cloud base height for cloud fraction > 50%, and this variable is from the local surface rather than from mean sea level. If no ceiling is found a value of 20,000m is used. * Fixes incorrect quotes in the use case documentation template for the gallery image path. * Fixes incorrect quotes in sphinx gallery image file path. * Switches back to Python embedding for the low_cloud_base_altitude since bonafide ceiling was not available in the METAR obs file we are using. * GHA/CI integration after adding sample data. * Updates to input dirs. * Correct input directory for obs. * Avoids namespace conflict with standard conf item. * Adjusts documentation for correct forecast leads. * Removes MET_PYTHON_EXE from config. * Removes verbosity settings from testing. * Removes setting OBTYPE. * Adds more comments to interpolation function. * Adds many comments about read_asos_ceil.py. * Adds info about the third Python embedding script gfs_025_interp_funcs.py. --------- Co-authored-by: Tracy Hertneky <hertneky@ucar.edu> Co-authored-by: j-opatz <jopatz@ucar.edu>
* initial implementation of conda recipe to install MET executables and METplus wrappers * add setup tools to fix incorrect METplus wrappers on install * prevent conflict when isatty is defined in multiple places * update version of MET to 12.0.1 to include pb2nc bugfix * remove specific versions of compilers that are no longer needed * use printf to properly add line break * turn off fortran optimization to prevent issues with NHC-provided code * remove setting z lib flag because it shouldn't be necessary to compile MET * remove fortran flags that are not needed * skip if attempting to build on windows * explicitly use gcc 12 for linux because atlas fails to compile using gcc 14. Clean up comments to be more clear why dependencies are needed * update website URL and change maintainer to GitHub username * prep for bugfix release * update MET v12.0.2 hashes * remove unused env vars and sym link of ar * add back env vars that are needed, clean up * set env var to easily access the parm directory for use case files * specify version for libopenblas because latest version 0.3.29 caused eckit to not find the library and created additional library dependencies that were not listed in the recipe * added more about info and changed maintainers item to correct name * update license info * add tests to ensure MET executables and required python imports are available * clean up comments so they can be easily removed when submitting to conda-forge * formatted changes based on conda-forge linter * add cmake because it was missing from conda-forge build * specify different dynamic library name for netcdf C++ to match what is available for osx-64 * change specific gcc compiler version line * revert change to try to handle differing netcdf c++4 dynamic library names * change wget to curl because wget is not available in conda-forge env * formatting for conda forge linter * skip stdlib c for osx-arm64 to prevent error, get latest netcdf-cxx4 to fix issue with lib naming on osx-64, specify python version as 3.10 * add license files for library dependencies * specify linux compiler versions * fixed typo in fortran version spec * add make and pip to build reqs * fix c stdlib logic by specifying versions and names, move pip to host reqs, capitalize True for skip windows * add install of run_metplus.py in path * clean up comments to be more accurate * changed format of exact version requirements * add instructions to build/test a conda recipe * set parm base based on python version * specify python version and numpy requirements * changes suggested by @JohnHalleyGotway in the pull request review * made formatting fixes suggested by @jprestop in the PR review * per suggestion from @jprestop code review, moved Conda Recipe chapter up to be chapter 3 * update c_stdlib verisons to match https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml * updates based on feedback from conda-forge PR * get eckit/atlas from metview on conda-forge instead of installing it * change tests back to checking if exes exist because they return 1 when run without arguments and cause a failure * add description
* Initial commit of Use Case * Adding use case image * Updating documentation paths * Updating processing * Fixed typo * Updated documentation typos * Trying to get use case tests to run * Fixed typo in use case name * Turned on MODE run for testing * More testing * Updating data path * Trying updated docker settings * Updating text in .conf file * Testing time for all mode runs * Moving use case to new area * Adding masking to the use case * Fixing mask location * Adding CI overrides * Updating process list * Fixing Wrong lead times * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * Update docs/use_cases/model_applications/clouds/GridStat_fcstGFS_obsGFS_cloudFracLayer.py Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu> * updating to be clouds #8 * Update use_case_groups.json * Updating to be clouds #8 * Some refactoring of the UserScript section. * Missed an indent. * Removing line with space confusing the RST formatter. * Minor clarification about three MODE entries and only one GridStat for the PROCESS_LIST. * Fixed error in use_case_groups.json * Updates to fix one sonar cube error * Changing use case to false * Minor changes to UserScript wording. * Addresses logger variable formatting issue. --------- Co-authored-by: Christina Kalb <kalb@seneca.rap.ucar.edu> Co-authored-by: Dan Adriaansen <dadriaan@ucar.edu>
…ons to install METplus analysis tools in conda envs locally and consolidated them in one place to reduce redundant instructions
* Per #2771, attempting to add MET User's Guide to ToC * Per #2771, updating link to MET User's Guid and adding link for MET Contributor's Guide * Per #2771, adding COMPONENTS section * Per #2771, playing with menu options * Per #2771, playing with menu options * Per #2771, try to add scroll bar functionality to ToC * Per #2771, attempt two to add scroll bar * Per #2771, attempt three add scroll bar for ToC * Per #2771, attempt four to get scroll bar for ToC * Per #2771, attempt to make logo area sticky * Per #2771, attempt to fix problems with sticky logo * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, attempting to fix overlap of sticky logo over ToC * Per #2771, ensures that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, attempt 2 - ensure that when an anchor is clicked, the selected element is not hidden behind the sticky header * Per #2771, menu updates * Per #2771, hide the sticky logo so that it behaves like the original (disappearing into the hamburger menu) * Per #2771, replace reference to feature branch with latest
* Update config file to match updated data * redirect output to OUTPUT_BASE/tmp_500hPa.png because the refactored script will save the output to the directory where the command was invoked. This may cause issues when running the use case in a CI environment * Update config file to match updated data * Replaced plot with new plot using updated data. * Updated plot using latest data * Updated for refactored code that doesn't require additional args from the command line. Also redirect output to the OUTPUT_BASE directory because the plotting script stores the output to the directory from where the script was invoked. * updates to conf files for the vertical profile plotting * Updated thumbnails for new physics tendency plots generated from new data * updates made to reflect changes due to new data * Modified instructions on location of output file * remove the redirection command, it does not work * Update UserScript_fcstFV3_fcstOnly_PhysicsTendency_Planview.py removed text that was erroneously copied * pass output directory to script to write output in correct location * remove unused imports --------- Co-authored-by: George McCabe <23407799+georgemccabe@users.noreply.github.com>
* sq fix * move duplicate script into common directory and update conf files to call script from new location -- removed duplicate code SQ complaints * handle SQ complaints for duplicate code and code smells * fix paths to new locations of common scripts in documentation * refactor to resolve SonarQube complaints * more SQ - specify exception and start to reduce cognitive complexity * clean up SQ complaints * remove a few more code smells * more code smell cleanup
…nd remove obs_window for PairStat wrapper (#2945)
…her python version unit test if one fails
* add release notes * update version for beta2 release * update beta2 release date * update info about DockerHub images for beta2 * update date of release * Update docs/Users_Guide/release-notes.rst Co-authored-by: Julie Prestopnik <jpresto@ucar.edu> * per suggestion from @jprestop, add info about python version upgrade to the upgrade instructions * disable doxygen by default * fix formatting of note --------- Co-authored-by: Julie Prestopnik <jpresto@ucar.edu>
* Per #2953, update Orion file for Python 3.12 * Per #2953, updating Existing Builds page for installation on Orion * Per #2953, updating Casper file for Python 3.12 * Per #2953, adding script for METplus-6.1 using Python 3.12 * Per #2953, updated installation information for Casper * Per #2953, updated compiler for Orion * Per #2953 updating compiler in Existing Builds for Orion * Per #2953, added qhull needed for atlas on hera * Per #2953, remove qhull as it was actually causing problems on hera * Per #2953, updating hera file for Python 3.12 * Per #2953, updating with hera information * Per #2953, updating jet file for Python 3.12 * Per * Per #2953, adding internal/scripts/installation to paths-ignore
* add quotation marks around file path if it includes spaces * update use case to read WRF fire field on subgrid directly instead of using Python Embedding * update instructions for updating web server data to use MET scripts * add a test for python embedding input * fix issue introduced where double quotes are added * prevent incorrect error when running PyEmbedIngest * fix URL link * update use case doc to note that Python Embedding is no longer used * remove keyword to set up Python Embedding in automated tests because it is no longer used * per PR review suggestion from Copilot, add comment to describe why variable is set * override output units/level Co-authored-by: John Halley Gotway <johnhg@ucar.edu> * fix formatting of set_attr options --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
… fixes/enhancements to diff util are used for automated testing (#3242)
…endering in the web UI. Consider adding a verbose option to get this information if desired
…it can be used by other components besides METplus
…ith --diff_dir. Also changed logic to remove diff file from 'output' directory after copying it to the diff_dir unless a command line override prevents it
* Per MET #3362, add logic to diff_util.py to check variable attributes for diffs. * Enhance logic to report differences in NetCDF attributes -- which attributes are found in one and not the other and which attributes have different values. Also continue to check for differences in data even if attributes differ * refactor to reduce cognitive complexity per SQ * added unit tests to test new diff util logic that compares NetCDF attributes --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* Per MET #3362, add logic to diff_util.py to check variable attributes for diffs. * Enhance logic to report differences in NetCDF attributes -- which attributes are found in one and not the other and which attributes have different values. Also continue to check for differences in data even if attributes differ * refactor to reduce cognitive complexity per SQ * added unit tests to test new diff util logic that compares NetCDF attributes * Improve NetCDF diff logic to properly handle attributes that are numpy arrays, NaN values. Only log NC variable info if there is a failure or if METPLUS_DIFF_VERBOSE is set. Added more unit tests to demonstrate tests that incorrectly failed in the METplus use case runs * allow override of dummy netcdf fill value * fix logic to handle NetCDF attribute diffs when nans are present * update versions of actions to resolve Node.js 20 action deprecation warnings * update version of action to scan docker images to resolve Node.js 20 deprecation --------- Co-authored-by: John Halley Gotway <johnhg@ucar.edu>
* adding use case material * corrected python env * updated MET path for masking * updated MET path again to be local specific * removed unused code * Apply suggestions from code review Co-authored-by: Andy Newman <anewman89@users.noreply.github.com> * adding last docs update --------- Co-authored-by: Andy Newman <anewman89@users.noreply.github.com>
…sting has completed with the next beta since that version will not be released in favor of switching all components to the same X.Y version going forward
…ifacts because of a bug where downloads can fail silently -- see actions/download-artifact#396
…ig var with a double slash does not properly preserve the double slash when requested, e.g. for a URL. Apply fix to allow the test to pass. (#3276)
github-project-automation
Bot
moved this from 🩺 Needs Triage
to 🏁 Done
in METplus-13.0 Development
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
12 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dtcenter/METplotpy#576 replaced plotly with matplotlib in METplotpy which caused some slight formatting differences in the plots generated by a few use cases
Created by @georgemccabe