Skip to content

Conversation

@lee1043
Copy link
Contributor

@lee1043 lee1043 commented May 5, 2025

Description

Checklist

Please confirm that this pull request has done the following:

  • Tests added
  • Documentation added (where applicable)
  • Changelog item added to changelog/

@lee1043 lee1043 mentioned this pull request May 5, 2025
3 tasks
* origin/main: (38 commits)
  Add changelog
  Bump sample data to v0.5.1
  Add regression test data
  chore: Remove old migration for metric_value columns
  test: Ensure that run is correctly mocked
  chore: Fix last test
  chore: Update cmec metric bundles
  revert: Roll back changing the data requirements for the modes of variability
  chore: revert some changes to avoid to many changes in this PR
  test: Add tests for remove_dimensions
  test: Fix tests
  fix: Add additional facets to variability modes
  fix: Add additional facets to annual_cycle
  chore: Remove excess dimensions and replace with selectors
  chore: Remove excess dimensions and replace with selectors
  test: Add ilamb integration test
  feat: Add a prepend routine
  chore: Check that metric bundles have the correct dimensions
  chore: Reorder functions
  docs: Changelog
  ...
@lewisjared
Copy link
Contributor

lewisjared commented May 18, 2025

@lee1043 This doesn't work for me. The regridding is always failing. I've used the decimated and complete files with the same errors.

Have you seen this before?

PET0.ESMF_LogFile

20250518 124932.399 ERROR            PET0 ESMF_FieldRegrid.F90:4605 checkGrid Invalid argument  -  some types of regridding (e.g. bilinear) are not supported on Grids that contain a DE of width 1.
20250518 124932.401 ERROR            PET0 ESMF_FieldRegrid.F90:3246 b_or_p_GridToMesh Invalid argument  - Internal subroutine call returned Error
20250518 124932.401 ERROR            PET0 ESMF_FieldRegrid.F90:1404 getMeshWithNodesOnFieldLoc Invalid argument  - Internal subroutine call returned Error
20250518 124932.401 ERROR            PET0 ESMF_FieldRegrid.F90:1026 ESMF_FieldRegridStoreNX Invalid argument  - Internal subroutine call returned Error
20250518 124932.401 ERROR            PET0 ESMF_Field_C.F90:1137 f_esmf_regridstore Invalid argument  - Internal subroutine call returned Error
20250518 124932.401 ERROR            PET0 ESMCI_Field.C:1468 ESMCI::Field::regridstore() Invalid argument  - Internal subroutine call returned Error
20250518 124932.401 ERROR            PET0 ESMC_Field.C:449 ESMC_FieldRegridStore() Invalid argument  - Internal subroutine call returned Error
20250518 124934.560 ERROR            PET0 ESMF_FieldRegrid.F90:4605 checkGrid Invalid argument  -  some types of regridding (e.g. bilinear) are not supported on Grids that contain a DE of width 1.
20250518 124934.560 ERROR            PET0 ESMF_FieldRegrid.F90:3246 b_or_p_GridToMesh Invalid argument  - Internal subroutine call returned Error
20250518 124934.560 ERROR            PET0 ESMF_FieldRegrid.F90:1404 getMeshWithNodesOnFieldLoc Invalid argument  - Internal subroutine call returned Error
20250518 124934.560 ERROR            PET0 ESMF_FieldRegrid.F90:1026 ESMF_FieldRegridStoreNX Invalid argument  - Internal subroutine call returned Error
20250518 124934.560 ERROR            PET0 ESMF_Field_C.F90:1137 f_esmf_regridstore Invalid argument  - Internal subroutine call returned Error
20250518 124934.560 ERROR            PET0 ESMCI_Field.C:1468 ESMCI::Field::regridstore() Invalid argument  - Internal subroutine call returned Error
20250518 124934.560 ERROR            PET0 ESMC_Field.C:449 ESMC_FieldRegridStore() Invalid argument  - Internal subroutine call returned Error
20250518 124937.536 ERROR            PET0 ESMF_FieldRegrid.F90:4605 checkGrid Invalid argument  -  some types of regridding (e.g. bilinear) are not supported on Grids that contain a DE of width 1.
20250518 124937.536 ERROR            PET0 ESMF_FieldRegrid.F90:3246 b_or_p_GridToMesh Invalid argument  - Internal subroutine call returned Error
20250518 124937.536 ERROR            PET0 ESMF_FieldRegrid.F90:1404 getMeshWithNodesOnFieldLoc Invalid argument  - Internal subroutine call returned Error
20250518 124937.536 ERROR            PET0 ESMF_FieldRegrid.F90:1026 ESMF_FieldRegridStoreNX Invalid argument  - Internal subroutine call returned Error
20250518 124937.536 ERROR            PET0 ESMF_Field_C.F90:1137 f_esmf_regridstore Invalid argument  - Internal subroutine call returned Error
20250518 124937.536 ERROR            PET0 ESMCI_Field.C:1468 ESMCI::Field::regridstore() Invalid argument  - Internal subroutine call returned Error
20250518 124937.536 ERROR            PET0 ESMC_Field.C:449 ESMC_FieldRegridStore() Invalid argument  - Internal subroutine call returned Error
20250518 124958.061 INFO             PET0  Finalizing ESMF

### PMP ENSO: Compute the metric collection ###

JSON file created: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/input_ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_processed.json
     ComputeCollection: metric = BiasSstLonRmse
     ComputeMetric: oneVarRMSmetric, BiasSstLonRmse = ACCESS-ESM1-5_r1i1p1f1 and ERA-Interim
          BiasSstLonRmse
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ts/gn/v20191115/ts_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_200001-201412.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                         shape 1: (temperature) (180, 1, 13)
                    after ReadAreaSelectRegion
                         areacell is None 
                    after ReadLandmaskSelectRegion
                         axes list 1: (temperature) ['lat', 'lon']
                         shape 1: (temperature) (1, 13)
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/obs4REF/obs4REF/ECMWF/ERA-INT/mon/ts/gn/v20210727/ts_mon_ERA-INT_PCMDI_gn_197901-201903.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('1979-1-16 12:0:0.0', '2018-12-16 12:0:0.0')
                         shape 1: (temperature) (480, 13, 161)
                    after ReadAreaSelectRegion
                         areacell is None 
                         NOTE: Estimated landmask applied
/Users/jared/code/Climate-REF/climate-ref/.ref/software/conda/pmp-09fc79daa7e05ca0dfac8feba9246e4bb5045381/lib/python3.10/site-packages/share/cdutil/navy_land.nc
     ComputeCollection: metric = BiasTauxLonRmse
     ComputeCollection: ENSO_proc, metric BiasTauxLonRmse not computed
          reason(s):
           no modeled taux given
'modelFile2'
     ComputeCollection: metric = EnsoAmpl
     ComputeMetric: oneVarmetric = ACCESS-ESM1-5_r1i1p1f1
          EnsoAmpl
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ts/gn/v20191115/ts_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_200001-201412.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                         shape 1: (temperature) (180, 1, 6)
                    after ReadAreaSelectRegion
                         areacell is None 
                    after ReadLandmaskSelectRegion
                         axes list 1: (temperature) ['lat', 'lon']
                         shape 1: (temperature) (1, 6)
               after PreProcessTS
                    axes list 1: (sst) ['time']
                    time bounds 1: (sst) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                    shape 1: (sst) (180,)
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ts/gn/v20191115/ts_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_200001-201412.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                         shape 1: (temperature) (180, 3, 17)
                    after ReadAreaSelectRegion
                         areacell is None 
                    after ReadLandmaskSelectRegion
                         axes list 1: (temperature) ['lat', 'lon']
                         shape 1: (temperature) (3, 17)
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ts/gn/v20191115/ts_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_200001-201412.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                         shape 1: (temperature) (180, 1, 13)
                    after ReadAreaSelectRegion
                         areacell is None 
                    after ReadLandmaskSelectRegion
                         axes list 1: (temperature) ['lat', 'lon']
                         shape 1: (temperature) (1, 13)
          after PreProcessTS
               axes list 1: (sst1) ['time', 'lat', 'lon']
               time bounds 1: (sst1) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
               shape 1: (sst1) (180, 3, 17)
               axes list 2: (sst2) ['time', 'lat', 'lon']
               time bounds 2: (sst2) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
               shape 2: (sst2) (180, 1, 13)
ESMC_FieldRegridStore failed with rc = 6. Please check the log files (named "*ESMF_LogFile").
     ComputeCollection: metric = EnsodSstOce_2
     ComputeCollection: ENSO_proc, metric EnsodSstOce_2 not computed
          reason(s):
           no modeled thf given
     ComputeCollection: metric = EnsoFbSshSst
     ComputeCollection: ENSO_proc, metric EnsoFbSshSst not computed
          reason(s):
           no modeled ssh given
     ComputeCollection: metric = EnsoFbSstTaux
     ComputeCollection: ENSO_proc, metric EnsoFbSstTaux not computed
          reason(s):
           no modeled taux given
     ComputeCollection: metric = EnsoFbSstThf
     ComputeCollection: ENSO_proc, metric EnsoFbSstThf not computed
          reason(s):
           no modeled thf given
     ComputeCollection: metric = EnsoFbTauxSsh
     ComputeCollection: ENSO_proc, metric EnsoFbTauxSsh not computed
          reason(s):
           no modeled taux given
           no modeled ssh given
     ComputeCollection: metric = EnsoSeasonality
     ComputeMetric: oneVarmetric = ACCESS-ESM1-5_r1i1p1f1
          EnsoSeasonality
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ts/gn/v20191115/ts_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_200001-201412.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                         shape 1: (temperature) (180, 1, 6)
                    after ReadAreaSelectRegion
                         areacell is None 
                    after ReadLandmaskSelectRegion
                         axes list 1: (temperature) ['lat', 'lon']
                         shape 1: (temperature) (1, 6)
               after PreProcessTS
                    axes list 1: (sst) ['time']
                    time bounds 1: (sst) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                    shape 1: (sst) (180,)
               after SeasonalMean
                    axes list 1: (sst_NDJ) ['time']
                    shape 1: (sst_NDJ) (14,)
                    axes list 2: (sst_NDJ) ['time']
                    shape 2: (sst_NDJ) (15,)
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ts/gn/v20191115/ts_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_200001-201412.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                         shape 1: (temperature) (180, 3, 17)
                    after ReadAreaSelectRegion
                         areacell is None 
                    after ReadLandmaskSelectRegion
                         axes list 1: (temperature) ['lat', 'lon']
                         shape 1: (temperature) (3, 17)
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ts/gn/v20191115/ts_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_200001-201412.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                         shape 1: (temperature) (180, 1, 13)
                    after ReadAreaSelectRegion
                         areacell is None 
                    after ReadLandmaskSelectRegion
                         axes list 1: (temperature) ['lat', 'lon']
                         shape 1: (temperature) (1, 13)
          after PreProcessTS: netcdf
               axes list 1: (sst1) ['time', 'lat', 'lon']
               time bounds 1: (sst1) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
               shape 1: (sst1) (180, 3, 17)
               axes list 2: (sst2) ['time', 'lat', 'lon']
               time bounds 2: (sst2) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
               shape 2: (sst2) (180, 1, 13)
               axes list 3: (sst3) ['time', 'lat', 'lon']
               time bounds 3: (sst3) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
               shape 3: (sst3) (180, 1, 13)
               after SeasonalMean: netcdf
                    axes list 1: (sst1_NDJ) ['time', 'lat', 'lon']
                    shape 1: (sst1_NDJ) (14, 3, 17)
                    axes list 2: (sst1_MAM) ['time', 'lat', 'lon']
                    shape 2: (sst1_MAM) (15, 3, 17)
                    axes list 3: (sst3_NDJ) ['time', 'lat', 'lon']
                    shape 3: (sst3_NDJ) (14, 1, 13)
                    axes list 4: (sst3_MAM) ['time', 'lat', 'lon']
                    shape 4: (sst3_MAM) (15, 1, 13)
ESMC_FieldRegridStore failed with rc = 6. Please check the log files (named "*ESMF_LogFile").
     ComputeCollection: metric = EnsoSstLonRmse
     ComputeMetric: oneVarRMSmetric, EnsoSstLonRmse = ACCESS-ESM1-5_r1i1p1f1 and ERA-Interim
          EnsoSstLonRmse
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ts/gn/v20191115/ts_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_200001-201412.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                         shape 1: (temperature) (180, 1, 6)
                    after ReadAreaSelectRegion
                         areacell is None 
                    after ReadLandmaskSelectRegion
                         axes list 1: (temperature) ['lat', 'lon']
                         shape 1: (temperature) (1, 6)
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/obs4REF/obs4REF/ECMWF/ERA-INT/mon/ts/gn/v20210727/ts_mon_ERA-INT_PCMDI_gn_197901-201903.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('1979-1-16 12:0:0.0', '2018-12-16 12:0:0.0')
                         shape 1: (temperature) (480, 13, 67)
                    after ReadAreaSelectRegion
                         areacell is None 
                         NOTE: Estimated landmask applied
/Users/jared/code/Climate-REF/climate-ref/.ref/software/conda/pmp-09fc79daa7e05ca0dfac8feba9246e4bb5045381/lib/python3.10/site-packages/share/cdutil/navy_land.nc
     ComputeCollection: metric = EnsoSstSkew
     ComputeMetric: oneVarmetric = ACCESS-ESM1-5_r1i1p1f1
          EnsoSstSkew
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ts/gn/v20191115/ts_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_200001-201412.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                         shape 1: (temperature) (180, 1, 6)
                    after ReadAreaSelectRegion
                         areacell is None 
                    after ReadLandmaskSelectRegion
                         axes list 1: (temperature) ['lat', 'lon']
                         shape 1: (temperature) (1, 6)
               after PreProcessTS
                    axes list 1: (sst) ['time']
                    time bounds 1: (sst) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                    shape 1: (sst) (180,)
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ts/gn/v20191115/ts_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_200001-201412.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                         shape 1: (temperature) (180, 3, 17)
                    after ReadAreaSelectRegion
                         areacell is None 
                    after ReadLandmaskSelectRegion
                         axes list 1: (temperature) ['lat', 'lon']
                         shape 1: (temperature) (3, 17)
                    Files
                         file name 1: (temperature) /Users/jared/code/Climate-REF/ref-sample-data/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ts/gn/v20191115/ts_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_200001-201412.nc
                         variable name 1: (temperature) ts
                    after ReadSelectRegionCheckUnits
                         axes list 1: (temperature) ['time', 'lat', 'lon']
                         time bounds 1: (temperature) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
                         shape 1: (temperature) (180, 1, 13)
                    after ReadAreaSelectRegion
                         areacell is None 
                    after ReadLandmaskSelectRegion
                         axes list 1: (temperature) ['lat', 'lon']
                         shape 1: (temperature) (1, 13)
          after PreProcessTS
               axes list 1: (sst1) ['time', 'lat', 'lon']
               time bounds 1: (sst1) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
               shape 1: (sst1) (180, 3, 17)
               axes list 2: (sst2) ['time', 'lat', 'lon']
               time bounds 2: (sst2) ('2000-1-16 12:0:0.0', '2014-12-16 12:0:0.0')
               shape 2: (sst2) (180, 1, 13)
ESMC_FieldRegridStore failed with rc = 6. Please check the log files (named "*ESMF_LogFile").
INFO::2025-05-18 12:49::pcmdi_metrics:: Results saved to a json file: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1.json
2025-05-18 12:49:52,845 [INFO]: base.py(write:429) >> Results saved to a json file: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1.json
2025-05-18 12:49:52,845 [INFO]: base.py(write:429) >> Results saved to a json file: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1.json
INFO::2025-05-18 12:49::pcmdi_metrics:: Results saved to a json file: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_diveDown.json
2025-05-18 12:49:58,055 [INFO]: base.py(write:429) >> Results saved to a json file: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_diveDown.json
2025-05-18 12:49:58,055 [INFO]: base.py(write:429) >> Results saved to a json file: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_diveDown.json
metrics: ['BiasSstLonRmse', 'BiasTauxLonRmse', 'EnsoAmpl', 'EnsodSstOce_2', 'EnsoFbSshSst', 'EnsoFbSstTaux', 'EnsoFbSstThf', 'EnsoFbTauxSsh', 'EnsoSeasonality', 'EnsoSstLonRmse', 'EnsoSstSkew']
met: BiasSstLonRmse
filename_nc: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_BiasSstLonRmse.nc
file not found: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_BiasSstLonRmse.nc
met: BiasTauxLonRmse
filename_nc: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_BiasTauxLonRmse.nc
file not found: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_BiasTauxLonRmse.nc
met: EnsoAmpl
filename_nc: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoAmpl.nc
file not found: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoAmpl.nc
met: EnsodSstOce_2
filename_nc: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsodSstOce_2.nc
file not found: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsodSstOce_2.nc
met: EnsoFbSshSst
filename_nc: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoFbSshSst.nc
file not found: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoFbSshSst.nc
met: EnsoFbSstTaux
filename_nc: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoFbSstTaux.nc
file not found: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoFbSstTaux.nc
met: EnsoFbSstThf
filename_nc: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoFbSstThf.nc
file not found: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoFbSstThf.nc
met: EnsoFbTauxSsh
filename_nc: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoFbTauxSsh.nc
file not found: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoFbTauxSsh.nc
met: EnsoSeasonality
filename_nc: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoSeasonality.nc
file not found: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoSeasonality.nc
met: EnsoSstLonRmse
filename_nc: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoSstLonRmse.nc
file not found: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoSstLonRmse.nc
met: EnsoSstSkew
filename_nc: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoSstSkew.nc
file not found: /Users/jared/code/Climate-REF/climate-ref/tests/test-data/regression/pmp/enso_proc/cmip6_historical_r1i1p1f1_ACCESS-ESM1-5__obs4mips_obs4MIPs/ENSO_proc_ACCESS-ESM1-5_historical_r1i1p1f1_EnsoSstSkew.nc

@lee1043
Copy link
Contributor Author

lee1043 commented May 18, 2025

@lewisjared thanks for testing. The error looks new to me. Let me see if I can reproduce and pin it down.

@bouweandela
Copy link
Contributor

The diagnostics provider for ENSO_perf metrics collection is turned off by commenting it out but confirmed it is working in my local test. @bouweandela and ESMValTool team will implement this part though ESMValTool. If they have other higher priority, I don't mind turning it back on.

Thanks for offering @lee1043! I would be keen to stick with the plan for now, so as not to disappoint the folks at ACCESS-NRI who worked hard to get their ENSO diagnostics ready for inclusion in the REF in time.

@lee1043 lee1043 linked an issue May 21, 2025 that may be closed by this pull request
@lewisjared
Copy link
Contributor

@lee1043 Did you get a chance to try and reproduce my error? If you couldn't could you note down when steps and data you used to get this to work?

@lee1043
Copy link
Contributor Author

lee1043 commented May 21, 2025

@lewisjared I added couple commits that resolves the error you had. Could you test when you get a chance? I also noticed that there is a conflict from pyproject.toml, can you help addressing it?

I will work on setting up the updated ref data pool.

lee1043 and others added 5 commits May 21, 2025 17:25
* origin/main: (60 commits)
  chore: Use sha1sums for hashes
  chore: Changelog
  chore: Add additional dataset to ilamb/iomb
  Bump version: 0.5.4 → 0.5.5
  chore: Move to the newer version of development dependencies
  chore: Unify pyproject classifiers
  docs: Changelog
  chore: Print more outputs
  docs: More basic documentation
  Recreate TCR regression test output
  chore: Cleanup aft regression test
  chore: regenerate regression output
  Add changelog
  Remove superfluous recipe keys in TCR diagnostic and rename diag
  Add changelog
  Avoid using 'cmip6' in diag name
  Add tests
  Remove superfluous recipe keys in ECS diagnostic
  chore: Fix integration tests
  chore: Add ilamb prefix everywhere
  ...
@lee1043
Copy link
Contributor Author

lee1043 commented May 22, 2025

With the updated reference dataset below, I think this PR is ready to be merged. It looks like the CI/tests is failing for python 3.13, but PMP is yet to support python 3.13 so I think it is expected. @lewisjared do you think it is okay to merge this PR?

https://portal.nersc.gov/cfs/m4581/PMP/reference_datasets/
e607167a08a2521b65e55eb186182003  obs4MIPs_monthly/hfls_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc
6f766ca0332a8e566c408d237571a924  obs4MIPs_monthly/hfss_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc
6970c22443e2097c45de5db8947318eb  obs4MIPs_monthly/pr_mon_GPCP-Monthly-3-2_RSS_gn_198301-202303.nc
897451ed566251135483d1413cd6bee3  obs4MIPs_monthly/rlds_mon_CERES-EBAF-4-2_RSS_gn_200003-202309.nc
750650025845fc89d9e56a3690deea21  obs4MIPs_monthly/rlus_mon_CERES-EBAF-4-2_RSS_gn_200003-202309.nc
5c33068dd11e6eb8d0bf6e2aa0335ef2  obs4MIPs_monthly/rsds_mon_CERES-EBAF-4-2_RSS_gn_200003-202309.nc
4f67c58186905e995a8b9497a49ecbf0  obs4MIPs_monthly/rsus_mon_CERES-EBAF-4-2_RSS_gn_200003-202309.nc
527ab1b9becf2a793df558532eccfe69  obs4MIPs_monthly/tas_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc
0822e2002e61472277116d38e5e19498  obs4MIPs_monthly/tauu_mon_TropFlux-1-0_PCMDI_gn_197901-201707.nc
66fb8cdf53ec0e073c565adfa57862b3  obs4MIPs_monthly/ts_mon_HadISST-1-1_PCMDI_gn_187001-202501.nc

@lewisjared
Copy link
Contributor

This still needs regression tests and I haven't been able to verify that it works locally yet so it isn't ready to merge. The failing test was due to the lack of coverage. I think that the decimated data breaks the regridding so we might need to ignore this in the integration tests. If it isn't easy to test then we need some locally generated regression output so that we can check that the results parsing works (that will help the coverage).

If you can zip up and send me some generated output that would be useful.

I don't want to merge this until we have either the right reference data or some regression output from the test and an understanding of what data is needed that isn't in the tests.

@lewisjared
Copy link
Contributor

I've noticed that some of these are newer versions of the other reference datasets you mentioned above or they have different hashes. Which ones should be used?

@lee1043
Copy link
Contributor Author

lee1043 commented May 23, 2025

I've noticed that some of these are newer versions of the other reference datasets you mentioned above or they have different hashes. Which ones should be used?

Datasets in the latest comment are the sole set needed for PMP ENSO. Once you have them and a model data with right variables, it will enable running local test. I will also zip up the output files that can be used for regression tests.

* origin/main:
  chore: Fix edge case of an empty catalog
  chore: Changelog
  feat: Exclude all but the latest version
  chore: Update count of datasets
  chore: Add additional obs data to the registry
  test: coverage
  test: Fix
  docs: Changelog
  feat: split file logging by process
  feat: Write out verbose log files for debugging
  chore: Changelog
  chore: Disable marking CV failures as failed executions
  test: Update tests
  chore: Enable marking executions as failed if they don't have the correct cv
  chore: Regenerate regression tests
@codecov
Copy link

codecov bot commented May 26, 2025

Codecov Report

❌ Patch coverage is 98.88889% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...te-ref-pmp/src/climate_ref_pmp/diagnostics/enso.py 98.68% 0 Missing and 1 partial ⚠️
Files with missing lines Coverage Δ
...s/climate-ref-core/src/climate_ref_core/logging.py 91.66% <ø> (ø)
...es/climate-ref-pmp/src/climate_ref_pmp/__init__.py 100.00% <100.00%> (ø)
...ef-pmp/src/climate_ref_pmp/diagnostics/__init__.py 100.00% <100.00%> (ø)
...c/climate_ref_pmp/diagnostics/variability_modes.py 89.28% <100.00%> (ø)
...ages/climate-ref/src/climate_ref/executor/local.py 93.75% <100.00%> (+0.32%) ⬆️
packages/climate-ref/src/climate_ref/testing.py 63.26% <100.00%> (+0.76%) ⬆️
...te-ref-pmp/src/climate_ref_pmp/diagnostics/enso.py 98.68% <98.68%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@lewisjared lewisjared merged commit 4871c13 into main May 26, 2025
13 checks passed
@lewisjared lewisjared deleted the 223_pmp-enso-2 branch May 26, 2025 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request metric Metrics

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implementation of PMP ENSO metrics

4 participants