esgdrs for cmip6plus

[NishkaDG]

Hi,

@tiantdk and I tried to run an esgdrs command for cmip6plus output, with the esgprep code release v3.1.0 .

This was the command:

esgdrs make list -p cmip6plus /data/project/dmiesgf/cmip6plus/CMIP6Plus/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r8i1p1f1/

I followed the guide here: https://esgf.github.io/esgf-prepare/getting_started.html , and changed "cmip6" to "cmip6plus", and I am not sure what I did differently.

This was the output for each file:

:: SKIPPED :: /data/project/dmiesgf/cmip6plus/CMIP6Plus/CMIP/EC-Earth-Consortium/EC-Earth3/historical/r8i1p1f1/SImonPt/sidivvel/gn/v20210701/sidivvel_SImonPt_EC-Earth3_historical_r8i1p1f1_gn_201402-201501.nc
Traceback (most recent call last):
  File "/data/project/dmiesgf/esgf-prepare/.venv/lib/python3.12/site-packages/esgvoc/api/projects.py", line 791, in get_project
    result = ProjectSpecs(**project.specs)  # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/dmiesgf/esgf-prepare/.venv/lib/python3.12/site-packages/pydantic/main.py", line 250, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 3 validation errors for ProjectSpecs
drs_specs.directory.regex
  Field required [type=missing, input_value={'type': 'directory', 'se..., 'is_required': True}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
drs_specs.file_name.regex
  Field required [type=missing, input_value={'type': 'file_name', 'se... 'is_required': False}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing
drs_specs.dataset_id.regex
  Field required [type=missing, input_value={'type': 'dataset_id', 's..., 'is_required': True}]}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.12/v/missing

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/project/dmiesgf/esgf-prepare/esgprep/drs/make.py", line 121, in __call__
    dg = DrsGenerator(self.project)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/dmiesgf/esgf-prepare/.venv/lib/python3.12/site-packages/esgvoc/apps/drs/validator.py", line 41, in __init__
    project_specs: ProjectSpecs | None = projects.get_project(project_id)
                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/project/dmiesgf/esgf-prepare/.venv/lib/python3.12/site-packages/esgvoc/api/projects.py", line 794, in get_project
    raise EsgvocDbError(msg) from e
esgvoc.core.exceptions.EsgvocDbError: unable to read specs in project 'cmip6plus'

For institutional reasons we are using pip instead of uv; here is the list of packages and versions that we have:

$ pip list
Package            Version
------------------ -----------
attrs              20.3.0
Babel              2.9.1
certifi            2024.8.30
cffi               1.14.5
cftime             1.6.4.post1
chardet            4.0.0
charset-normalizer 3.3.2
configobj          5.0.6
cryptography       36.0.1
dbus-python        1.2.18
decorator          4.4.2
distro             1.5.0
dnspython          2.3.0
docker             7.1.0
esgprep            2.7.3
fuzzywuzzy         0.18.0
github3.py         0.9.5
gpg                1.15.1
gssapi             1.6.9
hurry.filesize     0.9
idna               2.10
ipaclient          4.11.0
ipalib             4.11.0
ipaplatform        4.11.0
ipapython          4.11.0
Jinja2             2.11.3
jsonpatch          1.21
jsonpointer        2.0
jsonschema         3.2.0
jwcrypto           0.8
libcomps           0.1.18
lockfile           0.12.2
MarkupSafe         1.1.1
netaddr            0.8.0
netCDF4            1.7.2
netifaces          0.10.6
nftables           0.1
numpy              2.0.2
oauthlib           3.1.1
pip                21.3.1
ply                3.11
prettytable        0.7.2
pyasn1             0.4.8
pyasn1-modules     0.2.8
pycparser          2.20
PyGObject          3.40.1
pyrsistent         0.17.3
pyserial           3.4
PySocks            1.7.1
python-augeas      0.5.0
python-dateutil    2.8.1
python-ldap        3.4.3
python-yubico      1.3.3
pytz               2021.1
pyusb              1.0.2
PyYAML             5.4.1
qrcode             6.1
requests           2.32.3
rpm                4.16.1.3
selinux            3.6
sepolicy           3.6
setools            4.4.4
setuptools         53.0.0
six                1.15.0
SSSDConfig         2.9.4
systemd-python     234
tqdm               4.11.2
treelib            1.8.0
uritemplate.py     2.0.0
urllib3            1.26.5

How can we get esgprep to work on our CMIP6Plus data?

Thanks,

[ltroussellier]

it comes from esgvoc thats was break cause one of its dependency

I’ve just released a new esgvoc version (3.1.0).
You can check your current version with:

uv run esgvoc version
# or
esgvoc version

I believe this issue is resolved, but I’m not completely sure—could you please try again?

[NishkaDG]

Hi @ltroussellier , I get this:

$ esgvoc version
Usage: esgvoc [OPTIONS] COMMAND [ARGS]...
Try 'esgvoc --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ No such command 'version'.                                                                                                                                                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
$ esgvoc --help
                                                                                                                                                                                                                                                        
 Usage: esgvoc [OPTIONS] COMMAND [ARGS]...                                                                                                                                                                                                              
                                                                                                                                                                                                                                                        
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion          Install completion for the current shell.                                                                                                                                                                              │
│ --show-completion             Show completion for the current shell, to copy it or customize the installation.                                                                                                                                       │
│ --help                        Show this message and exit.                                                                                                                                                                                            │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ list-projects   List all available projects with their default configurations.                                                                                                                                                                       │
│ get             Retrieve a specific value from the database system.                                                                                                                                                                                  │
│ status          Command to display status i.e summary of version of usable ressources (between remote/cached)                                                                                                                                        │
│ valid           Validates one or more strings against specified Project:Collection:Term configurations.                                                                                                                                              │
│ install         Initialize default config and apply settings                                                                                                                                                                                         │
│ drsvalid        Validates DRS strings for a specific project and type.                                                                                                                                                                               │
│ drsgen          Generates DRS strings for a specific project and type based on input bag of terms.                                                                                                                                                   │
│ config                                                                                                                                                                                                                                               │
│ offline                                                                                                                                                                                                                                              │
│ clean                                                                                                                                                                                                                                                │
│ test                                                                                                                                                                                                                                                 │
│ find            Retrieve a specific value from the database system.                                                                                                                                                                                  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
$ git log --oneline
9e658b3 (HEAD -> main, tag: v3.1.0) Issue release 3 0 2 (#104)

I thought that since the tag of the last commit I have pulled says "v3.1.0" then I am on the correct version. Is there something else I should do for setup?

Though in #102 I also faced errors with esgvoc.

Thanks,

[ltroussellier]

hum really weird ...
i think the esgvoc you run .. is not the esgvoc from the directory you are in (thats why i use uv)
can you try

which esgvoc

[NishkaDG]

Hi,

When I am in the directory with the python venv activated, the esgvoc appears to be exactly in the same directory as the venv and the esgf-prepare scripts:

$ which esgvoc
/data/project/dmiesgf/esgf-prepare/.venv/bin/esgvoc
$ which python
/data/project/dmiesgf/esgf-prepare/.venv/bin/python
$ pwd
/data/project/dmiesgf/esgf-prepare

Thanks,

[ltroussellier]

i dont understand why

just to show you from 0 :

❯❯❯ cd tests
~/B/d/tests 
❯❯❯ mkdir temp6
~/B/d/tests 
❯❯❯ cd temp6
~/B/d/t/temp6 
❯❯❯ uv init
Initialized project `temp6`
~/B/d/t/temp6 
❯❯❯ uv add esgprep                                                        main ◼
Using CPython 3.14.0
Creating virtual environment at: .venv
Resolved 46 packages in 8.59s
Prepared 2 packages in 413ms
Installed 44 packages in 59ms
 + annotated-doc==0.0.4
 + annotated-types==0.7.0
 + cachetools==7.0.5
 + certifi==2026.2.25
 + cftime==1.6.5
 + charset-normalizer==3.4.7
 + click==8.3.2
 + esgprep==3.1.0
 + esgvoc==3.1.0
 + frozendict==2.4.7
 + fuzzywuzzy==0.18.0
 + greenlet==3.4.0
 + hurry-filesize==0.9
 + idna==3.11
 + jinja2==3.1.6
 + levenshtein==0.27.3
 + lockfile==0.12.2
 + lxml==6.0.4
 + markdown-it-py==4.0.0
 + markupsafe==3.0.3
 + mdurl==0.1.2
 + netcdf4==1.7.4
 + numpy==2.4.4
 + platformdirs==4.9.6
 + pydantic==2.13.0
 + pydantic-core==2.46.0
 + pygments==2.20.0
 + pyld==3.0.0
 + python-levenshtein==0.27.3
 + pyyaml==6.0.3
 + rapidfuzz==3.14.5
 + requests==2.33.1
 + rich==15.0.0
 + setuptools==82.0.1
 + shellingham==1.5.4
 + six==1.17.0
 + sqlalchemy==2.0.49
 + sqlmodel==0.0.38
 + toml==0.10.2
 + treelib==1.8.0
 + typer==0.24.1
 + typing-extensions==4.15.0
 + typing-inspection==0.4.2
 + urllib3==2.6.3
~/B/d/t/temp6 
❯❯❯ uv run esgvoc --help                                                                                                                             main ◼
                                                                                                                                                           
 Usage: esgvoc [OPTIONS] COMMAND [ARGS]...                                                                                                                 
                                                                                                                                                           
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --install-completion          Install completion for the current shell.                                                                                 │
│ --show-completion             Show completion for the current shell, to copy it or customize the installation.                                          │
│ --help                        Show this message and exit.                                                                                               │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ list-projects  List all available projects with their default configurations.                                                                           │
│ version        Show esgvoc version and optionally check for updates.                                                                                    │
│ get            Retrieve a specific value from the database system.                                                                                      │
│ status         Command to display status                                                                                                                │
│                i.e summary of version of usable ressources (between remote/cached)                                                                      │
│ valid          Validates one or more strings against specified Project:Collection:Term configurations.                                                  │
│ install        Initialize default config and apply settings                                                                                             │
│ drsvalid       Validates DRS strings for a specific project and type.                                                                                   │
│ drsgen         Generates DRS strings for a specific project and type based on input bag of terms.                                                       │
│ config                                                                                                                                                  │
│ offline                                                                                                                                                 │
│ clean                                                                                                                                                   │
│ test                                                                                                                                                    │
│ find           Retrieve a specific value from the database system.                                                                                      │
│ schema         Generate a JSON schema for a project's catalog specification.                                                                            │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

the esgvoc version 3.1.0 has the version sub command ^^. So i really dont know why you dont have it.
But it will not resolve your issue thought

can you send me the header of your file, i ll be able to create a "fake file" and try esgprep on it

[NishkaDG]

Hello Laurent,

Apologies for the late reply. Here is the header for the file in the error message:

netcdf sidivvel_SImonPt_EC-Earth3_historical_r8i1p1f1_gn_201402-201501 {
dimensions:
	time = UNLIMITED ; // (12 currently)
	j = 292 ;
	i = 362 ;
	bnds = 2 ;
	vertices = 4 ;
variables:
	double time(time) ;
		time:units = "days since 1850-01-01 00:00:00" ;
		time:calendar = "proleptic_gregorian" ;
		time:axis = "T" ;
		time:long_name = "time" ;
		time:standard_name = "time" ;
	int j(j) ;
		j:units = "1" ;
		j:long_name = "second spatial index for variables stored on an unstructured grid" ;
	int i(i) ;
		i:units = "1" ;
		i:long_name = "first spatial index for variables stored on an unstructured grid" ;
	double latitude(j, i) ;
		latitude:standard_name = "latitude" ;
		latitude:long_name = "latitude" ;
		latitude:units = "degrees_north" ;
		latitude:missing_value = 1.e+20 ;
		latitude:_FillValue = 1.e+20 ;
		latitude:bounds = "vertices_latitude" ;
	double longitude(j, i) ;
		longitude:standard_name = "longitude" ;
		longitude:long_name = "longitude" ;
		longitude:units = "degrees_east" ;
		longitude:missing_value = 1.e+20 ;
		longitude:_FillValue = 1.e+20 ;
		longitude:bounds = "vertices_longitude" ;
	double vertices_latitude(j, i, vertices) ;
		vertices_latitude:units = "degrees_north" ;
		vertices_latitude:missing_value = 1.e+20 ;
		vertices_latitude:_FillValue = 1.e+20 ;
	double vertices_longitude(j, i, vertices) ;
		vertices_longitude:units = "degrees_east" ;
		vertices_longitude:missing_value = 1.e+20 ;
		vertices_longitude:_FillValue = 1.e+20 ;
	float sidivvel(time, j, i) ;
		sidivvel:standard_name = "divergence_of_sea_ice_velocity" ;
		sidivvel:long_name = "Divergence of the Sea-Ice Velocity Field" ;
		sidivvel:comment = "Divergence of sea-ice velocity field (first shear strain invariant)" ;
		sidivvel:units = "s-1" ;
		sidivvel:original_name = "sidivvel" ;
		sidivvel:cell_methods = "area: mean where sea_ice (comment: mask=siconc) time: point" ;
		sidivvel:cell_measures = "area: areacello" ;
		sidivvel:history = "2021-06-21T12:35:20Z altered by CMOR: Reordered dimensions, original order: j i time." ;
		sidivvel:missing_value = 1.e+20f ;
		sidivvel:_FillValue = 1.e+20f ;
		sidivvel:coordinates = "latitude longitude" ;

// global attributes:
		:Conventions = "CF-1.7 CMIP-6.2" ;
		:activity_id = "CMIP" ;
		:branch_method = "standard" ;
		:branch_time_in_child = 0. ;
		:branch_time_in_parent = 252017. ;
		:contact = "cmip6-data@ec-earth.org" ;
		:creation_date = "2021-06-21T12:35:20Z" ;
		:data_specs_version = "01.00.33" ;
		:ece2cmor_git_revision = "0e866d8587c8b58dc91aaeb52e2751dcefca6880" ;
		:experiment_id = "historical" ;
		:external_variables = "areacello" ;
		:forcing_index = 1 ;
		:frequency = "monPt" ;
		:grid = "ORCA1L75" ;
		:grid_label = "gn" ;
		:initialization_index = 1 ;
		:institution_id = "EC-Earth-Consortium" ;
		:latest_applied_cmor_fixer_version = "None" ;
		:latest_ece2cmor_version = "v1.7.0" ;
		:nominal_resolution = "100 km" ;
		:parent_activity_id = "CMIP" ;
		:parent_experiment_id = "piControl" ;
		:parent_source_id = "EC-Earth3" ;
		:parent_time_units = "days since 1850-01-01" ;
		:parent_variant_label = "r1i1p1f1" ;
		:physics_index = 1 ;
		:product = "model-output" ;
		:realization_index = 8 ;
		:realm = "seaIce" ;
		:source_id = "EC-Earth3" ;
		:source_type = "AOGCM" ;
		:sub_experiment = "none" ;
		:sub_experiment_id = "none" ;
		:table_info = "Creation Date:(18 November 2020) MD5:7d98b629c7bbf47e2f5ee2866e1a18cf" ;
		:tracking_id = "hdl:21.14100/09c0d259-9e65-47d3-85cc-354d217f7a72" ;
		:variable_id = "sidivvel" ;
		:variant_label = "r8i1p1f1" ;
		:cmor_version = "3.5.0" ;
		:experiment = "all-forcing simulation of the recent past (CO2 concentration-driven)" ;
		:history = "2026-03-19T08:16:29 AMZ ; The convert-cmip6-to-cmip6plus.sh script has been applied.;\n",
			"2021-06-21T12:19:45Z ; CMOR rewrote data to be consistent with CMIP6, CF-1.7 CMIP-6.2 and CF standards.;\n",
			"processed by ece2cmor v1.7.0, git rev. 0e866d8587c8b58dc91aaeb52e2751dcefca6880\n",
			"" ;
		:institution = "EC-Earth-Consortium - EC-Earth-Consortium [consortium]" ;
		:mip_era = "CMIP6Plus" ;
		:parent_mip_era = "CMIP6" ;
		:source = "EC-Earth3 (2019): \n",
			"aerosol: none\n",
			"atmos: IFS cy36r4 (TL255, linearly reduced Gaussian grid equivalent to 512 x 256 longitude/latitude; 91 levels; top level 0.01 hPa)\n",
			"atmosChem: none\n",
			"land: HTESSEL (land surface scheme built in IFS)\n",
			"landIce: none\n",
			"ocean: NEMO3.6 (ORCA1 tripolar primarily 1 deg with meridional refinement down to 1/3 degree in the tropics; 362 x 292 longitude/latitude; 75 levels; top grid cell 0-1 m)\n",
			"ocnBgchem: none\n",
			"seaIce: LIM3)" ;
		:table_id = "SImonPt" ;
		:title = "EC-Earth3 output prepared for CMIP6" ;
		:license = "CMIP6Plus model data produced by EC-Earth-Consortium is licensed under a Creative Commons 4.0 (CC BY 4.0) License (https://creativecommons.org/). Consult https://pcmdi.llnl.gov/CMIP6Plus/TermsOfUse for terms of use governing CMIP6Plus output, including citation requirements and proper acknowledgment. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law." ;
		:comment = "This experiment was done as part of CMIP6 CMIP by DMI" ;
		:description = "CMIP6 historical" ;
}

Thanks,

[NishkaDG]

Hello @ltroussellier any update on this?

[ltroussellier]

It’s still on my to-do list (I was on leave last week). I’ll try to take a deeper look at it this week, though I’m not sure exactly when yet. I’ve been quite busy implementing the new ESGVOC release (planned soon), and it will likely change a few things.

Just to let you know, ESGPREP is the next priority, so I’ll be looking into this soon as well.

[NishkaDG]

Hi @ltroussellier , just checking in if you need any information about our setup to proceed with this. Or are there any changes/updates (e.g with esgvoc) that we should implement on our end?

Thanks,

[ltroussellier]

Hi @NishkaDG
On paper: there will be a new esgprep release next week, using esgvoc >= 4.x.x along with some fixes and new features. After that, we'll be able to see if this is still an issue. My intuition says it will be resolved, but no promises! — at worst, we should see things more clearly at that point