Input data backup from betzy

[MichaelSchulzMETNO]

goal: Save betzy NorESM input files to nird /projects/NS9560K/www/inputdata

steps:

1. Identify "NorESM input files" on betzy:/cluster/shared/noresm/inputdata of group owner noresm
   which are NOT in NCAR copy : /datalake/NS12077K/CESM-input-data
   Exclude some files: svn, lock, etc

2. Rsync those "NOT-in-NCAR NorESM input files" to /projects/NS9560K/www/inputdata
   2b) identify problems with permissions - communicate with sigma2 to make all files readable for NS9560K
   2c) rsync will only copy new files on www/inputdata
   2d) Only the rsync script should add files on www/inputdata

3. Users on betzy can add files to this process by making files and directories owned by group "noresm"

4. Make a repository under NorESMhub with scripts for steps 1+2 @jgriesfeller

5. Run script regularly (tbd) (Jan)

5b) Manually correct files on /projects/NS9560K/www/inputdata on demand with Data management group

6. Update NCAR files now and then (Marit?)

7. Look at input files from other machines , on demand

8. Move eventually www/inputdata to a specific input data project

Please comment @maritsandstad @gold2718 @tylov @jgriesfeller @DirkOlivie @monsieuralok @matsbn @lisesg

[gold2718]

One way to identify missing files is to run tests with DIN_LOC_ROOT set to an empty directory. This can be done both with standard regression tests and with a NorESM3 production setup. Any file that does not clear running ./check_input_data after ./case.setup needs to be copied to /projects/NS9560K/www/inputdata

One current issue with this method on Betzy is NorESMhub/ccs_config_noresm#70.

[MichaelSchulzMETNO]

Our idea was that the betzy directory would be the primary master copy of what we need. The www/inputdata directory would then get any missing files slowly also from betzy by the rsync script. So any file that is missing should be copied to betzy to my understanding. Or is that not logic?

[gold2718]

I think your logic is spot on (or at least matches mine). The hole is that there are some NorESM-only files that are created on Betzy and need to be copied by hand to the NIRD location. The test I outline above is just a well-tested method for finding any missing files (i.e., that are not on any server including NIRD).

[MichaelSchulzMETNO]

NorESM-only files will be copied to nird if they are owned by the group "noresm", using the unix attribute. So if one identifies missing files with your method, they should be copied to betzy and then the group ownership eventually has to be changed to noresm by the one finding missing files (or another script?)

[maritsandstad]

I have been doing 6 semi-sporadically @MichaelSchulzMETNO . Do you want some more regular update schedule?

[MichaelSchulzMETNO]

I am not sure how much that NCAR directory changes. Bimonthly sounds good to me.

[jgriesfeller]

while looking around a bit I noticed that all data in /datalake/NS12077K/CESM-input-data is owned by the group ns9560k and not by the group ns12077k. Doesn't that mean that the data is accounted to the ns9560k project? Or am I missing something?

[tylov]

No, I think it only matters under which project they are stored under (for who they are accounted to), but yes the wrong group ownership can be a problem regarding access.

[jgriesfeller]

Potential show stopper:
We want to automise the copying of new input data from betzy to nird using cron and rsync. But isn't the two factor authentication preventing that?
According to https://www.sigma2.no/documentation/getting_help/two_factor_authentication.html only scp can be run circumventing 2FA. This excludes the usage of rsync.

[TomasTorsvik]

nird is mounted on betzy. I use rsync regularly from betzy to nird without 2FA, using the local mount point. However, the permissions often get messed up, so sometimes I need to fix this afterwards on nird.

[jgriesfeller]

That will actually work.
But it means that betzy's login nodes will do the copying and not nird's as we wanted. Do we have any experince running cron jobs on betzy?

[TomasTorsvik]

Yes, that is right, so it is not optimal. I have never tried to run cron jobs on betzy.

[rosiealice]

@TomasTorsvik that's interesting. When I try rsync from betzy to nird it always asks for 2FA which is VERY annoying! What trick do you use to circumvent it?

[jgriesfeller]

the nird file systems are below /nird on betzy (login nodes only)

[TomasTorsvik]

If you look in the root folder on betzy, you will find a directory /nird that mirrors the nird storage servers.
(see https://documentation.sigma2.no/files_storage/nird/mounts_lmd.html)

NOTE1: /nird/datapeak is mounted on /nird/projects

NOTE2: The /nird mount point is not accessible on betzy work nodes, only login nodes, because the work nodes only see directories that are under the /cluster folder. At least, that is how it is on betzy.

[tylov]

NOTE1: /nird/datapeak is mounted on /nird/projects

You can also use /nird/projects from betzy. For scripts, you can use /nird/projects, /nird/datalake from both betzy and nird, and it will work on both machines (unless you use modules that is only on betzy).

[MichaelSchulzMETNO]

"Automising" is not the only goal. First priority is to rsync and secure the specific NorESM input data. Updating such securisation procedure comes after that.

[jgriesfeller]

Just for documentation:
The following input data directories are not readable by me although I was searching for group ownership for the noresm group only (betzy):

find: ‘/cluster/shared/noresm/inputdata/BlueAction/NHIST_f19_tn14_Init01_rest1900’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/old_output’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/HCLIMCNRMSSP370’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/HCLIMNORESMhistorical’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/HCLIMCNRMhistorical’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/ICONNORESMSSP370’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/ICONCNRMSSP370’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/HCLIMNORESMSSP370’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/PolarRES/RCM_forcing_files/ICONNORESMhistorical’: Permission denied
find: ‘/cluster/shared/noresm/inputdata/lnd/clm2/rawdata/.svn’: Permission denied

If someone feels responsible for those, please either adjust permissions (readable for the group noresm) or change group ownership to a group that is NOT noresm.
I will ask sigma2 to to adjust their permissions once my script has progressed more.

[MichaelSchulzMETNO]

@lisesg @JorgSchwinger @kjetilaas ??

[lisesg]

I think the owner of these files is Iris Mužić at CICERO. I can contact her.

[jgriesfeller]

the PolarRES data is readable now.

[jgriesfeller]

Some very preliminary info regarding the amount of input data files on betzy that don't have backup:

Info: 639 source files were found in the backup folder.
Info: 19025 source files were found in the NCAR archive folder.
Info: This leaves 203853 source files to be copied.

Is this high number expected?
I haven't checked this very intensively though, but the fact that some matching files are found sort of indicates that my algorithm is working...
At the moment files matching these patterns have been excluded already:

SOURCE_PATH_EXCLUDE_LIST = [".svn/*", "*/.svn/*", "*.lock"]

Please suggest more file patterns for removal...

[jgriesfeller]

applying that additional exclude criterion leaves this:

Info: 639 source files were found in the backup folder.
Info: 19025 source files were found in the NCAR archive folder.
Info: This leaves 132243 source files to be copied.

If noone comes up with further comments, I will run the copy script as soon as I have actually added the copying functionality... Still looking for the most efficient way to do that with Python...

[jgriesfeller]

just to be sure:
the directory /cluster/shared/noresm/inputdata/restart is supposed to be backed up as well?

[MichaelSchulzMETNO]

do we need that backupped?? maybe not ? @DirkOlivie @TomasTorsvik ??

[oyvindseland]

I do not think it needs backup since it is a copy of the original output in the first place.

[TomasTorsvik]

@oyvindseland - I agree, at least for the NOIIAJRA* data. I suppose for N20TRAERCN we only have one copy (in NS2345K)? I don't think we will be able to reproduce these datasets bit-for-bit in the future, if there ever is a need.

[oyvindseland]

I think it is copied to the research archive though.

[jgriesfeller]

Hopefully the last post before copying...

The following list is the files below the path /cluster/shared/noresm/inputdata that my script would copy to nird for the moment: files_to_copy.txt
It's all in all 33TB of data and 226390 files.

Please confirm that these are the files that are supposed to be copied to nird

dusage for ns9560k on nird (that's where the data would go) as additional information:

====================================================================================
Project      Resource  Type    Usage       Soft Limit   Hard Limit               
NS9560K      datalake  Disk    752.247 TiB  800 TiB      800 TiB                   
NS9560K      datalake  Files   6095194      6488064      8957952                   

This would leave around 12TB and roughly 166480 files (soft quota) of the ns9560k quota.

[TomasTorsvik]

It looks correct to me. I think we might have to clean up/remove some of the older data on the ns9560k project. Probably, most of the NorESM2.5_alphaXX datasets are outdated now, except for long reference runs?

[jgriesfeller]

It has been decided to backup the inputdata to from betzy to the path /nird/datalake/NS12077K/NorESMInputdata for now.
I will start the backup after some more tests shortly.

[jgriesfeller]

Next question:
While testing the backup I noticed that e.g. the file atm/cam/solar/polarForcing1995-2005avg_c160929.nc-3650289664-2613228.loc disappeared.
Can these .loc files be added to the ignore list?

[jgriesfeller]

How about PAMIP/N2000_PAMIP_6_2_v1/Buildconf or CaseDocs directories? Are these supposed to be in the inputdata directory?

[MichaelSchulzMETNO]

@lisesg can you comment possibly?

[lisesg]

@AleksiNummelin: we are backing up the Betzy input data, do you want PAMIP/N2000_PAMIP_6_2_v1/Buildconf and/or CaseDocs to be included?

[jgriesfeller]

Just for info:
The initial backup has been started. It should finish by Friday.

[jgriesfeller]

Lorand just made it clear that the inputdata backup should not be located within the ns12077k project.
But since I just got access to the S3 based NS16001K project, I will shift my efforts to that.

[jgriesfeller]

Copying the the data to the new location (/nird/datalake/NS16001B/cdl-ns16001b-NorESMInputdata) now.

[jgriesfeller]

The initial backup has finished. Please check and report errors.

Note that

• the data can only be altered by members of the group ns16001b
• all executable permissions on files have been removed
• all data is readable for the world as this is needed for future web access

I will now polish the code and the documentation and try to setup a cron job that runs the backup on a weekly basis.

[TomasTorsvik]

@jgriesfeller - I don't have read access to the NS16001B storage space on Nird. Is this supposed to be through SSH, or is there some other procedure for S3 access (I haven't used S3 before)? Or perhaps it is not open for read access?

[jgriesfeller]

Unfortunately /nird/datalake/NS16001B is not world readable as it should. Sorry for missing that.
@tylov : I guess it's you as project owner who has to ask sigma2 to open the project for world access. Can you maybe ask for it?
NS9560K is open btw, so there is an example.

[jgriesfeller]

Next question:
There's a lot of file with executable permissions below /cluster/shared/noresm/inputdata.
Should there be any executable file there? The name seems to indicate that thats only data and no scripts.
I tend to remove all executable bits at the backup path. Any subjections?

[TomasTorsvik]

I don't see a reason for executable permissions on data files, should be safe to remove the executable bits.

[jgriesfeller]

Just for info:
I have just created the initial working version of the input data management tool for betzy: https://github.com/NorESMhub/inputdatamanagement
Please read through the minimal documentation (the README.md file) and report potential errors.
I'm waiting for sigma2 to open the backup directory for world and for the web accessibility as we have on ns9560k.

[TomasTorsvik]

@jgriesfeller , @tylov
Now we have two repositories:

• https://github.com/NorESMhub/inputdatamanagement
• https://github.com/NorESMhub/inputdata_management
  It seems the second one of these is not needed?

[jgriesfeller]

Indeed. The second was the first try I guess...
I just deleted it

[tylov]

There are two folders in /datalake/NS16001B, where the second is referred to from the www folder:

cdl-ns16001b-noresminputdata
cdl-ns16001b-NorESMInputdata

[jgriesfeller]

What's the question? I'm still testing things and wasn't sure if the S3 stuff likes CamelCase...
I will remove everything not needed once the testing has finished.

[jgriesfeller]

Just for info:
The quota of the inputdata backup project (NS16001B) has been exhausted.
The sub directory cicero_mods contains 116TB alone (out of a total 150TB quota). Was that intended?

Looking a bit further, the main thing is the directory cicero_mods/ERA5-Land with 115TB including some directories that look redundant at first glance.

Please advise what's really needed in the backup. The script can exclude directory structures.

2,7T	cicero_mods/ERA5-Land/TPQWL3Hrly_old
683G	cicero_mods/ERA5-Land/Prec3Hrly_old
12T	cicero_mods/ERA5-Land/calculated_vars
683G	cicero_mods/ERA5-Land/Solar3Hrly_old
13T	cicero_mods/ERA5-Land/original_GRIB_files
6,6M	cicero_mods/ERA5-Land/prep_for_plotting
1,3T	cicero_mods/ERA5-Land/regrid_ERA5Land_to_CRUJRA
77T	cicero_mods/ERA5-Land/single_vars
8,0T	cicero_mods/ERA5-Land/masked_ocean
115T	cicero_mods/ERA5-Land

[maritsandstad]

@jgriesfeller For now maybe exclude all of cicero_mods from backup? There is also way too much here which needs to be cleared away, we are working on that offline.

[jgriesfeller]

fine with me. Just let me know once you want to add things to the backup again.

[jgriesfeller]

JFYI: This has been implemented and the backup works again. Current quota usage is 34.405 TiB

[MichaelSchulzMETNO]

@kjetilaas @maritsandstad can you clarify?

[maritsandstad]

See comment above @MichaelSchulzMETNO

[MichaelSchulzMETNO]

Documentation of input data saving/syncing :

@tylov @jgriesfeller could you add a section on the input data maybe on the wiki:
https://github.com/NorESMhub/noresm3_dev_simulations/wiki

and check / edit this one :
https://noresm-docs.readthedocs.io/en/noresm2.3/access/download_input.html#

we have now:
https://ns9560k.web.sigma2.no/datapeak/inputdata/atm/cam/
https://ns16001b.web.sigma2.no/datalake/inputdata/

/datalake/NS12077K/CESM-input-data
/datalake/NS16001B/inputdata
/datalake/NS16001B/cdl-ns16001b-NorESMInputdata
/projects/NS9560K/www/inputdata

betzy
/shared/noresm/inputdata

olivia
/cluster/work/projects/nn9560k/inputdata/

How do they hang together?

[oyvindseland]

drwxrws--- 3 root ns12077k 4096 nov. 21 14:14 NS12077K

so unaccessible. Any file list available?

[gold2718]

Can you browse https://ns16001b.web.sigma2.no/datalake/inputdata/ ?

[oyvindseland]

yes

[oyvindseland]

I see that there is potentially files missing from NCAR though. E.g sst_HadOIBl_bc_0.23x0.31_clim_c110526.nc

is calculated from a transient file that is not present. I was asked whether there are any higher resolution SST files for PD that could be used for refined grid simulation but a discussion of what NCAR files is potentially missing should be in another discussion I guess.

[gold2718]

Perhaps. If there is a list of files we are missing, I can try to find them from the svn repo or on Derecho.

[maritsandstad]

To my knowledge, some of this is also a bit in flux still @MichaelSchulzMETNO, especially for the Olivia / 16001B / CDL part... I sent an email to sigma2 last week about it, and they say it's a priority for them to get that coupling working, however, right now they are very busy with sorting out the allocations for 2026.1...

[monsieuralok]

@tylov @jgriesfeller @matsbn @maritsandstad @MichaelSchulzMETNO Inputdata on nird (/datalake/NS16001B/) is not complete copy of Betzy; I was planning to check functionality of CDL by putting inputdata in cache as presented by Dhanya and Lorand during meeting. I was able to put most data in cache folder but still, some data are missing on nird.

For testing purpose, it would be good if we have a complete copy of Betzy on nird in /datalake/NS16001B/. Please can be prioritise this task.

Thanks a lot
Alok

[jgriesfeller]

The goal was to only copy inputdata that belongs to the unix group noresm. That should be the case now and just to be sure I just started the job again.

[gold2718]

@monsieuralok, is it possible to generate a list of the missing files? What test shows missing files which are needed by current simulations?

[monsieuralok]

For NF2000, atleast this file is missing

/datalake/NS16001B/cdl-ns16001b-NorESMInputdata/lnd/clm2/snicardata/snicar_optics_5bnd_c013122.nc

I will start to update the list once I start to build other cases

[gold2718]

This file (and some others) did not have the noresm group set. They have been fixed and should get copied on the next script run.

[monsieuralok]

@jgriesfeller at least can we copy following file from Betzy inputdata to nird:

share/meshes/ne16pg3_ESMFmesh_cdf5_c20211018.nc

[jgriesfeller]

done

[gold2718]

@monsieuralok, can you ask @tto061 to change the group to noresm on these files?

[gold2718]

For the record, I looked to see who owns files in inputdata on Betzy which are not in the noresm group:
a2021
aal069
adagj
adelez
agu002
anu074
astridbg
carrasco
cgu025
evaler
ffr004
hra063
ingo
irismuz
jang
janko
jeani
jschwing
kjetisaa
lama
lisesg
mdeb
mifajd
mvertens
olivie
ovewh
oyvinds
pgchiu
rosief
tbo066
tjiputra
tomast
tto061
UNKNOWN
zhang

I don't know all these users. I happened to stumble on evaler being (I think) Eva Lieungh (or Eva Lieungh Eriksen) who left a bunch of files in inputdata related to her thesis. I'm pretty sure we don't want to carry these forward in perpetuity.

[TomasTorsvik]

I updated the list of files in subfolders that do not have the group noresm setting, using the one-liner (example for "share" folder):

find ./share -type f -not -group noresm -printf "%u %p\n" > not-noresm-share.txt

There are still quite a few files without the noresm group setting in atm, cdeps, cpl, glc, lnd, ocn, restart, share.
These are the ones I assume are for the most general use.

[oyvindseland]

I changed the group for my missing file

/cluster/shared/noresm/inputdata/share_alok/domains/domain.ocn.fv0.9x1.25_tnx1v4.170609_djlo.nc

I did however notice that the identical file already being owned by the noresm group exists in

/cluster/shared/noresm/inputdata/atm/cam/ocnfrac/

@monsieuralok
The share_alok domain looks like something that was done s a fix for something. Is it just an old copy of other files?

[monsieuralok]

/cluster/shared/noresm/inputdata/share_alok/domains/

@oyvindseland I will change group of /cluster/shared/noresm/inputdata/share_alok/domains/ as /cluster/shared/noresm/inputdata/share/ have atleast all the files those exists in share_alok.

[monsieuralok]

@matsbn @MichaelSchulzMETNO @lisesg @jgriesfeller @gold2718 It would be good if could decide that which files from inputdata should be copied from betzy to nird. Also, we should send message to users to change group to noresm on Betzy. I am checking for functionality presented by Lorand for Olivia; it wokrs but, still some files are missing and I also have few questions. We should also think about previous versions of NorESM for inputdata. From my part, I have changed all files to group noresm those I owned.

[JorgSchwinger]

Sorry, for being slow...

I checked the summary files prepared by @TomasTorsvik (thank you!). There were indeed a few essential ocean files that had group jschwing - I have changed that to noresm now (also for a couple of other important files that should be kept). These files should be transferred the next time the backup script is run.

[maritsandstad]

Just to say, some of the users on this list have data there (irismuz and janko at least) which are on purpose not included in the noresm group to not have large datasets that don't need to be copied over to the input data

[monsieuralok]

@jgriesfeller @gold2718 Has we synced again the inputdata directory from Betzy to nird as many others have changed group in last couple of weeks; as for Olivia testing it is required on PRIORITY basis..

[TomasTorsvik]

I updated the list of files in sub-directories under betzy:inputdata that do not have the noresm group setting. Here is the line count (will delete the *.old.txt files now)

$ wc -l ./not-noresm*
   744 ./not-noresm-atm.old.txt
   725 ./not-noresm-atm.txt
     1 ./not-noresm-cdeps.old.txt
     1 ./not-noresm-cdeps.txt
    39 ./not-noresm-cpl.old.txt
    34 ./not-noresm-cpl.txt
     2 ./not-noresm-glc.old.txt
     2 ./not-noresm-glc.txt
     0 ./not-noresm-ice.txt
   666 ./not-noresm-lnd.old.txt
   118 ./not-noresm-lnd.txt
   245 ./not-noresm-ocn.old.txt
   217 ./not-noresm-ocn.txt
     6 ./not-noresm-restart.old.txt
     6 ./not-noresm-restart.txt
     0 ./not-noresm-rof.txt
    32 ./not-noresm-share.old.txt
    32 ./not-noresm-share.txt
  2870 total
``

[jgriesfeller]

The inputdata backup script has completed. Please check and report errors.

[adagj]

Hi,
I'm trying to run an historical simulation with the newest CMIP7 forcing files on Olivia, but the volcanic aod file /cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc (which is available on Betzy), is not found

Loading input file list: 'Buildconf/dglc.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Checking server https://osdf-data.gdex.ucar.edu/ncar/gdex/d651077/cesmdata/inputdata/ with protocol wget
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using protocol wget with user  and passwd 
Connection established with nonzero code 8
Loading input file list: 'Buildconf/blom.input_data_list'
Loading input file list: 'Buildconf/cice.input_data_list'
Loading input file list: 'Buildconf/ctsm.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'
Model cam missing file prescribed_volcaero_file = '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc'
Trying to download file: 'atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' to path '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' using WGET protocol.
wget failed with output:  and errput --2026-04-26 00:36:03--  https://osdf-data.gdex.ucar.edu/ncar/gdex/d651077/cesmdata/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc
Resolving osdf-data.gdex.ucar.edu (osdf-data.gdex.ucar.edu)... 128.117.251.130
Connecting to osdf-data.gdex.ucar.edu (osdf-data.gdex.ucar.edu)|128.117.251.130|:443... connected.
WARNING: cannot verify osdf-data.gdex.ucar.edu's certificate, issued by ‘CN=ncar-osdf00.hpc.ucar.edu’:
  Self-signed certificate encountered.
WARNING: no certificate subject alternative name matches
	requested host name ‘osdf-data.gdex.ucar.edu’.
HTTP request sent, awaiting response... 404 Not Found
2026-04-26 00:36:04 ERROR 404: Not Found.

Loading input file list: 'Buildconf/dglc.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Checking server https://redoak.cs.toronto.edu/twitcher/ows/proxy/thredds/fileServer/datasets/CESM/inputdata/ with protocol wget
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using protocol wget with user  and passwd 
Connection established with nonzero code 8
Loading input file list: 'Buildconf/blom.input_data_list'
Loading input file list: 'Buildconf/cice.input_data_list'
Loading input file list: 'Buildconf/ctsm.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'
Model cam missing file prescribed_volcaero_file = '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc'
Trying to download file: 'atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' to path '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' using WGET protocol.
wget failed with output:  and errput --2026-04-26 00:36:06--  https://redoak.cs.toronto.edu/twitcher/ows/proxy/thredds/fileServer/datasets/CESM/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc
Resolving redoak.cs.toronto.edu (redoak.cs.toronto.edu)... 128.100.20.38
Connecting to redoak.cs.toronto.edu (redoak.cs.toronto.edu)|128.100.20.38|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2026-04-26 00:36:06 ERROR 404: Not Found.

Loading input file list: 'Buildconf/dglc.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Checking server https://osdf-data.gdex.ucar.edu/ncar/gdex/d651077/cesmdata/inputdata/ with protocol wget
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using protocol wget with user  and passwd 
Connection established with nonzero code 8
Loading input file list: 'Buildconf/blom.input_data_list'
Loading input file list: 'Buildconf/cice.input_data_list'
Loading input file list: 'Buildconf/ctsm.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'
Model cam missing file prescribed_volcaero_file = '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc'
Trying to download file: 'atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' to path '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' using WGET protocol.
wget failed with output:  and errput --2026-04-26 00:36:08--  https://osdf-data.gdex.ucar.edu/ncar/gdex/d651077/cesmdata/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc
Resolving osdf-data.gdex.ucar.edu (osdf-data.gdex.ucar.edu)... 128.117.251.130
Connecting to osdf-data.gdex.ucar.edu (osdf-data.gdex.ucar.edu)|128.117.251.130|:443... connected.
WARNING: cannot verify osdf-data.gdex.ucar.edu's certificate, issued by ‘CN=ncar-osdf00.hpc.ucar.edu’:
  Self-signed certificate encountered.
WARNING: no certificate subject alternative name matches
	requested host name ‘osdf-data.gdex.ucar.edu’.
HTTP request sent, awaiting response... 404 Not Found
2026-04-26 00:36:08 ERROR 404: Not Found.

Loading input file list: 'Buildconf/dglc.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Checking server https://ftp.cgd.ucar.edu/cesm/inputdata/ with protocol wget
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using protocol wget with user  and passwd 
Connection established with nonzero code 0
Loading input file list: 'Buildconf/blom.input_data_list'
Loading input file list: 'Buildconf/cice.input_data_list'
Loading input file list: 'Buildconf/ctsm.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'
Model cam missing file prescribed_volcaero_file = '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc'
Trying to download file: 'atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' to path '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' using WGET protocol.
wget failed with output:  and errput --2026-04-26 00:36:10--  https://ftp.cgd.ucar.edu/cesm/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc
Resolving ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)... 128.117.13.41
Connecting to ftp.cgd.ucar.edu (ftp.cgd.ucar.edu)|128.117.13.41|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2026-04-26 00:36:10 ERROR 404: Not Found.

Loading input file list: 'Buildconf/dglc.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Checking server https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/ with protocol wget
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using protocol wget with user  and passwd 
Connection established with nonzero code 0
Loading input file list: 'Buildconf/blom.input_data_list'
Loading input file list: 'Buildconf/cice.input_data_list'
Loading input file list: 'Buildconf/ctsm.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'
Model cam missing file prescribed_volcaero_file = '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc'
Trying to download file: 'atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' to path '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' using WGET protocol.
wget failed with output:  and errput --2026-04-26 00:36:12--  https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc
Resolving svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)... 128.117.13.55
Connecting to svn-ccsm-inputdata.cgd.ucar.edu (svn-ccsm-inputdata.cgd.ucar.edu)|128.117.13.55|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2026-04-26 00:36:13 ERROR 404: Not Found.

Loading input file list: 'Buildconf/dglc.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Checking server https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata with protocol svn
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Using protocol svn with user  and passwd 
Loading input file list: 'Buildconf/blom.input_data_list'
Loading input file list: 'Buildconf/cice.input_data_list'
Loading input file list: 'Buildconf/ctsm.input_data_list'
Loading input file list: 'Buildconf/cam.input_data_list'
Model cam missing file prescribed_volcaero_file = '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc'
Trying to download file: 'atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' to path '/cluster/work/projects/nn9560k/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' using SVN protocol.
svn export failed with output:  and errput svn: E170000: URL 'https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/atm/cam/volc/cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc' doesn't exist

Loading input file list: 'Buildconf/dglc.input_data_list'
Loading input file list: 'Buildconf/cpl.input_data_list'
Loading input file list: 'Buildconf/mosart.input_data_list'
Checking server None with protocol None
Setting resource.RLIMIT_STACK to -1 from (-1, -1)
Client protocol None not enabled
ERROR: Could not find all inputdata on any server
 

[DirkOlivie]

I see that Steve has copied the missing file to olivia (cmip7_volcaod_v20260105/UOEXETER-CMIP-2-2-1_175001-202312_intpwghtavg_v20260105.nc).
(I have now changed the group for the file on betzy so that it will be backed-up to nird)

[adagj]

Actually, I did it last night. I intended to copy it to my local dir (which is what I usually do) but I guess I was a bit tired, so I copied it to the main input directory instead. I hope I didn't mess up the input data workflow, or permissions. The simulation is running fine :-)

[jgriesfeller]

JFYI: The inputdata backup has been run once again this morning. The file mentioned (and others) above has been transferred to nird.

[monsieuralok]

@jgriesfeller please could you copy following files from Betzy to nird:
ocn/blom/bndcon/topo_beta_tnx1v4_20170622_depth_python_SYNBATH_V1.1_3minute_filt_max100km_nc64bitoffset.nc
share/meshes/T62_040121_ESMFmesh.nc
share/meshes/T62_040121_ESMFmesh_c20190714.nc as those not having noresm group;
@JensBDebernard please could you change group to noresm for following files share/meshes/T62_040121_ESMFmesh_c20190714.nc share/meshes/T62_040121_ESMFmesh.nc @AleksiNummelin please can you do for ocn/blom/bndcon/topo_beta_tnx1v4_20170622_depth_python_SYNBATH_V1.1_3minute_filt_max100km_nc64bitoffset.nc

[jgriesfeller]

The data has been copied:

(base) [jang@login-1.betzy .../NS16001B/inputdata]$ pwd
/nird/datalake/NS16001B/inputdata
(base) [jang@login-1.betzy .../NS16001B/inputdata]$ ll ocn/blom/bndcon/topo_beta_tnx1v4_20170622_depth_python_SYNBATH_V1.1_3minute_filt_max100km_nc64bitoffset.nc
-rw-rw-r-- 1 jang noresm 3326704 May 20  2024 ocn/blom/bndcon/topo_beta_tnx1v4_20170622_depth_python_SYNBATH_V1.1_3minute_filt_max100km_nc64bitoffset.nc
(base) [jang@login-1.betzy .../NS16001B/inputdata]$ ll share/meshes/T62_040121_ESMFmesh_c20190714.nc
-rw-rw-r-- 1 jang noresm 1014812 Feb  4  2022 share/meshes/T62_040121_ESMFmesh_c20190714.nc
(base) [jang@login-1.betzy .../NS16001B/inputdata]$ ll share/meshes/T62_040121_ESMFmesh.nc
-rw-rw-r-- 1 jang noresm 960668 Feb  4  2022 share/meshes/T62_040121_ESMFmesh.nc