ini_parent variables: restart files not copied

[tsemmler05]

On aleph I am trying to run a branch-off simulation from hi1950d with AWI-CM 3.

I am using esm_master version 6.19.3

I specified in the run script /proj/awiiccp5/test.yaml:

awicm3:
postprocessing: false
model_dir: /proj/awiiccp5/awicm3-frontiers-xios/
ini_parent_exp_id: "hi1950d"
ini_parent_date: "1990-12-31"
ini_parent_dir: "/scratch/awiiccp5/hi1950d/"

In addition, I tried to specify the ini_* variables in

oifs:

section but without success. The restart files are not copied into the new work directory /scratch/awiiccp5/test/run_19900101-19901231/work . It also didn't help to put as ini_parent_dir /scratch/awiiccp5/hi1950d/restart/oifs/19901231/ , i.e. put the whole path in which the OpenIFS restart files are located.

[mandresm]

Hi @tsemmler05,

oifs.lresume is set to false, that's problem number 1. Of course that means you need to define also the prev_run_config_file, but if you don't define that variable ESM-Tools gives you an explanation of why is it needed for OpenIFS and which value to give to this variable.

There is at least one problem more with your runscript. The way you specify ini_parent_dir in oifs is incorrect. The way of generally specifying the ini_parent_dir is:

fesom:
    ini_restart_dir: "/scratch/awiiccp5/hi1950d/restart/fesom/"
oifs:
    ini_restart_dir: "/scratch/awiiccp5/hi1950d/restart/oifs/"

Alternatively, you can define a general ini_restart_dir and invoke that in the model lines.

general:
    ini_restart_dir: "/scratch/awiiccp5/hi1950d/restart/"
fesom:
    ini_restart_dir: "${general.ini_restart_dir}/fesom/"
oifs:
    ini_restart_dir: "${general.ini_restart_dir}/oifs/"

For more details please refer to this chapter in the workshop: https://github.com/esm-tools/workshops/blob/main/202207_AWI_ESM-Tools/docs/pdf/7.4_restarting_a_simulation.pdf

Note that I substituted in all those variables parent with restart. Both will work, but we will remove the support for the parent ones in the future for clarity, so I strongly recommend you to switch those variables names.

And this is the typical way of restarting or branching off a simulation. However, let's notice in your runscript the following (I am assuming your runscript is very similar to the one for the parent simulation):

    choose_general.run_number:
        1:
            #lasttime: '86340'
            restart_in_sources:
                par_oce_restart: /scratch/awiiccp5/hi1950d/run_19900101-19901231/work/fesom.1990.oce.restart/*.nc
                par_ice_restart: /scratch/awiiccp5/hi1950d/run_19900101-19901231/work/fesom.1990.ice.restart/*.nc
        "*":
            remove_restart_in_files: [par_oce_restart, par_ice_restart]
            remove_restart_out_files: [par_oce_restart, par_ice_restart]
            add_ignore_sources:
                par_oce_restart: fesom.*.oce.restart/*
                par_ice_restart: fesom.*.ice.restart/*

If this was the same for your parent simulation, with the lines below "*" you are saying: ignore my parallel restarts, meaning they won't be available in the usual way in /scratch/awiiccp5/hi1950d/restart/fesom/, which is fine as long as you keep the the run folders around as you did, and as long as you add the lines under the 1: which you did. alternatively, you could remove the lines in 1" and just write there a dummy variable to hold the case for 1, and define the ini_parent_dir as /scratch/awiiccp5/hi1950d/run_19900101-19901231/work/:

fesom:
    ini_parent_dir: "/scratch/awiiccp5/hi1950d/run_19900101-19901231/work/"
    choose_general.run_number:
        1:
            placeholder: variable
        "*":
            remove_restart_in_files: [par_oce_restart, par_ice_restart]
            remove_restart_out_files: [par_oce_restart, par_ice_restart]
            add_ignore_sources:
                par_oce_restart: fesom.*.oce.restart/*
                par_ice_restart: fesom.*.ice.restart/*

Another point, you probably want to use the old rstas.nc and rstos.nc and not som default fluxes defined in the pool. To achieve that i recommend to define your oasis3mct.ini_restart_dir: /scratch/awiiccp5/hi1950d/run_19900101-19901231/work/ to ensure that the correct files are copied.

Summary

This is one of the many ways you can fix your runscript:

awicm3:
    postprocessing: false
    model_dir: /proj/awiiccp5/awicm3-frontiers-xios/
-    ini_parent_exp_id: "hi1950d"      # This three are not needed
-    ini_parent_date: "1990-12-31"
-    ini_parent_dir: "/scratch/awiiccp5/hi1950d/"

fesom:
    #setup_dir: "/proj/awiiccp5/awicm3-frontiers-xios/fesom-2.0/"
-    ini_parent_exp_id: "hi1950d"
+    ini_restart_exp_id: "hi1950d"
-    ini_parent_date: "1990-12-31"
+    ini_restart_date: "1990-12-31"
-    ini_parent_dir: "/scratch/awiiccp5/hi1950d/run_19900101-19901231/work/"
+    ini_restart_dir: "/scratch/awiiccp5/hi1950d/run_19900101-19901231/work/"
    resolution: "DART"
    mesh_dir: /proj/awi/input/fesom2/dart/
    lresume: true  # don't forget to check!
    restart_rate: 1
    restart_unit: "y"
    restart_first: 1
    choose_general.run_number:
        1:
            #lasttime: '86340'
            restart_in_sources:
                par_oce_restart: /scratch/awiiccp5/hi1950d/run_19900101-19901231/work/fesom.1990.oce.restart/*.nc
                par_ice_restart: /scratch/awiiccp5/hi1950d/run_19900101-19901231/work/fesom.1990.ice.restart/*.nc
        "*":
            remove_restart_in_files: [par_oce_restart, par_ice_restart]
            remove_restart_out_files: [par_oce_restart, par_ice_restart]
            add_ignore_sources:
                par_oce_restart: fesom.*.oce.restart/*
                par_ice_restart: fesom.*.ice.restart/*
    namelist_changes:
        namelist.config:
           machine:
             n_levels: 2
             n_part: [10, 72]
           restart_log:
             raw_restart_length: 1
             raw_restart_length_unit: "y"
    namelist_dir: /proj/awiiccp5/esm_tools/namelists/fesom2/2.0/awicm3/DART_niko_kpp/  
    add_file_movements:
        outdata:
            all_directions: move
    time_step: 240
    nproc: 4800

oifs:
-    ini_parent_exp_id: "hi1950d"
+    ini_restart_exp_id: "hi1950d"
-    ini_parent_date: "1990-12-31"
+    ini_restart_date: "1990-12-31"
-    ini_parent_dir: "/scratch/awiiccp5/hi1950d/restart/oifs/19901231/"
+    ini_restart_dir: "/scratch/awiiccp5/hi1950d/restart/oifs/"
    resolution: "TCO319"
    levels: "L138"
    prepifs_expid: a0iw
    input_expid: awi3
    wam: true 
-    lresume: false
+    lresume: true
+    prev_run_config_file: "/scratch/awiiccp5/hi1950d/config/hi1950d_finished_config.yaml_19900101-19901231"
    time_step: 900
    nproc: 1040
    omp_num_threads: 2
    namelist_changes:
        fort.4:
            naerad:
                NCMIPFIXYR: 0
                SSPNAME: "SSP5-8.5"
            namcumf:
                RPRCON: 1.0E-3
                ENTRORG: 1.25E-3
            namgwwms:
                GGAUSSB: -0.45

oasis3mct:
    lresume: true # Set to false to generate the rst files for first leg
+    ini_restart_exp_id: "hi1950d"
+    ini_restart_date: "1990-12-31"
+    ini_restart_dir: "/scratch/awiiccp5/hi1950d/run_19900101-19901231/work/"
    time_step: 3600

As non of this is a bug, I'm moving it to Discussions.

[tsemmler05]

Thanks for the clarification!

Probably, in the alternative way of fixing the problem, you mean:

general:
ini_parent_dir: "/scratch/awiiccp5/hi1950d/restart/"

(rather than ini_restart_dir)

[tsemmler05]

I had to further refine the solution that you gave in the summary:

For oasis3mct I had to specify ini_restart_dir as /scratch/awiiccp5/hi1950d/restart/oasis3mct/ probably because of the format the rstas und rstos files are stored (rstas.nc_19900101-19901231).

But other than that it worked.

Question: I hadn't found this information how to make a branch-off simulation in the readthedocs. Would it be possible to transfer the information from the course material into the readthedocs?

[mandresm]

Yes, we will add this to the documentation, once we get your simulation running.

[tsemmler05]

The only problem of this solution is: the wave files were not copied over and I had to manually copy them to the work directory (BLS*, LAW*). What was a bit annoying: I had to also write a little script to rename them so that the year in the filename would be correct. For the next time: would there be an easier way doing this? In principle the esm_tools should copy and rename the wave files automatically?

[mandresm]

Hi @tsemmler05, yes, ESM-Tools should do this automatically. See the reply below for the solution. You should never have to handle things by hand.

[tsemmler05]

Oh no, this workaround didn't even work - the wave model still crashed. Very annoying! Am I missing some files?

Maybe you could take a look at the script:
/proj/awiiccp5/awicm3-frontiers-xios-aleph-TCO319L137-DART-85_1950d-restartfromcoupled.yaml

and into the directory
/scratch/awiiccp5/ssp585d/run_20150101-20151231/work

[mandresm]

Hi Tido,

Appologies for the problems you are having, my mistake. I have managed to create a branchoff work folder that contains all the necessary files. The only thing I needed to change was:

oifs:
    ini_restart_exp_id: "hi1950d"
-    ini_restart_date: "2014-12-31"
+    ini_restart_date: "${prev_date}"

Why? OpenIFS restart logic it's very convoluted: if you remember, we need to trick OpenIFS into thinking that is always running in the second leg to get to run long simulations. For that the configs/components/oifs/oifs.yaml has quite some intricate calendar logic for renaming the BLS, LAW and srf restarts. For that logic one needs the ini_restart_date to include up to the seconds of the previous timestep: ini_restart_date: 2014-12-31T23:45:00, defined as it was it was interpreted as ini_restart_date: 2014-12-31T00:00:00 (beginning of the day, not real end of the leg). Getting the time right is important because otherwise ESM-Tools searches for the files with the wrong name. What ${prev_date} does is giving you the value of the current date of the simulation minus the time step, therefore you get the correct ini_restart_date.

An additional point, please make sure you use oasis3mct.ini_restart_dir: "/scratch/awiiccp5/hi1950d/run_20140101-20141231/work/". Defining that on my runscript I get a rstas/rstos.nc in the work folder. The reason for this change to be needed, is that ESM-Tools searches for rstas/rstos.nc in the restart folder, and in /scratch/awiiccp5/hi1950d/restart/oasis3mct/ those are soft links to rstas/rstos.nc_<DATE_OF_LATEST_LEG> which means, if you branchoff a simulation from one of the intermediate legs you'll be using the wrong rstas/rstos.nc files. This is a bug and it's documented in the issue #826. We should be working on a fix next week so that this problem is solved once for all.

A final recommendation: I would give a new expid to your branchoff experiment or delete your previous one (/scratch/awiiccp5/ssp585d/) so that you start from a clean slate and we make sure nothing weird is going on with the manually edited files there.

Experiment folder: /scratch/awiiccp/tido_941/run_20150101-20151231/work/
Runscript: /home/awiiccp/my_scripts/awicm3-frontiers-xios-aleph-TCO319L137-DART-85_1950d-restartfromcoupled.yaml

[JanStreffing]

I've been following this a little bit. I make a note here that I will condense the lessons learned here into a little how to section of the AWI-CM3 documentation.

[tsemmler05]

Thanks, @mandresm, the solution worked! And good idea @JanStreffing to add this to AWI-CM3 documentation.

[mandresm]

I will also add a more general description to the documentation on how to branchoff experiments in the ESM-Tools documentation.

[nwieters]

I can also help with such a description since I am currently working on such a branchoff experiment myself.

[mandresm]

If you don't mind, I'll take care of this. I'll ask for your review though so you can further contribute.