diff --git a/documentation/docs/pages/configurations/pan-Antartic/Detailed_panan_isf.md b/documentation/docs/pages/configurations/pan-Antartic/Detailed_panan_isf.md new file mode 100644 index 000000000..9222ad5f1 --- /dev/null +++ b/documentation/docs/pages/configurations/pan-Antartic/Detailed_panan_isf.md @@ -0,0 +1,292 @@ +# ACCESS‐OM3 config with ice shelves ‐ input files and config details + +In this wiki, we first describe some aspects of the ACCESS-OM3 pan-Antarctic model with ice shelves at 1/12th degree (approximately 4km around the Antarctic continental shelf) horizontal resolution. We then show how the files were made (drawing from notes on the [MOM6-SIS2 version](https://github.com/claireyung/mom6-panAn-iceshelf-tools/wiki/How-to-make-a-new-panan-MOM6%E2%80%90SIS2-config-with-ice-shelves), which was developed first), for reference and for future modified configurations. Also refer to the config without ice shelves, documented [here](https://access-om3-configs.access-hive.org.au/pr-previews/573/configurations/pan-Antartic/Overview/). + +## Configuration overview + +The 4km ACCESS-OM3 pan-Antarctic model with ice shelves is built on the ACCESS-OM3 25km development, using the same ocean model MOM6 and sea ice model CICE6 with a data atmosphere and runoff, and NUOPC coupler. The domain spans -86.5 degrees to -37.5 degrees S at 1/12th degree resolution, nominally 4km near Antarctica (but nominally 8km near the equator). Similar to previous MOM6-SIS2 pan-Antarctic models (e.g. [Schmidt et al. 2025](https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2024MS004621)), the northern boundary uses open boundary conditions forced by an ACCESS-OM2-01 repeat year forced simulation, with daily output from a simulation close to initial conditions. + +The configuration includes static ice shelves as part of the MOM6 model (see [Stern et al. 2017](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1002/2017MS001002), [Stern et al. 2019](https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2018JC014876), [Yung et al. 2024](https://tc.copernicus.org/articles/19/5827/2025/), [Yung et al. 2025a](https://egusphere.copernicus.org/preprints/2025/egusphere-2025-1942/) and [Yung et al. 2025b](https://essopenarchive.org/users/1005425/articles/1366578-assessment-of-a-finite-volume-discretization-of-the-horizontal-pressure-gradient-force-beneath-sloping-ice-shelves) for examples of idealised versions of MOM6 ice shelves and some details on the implementation). The vertical coordinate uses the Arbitrary Lagrangian-Eulerian coordinate of MOM6, with the target coordinate `SIGMA_SHELF_ZSTAR`, developed by [Stern et al. 2017](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1002/2017MS001002). This vertical coordinate is zstar (geopotential) in the open ocean, but follows the ice draft below ice shelves in a quasi-sigma coordinate fashion, with some layers outcropping into the bottom bathymetry but none into the ice shelf, except at the grounding line. 75 vertical levels are used with a top layer thickness of 1.08m in the open ocean, which is compressed beneath the ice shelf leading to upper layers beneath the ice shelf of ~0.1 m. + +As in the ACCESS-OM3 25km version, we use GEBCO 2024 data to create the bathymetry, which is stitched to [Charrassin et al. 2025](https://www.nature.com/articles/s41598-024-81599-1) bathymetry data where it is available (EPSG:3031 Polar Stereographic grid). Ice thickness is also from [Charrassin et al. 2025](https://www.nature.com/articles/s41598-024-81599-1). We do not use the dynamic ice sheet capability of MOM6, instead making the static ice shelf assumption where mass loss to the ice shelf by melting is instantaneously offset by snowfall and ice movement to keep the same ice shelf shape. Meltwater is added as a volume and heat flux, so sea level could rise. The [Holland and Jenkins (1999)](https://journals.ametsoc.org/view/journals/phoc/29/8/1520-0485_1999_029_1787_mtioia_2.0.co_2.xml) ice shelf basal melt parameterisation with [McPhee (1981)](https://link.springer.com/article/10.1007/BF00119277) stability parameter is used, with a larger drag coefficient tuned to achieve melt rates on the order of other simulations and satellite melt rate estimates. Initial conditions use [Yamazaki et al. 2025 data](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2024JC020920) and World Ocean Atlas 2023 below 2000 m and north of 45 degrees S. + +## Input files + +_Steps used to make input files for the ACCESS-OM3 ice shelf pan-Antarctic model._ + +Please note that file names refer to those made in the https://github.com/claireyung/mom6-panAn-iceshelf-tools/ repo. They have been moved and renamed in [this issue](https://github.com/ACCESS-NRI/access-om3-configs/issues/880). + +### 1. Grid +The first step is to generate a supergrid. We used the [ocean_model_grid_generator](https://github.com/ACCESS-NRI/ocean_model_grid_generator). Something like the following would generate a global grid at 1/12th resolution with a transition from Mercator to fixed latitude at 75S and no shifted South Pole. These can require a lot of memory, so a PBS or interactive PBS job is a good idea: + +``` +python ocean_grid_generator.py -r 12 --no_south_cap --ensure_nj_even --bipolar_lower_lat 65 --mercator_lower_lat -75 --mercator_upper_lat 65 --match_dy so --shift_equator_to_u_point --south_ocean_lower_lat -86.5 --lower-lon -280.0 -f /scratch/x77/cy8964/grids/ocean_hgrid_025_ext.nc +``` + +Discussion: https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/7 + +### 2. Topography +We used the [Charrassin et al. 2025](https://www.nature.com/articles/s41598-024-81599-1) bathymetry and ice products, since GEBCO didn't have the required data (see [this discussion](https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/2)). However, we still need the topography where Charrassin data is not available (it's on the EPSG:3051 polar stereographic grid). So, using the topography generation pipeline is still needed. Something like this [make_OM3_025deg_topo](https://github.com/ACCESS-NRI/make_OM3_025deg_topo) but [WITHOUT the topo edits](https://github.com/ACCESS-NRI/make_OM3_025deg_topo/blob/main/gen_topo.sh#L50) which are hardcoded for the 0.25deg config. This will generate a GEBCO topography for a global context with NaNs beneath the Antarctic ice sheet. + +The next step is to generate topography and ice thickness files for the new product. Similar steps are used if you want to use BedMachine. This is done in a few steps, explained in [this notebook](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/generate-draft/Generate-Charrassin-bathy.ipynb) +1. Download the data from https://datadryad.org/dataset/doi:10.5061/dryad.rbnzs7hkc +2. Read the polar stereographic data and assign lat/lon values to each point using pyproj, and save the file as a netcdf with lat and lon variables +3. Use the ESMF regridder in a PBS job to create regridding weights from the polar grid to the model grid +``` +qsub -I -P YOUR_PROJECT -q hugemem -l ncpus=192,mem=5880G,walltime=05:00:00,storage=gdata/hh5+gdata/ik11+gdata/vk83+gdata/YOUR_PROJECT +module load esmf/8.6.1 openmpi/4.1.7 + + +mpirun -np 192 ESMF_RegridWeightGen -p none -i --ignore_degenerate -s /g/data/x77/cy8964/Charrassin2025_Data/BED_ANTGG2022_lat_lon.nc -d /g/data/x77/ahg157/inputs/mom6/global-8km/topog.nc -w /g/data/x77/cy8964/mom6/input/input-8km/Regridd_Charrassin_to_8km_global_grid_Bilinear.nc -m bilinear --netcdf4 --src_regional --check +``` +4. Open the topography file you generated for the global grid +5. Regrid the polar depth data to the global grid, using precalculated weights. This will look something like the following +image + +6. Where data is not available in this regridded polar grid, fill with GEBCO topography we already generated +7. Save file and add metadata, maybe check if there is a discontinuity at the join. +8. Repeat for the variables `ICE_THICKNESS` and `SURFACE` (and maybe `WATER_HEIGHT`); skip the step of joining to existing topography because that doesn't exist. +This will leave you with topography files on the new grid, including in ice shelf cavities. It also has it for all of Antarctica, not just the ocean bits; this allows you to run the model with a vanished ocean and dynamic ice sheet if you so desire. + +### 3. Initial Conditions and Salt Restoring +We generate initial conditions by regridding Yamazaki et al. 2025 and WOA2023 data, see [this notebook](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/claire_working/initial-conditions/Yamazaki_IC_into_8km_grid-Copy1.ipynb). Alternatives could be using ACCESS-OM2-01 data from the Jan 1 after second year restart (similar to WOA) or WOA data, but one must be careful about which temperature/salinity variable you use, and we've found the WOA data is [quite cold on the Amundsen shelf](https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/45). While extrapolating southwards, we also manually extrapolate in a different direction for parts of some cavities (Weddell, Ross, Larsen C) where otherwise the southward-extrapolated initial conditions are very warm in the cavities. + +Screenshot 2026-01-13 at 3 19 07 pm + +Salt restoring [used a similar method](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/initial-conditions/salt-restoring.ipynb) and data based on WOA13. We use a [salt restoring mask](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/claire_working/generate-draft/process-topo-8k-minimal-topoedited.ipynb) defined by 1000m isobath (the continental shelf), saved as `salt_restore_mask.nc`. + +*Ensure your E-W boundaries are correct! And make sure your files don't have a NaN fill value of NaN, this can cause problems.* + +Discussion: https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/3 and https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/13 + +### 4. Boundary Conditions +We generated boundary conditions at our chosen boundary latitude using the same method as the COSIMA panan using [this script](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/generate-obcs/ACCESS-OM2_panan_boundary_forcing_8km.ipynb). This script uses daily output from the second year of ACCESS-OM2-01 output, potential temperature and practical salinity, to be consistent with the initial conditions. **Didn't work with recent conda environments, so may need a code update** + +### 5. Truncating files +If our files are on a global grid, then we need to truncate them to the pan-Antarctic domain. We need to ensure this is consistent with all the files. We can use `nco` as an efficient `netcdf` chopper. +``` +module load nco/5.0.5 +module load netcdf +``` +You can use `ncdump -h XX.nc` to check what the y coordinate name and size are. To crop the domain to approximately 37.5 degrees S, we did trial and error to get the right index. For `ocean_hgrid.nc`, it is on a supergrid so has twice the number of points (ish) as the other variables. + +``` +ncks -d nyp,0,2884 -d ny,0,2883 ocean_hgrid.nc ocean_hgrid_cropped.nc +ncdump -v "y" ocean_hgrid_cropped.nc | tail -5 +``` + +Note `nyp` number is one larger than `ny` number. The output of the second command will tell you the latitude it finishes at. Adjust the nyp and ny numbers until satisfied. + +Now we do the same for the smaller netcdf files, noting ny in `ocean_hgrid`=2 times ny in other files. These numbers do need to work out! +As before, this is a global grid, so we truncate to get the desired grid size. I chose the top boundary to be so that the top `nyp` `y` value is -37.4627. + +We do the same for `topog` with the different smaller numbers since it's not on the supergrid (and same for other files as necessary) +``` +ncks -d ny,0,1441 topog.nc topog_cropped.nc +``` + +I copied the vertical grid from the 25km model. +``` +cp ../input-25km/ocean_vgrid_cropped.nc . +``` + +### 5. NUOPC input +We need to provide the coupler with a mesh mask of where to apply coupling between atmosphere - sea ice - ocean. Similar to the pan-Antarctic [version without ice shelves](https://access-om3-configs.access-hive.org.au/pr-previews/573/configurations/pan-Antartic/Overview/), but using the [sea ice topography](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/claire_working/generate-draft/process-topo-8k-minimal-topoedited.ipynb) (topography file with only open ocean) since NUOPC shouldn't be exchanging fields between the atmosphere and ocean below ice shelves. This requires [code changes to MOM6](https://github.com/ACCESS-NRI/MOM6/issues/29) to allow this to run so ensure an appropriate MOM6 executable is being used. + +In the final version I used the following, which draws on these [python scripts for mesh generation](https://github.com/ACCESS-NRI/om3-scripts/tree/main/mesh_generation) +``` +python3 generate_mesh.py --grid-type=mom --grid-filename=/g/data/x77/cy8964/mom6/input/input-8km/ocean_hgrid_cropped.nc --mesh-filename=/g/data/x77/cy8964/mom6/input/input-8km/150925/access-om3-8km-ESMFmesh_Charrassin_opencavity_cropped.nc --mask-filename=/g/data/x77/cy8964/mom6/input/input-8km/150925/mask_Charrassin_sea_ice_from_iceelev_new150925.nc --wrap-lons + +python3 generate_mesh.py --grid-type=mom --grid-filename=/g/data/x77/cy8964/mom6/input/input-8km/ocean_hgrid_cropped.nc --mesh-filename=/g/data/x77/cy8964/mom6/input/input-8km/150925/access-om3-8km-nomask-ESMFmesh_Charrassin_nocavity_cropped.nc --wrap-lons + +python3 generate_rof_weights.py --mesh-filename=/g/data/x77/cy8964/mom6/input/input-8km/150925/access-om3-8km-ESMFmesh_Charrassin_nocavity_cropped.nc --weights_filename=/g/data/x77/cy8964/mom6/input/input-8km/150925/access-om3-8km-rof-remap-weights_Charrassin_nocavity_cropped.nc + +``` + +> [!IMPORTANT] +> When we run the model for real, we need to make sure the exact **sea ice** topography is used for making the coupling mesh files (i.e. the topography file but with 0s or NaNs in the ice shelf cavities, since we don't want sea ice or atmospheric fluxes in there). For now it doesn't really matter, since we will regenerate the file later. So we could probably get away with using the version of the file for the no ice shelf configuration here and replace it later. + +Copy these files to your input directory. + +### 6. Set up config + +The easiest place to start is by cloning an existing ice shelf config, e.g. the alpha release **when it is available **(https://github.com/ACCESS-NRI/access-om3-configs/pull/814) + +You can use `payu` to do this, for example +``` +module use /g/data/vk83/modules +module load payu +payu clone --new-branch expt --branch dev-MC_4km_jra_ryf+regionalpanan+isf https://github.com/ACCESS-NRI/access-om3-configs dev-MC_4km_jra_ryf+regionalpanan+isf +``` +This config has ice shelf parameters, links to input files, `diag_table` with diagnostics for ice shelves (make sure to save the `e` variable for plotting, and the `ice_shelf_model` diagnostics e.g. `melt_rate`), and modified runoff streams files `drof.streams.xml` without the liquid runoff file (remove the [`FRIVER` related lines](https://github.com/ACCESS-NRI/access-om3-configs/issues/728)). You can update `config.yaml` with the correct project, shortpath, input files etc. You may need to update other files elsewhere, e.g. in `MOM_input` the initial conditions, topo file etc, tidal amplitude file (see step 13 and turn `READ_TIDEAMP = False` for now), and other grid-related files in `ice_in`, `nuopc.runconfig`, `drof_in`, `datm_in`. If changing layout, it might be easiest to not use a mask table and just use the automatic layout method. + +Alternatively, you could start with a non-ice shelf config and modify it. See section 9 for ice shelf-related parameters I recommend. + +### 7. Quick run to get area + +We are going to quickly run the model to get area files, as this is needed for the ice shelf input and let's assume we haven't made ice shelf input files yet. + +If we cloned the ice shelf version, we'll need to turn ice shelves off if we don't have the ice shelf input file. Modify the MOM_input so that +``` +ICE_SHELF = False +TRIM_IC_FOR_P_SURF = False +``` +Also set the following, so that we get the area everywhere +``` +#override WRITE_GEOM = 1 +#override TOPO_CONFIG = "flat" +#override MAX_DEPTH = 6000 +``` +and then run the model: +``` +payu setup +``` +Check the files are in `work/INPUT` +``` +payu sweep +payu run +``` +The model will probably crash, but it should generate the file `work/ocean_geometry.nc`. If it doesn't crash, you can `qdel` the job. + +Save this file! It should have area of tracer cells everwhere in the variable `Ah`. You can just save one variable. +``` +ncks -v Ah work/ocean_geometry.nc /g/data/x77/cy8964/mom6/input/input-8km/area_everywhere.nc +``` +### 8. Make ice draft +Now that we have ice area, we can make an ice shelf input file, which requires ice thickness (!! not draft, thickness!!) and area of the cell. + +Following [this notebook](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/generate-draft/process-topo-8k-minimal.ipynb), we make an ice thickness file, setting it to be area where there is nonzero elevation in the `ICE_SURFACE` file we regridded earlier (if you use just thickness there might be some holes in it). We also create a sea ice topography file based on where the ice surface is zero. This means that the ice sheet and sea ice will NOT overlap. Both of these ocean and sea ice topography files need to have unconnected cells removed, as these unconnected cells crash both MOM6 and CICE6. + +I am not totally sure if this is essential, but I also make a `h_mask` variable in the ice file and change thicknesses that are zero where the elevation is nonzero (probably nunataks or non-ice covered rock above ground in Antarctica) to a very small amount. Otherwise, MOM makes an ice mask based on the value of the thickness, which might mean the ice coverage + sea ice coverage is not quite equal to ocean coverage. + +### 9. Ice shelf config parameters + +If you cloned an ice shelf version of the config you can ignore this step (and simply set `ICE_SHELF` and `TRIM_IC_FOR_P_SURF` back to True and get rid of the extra topo lines), but if you did not, here is a list of parameters you may want to add to a non-ice shelf config as a `MOM_override` file. + +- `input.nml`: Ensure that in `&MOM_input_nml` there is a line `input_filename = 'n'`. Also add the name of your `MOM_override_IS` as another parameter filename. +- [`MOM_override_IS`](https://github.com/claireyung/mom6-panan/blob/1_12_IS_ALE_working_faster/MOM_override_IS): Make this new file where we add ice shelf info +``` +ICE_SHELF = True ! turns on ice shelf +SHELF_INSULATOR = False ! do ice shelf heat conduction (extra term in melt parameterisation) +SHELF_THERMO = True ! melt/freeze the ice shelf +ICE_SHELF_TEMPERATURE = -20.0 ! temperature of the ice for heat conduction term +DENSITY_ICE = 917. ! ice density +USTAR_SHELF_BG = 0.0006 ! minimum background friction velocity, taken from Jourdain et al 2019 +ICE_PROFILE_CONFIG = "FILE" +ICE_THICKNESS_FILE ="ice_thickness_Charrassin_regridded_cropped_add_area_where_iceelev_noGL5m_ADDHMASKFILLTHICK_190825.nc" !name of ice thickness and area file +ICE_THICKNESS_VARNAME = "thick" !variable name +ICE_AREA_VARNAME = "area" !variable name +#override TRIM_IC_FOR_P_SURF = True !initialise ice shelf properly in ALE coordinates using SURFACE_PRESSURE_FILE +#override COL_THICK_MELT_THRESHOLD = 1e-3 ! don't melt when thickness less than 1mm +#override MIN_THICKNESS = 1.e-12 ! minimum ocean thickness +#override ANGSTROM = 1.e-15 ! not sure why, but seems to work +SURFACE_PRESSURE_FILE ="ice_thickness_Charrassin_regridded_cropped_add_area_where_iceelev_noGL5m_ADDHMASKFILLTHICK_190825.nc" !same file as ICE_THICKNESS_FILE +SURFACE_PRESSURE_VAR = "thick" !variable name +SURFACE_PRESSURE_SCALE = 8986.6 !! Ensure this is DENSITY_ICE X G_EARTH +#override REGRIDDING_COORDINATE_MODE = "SIGMA_SHELF_ZSTAR" !recommended coordinate for ice shelves as it does not vanish at the ice-ocean interface + +!Melt parameterisation modifications +HMIX_SFC_PROP = 1.08 ! sampling distance of temp and salt in melt param (set to open ocean top cell value since it also affects open ocean) +HMIX_UV_SFC_PROP = 1.08 ! sampling distance of u and v in melt param (set to open ocean top cell value since it also affects open ocean) +MINIMUM_FORCING_DEPTH = 1.08 ! meltwater flux distribution distance (set to open ocean top cell value since it also affects open ocean) +CDRAG_SHELF = 0.007 + +!Custom changes (https://github.com/ACCESS-NRI/MOM6/pull/38) +FRAZIL_NOT_UNDER_ICESHELF = True !don't use frazil under ice shelves +ICE_SHELF_TIDEAMP_SCALING_FACTOR = 0.66 !use a different scaled tidal velocity file + +ICE_SHELF_USTAR_FROM_VEL_BUGFIX = True !recent GFDL bug fix + +! Recommended pressure gradient ice shelf fixes (see https://essopenarchive.org/users/1005425/articles/1366578-assessment-of-a-finite-volume-discretization-of-the-horizontal-pressure-gradient-force-beneath-sloping-ice-shelves) +#override MASS_WEIGHT_IN_PRESSURE_GRADIENT_TOP = True +#override RESET_INTXPA_INTEGRAL = True +#override RESET_INTXPA_INTEGRAL_FLATTEST = True +#override MASS_WEIGHT_IN_PGF_VANISHED_ONLY = True +#override HARMONIC_BL_SCALE = 1 + +! Recommended to fix remapping and initialisation (particularly if using sigma coords) +#override INIT_BOUNDARY_EXTRAP = True +#override TRIMMING_USES_REMAPPING = True +#override TRIM_IC_Z_TOLERANCE = 1.0E-10 +#override REMAPPING_USE_OM4_SUBCELLS = False +#override Z_INIT_REMAP_GENERAL = True + +! init ice shelf bugfix nonlineos +#override FRAC_DP_AT_POS_NEGATIVE_P_BUGFIX = True + +! Add drag (can increases time before crashing but not always, depending on other parameters) +#override LINEAR_DRAG = True +#override DRAG_BG_VEL = 0.05 + +! Make sure DTBT is allowed to be small at the start of the run +#override DTBT_RESET_PERIOD = 0 + +! linear eqn of state freezing point dependence on pressure +#override DTFREEZE_DP = -7.75E-08 + +! generates ocean_geometry file +#override WRITE_GEOM = 1 + +! other options (off unless required) +!!! #override PRESSURE_RECONSTRUCTION_SCHEME = 2 ! use a higher order pressure gradient vertical reconstruction scheme (more expensive) +!!! #override HARMONIC_VISC = True ! can be helpful if having problems near grounding line, but otherwise don't use it + +``` +In the above files, make sure to edit `ICE_THICKNESS_FILE` and `SURFACE_PRESSURE_FILE` to be your new ice file. The above is for `SIGMA_SHELF_ZSTAR` coordinates (ice draft following but can incrop at the bottom, and is zstar in open ocean), `SIGMA` (terrain-following) probably also works but `ZSTAR` has [known problems](https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/5) beneath thermodynamically active ice shelves due to having vanished layers at the ice interface. + +### 10. Layer run to get grounding line + +For some reason, in my experience of this config, ALE mode requires no grounded ocean to initialise (i.e. ocean thickness always finite; it is odd since this is not requirement in idealised models). So, we initialise with layer mode to look at the output and determine where the cavity is grounded or not. +We add the following to `MOM_input` or `MOM_override_IS`, which we will remove later +``` +!Turn on layer mode +#override USE_REGRIDDING = False +#override ENERGETICS_SFC_PBL = False +#override BULKMIXEDLAYER = True +#override MIXEDLAYER_RESTRAT = True +#override HMIX_MIN = 10.0 +#override SAVE_INITIAL_CONDS = True +#override BULK_RI_ML = 0.05 + +``` +Also replace the `TOPO_CONFIG` that we changed earlier back to `TOPO_CONFIG = "file"` and make sure the `TOPO_FILE` is for the whole continent (`topog_Charrassin_nocavity_cropped.nc`) and so is `ICE_THICKNESS_FILE` = `SURFACE_PRESSURE_FILE` = `ice_thickness_Charrassin_regridded_cropped_add_area.nc` + +Run the model again with these parameters (`payu sweep` first). This should create a file `MOM_IC.nc` or several of them if the model is big. The model might run in layer mode but it also might crash! + +### 11. File making, again + +Using the initialised model, we can now make topography files, ice files and nuopc mesh and runoff files without a grounding line, as done in [this notebook](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/generate-draft/process-topo-8k-minimal-topoedited.ipynb). I haven't tested the sensitivity of parameters, but masking `MOM_IC.h.sum('Layer')>5` gives only floating ocean. Make sure to remove unconnected cells again. Save these new files, and then regenerate the nuopc coupler input using step 5 +> [!IMPORTANT] +> Use the sea ice topography when generating the nuopc mask! +> If you have a nonzero masking depth in `MOM_input`it might affect the mask, make sure the masks are all consistent. Sea ice coverage plus ice sheet coverage should exactly equal ocean coverage - you can check this with `ocean_geometry.nc` and `access-om3.cice.static.nc` + +### 12. Run for real in ALE mode! +Update the file paths in your `MOM_input` to use these files with no grounding line, and remove the lines that say to use `layer` mode that we added in step 10. Might need to update the `config.yaml` to point to the new files. + +Think carefully about the timestep as it's quite unstable from rest -- `DT=150` and `DT_THERM = 150` is usually pretty safe at the start with `nuopc.runseq` also having the coupling timestep which needs to be made the same time, and make sure `DT_BT_RESET_PERIOD` isn't hardcoded to be something large. Increasing the sea ice timestep ratio `ndtd` may also be helpful if you get sea ice errors. + +While it is running, you can track the progress with +``` +tail -f work/log/ocn.log +``` +And if it crashes `access-om3.err` will give the error message. + +If you want to run month 2, you can increase the timestep as it should be more stable (monitor the maxCFL in the `ocn.log` file). To restart, you will also need to remove the `input_filename = 'n'` line in `input.nml`. See more run instructions [here](https://access-om3-configs.access-hive.org.au/pr-previews/573/configurations/pan-Antartic/run_panan_isf/) + +### 13. Other files +We also use tidal velocity files for the internal tide mixing and melt parameterisation - https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/claire_working/generate-draft/tidal_roughness_extrap.ipynb and extrapolate the roughness file outside the cavities southward down to the critical latitude. The file outside the cavity can be generated with [om3-scripts](https://access-om3-configs.access-hive.org.au/pr-previews/573/configurations/pan-Antartic/Detailedpananinstructions/) + + +### 14. Debugging + +Common issues that get disguised by unhelpful MOM error messages like "saturation vapour pressure overflow" or "SSH beneath bathymetry" or "NaN in reproducing EPF overflow" +- NaNs in input files +- instabilities after intialisation - drop the timestep and try `HARMONIC_VISC = True` which should arrest flows into vanished layers +- have a look at the `ocean_geometry` file and `MOM_IC` files to check things are what you expected. Check the `work/INPUT` that the paths to the files is correct. +- check the masking coverage of ice/sea ice/ocean + +Alternatively, you might have a [random segfault](https://github.com/ACCESS-NRI/access-om3-configs/issues/863) that goes away if you simply submit the job again. + +image + +There are ongoing issues with the model config that require further development - see this list https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/49 diff --git a/documentation/docs/pages/configurations/pan-Antartic/Detailedpananinstructions.md b/documentation/docs/pages/configurations/pan-Antartic/Detailedpananinstructions.md new file mode 100644 index 000000000..9b6fd8cda --- /dev/null +++ b/documentation/docs/pages/configurations/pan-Antartic/Detailedpananinstructions.md @@ -0,0 +1,574 @@ + +# Instructions to generate pan-Antarctic regional domain from global ACCESS-OM3 Configuration + +Contents: + +1. Introduction +2. Make 25km panan from 25km OM3 and add boundary conditions +3. Convert to 8km pan-An using GEBCO bathy +4. Optimisation +5. Use Charrassin bathy instead +6. Use new parameters +7. Fix bugs + + +# 1. Introduction +This page outlines the steps followed to create the pan-Antarctic configurations (ACCESS-rOM3-panan). It also describes some of the bugs encountered and outlines remaining issues to investigate. The purpose of this document is to: +1. have a record of the steps followed to create the domain and, +2. Document an example workflow for creating a regional domain as a subset of a global domain. + +When modifying this workflow to create a user-defined regional domain, be aware that subsetting of the global domain will need to occur in the x- and y- directions for non-polar domains. +**Note**: The below docs are based off [this markdown file](https://github.com/claireyung/access-om3-configs/blob/8km_jra_ryf_obc2-sapphirerapid-Charrassin-newparams-rerun-Wright-spinup-accessom2IC-yr9/panantarctic_instructions.md) +# 2. Make 25km panan from 25km OM3 and add boundary conditions + +These instructions were used to make a test 25km pan-An regional config. This configuration is a test configuration used for development, including developing the workflow for creating a regional domain from a global domain. We start with a global domain, truncate it, and then change configuration files as required. + +First, load modules for `payu`: +``` +module use /g/data/vk83/modules +module load payu +``` + +Then clone the current dev 25km branch: + +``` +mkdir ~/access-om3-configs/ +cd ~/access-om3-configs/ +payu clone --new-branch expt --branch dev-MC_25km_jra_ryf https://github.com/ACCESS-NRI/access-om3-configs 25km_jra_ryf +``` + +`config,yaml`, in the folder `25km_jra_ryf`, shows the location of the input files. These need to be modified for the regional domain which can be done in a new folder: + +``` +mkdir /g/data/x77/cy8964/mom6/input +mkdir /g/data/x77/cy8964/mom6/input/input-25km +``` + +Then copy the files listed in the `config.yaml` so there is a local copy. Note that file names and paths may differ for different varients of the base global configuration. + +``` +cd /g/data/x77/cy8964/mom6/input/input-25km +cp /g/data/vk83/prerelease/configurations/inputs/access-om3/cice/grids/global.025deg/2025.02.17/kmt.nc . +cp /g/data/vk83/prerelease/configurations/inputs/access-om3/mom/grids/mosaic/global.025deg/2025.01.30/ocean_hgrid.nc . +cp /g/data/vk83/prerelease/configurations/inputs/access-om3/mom/grids/vertical/global.025deg/2025.03.12/ocean_vgrid.nc . +cp /g/data/vk83/prerelease/configurations/inputs/access-om3/mom/initial_conditions/global.025deg/2025.03.19/ocean_temp_salt.res.nc . +cp /g/data/vk83/prerelease/configurations/inputs/access-om3/mom/surface_salt_restoring/global.025deg/2025.01.30/salt_sfc_restore.nc . +cp /g/data/vk83/prerelease/configurations/inputs/access-om3/share/grids/2025.02.17/topog.nc . +``` + +In this case, `ocean_mask.nc` is not listed in `config.yaml` and if it can't be found then a new one needs generating + +### Generating `ocean_mask.nc` +Clone the `make_OM3_025deg_topo` tools. First `cd` into a suitable directory (e.g, your `home` drive) +``` +git clone git@github.com:ACCESS-NRI/make_OM3_025deg_topo.git +``` + +To download the import submodules,do: + +``` +git submodule update --init --recursive +``` + +which downloads the correct branch of the submodules. Note starting a new gadi terminal will fix conflict with the xp86 conda environemnt. + +We then follow the instructions on `https://github.com/ACCESS-NRI/make_OM3_025deg_topo` which requires modifying the `gen_topog.sh` script to 1. Add the gdata to your project in the `#PBS -l storage=` line and 2. Use `qsub` to submit: +``` +qsub -v INPUT_HGRID="/path/to/ocean_hgrid.nc",INPUT_VGRID="/path/to/ocean_vgrid.nc",INPUT_GBCO="/path/to/GEBCO_2024.nc" -P $PROJECT gen_topo.sh +``` +For the 25km Pan-an development this looked like: +``` +qsub -v INPUT_HGRID=/g/data/x77/cy8964/mom6/input/input-25km/ocean_hgrid.nc,INPUT_VGRID=/g/data/x77/cy8964/mom6/input/input-25km/ocean_vgrid.nc,INPUT_GBCO=/g/data/ik11/inputs/GEBCO_2024/GEBCO_2024.nc -P x77 gen_topo.sh +``` +This generates the files where you cloned the `make_OM3_025deg_topo` repo and needs to be copied to the folder containing the input files (e.g. `/g/data/x77/cy8964/mom6/input/input-25km`. + +Note that bathymetry-tools pipeline used here has built in topo edits for the 025 deg config and these topo edits will need removing. + +### Cropping netcdf files to regional domain + +We can now use `nco` as an efficient `netcdf` chopper. Load module: +``` +module load nco/5.0.5 +module load netcdf +``` +The command `ncdump -h XX.nc` (where 'XX.nc' is your netcdf file) can be used to check the y coordinate name and size. To crop the domain to approximately 37.5 degrees S, some trial and error may be needed get the right index. + +``` +ncks -d nyp,0,790 -d ny,0,789 ocean_hgrid.nc ocean_hgrid_cropped.nc +ncdump -v "y" ocean_hgrid_cropped.nc | tail -5 +``` + +The output of the second command will tell you the latitude that the model was cropped to. Adjust the nyp and ny numbers until satisfied with the cropping latitude. +Note that `ocean_hgrid.nc` it is on a supergrid so has twice the number of points as variables in other netcdf files not on the supergrid. +Note also that the `nyp` index is one larger than the `ny` index. + +Now do the same for the smaller netcdf files, noting ny in `ocean_hgrid`=2 times ny in other files. + +``` +ncks -d ny,0,394 kmt.nc kmt_cropped.nc +ncks -d ny,0,394 topog.nc topog_cropped.nc +ncks -d GRID_Y_T,0,394 ocean_temp_salt.res.nc ocean_temp_salt.res_cropped.nc +ncks -d lat,0,394 salt_sfc_restore.nc salt_sfc_restore_cropped.nc +ncks -d ny,0,394 ocean_mask.nc ocean_mask_cropped.nc +``` + +For consistency, rename the `ocean_vgrid.nc` file even though it has no y coordinate. + +``` +mv ocean_vgrid.nc ocean_vgrid_cropped.nc +``` + +There are some more input files related to meshes for the nuopc coupler. These need to be generated from the new files, they can't simply be cropped from the global version. Using the `om3-scripts` submodule in the `make_OM3_025deg_topo` repository, there are scripts to make them. + +If not already loaded: +``` +module use /g/data/xp65/public/modules +module load conda/analysis3 +``` + +Then generate the mesh files, first `access-om3-025deg-ESMFmesh_cropped.nc ` and then `access-om3-025deg-nomask-ESMFmesh_cropped.nc`. Modify path names as necessary to point to your input directory. + +``` +python3 /home/156/cy8964/model-tools/om3-scripts/mesh_generation/generate_mesh.py --grid-type=mom --grid-filename=/g/data/x77/cy8964/mom6/input/input-25km/ocean_hgrid_cropped.nc --mesh-filename=/g/data/x77/cy8964/mom6/input/input-25km/access-om3-025deg-ESMFmesh_cropped.nc --mask-filename=/g/data/x77/cy8964/mom6/input/input-25km/ocean_mask_cropped.nc --wrap-lons + +python3 /home/156/cy8964/model-tools/om3-scripts/mesh_generation/generate_mesh.py --grid-type=mom --grid-filename=/g/data/x77/cy8964/mom6/input/input-25km/ocean_hgrid_cropped.nc --mesh-filename=/g/data/x77/cy8964/mom6/input/input-25km/access-om3-025deg-nomask-ESMFmesh_cropped.nc --wrap-lons +``` + +Generating the `access-om3-025deg-rof-remap-weights_cropped.nc` file requires a PBS submission using a PBS script similar to this: + +``` +#!/usr/bin/env sh +# Copyright 2025 ACCESS-NRI and contributors. See the top-level COPYRIGHT file for details. +# SPDX-License-Identifier: Apache-2.0 + +#PBS -q normal +#PBS -l walltime=4:00:00,mem=10GB +#PBS -l wd +#PBS -l storage=gdata/hh5+gdata/ik11+gdata/x77+gdata/vk83 + +module use /g/data/hh5/public/modules +module load conda/analysis3 +module load nco + +set -x #print commands to e file +set -e #exit on error + +python3 ./om3-scripts/mesh_generation/generate_rof_weights.py --mesh_filename=/g/data/x77/cy8964/mom6/input/input-25km/access-om3-025deg-ESMFmesh_cropped.nc --weights_filename=/g/data/x77/cy8964/mom6/input/input-25km/access-om3-025deg-rof-remap-weights_cropped.nc + +``` +Adjust file names and project storage as appropriate. + +All input files have now been generated and the next steps are to modify the input configuration files. + +### Modifying namelist files +The next step is to modify the namelists in the run directory folder where we originally cloned the global model (`cd ~/access-om3-configs/25km_jra_ryf`). First, we modify `config.yaml`. This needs the following: +- project code +- set `runlog: true` + +Update input paths to where your new cropped input files are stored. For the pan-An 25 km configurations, this looks like: + +``` +input: + - /g/data/x77/cy8964/mom6/input/input-25km/kmt_cropped.nc + - /g/data/x77/cy8964/mom6/input/input-25km/ocean_hgrid_cropped.nc + - /g/data/x77/cy8964/mom6/input/input-25km/ocean_vgrid_cropped.nc + - /g/data/x77/cy8964/mom6/input/input-25km/ocean_temp_salt.res_cropped.nc + - /g/data/x77/cy8964/mom6/input/input-25km/salt_sfc_restore_cropped.nc + - /g/data/x77/cy8964/mom6/input/input-25km/topog_cropped.nc + - /g/data/x77/cy8964/mom6/input/input-25km/access-om3-025deg-ESMFmesh_cropped.nc + - /g/data/x77/cy8964/mom6/input/input-25km/access-om3-025deg-nomask-ESMFmesh_cropped.nc + - /g/data/vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/JRA55do-datm-ESMFmesh.nc + - /g/data/vk83/configurations/inputs/access-om3/share/meshes/share/2024.09.16/JRA55do-drof-ESMFmesh.nc + - /g/data/x77/cy8964/mom6/input/input-25km/access-om3-025deg-rof-remap-weights_cropped.nc + - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data +``` +Noting that the JRA55do atmosphere input stays the same and does not need to change. Also, we start from a deafult CICE initial condition; otherwise that could also be an input file. + +Next modify `datm_in` with the new y length (ny in the kmt_cropped file i.e. real grid size not supergrid): + +``` +model_maskfile = "./INPUT/access-om3-025deg-nomask-ESMFmesh_cropped.nc" +model_meshfile = "./INPUT/access-om3-025deg-nomask-ESMFmesh_cropped.nc" +nx_global = 1440 +ny_global = 395 +``` +Do the same changes in `drof_in`. + +In `MOM_input` we need to change `NJ_GLOBAL = 395` and `TRIPOLAR = False`. Additionally, change all the netcdf files (seach for "*.nc") to align with the new cropped name (e.g. `ocean_hgrid.nc` => `ocean_hgrid_cropped.nc`. + +For `ice_in`, the `history_chunksize` needs to be changed. Choose something proportional to the original i.e. decrease second number by ratio of new ny to old ny): +``` +history_chunksize = 720, 186 +``` + +Also change grid info: +``` + grid_type = "regional" + + ns_boundary_type = "open" + nx_global = 1440 + ny_global = 395 +``` +Where `ny_global` is the new non-supergrid y-dimention + +and file names: +``` +&grid_nml + bathymetry_file = "./INPUT/topog_cropped.nc" + grid_atm = "A" + grid_file = "./INPUT/ocean_hgrid_cropped.nc" + grid_format = "mom_nc" + grid_ice = "B" + grid_ocn = "A" + grid_type = "regional" + kcatbound = 0 + kmt_file = "./INPUT/kmt_cropped.nc" +``` + +`nuopc.runconfig` also needs the updated netcdf file names (search "*.nc") + +Then add into `MOM_override`: +``` +#override TOPO_FILE = "topog_cropped.nc" +``` +This is because the default MOM6 topo file name is topog.nc but we have changed it. + +To run, do `payu setup`, check file paths look right and work directory was made. Then +`payu sweep` +and `payu run`. You can check the status with the command `uqstat`. Errors will come up in the access-om3.eXXXX file. + +The 25km panAn ran in 2.5 hours and used 6.9 kSU. + +### CICE initial conditions +The default CICE IC is "default" which has full ice cover below 60S (and above 60N). Since the simulation starts in January, a zero ice cover initial condition may be more appropriate. This wasn't tested by we suggest it could be controlled in `ice_in` with + +`setup_nml`: `ice_ic` to `'none'` instread of `'deafult'`. + +### OBC instructions + +The boundary condition files can be generated by using [this script](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/generate-obcs/ACCESS-OM2_panan_boundary_forcing_8km.ipynb). However, this script is now depreciated and will need some minor updates to the xarray syntax to allow the notebook to run with updated xarray versions in newer analysis environments (the notebook previously ran on hh5). + +Before running, check that the files give sensible numbers e.g. temperature in celcius. + +Then add file to `config.yaml`: +``` +input: + - /g/data/x77/cy8964/mom6/input/input-25km/forcing_access_yr2_25km_fill.nc +``` + +Then add to `MOM_override`: +``` +! === module MOM_open_boundary === +OBC_NUMBER_OF_SEGMENTS = 1 +OBC_FREESLIP_VORTICITY = True +OBC_FREESLIP_STRAIN = True +!OBC_COMPUTED_VORTICITY = True +OBC_ZERO_BIHARMONIC = True + +OBC_SEGMENT_001 = "J=N,I=N:0,FLATHER,ORLANSKI,NUDGED" +OBC_SEGMENT_001_VELOCITY_NUDGING_TIMESCALES = .3, 360.0 ! inflow and outflow timescales +BRUSHCUTTER_MODE = True ! read data on supergrid +OBC_SEGMENT_001_DATA = "U=file:forcing_access_yr2_25km_fill.nc(u),V=file:forcing_access_yr2_25km_fill.nc(v),SSH=file:forcing_access_yr2_25km_fill.nc(eta_t),TEMP=file:forcing_access_yr2_25km_fill.nc(pot_temp),SALT=file:forcing_access_yr2_25km_fill.nc(salt)" +!RAMP_OBCS = True + +OBC_TRACER_RESERVOIR_LENGTH_SCALE_OUT = 30000 +OBC_TRACER_RESERVOIR_LENGTH_SCALE_IN = 3000 + +! sponges +SPONGE = False +``` +These parameter choices are copied from MOM6-SIS2 panantarctic: https://github.com/COSIMA/mom6-panan/blob/panan-005/MOM_input + + +# 3. Moving to 8km domain + +The above instructions create a [25km domain](https://github.com/claireyung/access-om3-configs/tree/25km_jra_ryf-obc) subsetted from the [global 25km dev model](https://github.com/ACCESS-NRI/access-om3-configs/tree/dev-MC_25km_jra_ryf). + +Now we want to use an 8km model domain. + +First, we use the 8km [global grid that Angus made](https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/7). Copy files from Angus `/g/data/x77/ahg157/inputs/mom6/global-8km/` + +As before, this is a global grid which needs truncating to get the desired grid size. The top boundary is chosen such that the top `nyp` `y` value is -37.4627 (to matche the [initial conditions](https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/3)). + +``` +ncks -d nyp,0,2884 -d ny,0,2883 ocean_hgrid.nc ocean_hgrid_cropped.nc +ncdump -v "y" ocean_hgrid_cropped.nc | tail -5 +``` +A similar operation is needed for for `ocean_mask` and `topog` except the different `ny` as these are not on the supergrid +``` +ncks -d ny,0,1441 topog.nc topog_cropped.nc +ncks -d ny,0,1441 ocean_mask.nc ocean_mask_cropped.nc +``` + +The vertical grid can be used from the 25km model (which was a renamed version of the global model vertical grid) +``` +cp ../input-25km/ocean_vgrid_cropped.nc . +``` + +The kmt (sea ice mask file) is in this case the same as ocean_mask (not that it won't be when ice shelf cavities are introduced) so rename it: +``` +ncrename -O -v mask,kmt ocean_mask_cropped.nc kmt_cropped.nc +ncks -O -x -v geolon_t,geolat_t kmt_cropped.nc kmt_cropped.nc +``` +Then follow the previous pipeline in `make_OM3_025deg_topo` to generate the new mesh files.This was done in a PBS script with the following commands. + +``` +python3 /home/156/cy8964/model-tools/om3-scripts/mesh_generation/generate_mesh.py --grid-type=mom --grid-filename=/g/data/x77/cy8964/mom6/input/input-8km/ocean_hgrid_cropped.nc --mesh-filename=/g/data/x77/cy8964/mom6/input/input-8km/access-om3-8km-ESMFmesh_cropped.nc --mask-filename=/g/data/x77/cy8964/mom6/input/input-8km/ocean_mask_cropped.nc --wrap-lons + +python3 /home/156/cy8964/model-tools/om3-scripts/mesh_generation/generate_mesh.py --grid-type=mom --grid-filename=/g/data/x77/cy8964/mom6/input/input-8km/ocean_hgrid_cropped.nc --mesh-filename=/g/data/x77/cy8964/mom6/input/input-8km/access-om3-8km-nomask-ESMFmesh_cropped.nc --wrap-lons + +python3 ./om3-scripts/mesh_generation/generate_rof_weights.py --mesh_filename=/g/data/x77/cy8964/mom6/input/input-8km/access-om3-8km-ESMFmesh_cropped.nc --weights_filename=/g/data/x77/cy8964/mom6/input/input-8km/access-om3-8km-rof-remap-weights_cropped.nc +``` + +The OBC conditions are created in a similar manner to the 25 km pan-An domain by using [this script](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/generate-obcs/ACCESS-OM2_panan_boundary_forcing_8km.ipynb). Note that this script is now depreciated and will need some minor updates to the xarray syntax before rerunning. + +Next step are to modify all the names in `config.yaml` to point to the correct, new files. +``` + - /g/data/tm70/cy8964/mom6/input/input-8km/ocean_hgrid_cropped.nc + - /g/data/tm70/cy8964/mom6/input/input-8km/ocean_vgrid_cropped.nc + - /g/data/tm70/cy8964/mom6/input/input-8km/ACCESS-OM2_IC_bfilled_smoothedland.nc + - /g/data/tm70/cy8964/mom6/input/input-8km/salt_restore_interpolated_nearest.nc + - /g/data/tm70/cy8964/mom6/input/input-8km/topog_Charrassin_nocavity_cropped.nc + - /g/data/tm70/cy8964/mom6/input/input-8km/access-om3-8km-ESMFmesh_Charrassin_nocavity_cropped.nc + - /g/data/tm70/cy8964/mom6/input/input-8km/access-om3-8km-nomask-ESMFmesh_Charrassin_nocavity_cropped.nc +``` +Then, change the latitude AND longitude size numbers in `datm_in`, `drof_in`, `ice_in`, `nuopc.runcofig` as well as update names. +In datm_in: +``` +ny_global = 1142 + model_maskfile = "./INPUT/access-om3-8km-nomask-ESMFmesh_Charrassin_nocavity_cropped.nc" + model_meshfile = "./INPUT/access-om3-8km-nomask-ESMFmesh_Charrassin_nocavity_cropped.nc" + nx_global = 4320 + ny_global = 1442 +``` + +In drof_in: +``` + model_maskfile = "./INPUT/access-om3-8km-nomask-ESMFmesh_Charrassin_nocavity_cropped.nc" + model_meshfile = "./INPUT/access-om3-8km-nomask-ESMFmesh_Charrassin_nocavity_cropped.nc" + nx_global = 4320 + ny_global = 1442 +``` +In ice_in: +``` + history_chunksize = 720, 186 + ndtd = 3 + bathymetry_file = "./INPUT/topog_Charrassin_nocavity_cropped.nc" + grid_file = "./INPUT/ocean_hgrid_cropped.nc" + grid_type = "regional" + kcatbound = 2 + kmt_file = "./INPUT/kmt_Charrassin_nocavity_cropped.nc" + ncat = 7 +``` + +For now, turn salt restoring OFF since as the file has not been generated. Alse decrease the timesteps `DT` and `DT_THERM` by 3 (25km/8km ~ 3). + +In MOM_override: + +``` +#override RESTORE_SALINITY = False + +#override FATAL_UNUSED_PARAMS = False ! because there are a bunch of salt restoring stuff in the MOM_input +#override DT = 360 +#override DT_THERM = 3600 +!#override DEBUG = True +``` + +Finally, `nuopc.runconfig` `PELAYOUT_attributes::` are updated with optimisation experinment layout. + +Since that has more processors going to ocean than current CPUs, increase thw number of CPUs to 2016 (a multiple of 48 but otherwise the number choice was somewhat arbitrary). + +Change `nuopc.runconfig` `CLOCK_attributes` section to have `nmonths` as the model runs slower than the global models. + +``` +restart_n = 1 +restart_option = nmonths +restart_ymd = -999 +rof_cpl_dt = 99999 #not used +start_tod = 0 +start_ymd = 19000101 +stop_n = 1 +stop_option = nmonths +``` + +To monitor the progress of the job after submitting it via `payu` use: +``` +tail -f work/log/ocn.log +``` +which updates with some information every day. + +# 4. Optimisation +Optimisation experinments that [improve runtime](https://github.com/claireyung/access-om3-configs/pull/1) have been pulled into the cascade lakes branch. Note the executable is not the most up to date MOM6 codebase. + +This configurations is slow and below are some more suggestions to help optimise the configuration and others like it. + +1. use sapphire rapids. This requires an extra lines in `config.yaml` + +``` +platform: + nodesize: 104 + nodemem: 512 +``` +Note that the ACCESS-NRI released executables are optised to work on saffhire rapids and we strongly suggest trying this. +2. try to monitor state using aps profiler. This requires adding `modules: load: - intel-vtune/2025.0.1` in `config.yaml`, and `exe_prefix: aps`. + +This seems to slow the model down.... but you can look at output by following instructions here https://www.intel.com/content/www/us/en/docs/vtune-profiler/cookbook/2025-0/profiling-mpi-applications.html + +``` +module load intel-vtune/2025.0.1 +aps --report ./work/aps_result_20250528_141890971.gadi-pbs +``` + +and then download the .html file produced and look at it. This will only work if the model does not crash during run. + +3. try sea ice settings +`ice.log` tells you about blocks. If block sizes are too small it could slow the model down. The ideal number is 3-8. + +To explore this option, change `domain_nml` block size in `ice_in` + +This is still experinmental as tests on these settings thus far have crashed. + + +# 5. Add Charrassin bathymetry (no ice shelves) + +Bathymetry files for the Charrissin bathymetry were created in [these](https://github.com/claireyung/mom6-panAn-iceshelf-tools/generate-draft/Generate-Charrassin-bathy.ipynb) [notebooks](https://github.com/claireyung/mom6-panAn-iceshelf-tools/generate-draft/process-topo.ipynb).These notebooks also generate files for ice shelf cavities opened when this configuration comes online. + +As before: truncate files, regenerate mesh files and update the netcdf name in the configuration files. No changes are needed to the grid-size parameters (`nx', `ny') when using the above 8k pan-An configuration as a stencil as the grid size is the same. + +Copy config dir +``` +cp -r 8km_jra_ryf_obc2-sapphirerapid 8km_jra_ryf_obc2-sapphirerapid-Charrassin +cd 8km_jra_ryf_obc2-sapphirerapid-Charrassin +module use /g/data/vk83/modules +module load payu +payu checkout -b 8km_jra_ryf_obc2-sapphirerapid-Charrassin +``` + +Now crop files (`cd /g/data/x77/cy8964/mom6/input/input-8km`) + +``` +ncks -d ny,0,1441 topog_Charrassin_nocavity.nc topog_Charrassin_nocavity_cropped.nc +ncks -d ny,0,1441 ocean_mask_Charrassin_nocavity.nc ocean_mask_Charrassin_nocavity_cropped.nc +ncks -d ny,0,1441 kmt_Charrassin_nocavity.nc kmt_Charrassin_nocavity_cropped.nc +``` + +and make rof, mask files etc (`cd /home/156/cy8964/model-tools/make_OM3_8k_topo`) through submission of below PBS script on Gadi. + +``` +# Copyright 2025 ACCESS-NRI and contributors. See the top-level COPYRIGHT file for details. +# SPDX-License-Identifier: Apache-2.0 + +#PBS -q normal +#PBS -l walltime=4:00:00,mem=10GB +#PBS -l wd +#PBS -l storage=gdata/hh5+gdata/ik11+gdata/tm70+gdata/vk83+gdata/x77 + +# Input files - Using the environment variables passed via -v +INPUT_HGRID=$INPUT_HGRID +INPUT_VGRID=$INPUT_VGRID +INPUT_GBCO=$INPUT_GBCO +# Minimum allowed y-size for a cell (in m) +CUTOFF_VALUE=6000 +# Output filenames +ESMF_MESH_FILE='access-om3-8km-ESMFmesh.nc' +ESMF_NO_MASK_MESH_FILE='access-om3-8km-nomask-ESMFmesh.nc' +ROF_WEIGHTS_FILE='access-om3-8km-rof-remap-weights.nc' + +# Build bathymetry-tools +./build.sh + +module purge +module use /g/data/hh5/public/modules +module load conda/analysis3 +module load nco + +set -x #print commands to e file +set -e #exit on error + +python3 /home/156/cy8964/model-tools/om3-scripts/mesh_generation/generate_mesh.py --grid-type=mom --grid-filename=/g/data/x77/cy8964/mom6/input/input-8km/ocean_hgrid_cropped.nc --mesh-filename=/g/data/x77/cy8964/mom6/input/input-8km/access-om3-8km-ESMFmesh_Charrassin_nocavity_cropped.nc --mask-filename=/g/data/x77/cy8964/mom6/input/input-8km/ocean_mask_Charrassin_nocavity_cropped.nc --wrap-lons + +python3 /home/156/cy8964/model-tools/om3-scripts/mesh_generation/generate_mesh.py --grid-type=mom --grid-filename=/g/data/x77/cy8964/mom6/input/input-8km/ocean_hgrid_cropped.nc --mesh-filename=/g/data/x77/cy8964/mom6/input/input-8km/access-om3-8km-nomask-ESMFmesh_Charrassin_nocavity_cropped.nc --wrap-lons + +python3 ./om3-scripts/mesh_generation/generate_rof_weights.py --mesh_filename=/g/data/x77/cy8964/mom6/input/input-8km/access-om3-8km-ESMFmesh_Charrassin_nocavity_cropped.nc --weights_filename=/g/data/x77/cy8964/mom6/input/input-8km/access-om3-8km-rof-remap-weights_Charrassin_nocavity_cropped.nc + +``` + +Move files to the input directory ('/g/data/x77/cy8964/mom6/input/input-8km') and update the netcdf names in `config.yaml`, `datm_in`, `ice_in`, `drof_in`, `nuopc.runconfig`, `MOM_override` + +# 6. Add new parameters +A few parameters were changed as outlined below. +1. The coupling timestep is set on line 2 of `nuopc.runseq`. This needs to not a ratio of 3x the timestep to produce restart files + +2. Parameters in `MOM_input` were altered to match the new OM3 parameters (similar to GFDL OM5). Note that this was done in `MOM_override` + +3. The new internal tide mixing scheme requires two new files with roughness and tidal amplitude. This is done by reproducung the method for creating these in ACCESS-OM3 models: +https://github.com/ACCESS-NRI/om3-scripts/pull/53 +``` +qsub -I -P x77 -q normalbw -l ncpus=28,mem=120G,walltime=05:00:00,storage=gdata/hh5+gdata/ik11+gdata/x77+scratch/x77+gdata/xp65 +module use /g/data/xp65/public/modules +module load conda/analysis3 + +cd /home/156/cy8964/model-tools/om3-scripts/external_tidal_generation + +mpirun -n 28 python3 generate_bottom_roughness_polyfit.py --topo-file=/g/data/ik11/inputs/SYNBATH/SYNBATH.nc --hgrid-file=/g/data/x77/cy8964/mom6/input/input-8km/ocean_hgrid_cropped.nc --mask-file=/g/data/x77/cy8964/mom6/input/input-8km/ocean_mask_Charrassin_nocavity_cropped.nc --chunk-lat=800 --chunk-lon=1600 --output=/g/data/x77/cy8964/mom6/input/input-8km/bottom_roughness_Charrassin_nocavity_cropped.nc + +python3 generate_tide_amplitude.py --hgrid-file=/g/data/x77/cy8964/mom6/input/input-8km/ocean_hgrid_cropped.nc --mask-file=/g/data/x77/cy8964/mom6/input/input-8km/ocean_mask_Charrassin_nocavity_cropped.nc --method=conservative_normed --data-path=/g/data/ik11/inputs/TPXO10_atlas_v2 --output=/g/data/x77/cy8964/mom6/input/input-8km/tideamp_Charrassin_nocavity_cropped.nc +``` +This takes about an hour to calculate bottom_roughness and about 30 minutes for tideamp. + + +Note that this [will not work in configurations with ice-shelf cavities] .(https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/8) + + +4. For stability the timestep is reduces although it might be able to be increased after a few days of spinup. Note that `dt_therm` cannot be [too big](https://bb.cgd.ucar.edu/cesm/threads/limit-on-dt_therm-in-regional-configs.8226/) + +5. The diag rho coordinates were copied from a [previous panantactic configuration](https://github.com/COSIMA/mom6-panan). + +6. Diag z coordinates were regridded (ocean_month_z) to address a warmer than expected western boundary currents. This issue did vanish so the regridding may not be needed. + + +# 7. Fix bugs + +### Minor bugs +There was a Northern boundary issue where fast velocities were generated due to SSH gradient at northern boundary. A few different changes were made in an attempt to resolve this: +1. Reverting to `WRIGHT_REDUCED` equation of state(EOS) (to make the initial and boundary conditions). This didn't fix the issue but does mean that the boundary inputs and internal model has consistent EOS. + +2. Using the ACCESS-OM2-01 second year output https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/claire_working/initial-conditions/ACCESSOM2_IC_into_8km_grid.ipynb + +We also changed to `WRIGHT_FULL`. See [here](https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/13) for details. + +We added Stewart Allen's suggested Antarctic ice thickness categories (He suggested ncat=7, with similar spacing (kcatbound=2), noting that this has it's highest at about 2.2m.) + +Salinity was restoring to zero at E-W boundary and this was deactivated. +### Sapphire rapid bugs +The model crashed when running on Sapphire rapid (SR) with a segfault as the model was unable to find an exisiting and previously findable file for seaice initialisation during the restart. This was resolved by moving to Minghang's optimised and working number of cores/layout for the [ACCESS-OM3 global 25km config](github.com/ACCESS-NRI/access-om3-configs/pull/591/files) (note the`MOM_parameter_doclayout` file can be ignored as it is for a different domain). + +The number of CPUs used in the run can be set by `ncpus` in `config.yaml`, to a multiple of 104 for SR. The `mem` can also change accordingly (500 GB max per SR node). Note that if you decrease `ncpus` below to be less than any values set in `PELAYOUT_attributes`'s (set in `nuopc.runconfig`) then you will get an error message (`PELAYOUT_attributes` distributes the CPUs to different components of the model) + +### Timestep +When running on cascade lake (CL), a lot of `bad_depature_points` errors were reported. These occurs when CICE goes over the CFL limit. To fix this, the timesteps were reduced to be +``` +DT = 450 +DT_THERM = 900 +nuopc.runseq corner number 450 (coupling timestep) +``` +Attempts were made to change `ndtd` and `dt` in `ice_in` to force CICE to use a different timestep (default is the coupling timestep, you can see this in `work/log/ice.log`). However this produced segfaults and the timestemps are not decoupled. The resulting ocean component has CFLs of 0.2ish which is quite low. + +Further attempts to decouple the ocean and ice timesteps were attempted after a year of simulation: +``` +ndtd = 3` +DT_THERM = 1200 +DT = 600 +coupling timestep 600 +``` +This worked but it is still unknown why the decoupling did not work in the first year of the simulation. + +Note that when varing timesteps during the simulation, we want to impose these restrictions: +1. timesteps are divisors of 1 day and ideally 1 hour since almost all runlengths will be integer multiples of those. So 450, 600, 900 all good. They also need to be multiples of each other and match at the restart time, otherwise you get a [buoyancy flux crash](https://github.com/ACCESS-NRI/access-om3-configs/pull/556#issuecomment-2939907442). +2. the coupling timestep [cannot be 3x dt](https://github.com/ACCESS-NRI/access-om3-configs/issues/380) this issue needs further investigation. +3. the thermodynamic timestep [cannot be too long](https://github.com/COSIMA/mom6-panan/issues/28) + + + + + diff --git a/documentation/docs/pages/configurations/pan-Antartic/Overview.md b/documentation/docs/pages/configurations/pan-Antartic/Overview.md new file mode 100644 index 000000000..22da265ed --- /dev/null +++ b/documentation/docs/pages/configurations/pan-Antartic/Overview.md @@ -0,0 +1,131 @@ + +Here, we describe a regional OM3 configuration covering a pan-Antarctic circumpolar domain from Antarctica to 37 degrees South. This configuration inherits some of the development of the [MOM6-SIS2 COSIMA pan-Antarctic configurations](https://github.com/COSIMA/mom6-panan). + +### Truncating a global setup +One of the easiest ways to set up a pan-Antarctic configuration is to truncate a global OM3 configuration. Detailed step-by-step instructions are provided [here](https://github.com/claireyung/access-om3-configs/blob/8km_jra_ryf_obc2-sapphirerapid-Charrassin-newparams-rerun-Wright-spinup-accessom2IC-yr9/panantarctic_instructions.md) but the main steps involve: + +1. Cloning a global model at desired resolution, and creating a new branch. The global model at the desired resolution may not exist and, in this case, you will need to create one first. + +2. Truncating netcdf files from the global model using `ncks` and ensuring buth supergrid and normal grid y coordinates are correct/match + +3. Generating mesh files with [`om3-scripts`](https://github.com/ACCESS-NRI/om3-scripts) + +4. Modifying namelists to use the correct cropped netcdf files (searching by `.nc` is helpful) and changing the y index size to be correct. This involves changing `datm_in`, `MOM_input`, `ice_in` (where `history_chunksize`, `grid_type = "regional"` and `ns_boundary_type = "open"` are also needed), `nuopc.runconfig`. + +5. Changing `config.yaml` file names if needed, and ensuring you are using a symmetric MOM6 memory exe (currently from an ACCESS-OM3 release) + +6. Open boundary conditions in MOM6 can be added using input from the MOM6-SIS2 COSIMA configuration, see [this notebook](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/generate-obcs/ACCESS-OM2_panan_boundary_forcing_8km.ipynb) for the OBC generation using ACCESS-OM2-01 data. Other datasets may be used for boundary conditions (such as ACCESS-OM3 outputs) but it is advisable to use the same product for initial and boundary conditions. + +7. Extra lines need to be added into `MOM_input` to define the boundary conditions and location of the boundary condition input files. + +8. It may be helpful to use `MOM_override` to avoid changing `MOM_input` and pick up upstream global model changes more easily. + +### 1/12th degree/4km pan-Antarctic setup +Here we describe a proposed ACCESS-NRI-supported 4km (1/12 degree) pan-Antarctic regional OM3 configuration. The configuration has a similar resolution to a future 8km global OM3, but due to its polar latitudes is better described as a 4km resolution model. Here we focus on the configuration with ice shelf cavities closed (i.e. Antarctic ice shelf melt is represented as surface runoff, and there is no circulation in cavities), though development of an equivalent model with ice shelf cavities open is ongoing. + +#### Grid +The grid was generated using the `ocean_grid_generator` at 1/12th degree resolution with no displaced pole and a transition from Mercator to fixed latitude cells at 75$^\circ$S. + +#### Topography +Bottom topography is generated from [Charrassin et al. (2025)](https://www.nature.com/articles/s41598-024-81599-1) product, stitched to GEBCO 2024 where the Charrassin product is not available. The topography generation process required regridding from the Antarctic polar grid EPSG:3031 to latitude-longitude coordinates and is described [here](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/generate-draft/Generate-Charrassin-bathy.ipynb) and [here](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/generate-draft/process-topo.ipynb). Unfortunately, the normal OM3 topography generation workflow cannot be applied here so we manually set maximum and minimum depths, perform deseas to get rid of inland lakes, and create masks. These notebooks are also used for ice shelf configuration in development. Once these new topography and masks are generated, the nuopc mesh files also need to be generated with `om3-scripts`, as well as the bottom roughness and tidal amplitude. + +#### Initial Conditions and Boundary Conditions +Initial conditions were generated using a restart at the start of year 2 of an ACCESS-OM2-01 RYF90-91 configuration spun up from WOA13, as used in the COSIMA MOM6-SIS2 pan-Antarctic models, see [this notebook](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/initial-conditions/ACCESSOM2_IC_into_8km_grid.ipynb). Boundary conditions use daily data from the second year of this run, see [this notebook](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/generate-obcs/ACCESS-OM2_panan_boundary_forcing_8km.ipynb). Salt restoring is regridded from ACCESS-OM2-01, as in the COSIMA MOM6-SIS2 models, see [this notebook](https://github.com/claireyung/mom6-panAn-iceshelf-tools/blob/main/initial-conditions/salt-restoring.ipynb). + +#### Model parameters + +MOM6 parameters are a combination of ACCESS-OM3 25km choices, MOM6-SIS2 COSIMA pan-Antarctic models and regional models from GFDL. We use a Wright equation of state, see [this issue](https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/13), and higher resolution density coordinates (taken from MOM6-SIS2 COSIMA pan-Antarctic model). These parameter choices are found in [`MOM_override_newparams`]( https://github.com/claireyung/access-om3-configs/blob/8km_jra_ryf_obc2-sapphirerapid-Charrassin-newparams-rerun-Wright-spinup-accessom2IC-yr9/MOM_override_newparams). + +The CICE sea ice categories were modified to be `ncat=7` with `kcatbound=2`. + +#### Model behaviour + +The simulation is generally stable and is being evaluated [here](https://github.com/claireyung/mom6-panAn-iceshelf-tools/issues/15). Often the model crashes with a segfault immediately on initialisation, but persistent resubmission can get over this (a `resub.sh` [script](https://github.com/COSIMA/01deg_jra55_iaf/blob/01deg_jra55v140_iaf_cycle4_rerun_from_2002/resub.sh) can be helpful). + +## Optimization of 1/12th degree/4km pan-Antarctic configuration + +### Timestepping +In this configuration, an ocean baroclinic timestep of 600s is used, with a relatively short thermodynamic timestep of 1200s. This was chosen because of known problems with the MOM6-SIS2 COSIMA pan-Antarctic boundary conditions if the tracer timestep is too long, [see this issue](https://github.com/COSIMA/mom6-panan/issues/28). The behaviour of the boundary condition under different thermodynamic timesteps has not been explored in this configuration. + +The coupling timestep, set by `nuopc.runseq`, is 600, and `ndtd = 3` is set in `ice_in` to increase the CICE dynamic timestep and avoid CFL-related errors with ice dynamics. There are restrictions on ratios of timesteps for NUOPC and timesteps must divide into the restart time. + +### IO + +In this config, IO consumes a non-trivial amount of time. And most of the IO comes from MOM6 +diagnostic and restart dumps. At this 8km resolution and if IO is serialized (i.e., performed by 1 +CPU), **For every run, MOM6 IO can take around 20mins to dump the diagnostic and restart +information**. Hence, a key optimization is to parallelize IO. + +Because `AUTO_MASKTABLE = True` is being used in this config, we rely on changing number of CPUs +allocated to MOM6 such that `NIPROC` and `NJPROC` (determined by MOM6 at runtime) share a common +integer factor. As the procedure by which `NIPROC` and `NJPROC` isn't yet understood, finding the +correct number of CPUs is done by guess-and-check by running OM3 for 2 mins (i.e., try 1300 CPUs, +check what MOM6 calculates `NIPROC/NJPROC` to be and if there isn't a common integer factor, keep +trying). Once number of CPUs is chosen such that `NIPROC/NJPROC` has a common integer factor, then +set `AUTO_IO_LAYOUT_FAC = ` in MOM_input/MOM_override. + +A challenge with this is that changing the cores allocated to MOM6, naturally also affects +computation time. + +**The parallelization of diagnostic information reduced IO time by a few minutes**, but most of the time +gain is from paralellizing restart files. **Parallelizing restart can take off around 15mins** for this +config. However, `payu` is unable to understand parallel restarts and fails in its post-processing +of the MOM6 output. Hence, `PARALLEL_RESTARTFILES = False` until `payu` is updated with this +ability and the time gain isn't obtained. + +If IO layout blocks are completely land covered along a given latitude or longitude, the combined +netcdf output may contain non-sensible output (NaNs) along that coordinate. Therefore it is recommended +that IO blocks are large enough that there are ocean cells in every block. This may require explicit +setting of the `IO_LAYOUT`. + + +### OM3 Compiler flags + +In this config, time is dominated by IO and MPI communications, so the effect of compiler flags can +be quite limited. Below discusses some the effects of some of the flags investigated with the `ifx` +compiler. Note that the **improvements reported are based on total time of a 5-day run, and doesn't +dilineate time spent in IO/MPI/compute/etc. **Note that changing compiler flags can result in bitwise +differing results.** + +| Flag(s) | purpose | performance change | +| --- | --- | --- | +| `-march=sapphirerapids -mtune=sapphirerapids` | This compiles code specifically for Gadi's newest (at time of writing) hardware. Code cannot be run on Gadi's `normal` queue. | This gave a roughly 2.5% improvement | +| `-march=cascadelake -mtune=sapphirerapids` | Allows the code to run on Gadi's older Cascade Lake nodes (in the `normal` queue), but tries to optimize for the newer hardware. | Relative to no `march` or `mtune` flags, this improves runtime by roughly 1.5% | +| `-O3` | Increases the optimization level (default is `-O2`). When MPI is not involved, this can increase compute performance of MOM6 by 15% | 1% performance improvement | +| `-flto -fuse-ld=lld` | Enables link-time optimization (inter-procedural optimization). `-fuse-ld=lld` is not an optimization, but is needed for `ifx -flto` to work. | ~2% performance improvement | +| `-qopt-prefetch` | A flag to enable prefetching (can improve performance in some memory bound programs). | ~1.2% improvement | + +Altogether, **these resulted in a ~5-6% improvement based on default flags. If not including IO, +the improvement is probably more like 6-7%**. These flags can be applied to all OM3 builds as well. + +### CICE + +A basic way to check time spent on CICE is to read the `ice.log` in the output logs. The [CICE docs](https://cice-consortium-cice.readthedocs.io/en/main/user_guide/ug_implementation.html#performance) +offer a few strategies to improve performance. The below config changes are controlled in the `ice_in` +file under the `domain_nml` section. + +#### Block Size + +Domain block size should be optimized such that there are roughly 6-7 blocks per process (`nx_global*ny_global/(block_size_x*block_size_y)`). In a config with 254 cores on CICE and +`nx_global = 4320, ny_global = 1440`, doubling block size in each direction from `block_size_x = 30, block_size_y = 27` resulted in 5-30% performance improvement, where 5 is for little-to-no ice, and +30% is when there is unrealistically high amounts of ice. + +#### Process Distribution + +Changing from `roundrobin` to `sectrobin` reduced CICE time by a flat 20s, regardless of ice level. +This is likely due to `sectrobin` having a slightly better way of organising processes to ensure +neighbours are closer. + +### Runsequence + +The nuopc run sequence (`nuopc.runseq`) can be modified so that components aren't unnecessarily waiting for eachother. +(I just copied a [run sequence from Minghang](https://github.com/ACCESS-NRI/access-om3-configs/pull/590), which improved runtime by about 8-9%). + +## Aditional reading: + +- +- + +## References + + diff --git a/documentation/docs/pages/configurations/pan-Antartic/run_panan_isf.md b/documentation/docs/pages/configurations/pan-Antartic/run_panan_isf.md new file mode 100644 index 000000000..6144ff966 --- /dev/null +++ b/documentation/docs/pages/configurations/pan-Antartic/run_panan_isf.md @@ -0,0 +1,41 @@ +# How to run ACCESS-OM3 ice shelf configurations + +Please see https://github.com/claireyung/mom6-panAn-iceshelf-tools/wiki for notes + +Also refer to previous notes on the ACCESS-rOM3 panantarctic with no ice shelves here: https://github.com/claireyung/access-om3-configs/blob/8km_jra_ryf_obc2-sapphirerapid-Charrassin-newparams-rerun-Wright-spinup-accessom2IC-yr9/panantarctic_instructions.md + +**Important**: requires small timestep at the start, may need more walltime, but with default number of cores should fit within normalsr 10 hr limit + +**Also important**: Code changes are required, see https://github.com/ACCESS-NRI/MOM6/issues/29 and branch https://github.com/ACCESS-NRI/MOM6/tree/ice-shelf-dev + +**Also important**: If changing files, think carefully about masks and coverage. See Claire's notes/contact her if you have questions. Otherwise you might allow the ocean to talk to the atmos in the ice shelf cavities, or have sea ice grow there... + +# Instructions for starting from rest +### Step 1: run for 1 month from rest +- In `MOM_input` set `DT = 150`, `DT_THERM = 150` +- In `input.nml` set `input_filename = 'n'` +- Change coupling timestep at corner of `nuopc.runseq` to also be 150 +- To save ICs, use `SAVE_INITIAL_CONDS = True` in `MOM_input` +- do `payu setup`, `payu sweep`, `payu run` etc + +### Step 2: run for February +- Comment out `input.nml` `input_filename = 'n'` +- Change `DT`, `DT_THERM` and coupling timestep to be 400 (should work - otherwise try something in between eg 300) +- You can also set `DT_THERM` to be larger, e.g. 800 +- Turn off IC saving with `SAVE_INITIAL_CONDS = False` in `MOM_input` +- run - now you can use `payu run -n XX` +- decrease walltime to 4 hours in config.yaml + + +# Instructions for starting from a restart +- Use `payu checkout --restart path/to/restart` (this should already be in the config.yaml) +- Jan 1 1901 restart from [this](https://github.com/claireyung/access-om3-configs/tree/ice_shelf_panan-nancheck3-150925files-notidalmix) run is saved here: `/g/data/ol01/cy8964/access-om3/restart/ice_shelf_panan-nancheck3-150925files-v2-notidalmix/restart012` +- E.g. +``` +payu checkout dev-MC_4km_jra_ryf+regionalpanan+isf+draft --restart /g/data/ol01/cy8964/access-om3/restart/ice_shelf_panan-nancheck3-150925files-v2-notidalmix/restart012 +``` +- Run - ensure DT is 400 (ish), input.nml has `input_filename = 'n'` commented out, CLOCK time in nuopc.runconfig is months + +# Other tips +- If it crashes, drop the timestep - can check MOM CFL issues in `work/log/ocn.log` +- Might find a resub script helpful - usually it takes about 5 segfaults straight after initialisation until it finally wants to run :( e.g. https://github.com/claireyung/access-om3-configs/blob/8km_jra_ryf_obc2-sapphirerapid-Charrassin-newparams-rerun-Wright-spinup-accessom2IC-yr9/resub.sh and add ` error: resub.sh run: rm -f resubmit.count` to `userscripts` section of `config.yaml` diff --git a/documentation/mkdocs.yml b/documentation/mkdocs.yml index 9c23b08d4..97dcfa100 100755 --- a/documentation/mkdocs.yml +++ b/documentation/mkdocs.yml @@ -112,7 +112,11 @@ nav: - MOM6-CICE 25km: pages/configurations/MC_25km.md - WW3 config: pages/configurations/MOM6-CICE6-WW3-quickstart.md - WW3 parameter choices: pages/configurations/WW3_parameter_choices.md - + - pan-Antarctic: + - Overview: pages/configurations/pan-Antartic/Overview.md + - Detailed noISF: pages/configurations/pan-Antartic/Detailedpananinstructions.md + - Detailed ISF: pages/configurations/pan-Antartic/Detailed_panan_isf.md + - Run ISF: pages/configurations/pan-Antartic/run_panan_isf.md - Infrastructure: - Architecture: pages/infrastructure/Architecture.md - NUOPC driver: pages/infrastructure/NUOPC-driver.md @@ -126,6 +130,7 @@ nav: #- Tips and Tricks: pages/Tips-and-tricks.md - Online Resources: pages/Online-resources.md + # Footer extra: generator: false