diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index ae5f6bd..90147a2 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -1,2 +1,384 @@ +# ANU-TUB, an idealised sector model -## Running anu-tub, an idealised sector model +The purpose of this page is to demonstrate how to: + +1. run an ANU-TUB model with a MOM6 ocean only executable provided by ACCESS-NRI; and/or +1. set up the ANU-TUB model up with a MOM6 executable of the user's choice on gadi. + +## ANU-TUB configurarion details + +The [ANU-TUB](https://github.com/AndyHoggANU/anu-tub) is a MOM6 configuration that aims to: + +>produce the simplest configuration that incorporates buoyancy and wind forcing, gyre circulation, overturning circulation and topographically-constrained overflow + +The configuration was setup by [Andy Hogg](https://github.com/AndyHoggANU) and [Angus Gibson](https://github.com/angus-g) for exploring vertical coordinate choice and design in MOM6 which uses [generalised vertical coordinates](https://mom6.readthedocs.io/en/dev-gfdl/api/generated/pages/General_Coordinate.html) thereby placing vertical coordinate choice in the users hands. +More specific details about the configuraiton, taken from the repository [README](https://github.com/AndyHoggANU/anu-tub?tab=readme-ov-file#anu-tub), are: + +- The sector is 40° wide, and goes from ~70.3°S to ~70.3°N. +- It has a 1/4° nominal resolution and a Mercator grid refinement. +- the default vertical coordinate is ZSTAR with 75 vertical levels. +- The bathymetry is simple, with a vertical wall to the north, an "Antarctic shelf and slope" in the south and sloping sidewalls on the east and west. +- The domain is periodic in the east-west direction, allowing zonal flow in a narrow "Drake Passage" between ~65°S and ~52°S. +- Surface momentum forcing is via a prescribed zonal wind field that is constant, but varies with latitude. +- Thermal forcing is through relation to a latitude-dependent SST profile, and there is (currently) no freshwater forcing so that salinity is constant. +- We use the WRIGHT equation of state, without frazil formation or sea ice. +- The model includes the PBL surface boundary layer scheme with a contant background diffusivity of $2 \times 10^{-5}$m$^{2}$s$^{-1}$. +- The MEKE eddy parameterisation scheme is turned off, as is the mixed layer restratification scheme. + +The model has $160 \times 800$ grid points, with a tile layout of $6 \times 40$ to run efficiently on 240 cores. +With a 1200-second timestep, the standard ZSTAR case takes 1:45 hours per year (~14 years/day) and consumes ~900 SU per model year. +The relatively low cost of the ANU-TUB configuration allows for a wide variety of idealised experiments to be run that can investigate vertical coordinates, new diagnostics and many other things! + +# Using a pre-built ANU-TUB configuration + +This will be done once the ACCESS-NRI mom6 ocean only build is done. + +# Building your own ANU-TUB configuration + +Below the steps for creating your own ANU-TUB configuration are outlined. +!!! tip + This is more advanced than just running the ANU-TUB configuration that is provided, however it does allow more flexibility especially with regards to changing the MOM6 source code the model uses. + +## Requirements + +The required components for setting up ANU-TUB are: + +- access to the relevant projects on gadi +- a MOM6 executable +- a fork or clone of the ANU-TUB repository. + +### Project access + +Talk to Chris about what to do and where to point people here. +Need payu so vk83? +Maybe with Andy's permission can put the restart files into the COSIMA directory and provide access that way. + +### MOM6 executable + +We reccommend [building a MOM6 executable](https://access-om3-idealised-configs.readthedocs.io/infrastructure/Building/). +This will provide the most flexibility as source code can be modified. +If you would like to use an existing standalone MOM6 executable just make sure you know the exact path that points to it. + +### Cloning the ANU-TUB repository + +ANU-TUB can either be cloned directly or, you can create a fork and clone that---either option is fine here so whatever suits you! +The following commands clone the repository into the `$USER` directory: + +```bash +cd /g/data/$PROJECT/$USER/ +git clone https://github.com/AndyHoggANU/anu-tub.git +``` + + + cd /g/data/$PROJECT/$USER + git clone https://github.com/AndyHoggANU/anu-tub.git + Cloning into anu-tub... + + +!!! tip + If you have forked the repository and wish to clone the fork, change `AndyHoggANU` to your github username in the url. + +## Setting up + +With the ANU-TUB repository cloned to gadi we change into the `zstar` directory and edit the necessary part of the `config.yaml` file +!!! tip + The same process below will work for any of the other vertical coordinate choice but `zstar` is a good place to start. + +```bash +cd anu-tub/control/zstar +vim config.yaml +``` + + + cd anu-tub/control/zstar + vim config.yaml + + +The contents of the `config.yaml` file should, without the comments indicating what lines need changing, look like: + +```yaml +ncpus: 240 +walltime: 12:00:00 +jobname: tub-new +project: x77 # replace with NCI project you are part of +queue: normal + +model: mom6 +shortpath: /scratch/x77 # replace with NCI project you are part of +exe: /g/data/x77/ahg157/exes/MOM6/AG-07f0a144 # replace with path to MOM6 executable +input: /g/data/x77/amh157/mom6/input/anu-tub # replace with path to input, see instructions below + +storage: + gdata: + - x77 # replace with NCI project you are part of + +collate: False + +mpi: + module: openmpi/4.1.2 + +qsub_flags: -W umask=027 +restart_freq: 1 # use tidy_restarts.py instead + +userscripts: + archive: qsub sync_output_to_gdata.sh +``` + +As indicated above the `project`, `shortpath`, `exe` and `input` information needs to been updated: + +- `project` needs to be changed to the project which will be used to run the job on gadi; +- `shortpath` needs to have `x77` changed to the same project number as `project`; +- `exe` needs to point to the compiled ocean only MOM6 executable to be used in the model, e.g. `exe: /g/data/$PROJECT/$USER/path/to/MOM6executable` +- `input` needs to point to where the input files are stored. These files are located at **add path once known** so change to `input: /g/data/path/to/files/when/known`. + +Once these changes have been made save and close vim. +!!! tip + To enter insert mode in vim press `i` and change the required fields. + To save and quit type `:wq`. + For further information or more commands view a [vim cheat sheet](https://vim.rtorr.com/). + +The experiment is designed to run on `scratch` so any output will be saved there. +This can then be manually moved to `/g/data` or the executable script `sync_output_to_gdata.sh` can be run. + +*Check with Chris if there is a preferable way to move data to `/g/data` or if we should advise removing these lines.* + +## Running ANU-TUB + +To run the ANU-TUB model we use [`payu`](https://payu.readthedocs.io/en/stable/install.html), a workflow management tool for running models on gadi. +This requires membership to project vk83 -- add details of how to become member here (or above). +We also need to get the necessary input file from **/path/to/stored/input/files.** + +### Getting the necessary input files (remove once files are saved on gadi) + +This can be done in two ways. +The simplest way is to edit the `input` field in `config.yaml` to point to where these files are located on gadi. +To do this change the `input` field to + +```yaml +input: /path/to/stored/input/files +``` + +Alternatively, the input files can be copied to a directory of your choice. +Then, add the path to where they have been copied to the `input` field. + +### Saving script to copy experiment output to `g/data/` + +**Ask Chris if we should provide this or just mention data is saved on scratch so need to be moved to `g/data` if you want to keep it for longer.** + +### Experiment setup with `payu` + +First, we load payu into our gadi environment +```bash +module use /g/data/vk83/modules +module load payu +``` + + + module use /g/data/vk83/modules + module load payu + Loading payu/1.2.0 + Loading requirement: singularity + + +Within the `zstar` directory, where the `config.yaml` file we have just edited lives, run: + +```bash +payu setup +``` + + + payu setup + laboratory path: /scratch/$PROJECT/$USER/mom6 + input path: /scratch/$PROJECT/$USER/mom6/input + work path: /scratch/$PROJECT/$USER/mom6/work + archive path: /scratch/$PROJECT/$USER/mom6/archive + Found experiment archive: /scratch/$PROJECT/$USER/mom6/archive/zstar-tag_or_uuid + payu: Found modules in /opt/Modules/v4.3.0 + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + Setting up mom6 + Checking exe, input and restart manifests + Writing manifests/restart.yaml + + +The `tag_or_uuid` is something that will be generated as part of `payu setup` for this experiment. +!!! tip + If you recieve a message from `payu` that it expects a git repository run `git init` in the `zstar` directory. + *Chris this is something I have had to do but maybe there are better ways around it?* + *Or we run `git tag` to tag the experiment?* + +Now we move the experiment to `scratch` with + +```bash +payu sweep +``` + + + payu sweep + laboratory path: /scratch/$PROJECT/$USER/mom6 + input path: /scratch/$PROJECT/$USER/mom6/input + work path: /scratch/$PROJECT/$USER/mom6/work + archive path: /scratch/$PROJECT/$USER/mom6/archive + Found experiment archive: /scratch/$PROJECT/$USER/mom6/archive/zstar-tag_or_uuid + payu: Found modules in /opt/Modules/v4.3.0 + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + Setting up mom6 + Checking exe, input and restart manifests + Removing work path /scratch/$PROJECT/$USER/mom6/work/zstar-tag_or_uuid + Removing symlink /g/data/$PROJECT/$USER/anu-tub/control/zstar/work + + +### Getting restart files + +Restart files are available that **get info about it.** +If you wish to use this restart, it needs to be copied into the experiment directory. +To do this, and check it has been copied to the directory type: + +```bash +cd /scratch/$PROJECT/$USER/mom6/work/zstar-tag_or_uuid +cp /path/to/restart . +ls -a +``` + + + + + cp path/to/restart . + ls -a + . .. manifests metadata.yaml payu_jobs pbs_logs restart441 + + +!!! tip + This is where restart files that are generated at the end of an experiment run are saved. + +### Run an experiment + +Lastly, we run the experiment with + +```bash +payu run +``` + + + payu run + laboratory path: /scratch/$PROJECT/$USER/mom6 + input path: /scratch/$PROJECT/$USER/mom6/input + work path: /scratch/$PROJECT/$USER/mom6/work + archive path: /scratch/$PROJECT/$USER/mom6/archive + Found experiment archive: /scratch/$PROJECT/$USER/mom6/archive/zstar-tag_or_uuid + payu: Found modules in /opt/Modules/v4.3.0 + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + payu: Found modules in /opt/Modules/v4.3.0 + qsub -q normal -P e14 -l walltime=43200 -l ncpus=240 -l mem=960GB -N tub-new -l wd -j n -v PAYU_PATH=/g/data/vk83/apps/base_conda/envs/payu-1.2.0/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/vk83/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/e14+gdata/vk83+gdata/x77+scratch/e14 -- /g/data/vk83/./apps/conda_scripts/payu-1.2.0.d/bin/launcher.sh /g/data/vk83/./apps/base_conda/envs/payu-1.2.0/bin/python3.10 /g/data/vk83/apps/base_conda/envs/payu-1.2.0/bin/payu-run + 155451836.gadi-pbs + + +!!! tip + By default, a new restart directory is created at the end of an experiment that is used to pick up from when the experiment is run again. + If you wish to not restart from this state remove the newly created restart directory, and the corresponding output directory. These will be named `restartXXX` and `outputXXX` where `XXX` is a string of numbers. + +To check the progress of the experiment type: + +```bash +qstat -swx +``` + +into the terminal. +Further details on [job monitoring](https://opus.nci.org.au/spaces/Help/pages/236880322/Job+monitoring...) are available in the using gadi section of the NCI documentation. + +## Changes to the experiment setup + +### Altering the `diag_table` + +By cloning the repository, we have a `diag_table`. +To edit this use: + +```bash +vim diag_table +``` + + + + vim diag_table + + +This is a standard `diag_table` for MOM6 so [controlling run time diagnostics](https://mom6.readthedocs.io/en/main/api/generated/pages/Diagnostics.html) can be handled like any MOM6 model run. + +### Changing model run time + +To change the model run time we need to edit the `input.nml` file within the `zstar` directory. +Running, + +```bash +vim input.nml +``` + +we get: + +```bash +&mom_input_nml + output_directory = './' + input_filename = 'r' + restart_input_dir = 'INPUT/' + restart_output_dir = 'RESTART/' + parameter_filename = 'MOM_input', 'MOM_override' +/ + +&fms_nml + clock_grain = 'MODULE' + clock_flags = 'NONE' + domains_stack_size = 1200000 +/ + +&ocean_domains_nml +/ + +&diag_manager_nml +/ + +&ocean_solo_nml + months = 60 + days = 0 + date_init = 1, 1, 1, 0, 0, 0 + hours = 0 + minutes = 0 + seconds = 0 + calendar = 'julian' +/ + +&mpp_io_nml + deflate_level = 5 + shuffle = 1 +/ +``` + +By default, the model is set to run for 60 months. +To change the length of time update the `months` and `dates` field accordingly. +!!! warning + Be sure not to change other parts of this file (unless you know what you are doing) as it will likely lead to the model not running. + +### Editing MOM6 source code + +If you have built your own MOM6 executable, as long as you re-build before, and the `$PATH` to the executable has not changed, running: + +```bash +payu setup +payu sweep +payu run +``` + +should be sufficient for the new changes to the MOM6 source code to be used. +If not please get in touch! + +# Other resources + +The following resources may be useful for changing configurations or experiment setup: + +- [MOM6 documentation](https://mom6.readthedocs.io/en/main/) +- [Running ACCESS models with `payu`](https://docs.access-hive.org.au/models/run_a_model/run_access-om3/) +- [Learning to use vim](https://github.com/iggredible/Learn-Vim?tab=readme-ov-file)