From c9b82af7e49ff414e61b02009e66a63b5e5d18ce Mon Sep 17 00:00:00 2001 From: jbisits Date: Tue, 25 Nov 2025 15:01:04 +1100 Subject: [PATCH 01/12] Skeleton of anu-tub setup --- .../docs/pages/configurations/anu-tub.md | 162 +++++++++++++++++- 1 file changed, 161 insertions(+), 1 deletion(-) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index ae5f6bd..7a4b67f 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -1,2 +1,162 @@ +# ANU-TUB, an idealised sector model -## Running anu-tub, an idealised sector model +The purpose of this page is to demonstrate how to set up the ANU-TUB model up with a MOM6 executable of the user's choice on gadi. + +## ANU-TUB configurarion details + +The [ANU-TUB](https://github.com/AndyHoggANU/anu-tub) is a MOM6 configuration that aims to: + +>produce the simplest configuration that incorporates buoyancy and wind forcing, gyre circulation, overturning circulation and topographically-constrained overflow + +The configuration was setup by [Andy Hogg](https://github.com/AndyHoggANU) and [Angus Gibson](https://github.com/angus-g) for exploring vertical coordinate choice and design in MOM6 which uses [generalised vertical coordinates](https://mom6.readthedocs.io/en/dev-gfdl/api/generated/pages/General_Coordinate.html) thereby placing vertical coordinate choice in the users hands. +More specific details about the configuraiton, taken from the repository [README](https://github.com/AndyHoggANU/anu-tub?tab=readme-ov-file#anu-tub), are: +* The sector is 40° wide, and goes from ~70.3°S to ~70.3°N. +* It has a 1/4° nominal resolution and a Mercator grid refinement. +* the default vertical coordinate is ZSTAR with 75 vertical levels. +* The bathymetry is simple, with a vertical wall to the north, an "Antarctic shelf and slope" in the south and sloping sidewalls on the east and west. +* The domain is periodic in the east-west direction, allowing zonal flow in a narrow "Drake Passage" between ~65°S and ~52°S. +* Surface momentum forcing is via a prescribed zonal wind field that is constant, but varies with latitude. +* Thermal forcing is through relation to a latitude-dependent SST profile, and there is (currently) no freshwater forcing so that salinity is constant. +* We use the WRIGHT equation of state, without frazil formation or sea ice. +* The model includes the PBL surface boundary layer scheme with a contant background diffusivity of $2 \times 10^{-5}$. +* The MEKE eddy parameterisation scheme is turned off, as is the mixed layer restratification scheme. + +The model has $160 \times 800$ grid points, with a tile layout of $6 \times 40$ to run efficiently on 240 cores. +With a 1200-second timestep, the standard ZSTAR case takes 1:45 hours per year (~14 years/day) and consumes ~900 SU per model year. +The relatively low cost of the ANU-TUB configuration allows for a wide variety of idealised experiments to be run that can investigate vertical coordinates, new diagnostics and many other things! + +## Requirements + +The required components for setting up ANU-TUB are: +* access to the relevant projects on gadi +* a MOM6 executable +* a fork or clone of the ANU-TUB repository. + +### Project access + +Talk to Chris about what to do and where to point people here. +Need payu so vk83? +Maybe with Andy's permission can put the restart files into the COSIMA directory and provide access that way. + +### MOM6 executable + +We reccommend [building a MOM6 executable](https://access-community-hub.github.io/access-iom3-configs/infrastructure/Building/). +This will provide the most flexibility as source code can be modified. +If you would like to use an existing standalone MOM6 executable just make sure you know the exact path that points to it. + +### Forking or cloning the ANU-TUB repository + +Either option is fine here so whatever suits you! +The following commands clone the repository into the `$USER` directory: + +```bash +cd /g/data/$PROJECT/$USER/ +git clone https://github.com/AndyHoggANU/anu-tub.git +``` +!!! tip + If you have forked the repository and wish to clone the fork, change the AndyHoggANU to your github username. + +## Setting up + +With the ANU-TUB repository cloned to gadi we change into the `zstar` directory and edit the necessary part of the `config.yaml` file +!!! tip + The same process below will work for any of the other vertical coordinate choices but `zstar` is a good place to start. + +```bash +cd anu-tub/control/zstar +vim config.yaml +``` + +The contents of the `config.yaml` file should look like: + +```yaml +ncpus: 240 +walltime: 12:00:00 +jobname: tub-new +project: x77 # replace with NCI project you are part of +queue: normal + +model: mom6 +shortpath: /scratch/x77 # replace with NCI project you are part of +exe: /g/data/x77/ahg157/exes/MOM6/AG-07f0a144 # replace with path to MOM6 executable +input: /g/data/x77/amh157/mom6/input/anu-tub # replace with path to input, possible COSIMA path Chris mentioned + +storage: + gdata: + - x77 # replace with NCI project you are part of + +collate: False + +mpi: + module: openmpi/4.1.2 + +qsub_flags: -W umask=027 +restart_freq: 1 # use tidy_restarts.py instead + +userscripts: + archive: qsub sync_output_to_gdata.sh +``` + +As indicated above the `project`, `exe` and `input` information needs to been updated. +Once these changes have been made save and close vim (with `:wq`). + +The experiment is designed to run on `scratch` so any output will be saved there. +This can then be manually moved to `/g/data` or the executable script `sync_output_to_gdata.sh` can be run. + +*Check with Chris if there is a preferable way to move data to `/g/data` or if we should advise removing these lines.* + +## Running ANU-TUB + +To run the ANU-TUB model we use [`payu`](https://payu.readthedocs.io/en/stable/install.html), a workflow management tool for running models on gadi. +This requires membership to project vk83 -- add details of how to become member here (or above). + +First, we load payu into our gadi environment +```bash +module use /g/data/vk83/modules +module load payu +``` + +Within the `zstar` directory, where the `config.yaml` file we have just edited lives, run: + +```bash +payu setup +``` + +*add terminal window* + +!!! tip + If you recieve a message from `payu` that it expects a git repository run `git init` in the `zstar` directory. + *Chris this is something I have had to do but maybe there are better ways around it?* + *Or we run `git tag` to tag the experiment?* + +Now we move the experiment to `scratch` with + +```bash +payu sweep +``` + +*add terminal window* + +Lastly, we run the experiment with + +```bash +payu run +``` + +!!! tip + By default, a restart directory is created that is used when the experiment is run again. + If you wish to not restart from this state it, and the corresponding output directory need to be removed within the experiment archive. + +## Changes to the experiment setup + +Mention: +- altering the `diag_table` +- changing run time +- editing MOM6 source code + +# Other resources + +The following resources may be useful for changing configurations or experiment setup: + +- [MOM6 documentation](https://mom6.readthedocs.io/en/main/) +- [running ACCESS models with `payu`](https://docs.access-hive.org.au/models/run_a_model/run_access-om3/) From c25a9e3acf7199c437bf3ab857908c1691d1b711 Mon Sep 17 00:00:00 2001 From: jbisits Date: Tue, 25 Nov 2025 15:08:50 +1100 Subject: [PATCH 02/12] Fix enumeration --- .../docs/pages/configurations/anu-tub.md | 32 ++++++++++--------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index 7a4b67f..b873bbc 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -10,16 +10,17 @@ The [ANU-TUB](https://github.com/AndyHoggANU/anu-tub) is a MOM6 configuration th The configuration was setup by [Andy Hogg](https://github.com/AndyHoggANU) and [Angus Gibson](https://github.com/angus-g) for exploring vertical coordinate choice and design in MOM6 which uses [generalised vertical coordinates](https://mom6.readthedocs.io/en/dev-gfdl/api/generated/pages/General_Coordinate.html) thereby placing vertical coordinate choice in the users hands. More specific details about the configuraiton, taken from the repository [README](https://github.com/AndyHoggANU/anu-tub?tab=readme-ov-file#anu-tub), are: -* The sector is 40° wide, and goes from ~70.3°S to ~70.3°N. -* It has a 1/4° nominal resolution and a Mercator grid refinement. -* the default vertical coordinate is ZSTAR with 75 vertical levels. -* The bathymetry is simple, with a vertical wall to the north, an "Antarctic shelf and slope" in the south and sloping sidewalls on the east and west. -* The domain is periodic in the east-west direction, allowing zonal flow in a narrow "Drake Passage" between ~65°S and ~52°S. -* Surface momentum forcing is via a prescribed zonal wind field that is constant, but varies with latitude. -* Thermal forcing is through relation to a latitude-dependent SST profile, and there is (currently) no freshwater forcing so that salinity is constant. -* We use the WRIGHT equation of state, without frazil formation or sea ice. -* The model includes the PBL surface boundary layer scheme with a contant background diffusivity of $2 \times 10^{-5}$. -* The MEKE eddy parameterisation scheme is turned off, as is the mixed layer restratification scheme. + +- The sector is 40° wide, and goes from ~70.3°S to ~70.3°N. +- It has a 1/4° nominal resolution and a Mercator grid refinement. +- the default vertical coordinate is ZSTAR with 75 vertical levels. +- The bathymetry is simple, with a vertical wall to the north, an "Antarctic shelf and slope" in the south and sloping sidewalls on the east and west. +- The domain is periodic in the east-west direction, allowing zonal flow in a narrow "Drake Passage" between ~65°S and ~52°S. +- Surface momentum forcing is via a prescribed zonal wind field that is constant, but varies with latitude. +- Thermal forcing is through relation to a latitude-dependent SST profile, and there is (currently) no freshwater forcing so that salinity is constant. +- We use the WRIGHT equation of state, without frazil formation or sea ice. +- The model includes the PBL surface boundary layer scheme with a contant background diffusivity of $2 \times 10^{-5}$. +- The MEKE eddy parameterisation scheme is turned off, as is the mixed layer restratification scheme. The model has $160 \times 800$ grid points, with a tile layout of $6 \times 40$ to run efficiently on 240 cores. With a 1200-second timestep, the standard ZSTAR case takes 1:45 hours per year (~14 years/day) and consumes ~900 SU per model year. @@ -28,9 +29,10 @@ The relatively low cost of the ANU-TUB configuration allows for a wide variety o ## Requirements The required components for setting up ANU-TUB are: -* access to the relevant projects on gadi -* a MOM6 executable -* a fork or clone of the ANU-TUB repository. + +- access to the relevant projects on gadi +- a MOM6 executable +- a fork or clone of the ANU-TUB repository. ### Project access @@ -54,7 +56,7 @@ cd /g/data/$PROJECT/$USER/ git clone https://github.com/AndyHoggANU/anu-tub.git ``` !!! tip - If you have forked the repository and wish to clone the fork, change the AndyHoggANU to your github username. + If you have forked the repository and wish to clone the fork, change AndyHoggANU to your github username in the url. ## Setting up @@ -67,7 +69,7 @@ cd anu-tub/control/zstar vim config.yaml ``` -The contents of the `config.yaml` file should look like: +The contents of the `config.yaml` file should look like (without the comments that indicate what lines need changing): ```yaml ncpus: 240 From 40a5877f311cd7dde5152f3b71bba8de874363ba Mon Sep 17 00:00:00 2001 From: jbisits Date: Tue, 25 Nov 2025 15:12:10 +1100 Subject: [PATCH 03/12] Missed a list --- documentation/docs/pages/configurations/anu-tub.md | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index b873bbc..fc801fe 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -152,6 +152,7 @@ payu run ## Changes to the experiment setup Mention: + - altering the `diag_table` - changing run time - editing MOM6 source code From 142fb61590caef8ca82d65aa432a9c0cde3662fa Mon Sep 17 00:00:00 2001 From: jbisits Date: Thu, 27 Nov 2025 12:04:28 +1100 Subject: [PATCH 04/12] add terminal windows --- .../docs/pages/configurations/anu-tub.md | 79 +++++++++++++++++-- 1 file changed, 74 insertions(+), 5 deletions(-) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index fc801fe..417a3c0 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -55,8 +55,15 @@ The following commands clone the repository into the `$USER` directory: cd /g/data/$PROJECT/$USER/ git clone https://github.com/AndyHoggANU/anu-tub.git ``` + + + cd /g/data/$PROJECT/$USER + git clone https://github.com/AndyHoggANU/anu-tub.git + Cloning into anu-tub... + + !!! tip - If you have forked the repository and wish to clone the fork, change AndyHoggANU to your github username in the url. + If you have forked the repository and wish to clone the fork, change `AndyHoggANU` to your github username in the url. ## Setting up @@ -69,6 +76,11 @@ cd anu-tub/control/zstar vim config.yaml ``` + + cd anu-tub/control/zstar + vim config.yaml + + The contents of the `config.yaml` file should look like (without the comments that indicate what lines need changing): ```yaml @@ -100,7 +112,11 @@ userscripts: ``` As indicated above the `project`, `exe` and `input` information needs to been updated. -Once these changes have been made save and close vim (with `:wq`). +Once these changes have been made save and close vim. +!!! tip + To enter insert mode in vim press `i` and change the required fields. + To save and quit type `:wq`. + For further information or more commands view a [vim cheat sheet](https://vim.rtorr.com/). The experiment is designed to run on `scratch` so any output will be saved there. This can then be manually moved to `/g/data` or the executable script `sync_output_to_gdata.sh` can be run. @@ -118,14 +134,36 @@ module use /g/data/vk83/modules module load payu ``` + + module use /g/data/vk83/modulest + module load payu + Loading payu/1.2.0 + Loading requirement: singularity + + Within the `zstar` directory, where the `config.yaml` file we have just edited lives, run: ```bash payu setup ``` -*add terminal window* - + + payu setup + laboratory path: /scratch/$PROJECT/$USER/mom6 + input path: /scratch/$PROJECT/$USER/mom6/input + work path: /scratch/$PROJECT/$USER/mom6/work + archive path: /scratch/$PROJECT/$USER/mom6/archive + Found experiment archive: /scratch/$PROJECT/$USER/mom6/archive/zstar-tag_or_uuid + payu: Found modules in /opt/Modules/v4.3.0 + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + Setting up mom6 + Checking exe, input and restart manifests + Writing manifests/restart.yaml + + +The `tag_or_uuid` is something that will be generated as part of `payu setup` for this experiment. !!! tip If you recieve a message from `payu` that it expects a git repository run `git init` in the `zstar` directory. *Chris this is something I have had to do but maybe there are better ways around it?* @@ -137,7 +175,22 @@ Now we move the experiment to `scratch` with payu sweep ``` -*add terminal window* + + payu sweep + laboratory path: /scratch/$PROJECT/$USER/mom6 + input path: /scratch/$PROJECT/$USER/mom6/input + work path: /scratch/$PROJECT/$USER/mom6/work + archive path: /scratch/$PROJECT/$USER/mom6/archive + Found experiment archive: /scratch/$PROJECT/$USER/mom6/archive/zstar-tag_or_uuid + payu: Found modules in /opt/Modules/v4.3.0 + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + Setting up mom6 + Checking exe, input and restart manifests + Removing work path /scratch/$PROJECT/$USER/mom6/work/zstar-tag_or_uuid + Removing symlink /g/data/$PROJECT/$USER/anu-tub/control/zstar/work + Lastly, we run the experiment with @@ -145,6 +198,22 @@ Lastly, we run the experiment with payu run ``` + + payu run + laboratory path: /scratch/$PROJECT/$USER/mom6 + input path: /scratch/$PROJECT/$USER/mom6/input + work path: /scratch/$PROJECT/$USER/mom6/work + archive path: /scratch/$PROJECT/$USER/mom6/archive + Found experiment archive: /scratch/$PROJECT/$USER/mom6/archive/zstar-tag_or_uuid + payu: Found modules in /opt/Modules/v4.3.0 + Loading input manifest: manifests/input.yaml + Loading restart manifest: manifests/restart.yaml + Loading exe manifest: manifests/exe.yaml + payu: Found modules in /opt/Modules/v4.3.0 + qsub -q normal -P e14 -l walltime=43200 -l ncpus=240 -l mem=960GB -N tub-new -l wd -j n -v PAYU_PATH=/g/data/vk83/apps/base_conda/envs/payu-1.2.0/bin,MODULESHOME=/opt/Modules/v4.3.0,MODULES_CMD=/opt/Modules/v4.3.0/libexec/modulecmd.tcl,MODULEPATH=/g/data/vk83/modules:/etc/scl/modulefiles:/opt/Modules/modulefiles:/opt/Modules/v4.3.0/modulefiles:/apps/Modules/modulefiles -W umask=027 -l storage=gdata/e14+gdata/vk83+gdata/x77+scratch/e14 -- /g/data/vk83/./apps/conda_scripts/payu-1.2.0.d/bin/launcher.sh /g/data/vk83/./apps/base_conda/envs/payu-1.2.0/bin/python3.10 /g/data/vk83/apps/base_conda/envs/payu-1.2.0/bin/payu-run + 155451836.gadi-pbs + + !!! tip By default, a restart directory is created that is used when the experiment is run again. If you wish to not restart from this state it, and the corresponding output directory need to be removed within the experiment archive. From 92f71e3c75e0d56dc67e79eb91032b572a1bacf6 Mon Sep 17 00:00:00 2001 From: jbisits Date: Thu, 27 Nov 2025 12:10:34 +1100 Subject: [PATCH 05/12] Fix typo --- documentation/docs/pages/configurations/anu-tub.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index 417a3c0..a1fa19c 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -135,7 +135,7 @@ module load payu ``` - module use /g/data/vk83/modulest + module use /g/data/vk83/modules module load payu Loading payu/1.2.0 Loading requirement: singularity From 6c4ee02b861de846566c289f5d9456d4192a6d7c Mon Sep 17 00:00:00 2001 From: jbisits Date: Thu, 27 Nov 2025 20:17:43 +1100 Subject: [PATCH 06/12] Few more additions --- .../docs/pages/configurations/anu-tub.md | 93 +++++++++++++++++-- 1 file changed, 85 insertions(+), 8 deletions(-) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index a1fa19c..ad70c1e 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -46,9 +46,9 @@ We reccommend [building a MOM6 executable](https://access-community-hub.github.i This will provide the most flexibility as source code can be modified. If you would like to use an existing standalone MOM6 executable just make sure you know the exact path that points to it. -### Forking or cloning the ANU-TUB repository +### Cloning the ANU-TUB repository -Either option is fine here so whatever suits you! +ANU-TUB can either be cloned directly or, you can create a fork and clone that---either option is fine here so whatever suits you! The following commands clone the repository into the `$USER` directory: ```bash @@ -81,7 +81,7 @@ vim config.yaml vim config.yaml -The contents of the `config.yaml` file should look like (without the comments that indicate what lines need changing): +The contents of the `config.yaml` file should, without the comments indicating what lines need changing, look like: ```yaml ncpus: 240 @@ -220,15 +220,92 @@ payu run ## Changes to the experiment setup -Mention: +### Altering the `diag_table` -- altering the `diag_table` -- changing run time -- editing MOM6 source code +By cloning the repository, we have a `diag_table`. +To edit this use: + +```bash +vim diag_table +``` + + + + vim diag_table + + +This is a standard `diag_table` for MOM6 so [controlling run time diagnostics](https://mom6.readthedocs.io/en/main/api/generated/pages/Diagnostics.html) can be handled like any MOM6 model run. + +### Changing model run time + +To change the model run time we need to edit the `input.nml` file within the `zstar` directory. +Running, + +```bash +vim input.nml +``` + +we get: + +```bash +&mom_input_nml + output_directory = './' + input_filename = 'r' + restart_input_dir = 'INPUT/' + restart_output_dir = 'RESTART/' + parameter_filename = 'MOM_input', 'MOM_override' +/ + +&fms_nml + clock_grain = 'MODULE' + clock_flags = 'NONE' + domains_stack_size = 1200000 +/ + +&ocean_domains_nml +/ + +&diag_manager_nml +/ + +&ocean_solo_nml + months = 60 + days = 0 + date_init = 1, 1, 1, 0, 0, 0 + hours = 0 + minutes = 0 + seconds = 0 + calendar = 'julian' +/ + +&mpp_io_nml + deflate_level = 5 + shuffle = 1 +/ +``` + +By default, the model is set to run for 60 months. +To change the length of time update the `months` and `dates` field accordingly. +!!! warning + Be sure not to change other parts of this file (unless you know what you are doing) as it will likely lead to the model not running. + +### Editing MOM6 source code + +If you have built your own MOM6 executable, as long as you re-build before, and the `$PATH` to the executable has not changed, running: + +```bash +payu setup +payu sweep +payu run +``` + +should be sufficient for the new changes to the MOM6 source code to be used. +If not please get in touch! # Other resources The following resources may be useful for changing configurations or experiment setup: - [MOM6 documentation](https://mom6.readthedocs.io/en/main/) -- [running ACCESS models with `payu`](https://docs.access-hive.org.au/models/run_a_model/run_access-om3/) +- [Running ACCESS models with `payu`](https://docs.access-hive.org.au/models/run_a_model/run_access-om3/) +- [Learning to use vim](https://github.com/iggredible/Learn-Vim?tab=readme-ov-file) From bf0c5f9964ee6c1bd00d71c8579fb220ede87cfc Mon Sep 17 00:00:00 2001 From: jbisits Date: Thu, 27 Nov 2025 20:23:42 +1100 Subject: [PATCH 07/12] Fix typos --- documentation/docs/pages/configurations/anu-tub.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index ad70c1e..2b2e7a3 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -19,7 +19,7 @@ More specific details about the configuraiton, taken from the repository [README - Surface momentum forcing is via a prescribed zonal wind field that is constant, but varies with latitude. - Thermal forcing is through relation to a latitude-dependent SST profile, and there is (currently) no freshwater forcing so that salinity is constant. - We use the WRIGHT equation of state, without frazil formation or sea ice. -- The model includes the PBL surface boundary layer scheme with a contant background diffusivity of $2 \times 10^{-5}$. +- The model includes the PBL surface boundary layer scheme with a contant background diffusivity of $2 \times 10^{-5}$m$^{2}$s$^{-1}$. - The MEKE eddy parameterisation scheme is turned off, as is the mixed layer restratification scheme. The model has $160 \times 800$ grid points, with a tile layout of $6 \times 40$ to run efficiently on 240 cores. @@ -69,7 +69,7 @@ git clone https://github.com/AndyHoggANU/anu-tub.git With the ANU-TUB repository cloned to gadi we change into the `zstar` directory and edit the necessary part of the `config.yaml` file !!! tip - The same process below will work for any of the other vertical coordinate choices but `zstar` is a good place to start. + The same process below will work for any of the other vertical coordinate choice but `zstar` is a good place to start. ```bash cd anu-tub/control/zstar From 55589f357e87befef8f7261fb3b0b1ce041e11f0 Mon Sep 17 00:00:00 2001 From: jbisits Date: Fri, 28 Nov 2025 09:51:50 +1100 Subject: [PATCH 08/12] More info about getting necessary files --- .../docs/pages/configurations/anu-tub.md | 58 +++++++++++++++++-- 1 file changed, 54 insertions(+), 4 deletions(-) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index 2b2e7a3..3ff739d 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -93,7 +93,7 @@ queue: normal model: mom6 shortpath: /scratch/x77 # replace with NCI project you are part of exe: /g/data/x77/ahg157/exes/MOM6/AG-07f0a144 # replace with path to MOM6 executable -input: /g/data/x77/amh157/mom6/input/anu-tub # replace with path to input, possible COSIMA path Chris mentioned +input: /g/data/x77/amh157/mom6/input/anu-tub # replace with path to input, see instructions below storage: gdata: @@ -111,7 +111,7 @@ userscripts: archive: qsub sync_output_to_gdata.sh ``` -As indicated above the `project`, `exe` and `input` information needs to been updated. +As indicated above the `project`, `shortpath`, `exe` and `input` information needs to been updated. Once these changes have been made save and close vim. !!! tip To enter insert mode in vim press `i` and change the required fields. @@ -127,6 +127,22 @@ This can then be manually moved to `/g/data` or the executable script `sync_outp To run the ANU-TUB model we use [`payu`](https://payu.readthedocs.io/en/stable/install.html), a workflow management tool for running models on gadi. This requires membership to project vk83 -- add details of how to become member here (or above). +We also need to get the necessary input file from **/path/to/stored/input/files.** + +### Getting the necessary input files + +This can be done in two ways. +The simplest way is to edit the `input` field in `config.yaml` to point to where these files are located on gadi. +To do this change the `input` field to + +```yaml +input: /path/to/stored/input/files +``` + +Alternatively, the input files can be copied to a directory of your choice. +Then, add the path to where they have been copied to the `input` field. + +### Experiment setup with `payu` First, we load payu into our gadi environment ```bash @@ -192,6 +208,31 @@ payu sweep Removing symlink /g/data/$PROJECT/$USER/anu-tub/control/zstar/work +### Getting restart files + +Restart files are available that **get info about it.** +If you wish to use this restart, it needs to be copied into the experiment directory. +To do this, and check it has been copied to the directory type: + +```bash +cd /scratch/$PROJECT/$USER/mom6/work/zstar-tag_or_uuid +cp /path/to/restart . +ls -a +``` + + + + + cp path/to/restart . + ls -a + . .. manifests metadata.yaml payu_jobs pbs_logs restart441 + + +!!! tip + This is where restart files that are generated at the end of an experiment run are saved. + +### Run an experiment + Lastly, we run the experiment with ```bash @@ -215,8 +256,17 @@ payu run !!! tip - By default, a restart directory is created that is used when the experiment is run again. - If you wish to not restart from this state it, and the corresponding output directory need to be removed within the experiment archive. + By default, a new restart directory is created at the end of an experiment that is used to pick up from when the experiment is run again. + If you wish to not restart from this state remove the newly created restart directory, and the corresponding output directory. These will be named `restartXXX` and `outputXXX` where `XXX` is a string of numbers. + +To check the progress of the experiment type: + +```bash +qstat -swx +``` + +into the terminal. +Further details on [job monitoring](https://opus.nci.org.au/spaces/Help/pages/236880322/Job+monitoring...) are available in the using gadi section of the NCI documentation. ## Changes to the experiment setup From c6f99b59ddb296ef43431c546a155fa4977b8969 Mon Sep 17 00:00:00 2001 From: jbisits Date: Fri, 28 Nov 2025 09:53:49 +1100 Subject: [PATCH 09/12] Saving script info --- documentation/docs/pages/configurations/anu-tub.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index 3ff739d..2b054db 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -142,6 +142,10 @@ input: /path/to/stored/input/files Alternatively, the input files can be copied to a directory of your choice. Then, add the path to where they have been copied to the `input` field. +### Saving script to copy experiment output to `g/data/` + +**Ask Chris if we should provide this or just mention data is saved on scratch so need to be moved to `g/data` if you want to keep it for longer.** + ### Experiment setup with `payu` First, we load payu into our gadi environment From baa8d298d1b99996583995aee890f7e2f08a2435 Mon Sep 17 00:00:00 2001 From: jbisits Date: Thu, 4 Dec 2025 09:19:28 +1100 Subject: [PATCH 10/12] Updates plus prepare section for pre-build config --- .../docs/pages/configurations/anu-tub.md | 24 ++++++++++++++++--- 1 file changed, 21 insertions(+), 3 deletions(-) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index 2b054db..3882791 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -1,6 +1,8 @@ # ANU-TUB, an idealised sector model -The purpose of this page is to demonstrate how to set up the ANU-TUB model up with a MOM6 executable of the user's choice on gadi. +The purpose of this page is to demonstrate how to: +1. run an ANU-TUB model with a MOM6 ocean only executable provided by ACCESS-NRI; and/or +1. set up the ANU-TUB model up with a MOM6 executable of the user's choice on gadi. ## ANU-TUB configurarion details @@ -26,6 +28,16 @@ The model has $160 \times 800$ grid points, with a tile layout of $6 \times 40$ With a 1200-second timestep, the standard ZSTAR case takes 1:45 hours per year (~14 years/day) and consumes ~900 SU per model year. The relatively low cost of the ANU-TUB configuration allows for a wide variety of idealised experiments to be run that can investigate vertical coordinates, new diagnostics and many other things! +# Using a pre-built ANU-TUB configuration + +This will be done once the ACCESS-NRI mom6 ocean only build is done. + +# Building your own ANU-TUB configuration + +Below the steps for creating your own ANU-TUB configuration are outlined. +!!! tip + This is more advanced than just running the ANU-TUB configuration that is provided, however it does allow more flexibility especially with regards to changing the MOM6 source code the model uses. + ## Requirements The required components for setting up ANU-TUB are: @@ -111,7 +123,13 @@ userscripts: archive: qsub sync_output_to_gdata.sh ``` -As indicated above the `project`, `shortpath`, `exe` and `input` information needs to been updated. +As indicated above the `project`, `shortpath`, `exe` and `input` information needs to been updated: + +- `project` needs to be changed to the project which will be used to run the job on gadi; +- `shortpath` needs to have `x77` changed to the same project number as `project`; +- `exe` needs to point to the compiled ocean only MOM6 executable to be used in the model, e.g. `exe: /g/data/$PROJECT/$USER/path/to/MOM6executable` +- `input` needs to point to where the input files are stored. These files are located at **add path once known** so change to `input: /g/data/path/to/files/when/known`. + Once these changes have been made save and close vim. !!! tip To enter insert mode in vim press `i` and change the required fields. @@ -129,7 +147,7 @@ To run the ANU-TUB model we use [`payu`](https://payu.readthedocs.io/en/stable/i This requires membership to project vk83 -- add details of how to become member here (or above). We also need to get the necessary input file from **/path/to/stored/input/files.** -### Getting the necessary input files +### Getting the necessary input files (remove once files are saved on gadi) This can be done in two ways. The simplest way is to edit the `input` field in `config.yaml` to point to where these files are located on gadi. From e6a6adcdcc680e7ae34188785454700554cd82f3 Mon Sep 17 00:00:00 2001 From: jbisits Date: Thu, 4 Dec 2025 10:57:24 +1100 Subject: [PATCH 11/12] Space before enumerations --- documentation/docs/pages/configurations/anu-tub.md | 1 + 1 file changed, 1 insertion(+) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index 3882791..85e5b8c 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -1,6 +1,7 @@ # ANU-TUB, an idealised sector model The purpose of this page is to demonstrate how to: + 1. run an ANU-TUB model with a MOM6 ocean only executable provided by ACCESS-NRI; and/or 1. set up the ANU-TUB model up with a MOM6 executable of the user's choice on gadi. From 48ee503f15a8ed808678e9520501537bec33de09 Mon Sep 17 00:00:00 2001 From: jbisits Date: Tue, 3 Feb 2026 16:12:43 +1100 Subject: [PATCH 12/12] update link to MOM6 building --- documentation/docs/pages/configurations/anu-tub.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/docs/pages/configurations/anu-tub.md b/documentation/docs/pages/configurations/anu-tub.md index 85e5b8c..90147a2 100644 --- a/documentation/docs/pages/configurations/anu-tub.md +++ b/documentation/docs/pages/configurations/anu-tub.md @@ -55,7 +55,7 @@ Maybe with Andy's permission can put the restart files into the COSIMA directory ### MOM6 executable -We reccommend [building a MOM6 executable](https://access-community-hub.github.io/access-iom3-configs/infrastructure/Building/). +We reccommend [building a MOM6 executable](https://access-om3-idealised-configs.readthedocs.io/infrastructure/Building/). This will provide the most flexibility as source code can be modified. If you would like to use an existing standalone MOM6 executable just make sure you know the exact path that points to it.