Skip to content

Current expt time for running jobs in payu status#702

Open
Qian-HuiChen wants to merge 8 commits intopayu-org:masterfrom
ACCESS-NRI:701-model-run-time
Open

Current expt time for running jobs in payu status#702
Qian-HuiChen wants to merge 8 commits intopayu-org:masterfrom
ACCESS-NRI:701-model-run-time

Conversation

@Qian-HuiChen
Copy link
Copy Markdown
Collaborator

In payu status, display current experiment time for running jobs, as:

  Job ID:            xxx.gadi-pbs
  Run ID:            xxx
  Stage:             model-run
  Total queue time:  0h 1m 4s
  Current Expt Time: 1901-02-20T00:00:00
  ::

This is implemented in models:

  • OM2
  • ESM1.6
  • MOM6
  • OM3
    For other models, no current expt time will be displayed.

In payu status, display current model finish time for archived jobs, e.g.,

Job ID:            xxx.gadi-pbs
 Run ID:            xxx
 Stage:             archive
 Total queue time:  0h 0m 22s
 Model Finish Time: 1900-05-01T00:00:00
 ::

Closes #701 as part of #640.

@Qian-HuiChen Qian-HuiChen requested a review from jo-basevi March 12, 2026 03:11
@codecov
Copy link
Copy Markdown

codecov bot commented Mar 12, 2026

Codecov Report

❌ Patch coverage is 94.07407% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.04%. Comparing base (0c05697) to head (b50a431).

Files with missing lines Patch % Lines
payu/status.py 80.95% 4 Missing ⚠️
payu/models/access_esm1p6.py 75.00% 1 Missing ⚠️
payu/models/mom6.py 96.00% 1 Missing ⚠️
payu/models/um.py 96.15% 1 Missing ⚠️
payu/subcommands/status_cmd.py 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #702      +/-   ##
==========================================
+ Coverage   63.30%   64.04%   +0.74%     
==========================================
  Files          67       67              
  Lines        5426     5541     +115     
==========================================
+ Hits         3435     3549     +114     
- Misses       1991     1992       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Qian-HuiChen
Copy link
Copy Markdown
Collaborator Author

Good morning @anton-seaice @blimlim @ezhilsabareesh8,
We are trying to display current model time (e.g., 1901-05-01T00:00:00) in payu status, which could be helpful during a long experiment run.

It will be highly appreciated if you could provide some feedback from a user perspective:

  • Is this information useful?
  • Is the current mode of presentation clear?
  • Any other feedback :)

Thanks!

@jo-basevi
Copy link
Copy Markdown
Collaborator

jo-basevi commented Mar 17, 2026

Thanks for adding this in! Great to have this implemented for so many model drivers.

I was testing running this branch with a pre-existing MOM6 configuration, and I am getting a warning when running payu status:

~/payu_development/payu_fork/payu/models/mom6.py:164: UserWarning: Could not find required files: /scratch/tm70/USER/mom6/work/mom6_double_gyre-expt-d9cb6783/input.nml or /scratch/tm70/USER/mom6/work/mom6_double_gyre-expt-d9cb6783/ocean.stats

As the run has completed, the work directory no longer exists so should change the code to only query the current run time only when the model is running.

The other initial thing I noticed was payu status --json also prints warnings before printing a json blob. In this case the output probably should only be json so it can still be machine readable.
UPDATE: Ok it's still possible to parse the json even with warnings as the warnings I think go to stderr. (e.g. payu status --json | jq '.experiment_uuid' seems to parse fine)

Copy link
Copy Markdown
Contributor

@blimlim blimlim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Qian-HuiChen, I've had a look through the changes and I think they're looking really good. Being able to query the run date like this would be very useful.

I've added a few small comments and questions throughout.

For the ESM1.6 implementation, all the relevant files (work/atmosphere/namelists, work/atmosphere/atm.fort6.pe0, and the suggested work/atmosphere/um.res.yaml) are all used and produced specifically by the UM submodel. Because of this, I'm wondering whether it could make sense to move the implementation of get_curr_exp_time from access_esm1p6.py to um.py, and then access it using the get_cur_expt_time_using_submodel method in ESM1.6?

Comment on lines +338 to +349
def read_start_date(self, nl_path):
"""Read the start date from the namelist file"""
try:
namelist = f90nml.read(nl_path)
model_basis_time = namelist.get('nlstcall', {}).get('model_basis_time', None)
if model_basis_time is None:
warnings.warn(f"model_basis_time not found in {nl_path}")
return None
return datetime(*model_basis_time)
except Exception as e:
warnings.warn(f"Could not read file {nl_path}: {e}")
return None
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies @Qian-HuiChen, I'm think my previous suggestion for reading the start date from the work/atmosphere/namelists file might not be the best one! It should work perfectly as is, but I'm thinking there might be existing methods which could be used to reduce duplication.

Each UM restart directory contains a calendar yaml file um.res.yaml, which holds a copy of the restart date (see /g/data/vk83/prerelease/configurations/inputs/access-esm1p6/modern/pre-industrial/restart/2026.02.20/atmosphere for an example).

I'd forgotten that we'd previously added a method to the um.py driver, get_restart_datetime which reads the date from this file, and uses it for the date-based restart pruning.

I'm wondering if it would make sense to reuse this method to get the start date, as it could be pointed to the copy of um.res.yaml in the atmosphere/work directory.

One issue is that the um.res.yaml is technically not required to run the model, and it's possible for it to be missing. This would nearly never be the case though, but there would have to be a safety check/warning in case it is missing.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing this out! It is better to reuse the get_restart_datetime than writing a new one.
I have changed the code to read out the starting date from work/atmosphere/um.res.yaml.

)

if json_output:
print(json.dumps(data, indent=4))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the current experiment time also be written to the json output, or would there not really be any uses for it?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree it will be good to have the current expt time here as well. I have added it in the latest commit.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If adding current experiment time to the json and the formatted display then it might make sense to add it in build_job_info() to reduce duplication and use status == model-run

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While testing this out I just noticed that payu status --json also doesn't print out the model finish time after the run has completed:

pay========================================
Run: 0
  Job ID:            163732205.gadi-pbs
  Run ID:            cd891450d95f0b8454f880b80bbb9fed40c034b4
  Stage:             archive
  Total queue time:  0h 1m 27s
  Model Finish Time: 0001-01-06T00:00:00
  Exit Status:       0 (Success)
  Model Exit Code:   0 (Success)
  Output Log:        /home/565/sw6175/esm1.6/misc/gettime-test/pre-industrial.o163732205
  Error Log:         /home/565/sw6175/esm1.6/misc/gettime-test/pre-industrial.e163732205
  Job File:          /scratch/tm70/sw6175/access-esm/archive/gettime-test-dev-preindustrial+concentrations-d33399d8/payu_jobs/0/run/163732205.gadi-pbs.json
========================================
u(payu-env) [sw6175@gadi-login-01 gettime-test]$ payu status --json
{
    "experiment_uuid": "d33399d8-053d-4eb6-aa5b-2b7e0ff783e5",
    "runs": {
        "0": {
            "run": [
                {
                    "job_id": "163732205.gadi-pbs",
                    "run_id": "cd891450d95f0b8454f880b80bbb9fed40c034b4",
                    "stage": "archive",
                    "exit_status": 0,
                    "model_exit_status": 0,
                    "stdout_file": "/home/565/sw6175/esm1.6/misc/gettime-test/pre-industrial.o163732205",
                    "stderr_file": "/home/565/sw6175/esm1.6/misc/gettime-test/pre-industrial.e163732205",
                    "job_file": "/scratch/tm70/sw6175/access-esm/archive/gettime-test-dev-preindustrial+concentrations-d33399d8/payu_jobs/0/run/163732205.gadi-pbs.json",
                    "start_time": "2026-03-23T16:23:21.697106"
                }
            ]
        }
    }
}

I'm wondering whether it would also work to add the finish time in build_job_info(), though I don't think I properly understand yet how the finish time is set.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

though I don't think I properly understand yet how the finish time is set.

model_finish_time is parsed from restart files by get_model_restart_datetimes() in experiment.py when stage is changed to archive.

whether it would also work to add the finish time in build_job_info()

Sounds good! I have implemented this in the latest commit. Now payu status --json should return something like:

{
    "experiment_uuid": "eaxxxx7",
    "runs": {
        "5": {
            "run": [
                {
                    ::
                    "stage": "archive",
                    ::
                    "start_time": "2026-03-27Txxx",
                    "model_finish_time": "0012-02-11T00:00:00"
                }
            ]
        }
    }
}

I also confirm that model_finish_time is not written twice in job file.

if os.path.exists(log_path):
with open(log_path, 'r') as f:
for line in reversed(f.readlines()):
if line.startswith(" memory_write: model date"):
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like its currently hardcoded to write this every model day:

https://github.com/access-nri/cmeps/blob/988c698ed6973c9d3c3cd7a14f497468ba94519f/mediator/med_phases_profile_mod.F90#L96-L98

as long as

med_phases_profile

is run.

So this seems fairly robust that "memory_write: model date" would always be available

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for checking this in the model code 👍

metadata.setup()
expt = Experiment(lab, config_path=config_path)
expt.init_models()
cur_expt_time = expt.get_model_cur_expt_time()
Copy link
Copy Markdown
Contributor

@anton-seaice anton-seaice Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we skip get_model_cur_expt_time() this step if stage is payu setup:

e.g.

$ payu status
/g/data/tm70/as2285/payu-dev/payu-dev/lib64/python3.11/site-packages/payu/models/cesm_cmeps.py:439: UserWarning: Log file /scratch/tm70/as2285/access-om3/work/1112-update-25km-topo/log/med.log does not exist or does not contain current model time.
========================================
Run: 0
  Job ID:            163373011.gadi-pbs
  Stage:             setup
  Job File:          /scratch/tm70/as2285/access-om3/archive/1112-update-25km-topo/payu_jobs/0/run/163373011.gadi-pbs.json

shows a warning when it's the expected behaviour

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or job is in the queue still

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this could be deferred to a later point when we know the the current stage is "model-run"

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up!
Following the suggestion from @jo-basevi , I have changed payu to run get_model_cur_expt_time() only during model-run stage in the latest commit.

@anton-seaice
Copy link
Copy Markdown
Contributor

For OM3 - this appears to work ok.

It does take a while for the model to initialise and run the first day, is it unclear that the warning always gets returned during this initialisation"

$ payu status
/g/data/tm70/as2285/payu-dev/payu-dev/lib64/python3.11/site-packages/payu/models/cesm_cmeps.py:439: UserWarning: Log file /scratch/tm70/as2285/access-om3/work/1112-update-25km-topo-1028-MC_25km_jra_iaf+wombatlite-15d683d3/log/med.log does not exist or does not contain current model time.
========================================
Run: 0
  Job ID:            163380214.gadi-pbs
  Run ID:            e5ff7b27fa5b0fc7de126a8b8d10e19646b24b10
  Stage:             model-run
  Job File:          /scratch/tm70/as2285/access-om3/archive/1112-update-25km-topo-1028-MC_25km_jra_iaf+wombatlite-15d683d3/payu_jobs/0/run/163380214.gadi-pbs.json
========================================

Copy link
Copy Markdown
Collaborator

@jo-basevi jo-basevi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just added a couple more comments.

I wonder if each get_cur_expt_time() model driver method could return a cftime.datetime object. It would catch if a valid date was parsed from the files correctly, and it would also store calendar information that could be used at a later point

As there may (?) be updates to models and their file contents in future, I wonder if we could add a optional command-line flag to silence warnings?

Another option to protect payu status erroring out in the future would be to have a higher level exception catch, e.g. something like:

try:
   current_model_time = self.model.get_cur_expt_time()
except Exception:
   warnings.warn("Unexpected error parsing current experiment time")

Could also raise Exceptions for FileNotFound, ValueErrors etc during parsing for current run time, and let those exceptions bubble up and then catch exceptions at a higher level and then display a warning of the error. The benefits of this is that you don't need to return and handle None values as much.

metadata.setup()
expt = Experiment(lab, config_path=config_path)
expt.init_models()
cur_expt_time = expt.get_model_cur_expt_time()
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this could be deferred to a later point when we know the the current stage is "model-run"

def get_cur_expt_time(self):
"""For model not implemented experiment time calculate/read-out,
leaves a warning and returns None."""
print("Getting current experiment time is not yet implemented.")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could leave this print statement out or change it to a warning?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After further consideration, I agree that a persistent warning is too intrusive, especially since payu status remains functional to display all other info, even if the experiment time isn't available.

In the latest commit, I have moved these messages to logger.debug(). This suppresses the 'noise' for standard users while ensuring the status output remains clean (see below). If a developer needs to troubleshoot why a specific model's time isn't fetched, they can simply run with the debug logging level enabled.

========================================
Run: 0
  Job ID:            100000.gadi-pbs
  Run ID:            aaaaa0000000
  Stage:             model-run
  Job File:          /scratch/tm70/...
========================================

What do you think?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that could work. We don't have command-line flags currently that enables debug logging easily however. I think there was some talk about that in this issue: #673.
As this PR already has a number of changes, an option could be to use warnings now and then change it in a future PR?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right. I change this back to a print statement. When I use warnings.warn, it includes the file path and line numbers in the output. I want to avoid these technical "noise" in payu status display, so I left it as a print statement.

warnings.warn(f"Log file {log_path} does not exist or does not contain current model time.")
return None

except KeyError as e:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What raises the KeyError here? If 'cur_exp-datetime' is in a string but not a top-level key in the json?

Could also guard against errors with json.loads(line) where line isn't valid json.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking if line cannot be parsed as json or 'cur_exp-datetime' is not the top key.
I changed it to:

 if 'cur_exp-datetime' in line:
    line_json = json.loads(line)
    time_str = line_json.get('cur_exp-datetime', None)
    if time_str is not None:
        return cftime.datetime.strptime(time_str, '%Y-%m-%dT%H:%M:%S')

and let the error bubble up and then the top level will catch the error.

warn(f"Log file {log_path} does not exist or does not contain current model time.")
return None

except KeyError as e:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be an IndexError for line.split()[4]

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right! It is good to correctly specify the error.

@anton-seaice
Copy link
Copy Markdown
Contributor

As there may (?) be updates to models and their file contents in future, I wonder if we could add a optional command-line flag to silence warnings?

I also wondered if the whole change could be configured in config.yaml, e.g. the filename, the search string etc. But payu already has lots of configuration items hard coded in, so it seemed unnecessarily complicated !

…ur_expt_time to um; log error at debug level.
@Qian-HuiChen
Copy link
Copy Markdown
Collaborator Author

Thanks @jo-basevi for pointing out the exception handling. I totally agree that we should use try::except in the top level and let the error bubble up. In the latest commit, in status.py:

if run_info.get("stage") == "model-run":
    cur_expt_time = None
    try:
        cur_expt_time = expt.get_model_cur_expt_time()
    except (FileNotFoundError, IndexError, OSError, json.JSONDecodeError, UnboundLocalError) as e:
        logger.debug(f"Cannot parse current experiment time: {e}")
    except Exception as e:
        logger.warning(f"Unexpected error getting current experiment time: {e}")

    if cur_expt_time is not None:
        print(f"  {'Current Expt Time:':<18} {cur_expt_time.isoformat()}")

so it simplifies the code without returning a trillion amount of None.

I wonder if each get_cur_expt_time() model driver method could return a cftime.datetime object.

This is also a great idea. I have changed the driver to return cftime.datetime object and transform it into string right before the printing.

Copy link
Copy Markdown
Collaborator

@jo-basevi jo-basevi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for adding in all those changes! Just added a few more comments.

Comment on lines +18 to 22
import logging
logger = logging.getLogger(__name__)

from payu.models.model import Model

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import logging
logger = logging.getLogger(__name__)
from payu.models.model import Model
import logging
from payu.models.model import Model
logger = logging.getLogger(__name__)

Just a small change to move setting the logger to after the imports. I think this applies to a couple other files

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good! I have moved all logger to after the imports in the latest commit. Some of the model drivers now raise an error which will bubble up to get_model_cur_expt_time() in build_job_info(), so logger is no longer needed and removed in those.

return cftime.datetime.strptime(time_str, '%Y-%m-%dT%H:%M:%S')

logger.debug(f"cur_exp-datetime not found in {log_path}")
return None
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If catching exceptions higher up, could raise a ValueError here instead?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes so much more sense. Thanks!

Comment on lines +458 to +459
logger.debug("Current experiment time not implemented for this model.")
return None
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could raise NotImplementedError("Current experiment time not implemented for this model.")?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. I have improved this in the latest commit.

3: "proleptic_gregorian",
4: "noleap"
}
return cftime_calendars[calendar_int]
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could move this function to the mom_mixin.py class to keep the ocean_solo.res logic together? Then the CFTIME_CALENDARS can be a global variable used by both get_restart_datetime and get_calendar

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good! It does make the code much cleaner.

Comment on lines +164 to +165
logger.debug(f"date_init not found in {input_path}")
return None
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could raise another ValueError here?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. Implemented in the latest commit.

config_path=config_path)
metadata.setup()
expt = Experiment(lab, config_path=config_path)
expt.init_models()
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
expt.init_models()

No need to run this as init_models() is already run as part of Experiment initialisation.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh I see. Thanks for pointing this out :)

)

if json_output:
print(json.dumps(data, indent=4))
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If adding current experiment time to the json and the formatted display then it might make sense to add it in build_job_info() to reduce duplication and use status == model-run

Comment on lines +498 to +499
timings: Optional[dict[str, Any]] = None,
cur_expt_time: Optional[str] = None
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
timings: Optional[dict[str, Any]] = None,
cur_expt_time: Optional[str] = None
timings: Optional[dict[str, Any]] = None

This code can safely be removed now I think

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doneeee.

def get_cur_expt_time(self):
"""For model not implemented experiment time calculate/read-out,
leaves a warning and returns None."""
print("Getting current experiment time is not yet implemented.")
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that could work. We don't have command-line flags currently that enables debug logging easily however. I think there was some talk about that in this issue: #673.
As this PR already has a number of changes, an option could be to use warnings now and then change it in a future PR?

payu/status.py Outdated
cur_expt_time = None
try:
cur_expt_time = expt.get_model_cur_expt_time()
except (FileNotFoundError, IndexError, OSError, json.JSONDecodeError, UnboundLocalError) as e:
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What causes the UnboundLocalError?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There was somewhere that could potentially call cur_expt_time before assigning. But I think it has been changed now. I have removed UnboundLocalError in the lates commit.

…build_job_info; raise error and bubble up in build_job_info.
@Qian-HuiChen
Copy link
Copy Markdown
Collaborator Author

If adding current experiment time to the json and the formatted display then it might make sense to add it in build_job_info() to reduce duplication and use status == model-run

Thanks @jo-basevi. It sounds good! I have moved the calling of get_model_cur_expt_time into build_job_info().

Copy link
Copy Markdown
Contributor

@blimlim blimlim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Qian-HuiChen for putting in those changes, they're looking really good! I've just had another read through, and added some small comments

import os
import shutil
import string
import cftime
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this one isn't directly being used!

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Thanks :)

"""Get the current experiment time from file"""
log_path = os.path.join(self.expt.work_path, 'atmosphere', 'atm.fort6.pe0')

start_date = self.get_restart_datetime(self.expt.work_path + '/atmosphere')
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably just being a bit pedantic here, but it could be worth consistently using os.path.join as with the log_path = line further above

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to keep it consistent.


Returns:
--------
Current experiment time in string (e.g., 1900-01-01T01:00:00)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the latest changes, the docstring can be updated to specify that it returns a cftime.datetime rather than string

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Thanks :)

)

if json_output:
print(json.dumps(data, indent=4))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While testing this out I just noticed that payu status --json also doesn't print out the model finish time after the run has completed:

pay========================================
Run: 0
  Job ID:            163732205.gadi-pbs
  Run ID:            cd891450d95f0b8454f880b80bbb9fed40c034b4
  Stage:             archive
  Total queue time:  0h 1m 27s
  Model Finish Time: 0001-01-06T00:00:00
  Exit Status:       0 (Success)
  Model Exit Code:   0 (Success)
  Output Log:        /home/565/sw6175/esm1.6/misc/gettime-test/pre-industrial.o163732205
  Error Log:         /home/565/sw6175/esm1.6/misc/gettime-test/pre-industrial.e163732205
  Job File:          /scratch/tm70/sw6175/access-esm/archive/gettime-test-dev-preindustrial+concentrations-d33399d8/payu_jobs/0/run/163732205.gadi-pbs.json
========================================
u(payu-env) [sw6175@gadi-login-01 gettime-test]$ payu status --json
{
    "experiment_uuid": "d33399d8-053d-4eb6-aa5b-2b7e0ff783e5",
    "runs": {
        "0": {
            "run": [
                {
                    "job_id": "163732205.gadi-pbs",
                    "run_id": "cd891450d95f0b8454f880b80bbb9fed40c034b4",
                    "stage": "archive",
                    "exit_status": 0,
                    "model_exit_status": 0,
                    "stdout_file": "/home/565/sw6175/esm1.6/misc/gettime-test/pre-industrial.o163732205",
                    "stderr_file": "/home/565/sw6175/esm1.6/misc/gettime-test/pre-industrial.e163732205",
                    "job_file": "/scratch/tm70/sw6175/access-esm/archive/gettime-test-dev-preindustrial+concentrations-d33399d8/payu_jobs/0/run/163732205.gadi-pbs.json",
                    "start_time": "2026-03-23T16:23:21.697106"
                }
            ]
        }
    }
}

I'm wondering whether it would also work to add the finish time in build_job_info(), though I don't think I properly understand yet how the finish time is set.


import pytest
import f90nml
import logging
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this one might not be needed here

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Thanks :)

cur_expt_time = expt.get_model_cur_expt_time()


def test_read_start_date(caplog):
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wanted to check if the caplog fixture is required here, and similarly for test_read_timestep

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it was used in previous versions but the new tests didn't use caplog anymore. I removed them now.

import cftime
import yaml
import f90nml
import logging
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like these two aren't currently being used

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Thanks :)

parsed_run_dt = expt.model.get_restart_datetime(restart_path)
assert parsed_run_dt == date

def test_convert_timestep(caplog):
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like caplog isn't currently being used here and could be removed from the test

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Thanks :)

Comment on lines +322 to +326
def get_cur_expt_time(self):
""" Use UM submodel to get the current experiment time."""
model_types = ['um']
return self.get_cur_expt_time_using_submodel(model_types)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably not super important as people will be moving to ESM1.6, however since ESM1.5 also uses the UM, it would be simple to add this feature to it. I think it would just involve copying these lines into access.py

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have implemented this in access-esm1.5 in the latest commit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add current model time for running jobs

4 participants