Implement interactive prompts for payu clone settings.

[Qian-HuiChen]

To reduce complexity and make payu easier for beginner users (see #655), this PR introduces an interactive mode in payu clone.

Users can simply run

payu clone

to start the guided prompt process.

Alternatively, users can provide known arguments (such as the repository, local directory, source branch, or commit) and enable the interactive prompts with:

payu clone -I [<repo>] [<local_dir>]

In this case, the wizard will only prompt for any missing information.

[codecov]

Codecov Report

❌ Patch coverage is 69.28571% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.67%. Comparing base (0c05697) to head (9524488).
⚠️ Report is 13 commits behind head on master.

Files with missing lines	Patch %	Lines
payu/subcommands/clone_cmd.py	70.31%	38 Missing ⚠️
payu/cli.py	58.33%	5 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #703      +/-   ##
==========================================
+ Coverage   63.30%   63.67%   +0.36%     
==========================================
  Files          67       67              
  Lines        5426     5561     +135     
==========================================
+ Hits         3435     3541     +106     
- Misses       1991     2020      +29     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Qian-HuiChen]

This is a diagram showing current workflow of payu clone prompts.

[blimlim]

This is looking really great @Qian-HuiChen. I'm finding it intuitive and like that the interactive mode automatically activates when you don't supply any arguments. The autocomplete is a great feature too.

I just had a couple comments from my testing so far:

1. I wonder if it would be informative to print out the equivalent one-line payu clone command just before it's run, to show the user how their choices map to a single command. E.g:

   ? Please enter the URL of the repository, or the local path of a configuration you want to clone:  (e.g., https://github.com/payu-org/bowl1.git, or /path/to/local/experiment) https://github.com/ACCESS-NRI/access-esm1.6-configs
   ? Do you want to clone the repo based on: An existing branch
   ? Please enter the name of the branch you want to clone ('Tab' to browse all branches): dev-preindustrial+concentrations
   ? How would you like to name your local experiment directory? devpi
   ? Is this a new experiment? (If yes, payu will create a new branch.) Yes
   ? Please enter the name of the new branch you want to create: test-1
   ? Are you starting from an existing restart path? No
   
   Running: 'payu clone  https://github.com/ACCESS-NRI/access-esm1.6-configs devpi -B dev-preindustrial+concentrations -b test-1 devpi'
   
   Cloned repository from https://github.com/ACCESS-NRI/access-esm1.6-configs to directory: /g/data/tm70/sw6175/development/temp/clone-tests/devpi
   

   I'm not sure whether this would actually be useful, so it's probably worth some discussion with others too!

2. The one bit of wording I found slightly harder to understand was Are you starting from an existing restart path? (Y/n). In the esm1.5/1.6 world, the model always starts from a restart file located somewhere on Gadi. A default is generally specified in the config.yaml, which can then be overwritten using the payu clone -r option (equivalent to selecting Y to the above option). I think the option is a bit less clear in this case as it uses an existing restart with either choice, though I might just be being a bit pedantic!

   I understand this is a bit different in OM2/OM3, where the models don't start from an existing restart by default, and the wording would apply quite well there.

   I wonder if something like Are you starting from a custom restart? (if N, the default restart/initial conditions will be used) could apply to both?

   It's worth checking with some others too though – not sure that my wording is any clearer.

3. The changes appear to add restrictions on the order of arguments when using the payu clone command. For no particular reason, I normally do

   payu clone https://github.com/ACCESS-NRI/access-esm1.6-configs -B dev-preindustrial+concentrations local-dir
   

   which longer works:

   payu clone https://github.com/ACCESS-NRI/access-esm1.6-configs -B dev-preindustrial+concentrations local-dir
   usage: payu [-h] [--version] {archive,branch,checkout,clone,collate,ghsetup,init,list,profile,push,run,setup,status,sweep,sync} ...
   payu: error: unrecognized arguments: local-dir
   

   I think this is related to the changes to nargs and default for repository and local directory.

Not sure how much of an issue this is, as possibly I'd gotten into a habit of using an odd order, but thought it's worth being aware of.

[Qian-HuiChen]

Thanks @blimlim!
In the latest commit,

• payu clone will show the complete command after users finish all prompts.
• In restart Y/n selection, the question is improved as "Do you want to specify a custom restart path? (If no, the default restart/initial conditions will be used.)"

Note for breaking change (SORRY!):

payu clone https://github.com/ACCESS-NRI/access-esm1.6-configs -B dev-preindustrial+concentrations local-dir

will no longer work after I updated <repo> and <local_dir> as optional arguments. The <repo> and <local_dir> must now be provided together, and no flags can appear between them. For example: payu clone <repo> -B <branch> <local_dir> will break.

@aidanheerdegen what do you think of this problem? Do you have any idea that could handle this more elegant?

[aidanheerdegen]

• payu clone will show the complete command after users finish all prompts.

Great. I think this is a good idea, and helps to inform users how to use the direct clone command.

[aidanheerdegen]

@aidanheerdegen what do you think of this problem? Do you have any idea that could handle this more elegant?

My suggestion would be to use mutually exclusive groups and not allow the -I option to have any other argument.

Though this doesn't allow a user to pre-fill some of the fields, it is conceptually simpler: payu clone is either interactive or it is not.

I don't think a lot of utility is lost, as the user likely has the url to clone in their paste buffer in any case, and the argument completion is so handy that it would likely be easier than looking up a branch name and adding it to the command line directly.

[Qian-HuiChen]

@aidanheerdegen Thank you for your comments and suggestions!
Unfortunately mutually_exclusive_groups have limitations on optional arguments. After some testing, it did not work as expected.

After some thinking on the design and the goal, I adopted a new method:

• keep and <local_dir> as required arguments --> so that all existing behavior remains unchanged. payu clone repo -B branch local_dir now should work. @blimlim could you please help me test it on your side again?
• when -I flag is presented, dynamically update required = False inside the parser --> so that payu clone -I doesn't require any arguments to work.

One advantage of this approach is that it can reusable for other payu command as well 😃

[aidanheerdegen]

This is a really nice addition, and thanks for getting and incorporating all the feedback.

[aidanheerdegen]

It might be worth taking a look at the codecov report to see if there are any tests that could be easily added

[blimlim]

keep and <local_dir> as required arguments --> so that all existing behavior remains unchanged. payu clone repo -B branch local_dir now should work. @blimlim could you please help me test it on your side again?

Apologies I was a bit slow to get to this, but with the latest changes it's working well!

> payu clone https://github.com/ACCESS-NRI/access-esm1.6-configs -B dev-preindustrial+concentrations newtest
> Cloned repository from https://github.com/ACCESS-NRI/access-esm1.6-configs to directory: /g/data/tm70/sw6175/development/temp/clone-tests/newtest
> Checked out branch: dev-preindustrial+concentrations
> laboratory path:  /scratch/tm70/sw6175/access-esm
> binary path:  /scratch/tm70/sw6175/access-esm/bin
> input path:  /scratch/tm70/sw6175/access-esm/input
> work path:  /scratch/tm70/sw6175/access-esm/work
> archive path:  /scratch/tm70/sw6175/access-esm/archive
> Updated metadata. Experiment UUID: 9ea24cf3-9efa-43b9-94ad-fc4a8b2c4663
> Added archive symlink to /scratch/tm70/sw6175/access-esm/archive/newtest-dev-preindustrial+concentrations-9ea24cf3
> To change directory to control directory run:
  cd newtest

[Qian-HuiChen]

@aidanheerdegen sounds good.

I added a few tests on clone_cmd.
I end up re-wrote the test_cli.py because it was using global parser and the tests were not stable. I re-wrote it with pytest.fixture.