Large Ensemble System Test for DART

[kdraeder]

The background for this issue is in CAM issue #1482.

I'm following @billsacks suggestion to define a new multi_instance_tests.py, which will have most of
the functionality of nck.py. Nck.py and a new nle.py (large ensemble) will then use multi_instance_tests.py.
It's not too late to go in a different direction, if a better one is available.

I'll test this in a low resolution FHIST compset first, but eventually will want to test a standard resolution BHIST
(ne30 + oi%tx2_3v2)

Here are a few pieces of the CAM issue. Let me know if I should import more of them for easier reference.

---

@cacraig recommended attaching the CAM label "Must have CESM3" to the CAM issue.
I don't see a similar label in cime, but I'm still hoping to have this included in the CESM3.0 code.

Is there anyone in particular you want to be part of this conversation?

@jedwards4b has been shepherding these DART related issues through all the pieces of CESM.
@billsacks has provided large amounts of useful guidance.
@cacraigucar has been indispensible in getting these tests through the git(hub) processes.

Will this change (regression test) answers?
No

Will you be implementing this enhancement yourself?
Yes, with help.

[kdraeder]

@kdraeder - also, if you feel it would take you a while to follow that plan for the extension of NCK to NLE, I can help with that piece.

@billsacks Thanks for your offer!
I've coded something that looks plausible to me in

/glade/work/raeder/Models/cesm3.0_testing_PRs2/cime/CIME/SystemTests:
   multi_instance_tests.py
   nck.py
   nle.py

I haven't tested it in any way, since I want to know whether I'm on the right track.
Also, using black is very cumbersome for me, so I don't want to waste time on that
if the code algorithm is misguided.

[kdraeder]

A small tangent: I'd like this test to write out auxiliary cpl hist files from the multi-instance case.
My best guess is that I would do this via a new test_mod, possibly in
$cesm/components/cmeps/cime_config/testdefs/testmods_dirs/drv/auxcplhist/user_nl_cpl

The main use of this would be with an FHIST compset, but we'll probably want a variant from a BHIST,
which would have additional files (ocn -> atm forcing).

For FHIST I think that the list is

histaux_rof2med_file1_enabled = .true.
histaux_rof2med_file1_history_option = "nhours"
histaux_rof2med_file1_history_n = 6
histaux_rof2med_file1_auxname = 'rof.6h.avrg'

histaux_atm2med_file1_enabled = .true.
histaux_atm2med_file2_enabled = .true.
histaux_atm2med_file4_enabled = .true.
histaux_atm2med_file5_enabled = .true.
histaux_atm2med_file5_history_option = 'nhours'
histaux_atm2med_file5_history_n = 6
histaux_atm2med_file5_auxname = 'atm.6h.avrg'

[billsacks]

@kdraeder - Thanks for getting this started. I took a quick look at your draft code. Broadly speaking, yes, what you've set up looks like what I imagined - thanks!

A couple of comments:

• Since you initialize self._comp_classes in MultiInstanceTest.__init__, you can delete its initialization in the nck and nle init methods.
• I'd suggest passing ninst to the MultiInstanceTest.__init__ method from each subclass (nck/nle) and setting self._ninst there. That makes things more explicit and you don't risk forgetting to set it in some subclass.
• There are some hard-coded 2 values throughout the test definition that I think should be changed to use self._ninst.

[kdraeder]

@billsacks Thanks for those suggestions. They're embarrassingly obvious in retrospect.
And they were completely on target. A test of nck seems to have run to completion!
/glade/work/raeder/Exp/CESM+DART_testing/NCK.ne3.FHISTC_LT.outfrq9s_leapday_base
and
/glade/derecho/scratch/raeder/NCK.ne3.FHISTC_LT.outfrq9s_leapday_base.

Are there more detailed evaluations that I need to do, besides running the cs.status script?

[kdraeder]

One major flaw in the test is that neither case wrote out instances of any file type.
There should be, e.g., ...cam_0001.i.... cam_0002, ... but there's only cam.i
So it looks like the ninst specified in nck.py is not propagating to cime, and a default value of 1 is being used.

[jedwards4b]

I think that the only way to set the number of instances in the case is by using the test name modifier
_C2.

[billsacks]

I think that the only way to set the number of instances in the case is by using the test name modifier _C2.

Hmmm, interesting. Indeed, I just checked a recently-run NCK test (/glade/derecho/scratch/fischer/t/cesm3_0_alpha08f_i/NCK_Ld1.f10_f10_mg37.I2000Clm50Sp.derecho_intel.clm-default--clm-nofireemis.GC.c3_0_a08f_int) and see that it isn't actually running multi-instance in either case!

This is in the nck test:

self._case.set_value("NINST_{}".format(comp), 2)

Does that not work for some reason?

Will the reverse (self._case.set_value("NINST_{}".format(comp), 1)) work if the test is set up with _C2?

So I guess step 1 is getting the NCK test working right, then step 2 is generalizing it.