ecFlow tool for early discussion

[christinaholtNOAA]

Synopsis

This is a very early prototype of a potential basis for an ecFlow tool that translates YAML to an ecFlow definition. I wanted to get some feedback before I go too much further.

In my working example, I'm testing with the following YAML:

workflow:
  extern: []
  vars: []
  suites_#type#:
    repeat:
      type: [prod] 
    family_primary:
      families_#synop#:
        repeat:
          synop: !list '[{% for h in range(0, 23, 23) %}"{{ "%02d" % h }}",{% endfor %} ]'
        family_aigfs:
          vars:
            PROJ: AIGFS
            PACKAGEHOME: /scratch3/BMC/wrfruc/cholt/aigfs/aigfs/ecf
            EVAL: YES
          family_v1.0:
            family_forecast:
              task_jaigfs_prep: {}
              task_jaigfs_forecast:
                trigger: jaigfs_prep == complete
                events: !list '[{% for h in range(0, 13, 6) %}"{{ "release_f%03d" % h }}",{% endfor %} ]'
              family_post:
                tasks_post_f#fhr#:
                  repeat:
                    fhr: !list '[{% for h in range(0, 13, 6) %}"{{ "%03d" % h }}",{% endfor %} ]'
                  trigger: "../jaigfs_forecast:release_f#fhr#"
                  vars:
                    FHR: "#fhr#"

And the result is:

#5.15.2
suite prod
  family primary
    family 00
      family aigfs
        family v1.0
          family forecast
            task jaigfs_prep
            task jaigfs_forecast
            family post
              task post_f000
              task post_f006
              task post_f012
            endfamily
          endfamily
        endfamily
      endfamily
    endfamily
  endfamily
endsuite
# enddef

At this point, I've added support for Suites, Families, and Tasks, but nothing else. The end goal is to have it look like this (more or less):

#5.15.1
suite prod
  family primary
    family 00
      family aigfs
        defstatus complete
        edit PROJ 'AIGFS'
        edit PACKAGEHOME '/scratch3/BMC/wrfruc/cholt/aigfs/aigfs/ecf'
        edit EVAL 'YES'
        family v1.0
          family forecast
            task jaigfs_prep
            task jaigfs_forecast
              trigger jaigfs_prep == complete
              event release_f000
              event release_f006
              event release_f012
            family post
              task jaigfs_post_f000
                trigger ../jaigfs_forecast:release_f000
                edit FHR '000'
              task jaigfs_post_f006
                trigger ../jaigfs_forecast:release_f006
                edit FHR '006'
              task jaigfs_post_f012
                trigger ../jaigfs_forecast:release_f012
                edit FHR '012'
            endfamily
          endfamily
        endfamily
      endfamily
    endfamily
  endfamily
endsuite
# enddef

One key aspect that I haven't seen a solution for yet is the repeat support that essentially behaves like a Rocoto metatask. I even stole the syntax of #var# as a reference to those repeated variables. I don't yet have the ability to reference those variables throughout, but I think it will be a relatively easy add as I am passing a references dict to all sub-levels of the tree.

I've been testing with a super simple script:

from pathlib import Path
from uwtools.ecflowtool import _ecFlowDef

config = Path("aigfs/ush/wflow.yaml")
e = _ecFlowDef(config)
print(e)

Type

• Bug fix (corrects a known issue)
• Code maintenance (refactoring, etc. without behavior change)
• Documentation
• Enhancement (adds new functionality)
• Tooling (CI, code-quality, packaging, revision-control, etc.)

Impact

• This is a breaking change (changes existing functionality)
• This is a non-breaking change (existing functionality continues to work as expected)

Checklist

• I have added myself and any co-authors to the PR's Assignees list.
• I have reviewed the documentation and have made any updates necessitated by this change.

[christinaholtNOAA]

Using Jinja2 expressions and dereferencing made things 100x easier! I added support for variables under Tasks, too, just to make sure I was getting the full benefit. Using an instance variable for these refs also means that a user can reference higher-level variables at any level, and we don't need to pass around that dict. I'm not sure that NCO typically wants to do that, but it is a pattern that's used in Rocoto, so it might come in handy for some users.

Here's the updated input YAML:

workflow:
  extern: []
  vars: []
  suites_{{ ec.type }}:
    repeat:
      type: [prod]
    family_primary:
      families_{{ ec.synop }}:
        repeat:
          synop: !list '[{% for h in range(0, 23, 23) %}"{{ "%02d" % h }}",{% endfor %} ]'
        family_aigfs:
          vars:
            PROJ: AIGFS
            PACKAGEHOME: /scratch3/BMC/wrfruc/cholt/aigfs/aigfs/ecf
            EVAL: YES
          family_v1.0:
            family_forecast:
              task_jaigfs_prep: {}
              task_jaigfs_forecast:
                trigger: jaigfs_prep == complete
                events: !list '[{% for h in range(0, 13, 6) %}"{{ "release_f%03d" % h }}",{% endfor %} ]'
              family_post:
                tasks_{{ ec.type }}_{{ ec.synop }}_post_f{{ ec.fhr }}:
                  repeat:
                    fhr: !list '[{% for h in range(0, 13, 6) %}"{{ "%03d" % h }}",{% endfor %} ]'
                  trigger: "../jaigfs_forecast:release_f{{ ec.fhr }}"
                  vars:
                    FHR: "{{ ec.fhr }}"

And the output suite definition where I've referenced synop and suite type in the task names:

#5.15.2
suite prod
  family primary
    family 00
      family aigfs
        family v1.0
          family forecast
            task jaigfs_prep
            task jaigfs_forecast
            family post
              task prod_00_post_f000
                edit FHR '000'
              task prod_00_post_f006
                edit FHR '006'
              task prod_00_post_f012
                edit FHR '012'
            endfamily
          endfamily
        endfamily
      endfamily
    endfamily
  endfamily
endsuite
# enddef

[christinaholtNOAA]

This logic got SO MUCH nicer with Jinja2 expressions!