Skip to content

Separate GetObs into individual tasks per observation #724

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mranst wants to merge 20 commits into
base: develop
Choose a base branch
from
Draft

Separate GetObs into individual tasks per observation #724

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
af3d6ba
Separate each fetch into its own task
mranst Mar 5, 2026
e616d67
Fix cylc templating
mranst Mar 5, 2026
12ed178
Fix task file parsing
mranst Mar 5, 2026
7ee4129
Up to 3dvar_atmos
mranst Mar 5, 2026
4ead142
Fix queue
mranst Mar 5, 2026
6a83ca3
Add all suites
mranst Mar 5, 2026
d99b12e
Fix bug in metop c yaml
mranst Mar 5, 2026
a4a95a4
Handle observation specification using arguments
mranst Mar 5, 2026
121a1a5
add localensembleda and 3dfgat_cycle
mranst Mar 5, 2026
20ccbcf
add key for concurrent limit
mranst Mar 5, 2026
7a670a9
fixes
mranst Mar 5, 2026
bd050ff
add 3dfgat_atmos
mranst Mar 5, 2026
6227961
add hofx_cf
mranst Mar 5, 2026
a08bf59
Merge branch 'develop' into feature/mranst/individual_obs
mranst Mar 6, 2026
afd1feb
Add GetObservations group to hofx_cf
mranst Mar 6, 2026
04220b4
Group GetObs tasks
mranst Mar 6, 2026
d99a8c6
Fix GetObs group
mranst Mar 12, 2026
17e3345
Group in 3dvar
mranst Mar 12, 2026
995e11f
Merge branch 'develop' into feature/mranst/individual_obs
mranst Mar 12, 2026
93ad28d
Add comma
mranst Mar 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 29 additions & 6 deletions src/swell/suites/3dfgat_atmos/flow.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@
initial cycle point = {{start_cycle_point}}
final cycle point = {{final_cycle_point}}
runahead limit = {{runahead_limit}}
[[queues]]
[[[default]]]
limit = {{concurrent_limit}}

[[graph]]
R1 = """
Expand Down Expand Up @@ -58,13 +61,17 @@
{% if cycling_varbc %}
# Cycling VarBC is active, biases from the previous cycle will be used

RunJediVariationalExecutable-{{model_component}}[-PT6H] => GetObservations-{{model_component}}
RunJediVariationalExecutable-{{model_component}}[-PT6H] => GetObservationsStart
{% else %}

# Cycling VarBC is inactive, static bias files will be used
GetObservations-{{model_component}}
GetObservationsStart
{% endif %}

{% for observation in models[model_component]['observations'] %}
GetObservationsStart => GetObservations-{{observation}}-{{model_component}} => GetObservationsFinish
{% endfor %}

# Perform staging that is cycle dependent
StageJediCycle-{{model_component}}

Expand All @@ -73,10 +80,9 @@
CloneJedi[^] => StageJediCycle-{{model_component}}
StageJediCycle-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
GetBackgroundGeosExperiment-{{model_component}}? | GetBackground-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
GetObsNotInR2d2-{{model_component}}: fail? => GetObservations-{{model_component}}
GetObsNotInR2d2-{{model_component}}? | GetObservations-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
GetObsNotInR2d2-{{model_component}}: fail? => GetObservationsStart
GetObsNotInR2d2-{{model_component}}? | GetObservationsFinish => RenderJediObservations-{{model_component}}
GenerateObservingSystemRecords-{{model_component}} => RenderJediObservations-{{model_component}}
GetObservations-{{model_component}} => RenderJediObservations-{{model_component}}

RenderJediObservations-{{model_component}} => RunJediVariationalExecutable-{{model_component}}

Expand Down Expand Up @@ -152,8 +158,25 @@
[[GetBackgroundGeosExperiment-{{model_component}} ]]
script = "swell task GetBackgroundGeosExperiment $config -d $datetime -m {{model_component}}"

[[GetObservationsStart]]
script = "true"

[[GetObservationsFinish]]
script = "true"

# ---------------------------------------------------------------------------------------------

[[GetObservations-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"

# ---------------------------------------------------------------------------------------------

{% for observation in models[model_component]['observations'] %}
[[GetObservations-{{observation}}-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}} -a {{observation}}"
inherit = GetObservations-{{model_component}}
{% endfor %}
# ---------------------------------------------------------------------------------------------

[[RenderJediObservations-{{model_component}}]]
script = "swell task RenderJediObservations $config -d $datetime -m {{model_component}}"
Expand Down
24 changes: 22 additions & 2 deletions src/swell/suites/3dfgat_marine_cycle/flow.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@
initial cycle point = {{start_cycle_point}}
final cycle point = {{final_cycle_point}}
runahead limit = {{runahead_limit}}
[[queues]]
[[[default]]]
limit = {{concurrent_limit}}

[[graph]]
R1 = """
Expand Down Expand Up @@ -76,7 +79,13 @@
LinkCoupledGeosOutput-{{model_component}} => GenerateBClimatology-{{model_component}}

# Data assimilation preperation
GetObservations-{{model_component}} => RenderJediObservations-{{model_component}}
StageJediCycle-{{model_component}} => RunJediFgatExecutable-{{model_component}}

GenerateBClimatology-{{model_component}} => RunJediFgatExecutable-{{model_component}}
{% for observation in models[model_component]['observations'] %}
GetObservations-{{observation}}-{{model_component}} => RenderJediObservations-{{model_component}}
{% endfor %}

RenderJediObservations-{{model_component}} => RunJediFgatExecutable-{{model_component}}

LinkCoupledGeosOutput-{{model_component}} => RunJediFgatExecutable-{{model_component}}
Expand Down Expand Up @@ -188,8 +197,19 @@
[[StageJediCycle-{{model_component}}]]
script = "swell task StageJedi $config -d $datetime -m {{model_component}}"

# ---------------------------------------------------------------------------------------------

[[GetObservations-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"

# ---------------------------------------------------------------------------------------------

{% for observation in models[model_component]['observations'] %}
[[GetObservations-{{observation}}-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}} -a {{observation}}"
inherit = GetObservations-{{model_component}}
{% endfor %}
# ---------------------------------------------------------------------------------------------

[[GenerateBClimatology-{{model_component}}]]
script = "swell task GenerateBClimatology $config -d $datetime -m {{model_component}}"
Expand Down
27 changes: 24 additions & 3 deletions src/swell/suites/3dvar_atmos/flow.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@
initial cycle point = {{start_cycle_point}}
final cycle point = {{final_cycle_point}}
runahead limit = {{runahead_limit}}
[[queues]]
[[[default]]]
limit = {{concurrent_limit}}

[[graph]]
R1 = """
Expand Down Expand Up @@ -58,11 +61,11 @@
{% if cycling_varbc %}
# Cycling VarBC is active, biases from the previous cycle will be used

RunJediVariationalExecutable-{{model_component}}[-PT6H] => GetObservations-{{model_component}}
RunJediVariationalExecutable-{{model_component}}[-PT6H] => GetObservationsStart
{% else %}

# Cycling VarBC is inactive, static bias files will be used
GetObsNotInR2d2-{{model_component}}: fail? => GetObservations-{{model_component}}
GetObsNotInR2d2-{{model_component}}: fail? => GetObservationsStart
{% endif %}

# Perform staging that is cycle dependent
Expand All @@ -73,7 +76,10 @@
CloneJedi[^] => StageJediCycle-{{model_component}}
StageJediCycle-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
GetBackgroundGeosExperiment-{{model_component}}? | GetBackground-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
GetObsNotInR2d2-{{model_component}}? | GetObservations-{{model_component}} => RenderJediObservations-{{model_component}}
{% for observation in models[model_component]['observations'] %}
GetObservationsStart => GetObservations-{{observation}}-{{model_component}} => GetObservationsFinish
{% endfor %}
GetObsNotInR2d2-{{model_component}}? | GetObservationsFinish => RenderJediObservations-{{model_component}}
RenderJediObservations-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
GenerateObservingSystemRecords-{{model_component}} => RunJediVariationalExecutable-{{model_component}}

Expand Down Expand Up @@ -129,6 +135,12 @@
--{{key}} = {{value}}
{%- endfor %}

[[GetObservationsStart]]
script = "true"

[[GetObservationsFinish]]
script = "true"

{% for model_component in model_components %}

[[CloneGeosMksi-{{model_component}}]]
Expand All @@ -149,9 +161,18 @@
[[GetBackgroundGeosExperiment-{{model_component}} ]]
script = "swell task GetBackgroundGeosExperiment $config -d $datetime -m {{model_component}}"

# ---------------------------------------------------------------------------------------------

[[GetObservations-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"

{% for observation in models[model_component]['observations'] %}
[[GetObservations-{{observation}}-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}} -a {{observation}}"
inherit = GetObservations-{{model_component}}
{% endfor %}
# ---------------------------------------------------------------------------------------------

[[RenderJediObservations-{{model_component}}]]
script = "swell task RenderJediObservations $config -d $datetime -m {{model_component}}"

Expand Down
18 changes: 16 additions & 2 deletions src/swell/suites/3dvar_marine/flow.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@
initial cycle point = {{start_cycle_point}}
final cycle point = {{final_cycle_point}}
runahead limit = {{runahead_limit}}
[[queues]]
[[[default]]]
limit = {{concurrent_limit}}

[[graph]]
R1 = """
Expand Down Expand Up @@ -60,7 +63,9 @@
GetBackground-{{model_component}} => RunJediVariationalExecutable-{{model_component}}

GenerateBClimatology-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
GetObservations-{{model_component}} => RenderJediObservations-{{model_component}}
{% for observation in models[model_component]['observations'] %}
GetObservations-{{observation}}-{{model_component}} => RenderJediObservations-{{model_component}}
{% endfor %}
RenderJediObservations-{{model_component}} => RunJediVariationalExecutable-{{model_component}}

# EvaObservations
Expand Down Expand Up @@ -125,8 +130,17 @@
[[ GetBackground-{{model_component}} ]]
script = "swell task GetBackground $config -d $datetime -m {{model_component}}"

# ---------------------------------------------------------------------------------------------

[[GetObservations-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"

{% for observation in models[model_component]['observations'] %}
[[GetObservations-{{observation}}-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}} -a {{observation}}"
inherit = GetObservations-{{model_component}}
{% endfor %}
# ---------------------------------------------------------------------------------------------

[[GenerateBClimatology-{{model_component}}]]
script = "swell task GenerateBClimatology $config -d $datetime -m {{model_component}}"
Expand Down
20 changes: 18 additions & 2 deletions src/swell/suites/3dvar_marine_cycle/flow.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,9 @@

initial cycle point = {{start_cycle_point}}
final cycle point = {{final_cycle_point}}
[[queues]]
[[[default]]]
limit = {{concurrent_limit}}

[[graph]]
R1 = """
Expand Down Expand Up @@ -78,7 +81,9 @@
StageJediCycle-{{model_component}} => RunJediVariationalExecutable-{{model_component}}

GenerateBClimatology-{{model_component}} => RunJediVariationalExecutable-{{model_component}}
GetObservations-{{model_component}} => RenderJediObservations-{{model_component}}
{% for observation in models[model_component]['observations'] %}
GetObservations-{{observation}}-{{model_component}} => RenderJediObservations-{{model_component}}
{% endfor %}
RenderJediObservations-{{model_component}} => RunJediVariationalExecutable-{{model_component}}

# Run analysis diagnostics
Expand Down Expand Up @@ -187,8 +192,19 @@
[[StageJediCycle-{{model_component}}]]
script = "swell task StageJedi $config -d $datetime -m {{model_component}}"

# ---------------------------------------------------------------------------------------------

[[GetObservations-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"

# ---------------------------------------------------------------------------------------------

{% for observation in models[model_component]['observations'] %}
[[GetObservations-{{observation}}-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}} -a {{observation}}"
inherit = GetObservations-{{model_component}}
{% endfor %}
# ---------------------------------------------------------------------------------------------

[[GenerateBClimatology-{{model_component}}]]
script = "swell task GenerateBClimatology $config -d $datetime -m {{model_component}}"
Expand Down
29 changes: 26 additions & 3 deletions src/swell/suites/hofx/flow.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@
initial cycle point = {{start_cycle_point}}
final cycle point = {{final_cycle_point}}
runahead limit = {{runahead_limit}}
[[queues]]
[[[default]]]
limit = {{concurrent_limit}}

[[graph]]
R1 = """
Expand Down Expand Up @@ -55,7 +58,7 @@
GetBackgroundGeosExperiment-{{model_component}} :fail? => GetBackground-{{model_component}}

# Get observations
GetObsNotInR2d2-{{model_component}}: fail? => GetObservations-{{model_component}}
GetObsNotInR2d2-{{model_component}}: fail? => GetObservationsStart

# Perform staging that is cycle dependent
StageJediCycle-{{model_component}}
Expand All @@ -65,7 +68,10 @@
CloneJedi[^] => StageJediCycle-{{model_component}}
StageJediCycle-{{model_component}} => RunJediHofxExecutable-{{model_component}}
GetBackgroundGeosExperiment-{{model_component}}? | GetBackground-{{model_component}} => RunJediHofxExecutable-{{model_component}}
GetObsNotInR2d2-{{model_component}}? | GetObservations-{{model_component}} => RenderJediObservations-{{model_component}}
{% for observation in models[model_component]['observations'] %}
GetObservationsStart => GetObservations-{{observation}}-{{model_component}} => GetObservationsFinish
{% endfor %}
GetObsNotInR2d2-{{model_component}}? | GetObservationsFinish => RenderJediObservations-{{model_component}}
RenderJediObservations-{{model_component}} => RunJediHofxExecutable-{{model_component}}
GenerateObservingSystemRecords-{{model_component}} => RunJediHofxExecutable-{{model_component}}

Expand Down Expand Up @@ -114,6 +120,12 @@
--{{key}} = {{value}}
{%- endfor %}

[[GetObservationsStart]]
script = "true"

[[GetObservationsFinish]]
script = "true"

{% for model_component in model_components %}

[[CloneGeosMksi-{{model_component}}]]
Expand All @@ -131,8 +143,19 @@
[[GetBackgroundGeosExperiment-{{model_component}} ]]
script = "swell task GetBackgroundGeosExperiment $config -d $datetime -m {{model_component}}"

# ---------------------------------------------------------------------------------------------

[[GetObservations-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"

# ---------------------------------------------------------------------------------------------

{% for observation in models[model_component]['observations'] %}
[[GetObservations-{{observation}}-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}} -a {{observation}}"
inherit = GetObservations-{{model_component}}
{% endfor %}
# ---------------------------------------------------------------------------------------------

[[GetObsNotInR2d2-{{model_component}}]]
script = "swell task GetObsNotInR2d2 $config -d $datetime -m {{model_component}}"
Expand Down
20 changes: 17 additions & 3 deletions src/swell/suites/hofx_cf/flow.cylc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@
initial cycle point = {{start_cycle_point}}
final cycle point = {{final_cycle_point}}
runahead limit = {{runahead_limit}}
[[queues]]
[[[default]]]
limit = {{concurrent_limit}}

[[graph]]
R1 = """
Expand Down Expand Up @@ -48,8 +51,9 @@
# Get background
GetBackground-{{model_component}}

# Get observations
GetObservations-{{model_component}}
{% for observation in models[model_component]['observations'] %}
GetObservations-{{observation}}-{{model_component}} => RenderJediObservations-{{model_component}}
{% endfor %}

# Perform staging that is cycle dependent
StageJediCycle-{{model_component}}
Expand All @@ -59,7 +63,6 @@
StageJediCycle-{{model_component}} => RunJediHofxExecutable-{{model_component}}
GetBackground-{{model_component}} => RunJediHofxExecutable-{{model_component}}

GetObservations-{{model_component}} => RenderJediObservations-{{model_component}}
RenderJediObservations-{{model_component}} => RunJediHofxExecutable-{{model_component}}

# EvaObservations
Expand Down Expand Up @@ -115,8 +118,19 @@
[[GetBackground-{{model_component}}]]
script = "swell task GetBackground $config -d $datetime -m {{model_component}}"

# ---------------------------------------------------------------------------------------------

[[GetObservations-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}}"

# ---------------------------------------------------------------------------------------------

{% for observation in models[model_component]['observations'] %}
[[GetObservations-{{observation}}-{{model_component}}]]
script = "swell task GetObservations $config -d $datetime -m {{model_component}} -a {{observation}}"
inherit = GetObservations-{{model_component}}
{% endfor %}
# ---------------------------------------------------------------------------------------------

[[RenderJediObservations-{{model_component}}]]
script = "swell task RenderJediObservations $config -d $datetime -m {{model_component}}"
Expand Down
Loading
Loading