Separate GetObs into individual tasks per observation

[mranst]

This is an alternative to #721, we can look at merging either one or the other depending on which makes the most sense. Looping through observations has been removed in favor of fetching each observation as its own task

Pros:
Much simpler to see which observations are failing to fetch and why
Can re-trigger failed tasks easily

Cons:
May complicate flow.cylc somewhat
Implementation may be a bit hacky

In order to limit the amount of concurrent tasks, I introduced a key for this cylc parameter that is limited to 10 by default

[mer-a-o]

I added two lines to hofx_cf suite to make GetObservations nested like the image below:

It'll make the tui a little cleaner. Is this something that we can use for other suites too?

[mranst]

It'll make the tui a little cleaner. Is this something that we can use for other suites too?

Cool, I didn't know inherit would group tasks like that in the tui, thanks!

[Dooruk]

I really like the ability to see individual observations, it would tremendously help in a monitoring setting. It's better than #721 in that regard.

One downside is now we would have 30 log files instead of 1 for each GetObs task, which is an IOnode issue over multiple timesteps. But we could look into some other solution for that. I agree it complicates flow.cylc but not too bad IMO.

Does it help with the execution time? I can test it when I'm not abusing R2D2 API calls but maybe you already made comparisons?

[mranst]

I really like the ability to see individual observations, it would tremendously help in a monitoring setting. It's better than #721 in that regard.

One downside is now we would have 30 log files instead of 1 for each GetObs task, which is an IOnode issue over multiple timesteps. But we could look into some other solution for that. I agree it complicates flow.cylc but not too bad IMO.

Does it help with the execution time? I can test it when I'm not abusing R2D2 API calls but maybe you already made comparisons?

IOnodes, always a concern... I'm not sure there's a good solution to this, since having each have their separate log file is part of the appeal of doing it like this

It's a little difficult to gauge because there's not the clean time readout at the end, but from what I can tell from the scheduling log it takes over twice as long as #721 🙃