Adding cycling to 3dvar_cf suite

[mer-a-o]

Description

This PR adds a new cycling 3DVar suite for GEOS-CF. The experiment is designed to run a 12h forecast starting from the beginning of the assimilation window and apply the increment from the middle of the window using IAU. The forecast length and frequency can be set using forecast_length and forecast_output_frequency but I didn't test changing the forecast length.

Summary of changes:

• RC files that get templated with SWELL are in src/swell/configuration/jedi/interfaces/geos_cf/namelists. The remaining RC files (static) are copied into the scratch directory from geos_cf_run_dir.

• get_background task fetches backgrounds from R2D2 using background_experiment in the config as the experiment ID. In cycling experiments (those with "cycle" in their name), the experiment ID is set to background_experiment for the first cycle. For subsequent cycles, it is set to the r2d2_experiment_id of the current experiment, so that backgrounds from the previous cycle are fetched for the variational task in the current one.

• save_forecast task stores forecasts in R2D2 using r2d2_experiment_id as the experiment ID. These files are then fetched from R2D2 during the get_background task.

• Restarts are saved (save_restart) and fetched (get_restart) from R2D2. To save space, restart files are not stored on R2D2 at every cycle. The rst_store_interval key controls how many cycles pass before restart files are stored as real files rather than symlinks. In intermediate cycles, restarts are saved as symlinks.

• prep_forecast prepares the scratch directory within the current cycle's run directory:

  • Copies static RC files from geos_cf_run_dir and copies/edits RC files from src/swell/configuration/jedi/interfaces/geos_cf/namelists.
  • Copies GEOS-FP files for replay into the scratch directory.
  • Changes the format of the JEDI increment file using inc_template.

• run_forecast submits gcm_run_geoscf.j (a lighter version of gcm_run.j used for running GEOS-CF) to the queue and waits until the job finishes. One limitation: the Cylc interface shows this job as "running" whether it is waiting in the queue or actually executing. There may be better approaches for handling gcm_run.j execution.

• clean_cycle.py now cleans the scratch directory of the previous cycle after the current cycle completes. This way, the restart files from the previous cycle that are saved as symlinks are kept until the current cycle uses them.