Option to execute tasks in scratch area

[mranst]

This PR adds the option to send intermediate and output files to a scratch area such as TSE_staging (addressing #490). This can potentially give better performance by leveraging a faster intermediate directory. At the end of the run, all eligible files (those not deleted by the CleanCycle task), are moved back to the main experiment directory, and deleted from the temporary scratch directory.

In order to try this feature, set the run_in_scratch flag in experiment.yaml to true, and specify the scratch_root (this will default to TSE_staging in advda)

[ashiklom]

Why not use shutil.copy or shutil.copytree here?

https://docs.python.org/3/library/shutil.html#shutil.copytree

[ashiklom]

Haven't tested, but generally looks OK to me. I would just recommend replacing subprocess calls with Python functions (shutil.copy / shutil.copytree) which are more portable across systems and should have cleaner error handling.

[Dooruk]

Sorry this totally fell out of my radar, I don't think everyone has access to a TSE_staging area, only certain project groups have that. So for most people default advda won't work.

Does the TSE survive until the full suite/workflow is completed?

[mranst]

Sorry this totally fell out of my radar, I don't think everyone has access to a TSE_staging area, only certain project groups have that. So for most people default advda won't work.

Does the TSE survive until the full suite/workflow is completed?

Unless the suite fails, the experiment contents are scrubbed after the run. Instead of TSE_staging we could also do TSE_TMPDIR, which everyone on discover should have access to