Effective file transfer mechanism?

[mapsacosta]

TODO: No inbound, hangs, FNAL site firewall blocks this, maybe we can do it from a WMAgent, need to test this with cmsgwms-submit1?. Would be wonderful if script would just scp everything before HPC/COBALT job is submitted.

╰─ [$] scp -vvv root@fermicloud510.fnal.gov:/home/cmsdataops/macosta/CMSSW-mcgen-testjob.sh /home/macosta/fnalhpc_startd/lumberjack/hpc/cobalt-cms-13499/sandbox/CMSSW-mcgen-testjob.sh
Executing: program /usr/bin/ssh host fermicloud510.fnal.gov, user root, command scp -v -f /home/cmsdataops/macosta/CMSSW-mcgen-testjob.sh
OpenSSH_7.9p1, OpenSSL 1.1.0i-fips  14 Aug 2018
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 2: Applying options for *
debug2: resolving "fermicloud510.fnal.gov" port 22
debug2: ssh_connect_direct
debug1: Connecting to fermicloud510.fnal.gov [131.225.152.148] port 22.

In the meantime, login node has SSH inbound connectivity and can be reached by the Schedd, so we need to scp the files over. This can't be automated in any way since SSH login to ALCF requires MFA. But the script will generate the command... or maybe write it to a mini script. Will have to implement that as it sounds useful.