CMSSW in REANA / snakemake / htcondor

[katilp]

We have following use cases:

Environment

• CMSSW standalone images with ready-made release area
  • the recent 10_6 and 8_X images require more space than available in the REANA cluster node (the images 30G)
  • more resources can be requested but they do not get allocated (at least in some hours)
  • the old reco 5_3 images run on the REANA cluster
• CMSSW release area built on top of a base image
  • Running

Releases:

• Sim: CMSSW_10_6_X, CMSSW_8_X_X (hlt)
• HI: CMSSW_5_3_8_HI_patch2 (PA), CMSSW_5_3_8_patch3 (pp)

Compute backend:

• REANA cluster
  • standalone won't run sim worflows, see above
  • workflows can run using the base image and building the CMSSW from cvmfs, but CMSSW_5_3_8_HI_patch2 is to be tested
• htcondor
  • htcondor backend is specified in the Snakefile resources,
  • if Snakefile present, it is not enough to define it in reana.yaml
  • sim workflows can run using htcondor writing to the REANA storage but not when writing authenticated to eos because it does not find kerberos cache
  • reco workflows (CMSSW_5_3_) either the job_wrapper called and not found from entrypoint or segmentation

What works? What does not work?

What works:

• HI workflows using the standalone images (both CMSSW versions) on the REANA cluster writing to eos
• HI workflows using the image to be tested
• sim workflows using the base image on the REANA cluster writing to eos

To do:

• run plain snakemake (without reana) on htcondor
  • https://oer-particle-physics.github.io/snakemake-particle-physics/

[KhaledAbuQ]

With the CMSSW release area built on top of a base image, to reproduce the 3 scenarios.

The reana.yaml file is consistent in the 3 cases:

version: 0.9.4
inputs:
  files:
  - Snakefile
workflow:
  file: Snakefile
  type: snakemake
  
  resources:
    kerberos: True
    cvmfs:
    - cms.cern.ch

1. Writing to EOS without HTCondor compute backend:
   Snakefile

eos_dir = "/eos/user/k/kabuquti/testing"

rule:
    input:
        "test.txt"

rule test_eos:
    output:
        "test.txt"
    container:
        "docker://gitlab-registry.cern.ch/cms-cloud/cmssw-docker/cc7-cms"
    shell:
        """
        set -e

        echo "Sourcing CMS environment to get ROOT..."
        export SCRAM_ARCH=slc7_amd64_gcc700
        source /cvmfs/cms.cern.ch/cmsset_default.sh
        
        scram p CMSSW CMSSW_10_6_30
        cd CMSSW_10_6_30/src/
        eval `scramv1 runtime -sh`
        cd ../../

        echo "Writing to EOS without HTCondor" > test.txt 

        mkdir -p {eos_dir}
        cp test.txt {eos_dir}
        """

Which runs fine and copies the test.txt file to my eos directory.

2. Writing to the REANA workspace using HTCondor
   Snakefile

eos_dir = "/eos/user/k/kabuquti/testing"

rule:
    input:
        "test.txt"

rule test_htcondor:
    output:
        "test.txt"
    container:
        "docker://gitlab-registry.cern.ch/cms-cloud/cmssw-docker/cc7-cms"
    resources:
        compute_backend="htcondorcern",
        htcondor_max_runtime="espresso"
    shell:
        """
        set -e

        echo "Sourcing CMS environment to get ROOT..."
        export SCRAM_ARCH=slc7_amd64_gcc700
        source /cvmfs/cms.cern.ch/cmsset_default.sh
        
        scram p CMSSW CMSSW_10_6_30
        cd CMSSW_10_6_30/src/
        eval `scramv1 runtime -sh`
        cd ../../

        echo "Writing to REANA workspace with HTCondor" > test.txt
        """

This also works fine and writes the test.txt file onto the REANA workspace

3. Attempting to write to EOS with HTCondor
   Snakefile

eos_dir = "/eos/user/k/kabuquti/testing"

rule:
    input:
        "test.txt"

rule test_htcondor_eos:
    output:
        "test.txt"
    container:
        "docker://gitlab-registry.cern.ch/cms-cloud/cmssw-docker/cc7-cms"
    resources:
        compute_backend="htcondorcern",
        htcondor_max_runtime="espresso"
    shell:
        """
        set -e

        echo "Sourcing CMS environment to get ROOT..."
        export SCRAM_ARCH=slc7_amd64_gcc700
        source /cvmfs/cms.cern.ch/cmsset_default.sh
        
        scram p CMSSW CMSSW_10_6_30
        cd CMSSW_10_6_30/src/
        eval `scramv1 runtime -sh`
        cd ../../

        echo "Copying to EOS with HTCondor" > test.txt
        ls {eos_dir}
        cp test.txt {eos_dir}

     """

With the above attempt, the workflow fails at the ls attempt giving:

ls: cannot access /eos/user/k/kabuquti/testing: No such file or directory

Showing that when htcondor is used in a rule, it can't access EOS.

This prompted attempting to use xrdcp and xrdfs to force the attempt of writing to EOS

eos_dir = "/eos/user/k/kabuquti/testing"
eos_server = "root://eosuser.cern.ch"
rule:
    input:
        "test.txt"

rule test_htcondor_eos:
    output:
        "test.txt"
    container:
        "docker://gitlab-registry.cern.ch/cms-cloud/cmssw-docker/cc7-cms"
    resources:
        compute_backend="htcondorcern",
        htcondor_max_runtime="espresso"
    shell:
        """
        set -e

        echo "Sourcing CMS environment to get ROOT..."
        export SCRAM_ARCH=slc7_amd64_gcc700
        source /cvmfs/cms.cern.ch/cmsset_default.sh
        
        scram p CMSSW CMSSW_10_6_30
        cd CMSSW_10_6_30/src/
        eval `scramv1 runtime -sh`
        cd ../../

        echo "Copying to EOS with HTCondor" > test.txt
        
        xrdfs {eos_server} mkdir -p {eos_dir}
        xrdcp -f test.txt {eos_server}/{eos_dir}/test.txt
        """

The workflow then fails and the Job Logs provide the following error:
[ERROR] Server responded with an error: [3010] Unable to give access - user access restricted - unauthorized identity used: Permission denied;

[RedknanRonin]

Using plain snakemake, it is possible to on HTCondor run:

ReReco workflows using both CMSSW_5_3_8_HI_patch2 a CMSSW_5_3_8_patch3 through:

• building on top of /cvmfs/unpacked.cern.ch/gitlab-registry.cern.ch/cms-cloud/cmssw-docker/slc5-cms:latest with cmsrel
• Pulling and running on docker://gitlab-registry.cern.ch/cms-cloud/cmssw-docker/cmssw_5_3_8_XXX-slc5_amd64_gcc462:latest, though it requires specifying
  APPTAINER_CACHEDIR to for example eos and APPTAINER_TMPDIR=/tmp/$USER/apptainer_tmp to not face issues pulling ~15Gb image.

Uploading results to eos is also possible.