Dev/automation

.gitignore

@@ -0,0 +1,4 @@
+**/__pycache__/** 
+/automation/logs/**
+/automation/queue.txt
+**.log

automation/README.md

@@ -0,0 +1,46 @@
+# Automation Tool Kit
+Tool kit to (semi) automize production of DQM plot from NanoAODs stored on tier 0. The automation is based on the cron job scheduler which executes bash scripts and commands periodically. The following commands will be useful:
+
+- show list of scheduled cron jobs: `acrontab -l`
+- remove all scheduled cron jobs: `acrontab -r`
+- open cron job editior: `acrontab -e`
+
+last acron command:
+```
+0 * * * * lxplus cd /afs/cern.ch/user/l/lebeling/MacrosNtuples/automation && sh cron_job.sh >>cron.log 2>&1
+```
+
+
+Inside the cron jib editior:
+- save changes via ctrl+o
+- close editior via ctrl+x
+
+Before running the automation tool kit, adjust the output path (i.e. the directory in which all plots and histogram are deployed) in: `utils.py` -> `dqm_prefix`
+
+To run the automation tool kit, open the cron editor and paste the following command (replace *PATH* by the actual installation path on lxplus):
+```
+* */1 * * * lxplus cd /PATH/MacrosNtuples/automation && sh cron_job.sh >>cron.log 2>&1
+```
+Cron will execute the command once every hour saving the output messages (and errors) into the *cron.log* logfile. More details on how to configure the timing of a cron job, can be found [here](https://crontab.guru).
+
+
+## automation steps
+The different processing steps are summarized as: 
+
+1) histograms
+Run `python3 make_hists.py` to produce `.root` files containing histograms for all data types (i.e. EGamma, Muon, JetMet). Which selections are run is specified in `config.yaml` (see scritps). If the respective output file already exists, the histogram production is skipped. 
+
+2) merge per run
+Run `python3 merge_per_run.py` to merge (i.e. hadd) the histogram files per run. If the respective output file already exists and is newer than all base histogram files, the merging is skipped. 
+
+3) merge per era/week
+Run `python3 merge_per_era.py` to further merge (i.e. hadd) the histograms per era (i.e. Run2024H) and per week using the merged histograms per run. If the respective output file already exists and is newer than all base histogram files, the merging is skipped. 
+
+4) merge per type
+Run `python3 merge_total.py` to merge (i.e. hadd) all histograms of one data type (i.e. EGamma, Muon, JetMet) using the merged histograms per era. If the respective output file already exists and is newer than all base histogram files, the merging is skipped. 
+
+5) plotting
+Run `make_plots.py` to produce png/pdf plots from all merged histograms (merge per run/era/week/total). The plotting repective scripts are specified `condfig.yaml`. If the png/pdf files already exist and are newer than the histogram files, the plotting is skipped.  
+
+## htcondor setup
+All prduction steps listed above can be run on htcondor. Using the flag `--htcondor`, the repective plotting scripts are not directly executed but instead written into the `queue.txt` file. With `condor_submit submit.txt`, all commands in the queue are submitted to htcondor. This mode is highly recommended to (re-)run all files currently stored on tier 0. 

automation/config.yaml

@@ -0,0 +1,45 @@
+JetMET:
+    datasets:
+        - 'JetMET0'
+        - 'JetMET1'
+    eras:
+        - 'Run2025*'
+    scripts:
+        - 'python3 ../l1macros/performances_nano.py -i $INFILE -o $OUTDIR/all_DiJet.root -c DiJet'  
+    plotting:
+        - 'python3 ../plotting/make_DiJet_plots.py --dir $OUTDIR --config ../config_cards/full_DiJet.yaml'
+
+EGamma:
+    datasets:
+        - 'EGamma0'
+        - 'EGamma1'
+        - 'EGamma2'
+        - 'EGamma3'
+    eras: 
+        - 'Run2025*'
+    scripts:
+        - 'python3 ../l1macros/performances_nano.py -i $INFILE -o $OUTDIR/all_PhotonJet.root -c PhotonJet'
+        - 'python3 ../l1macros/performances_nano.py -i $INFILE -o $OUTDIR/all_ZToEE.root -c ZToEE'
+        - 'python3 ../l1macros/performances_nano_dqmoff.py -i $INFILE -o $OUTDIR/oug_zee_dqmoff.root -c ZToEEDQMOff'
+    plotting:
+        - 'python3 ../plotting/make_ZToEE_plots.py --dir $OUTDIR --config ../config_cards/full_ZToEE.yaml'
+        - 'python3 ../plotting/make_PhotonJet_plots.py --dir $OUTDIR --config ../config_cards/full_PhotonJet.yaml'
+
+Muon:
+    datasets:
+        - 'Muon0'
+        - 'Muon1'
+    eras:
+        - 'Run2025*'
+    scripts:
+        - 'python3 ../l1macros/performances_nano.py -i $INFILE -o $OUTDIR/all_ZToMuMu.root -c ZToMuMu'
+        - 'python3 ../l1macros/performances_nano.py -i $INFILE -o $OUTDIR/all_MuonJet.root -c MuonJet' #TODO not working
+        - 'python3 ../l1macros/performances_nano.py -i $INFILE -o $OUTDIR/all_ZToTauTau.root -c ZToTauTau'
+        - 'python3 ../l1macros/performances_nano_dqmoff.py -i $INFILE -o $OUTDIR/out_zmumu_dqmoffl.root -c ZToMuMuDQMOff'
+        - 'python3 ../l1macros/performances_nano_dqmoff.py -i $INFILE -o $OUTDIR/out_jets_dqmoff.root -c JetsDQMOff'
+        - 'python3 ../l1macros/performances_nano_dqmoff.py -i $INFILE -o $OUTDIR/out_ztautau_dqmoff.root -c ZToTauTauDQMOff'
+        - 'python3 ../l1macros/performances_nano_dqmoff.py -i $INFILE -o $OUTDIR/out_etsum_dqmoff.root -c EtSumDQMOff'
+    plotting:
+        - 'python3 ../plotting/make_ZToMuMu_plots.py --dir $OUTDIR --config ../config_cards/full_ZToMuMu.yaml'
+        - 'python3 ../plotting/make_ZToTauTau_plots.py --dir $OUTDIR --config ../config_cards/full_ZToTauTau.yaml'
+        - 'python3 ../plotting/make_MuonJet_plots.py --dir $OUTDIR --config ../config_cards/full_MuonJet.yaml'

automation/cron_job.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+# python3 make_hists.py
+# python3 merge_per_run.py
+# python3 merge_per_era.py
+# python3 merge_total.py
+# python3 make_plots.py
+date
+echo "All done!"

automation/make_hists.py

@@ -0,0 +1,52 @@
+#!/bin/python3
+import yaml, os
+from glob import glob
+from utils import htcondor_flag, parse_file, run_command, write_queue, tier0, dqm_prefix
+
+config_file = yaml.safe_load(open('config.yaml', 'r'))
+
+htcondor = htcondor_flag()
+
+
+for label, config in config_file.items():
+
+    #step 1 - find all files on tier 0
+    #fnames = [glob(f"{tier0}/{era}/{dataset}/NANOAOD/PromptReco-v*/*/*/*/*/*.root") for era in config["eras"] for dataset in config["datasets"]]
+    fnames = []
+    for era in config["eras"]:
+        for dataset in config["datasets"]:
+            files = glob(f"{tier0}/{era}/{dataset}/NANOAOD/PromptReco-v*/*/*/*/*/*.root")
+            #print(files)
+            for f in files:
+                part = f.split("/")
+                run_str = part[-4] + part[-3]
+                run_num = int(run_str)
+                if run_num >= 392241:
+                    fnames.append(f)
+    #fnames = [item for sublist in fnames1 for item in sublist]
+    #step 2 - remove files that have already been processed
+    for file in fnames:
+        output_path = dqm_prefix + parse_file(file)
+        num_root_files = len(glob(f"{output_path}/*.root"))
+        if num_root_files > 0: 
+            fnames.remove(file)
+            print(file + " not processed")
+
+
+    #step 3 - run scripts
+    #enumerate over all files
+    for i, file in enumerate(fnames):
+        #if not htcondor and i == 10: break
+
+        print(f"Processing file {file}")
+
+        output_path = dqm_prefix + parse_file(file)
+
+        for cmd in config["scripts"]:
+            cmd = cmd.replace("$OUTDIR", output_path)
+            cmd = cmd.replace("$INFILE", file)
+
+            os.makedirs(output_path, exist_ok=True)
+
+            if htcondor: write_queue(cmd)
+            else: run_command(cmd, output_path + "/log.txt")

automation/make_plots.py

@@ -0,0 +1,35 @@
+#!/bin/python3
+
+import os, argparse, yaml
+from glob import glob
+from utils import run_command, write_queue, htcondor_flag, dqm_prefix
+
+# load config
+config_dict = yaml.safe_load(open('config.yaml', 'r'))
+
+htcondor = htcondor_flag()
+
+# main logic: glob files merged root files and make plots
+for label, config in config_dict.items():
+    pattern = os.path.join(dqm_prefix, '**', label,'**', 'merged')
+    merged_dirs = glob(pattern, recursive=True)
+
+    for merged_dir in merged_dirs:
+
+        # abort plotting if all .png files are newer than all .root files
+        t_newest, t_oldest = 0, 0
+        root_files = glob(f"{merged_dir}/*.root")
+        png_files = glob(f"{merged_dir}/plotsL1Run3/*.png")
+        if len(root_files) > 0: t_newest = max(os.path.getctime(f) for f in root_files)
+        if len(png_files) > 0: t_oldest = min(os.path.getctime(f) for f in png_files)
+        if t_oldest > t_newest: 
+            print('skipping: ' + merged_dir)
+            continue
+
+        for cmd in config["plotting"]:
+            print(80*"#"+'\n'+f"plotting for {merged_dir}")
+            os.makedirs(merged_dir + '/plotsL1Run3', exist_ok=True)
+            cmd = cmd.replace("$OUTDIR", merged_dir)
+            print(cmd)
+            if htcondor: write_queue(cmd) 
+            else: run_command(cmd, merged_dir + '/log.txt')

automation/merge_per_era.py

@@ -0,0 +1,39 @@
+#!/bin/python3
+
+from glob import glob
+from utils import hadd, get_weeks, htcondor_flag, dqm_prefix
+
+htcondor = htcondor_flag()
+
+# collect all histogram root files merged by run
+all_files = glob(f"{dqm_prefix}/*/*/*/*/*/merged/*.root") #change later to dqm_prefix 
+
+weeks = get_weeks()
+
+# group files by week and era
+file_groups = {}
+for file in all_files:
+    parts = file.split('/')
+    filename = parts[-1]
+    run = int(parts[-3])
+    era = parts[-6]
+    label = parts[-7]
+
+    # group by week - not all run in list? 
+    if run in weeks.keys():
+        week = weeks[run]
+        target = f"{dqm_prefix}/Weekly/{week}/{label}/merged/{filename}"
+        if target not in file_groups:
+            file_groups[target] = []
+        file_groups[target].append(file)
+
+    # group by era
+    target = f"{dqm_prefix}/{label}/{era}/merged/{filename}"
+    if target not in file_groups:
+        file_groups[target] = []
+    file_groups[target].append(file)
+
+
+# Hadd grouped files
+for target, files in file_groups.items():
+    hadd(target, files, htcondor)

automation/merge_per_run.py

@@ -0,0 +1,33 @@
+#!/bin/python3
+
+from glob import glob
+from utils import hadd, clean, htcondor_flag, dqm_prefix
+
+# parse arguments
+htcondor = htcondor_flag()
+
+# collect all base histogram root files
+all_files = glob(f"{dqm_prefix}/*/*/*/*/*/*/*.root")
+all_files = [f for f in all_files if 'merged' not in f]
+cleaned_files = clean(all_files)
+
+# group files by runnum, by era, and by year
+file_groups = {}
+for file in cleaned_files:
+    parts = file.split('/') 
+    run = int(parts[-3])
+    era = parts[-6]
+    dataset = parts[-7]
+
+    filename = parts[-1]
+    filehash = parts[-2]
+
+    # group by runnum 
+    target = file.replace(filehash, "merged")
+    if target not in file_groups:
+        file_groups[target] = []
+    file_groups[target].append(file)
+
+# Hadd grouped files
+for target, files in file_groups.items():
+    hadd(target, files, htcondor)

automation/merge_total.py

@@ -0,0 +1,29 @@
+#!/bin/python3
+
+from glob import glob
+from utils import hadd, clean, get_weeks, htcondor_flag, dqm_prefix
+
+htcondor = htcondor_flag()
+
+# collect all histogram root files merged by run
+all_files = glob(f'{dqm_prefix}/*/*/merged/*.root') #change later to dqm_prefix 
+cleaned_files = clean(all_files)
+
+# group by type (i.e. Muon, EGamma, JetMet, etc.)
+file_groups = {}
+for file in cleaned_files:
+    parts = file.split('/')
+    filename = parts[-1]
+    era = parts[-3]
+    label = parts[-4]
+
+    target = f"{dqm_prefix}/{label}/merged/{filename}"
+    if target not in file_groups:
+        file_groups[target] = []
+    file_groups[target].append(file)
+
+
+# Hadd grouped files
+for target, files in file_groups.items():
+    hadd(target, files, htcondor)
+    #print(target, files)

automation/submit.txt

@@ -0,0 +1,11 @@
+executable              = wrapper.py
+arguments               = $(cmd)
+output                  = logs/$(ClusterId).$(ProcId).out
+error                   = logs/$(ClusterId).$(ProcId).err
+log                     = logs/$(ClusterId).$(ProcId).log
++JobFlavour             = espresso
+
+should_transfer_files = yes
+when_to_transfer_output = on_exit
+
+queue cmd from queue.txt