DQM-Integration

Repository to store input files for the CMSSW DQM/Integration package.

Streamer Files

For the moment this repository is used to store streamer files that are needed as input for the unitTest.

Currently the unitTest that make use of input streamer files are:

the onlinebeammonitor_dqm_sourceclient_cfg.py client, it reads the streamDQM (prepared such that only the hltOnlineBeamSpot is present in the file) streamer files regenerated from run 381594 (from 2024E pp run OMS link):
```
run381594/run381594_ls1000_streamDQM_pid1388480.dat
run381594/run381594_ls1000_streamDQM_pid1388480.jsn
```
the beamhlt_dqm_sourceclient-live_cfg.py client, it reads the streamDQMOnlineBeamspot streamer files regenerated from run 381594 (from 2024E pp run OMS link):
```
run381594/run381594_ls1000_streamDQMOnlineBeamspot_pid1752643.dat
run381594/run381594_ls1000_streamDQMOnlineBeamspot_pid1752643.jsn
```
the ecalgpu_dqm_sourceclient-live_cfg.py, hcalgpu_dqm_sourceclient-live_cfg.py, pixelgpu_dqm_sourceclient-live_cfg.py and pfgpu_dqm_sourceclient-live_cfg.py read the streamDQMGPUvsCPU streamer files regenerated from run 397813 (from Run2025G pp run OMS link):
```
run397813/run397813_ls0131_streamDQMGPUvsCPU.dat
run397813/run397813_ls0131_streamDQMGPUvsCPU.jsn
```
the sistrip_approx_dqm_sourceclient-live_cfg.py reads the streamDQM streamer files regenerated from run 362321 (from 2022 HI run, OMS link, though they have been re-HLT'ed, see for more details at CMSHLT-2884):
```
run362321/run362321_ls0231_streamHIDQM_pid276864.dat
run362321/run362321_ls0231_streamHIDQM_pid276864.jsn
```
the scouting_dqm_sourceclient-live_cfg.py reads the streamDQMOnlineScouting streamer files generated from run 392642 (from Run2025C pp run, OMS link):
```
run392642/run392642_ls0174_streamDQMOnlineScouting_pid1152594.dat
run392642/run392642_ls0174_streamDQMOnlineScouting_pid1152594.jsn
```

Recipe to regenerate Streamer files (when streamer layout gets broken)

In repsonse to issue cms-sw/cmssw#45224, streamer files have been regenerated in a release that contains cms-sw/cmssw#44978 (in this case CMSSW_14_0_9_MULTIARCHS. This was done using the following script for the pp data:

#!/bin/bash -ex

RUNNUMBER=381594
LUMISECTION=1000

# cmsrel CMSSW_14_0_9_MULTIARCHS
# cd CMSSW_14_0_9_MULTIARCHS/src
# cmsenv
# scram b

INPUTFILE=root://eoscms.cern.ch//store/express/Run2024E/ExpressPhysics/FEVT/Express-v1/000/381/594/00000/1e2c895f-a250-45be-a7ff-ee95e636a6e9.root
rm -rf run${RUNNUMBER}*

# run on 300 events of LS 1000, with 300 events per input file
convertToRaw -f 300 -l 300 -r ${RUNNUMBER}:${LUMISECTION} -o . -- "${INPUTFILE}"

tmpfile=$(mktemp)
hltConfigFromDB --runNumber "${RUNNUMBER}" > "${tmpfile}"
cat <<@EOF >> "${tmpfile}"
process.load("run${RUNNUMBER}_cff")
process.hltOnlineBeamSpotESProducer.timeThreshold = int(1e6)

# to run without any HLT prescales
del process.PrescaleService
del process.MessageLogger
process.load('FWCore.MessageLogger.MessageLogger_cfi')

process.options.numberOfThreads = 32
process.options.numberOfStreams = 32

process.options.wantSummary = True
# # to run using the same HLT prescales as used online in LS 1000
# process.PrescaleService.forceDefault = True
@EOF
edmConfigDump "${tmpfile}" > hlt.py

cmsRun hlt.py &> hlt.log

While the following script for the HIon data:

#!/bin/bash -ex

# cmsrel CMSSW_14_0_9_MULTIARCHS
# cd CMSSW_14_0_9_MULTIARCHS/src
# cmsenv
# scram b

# run 362321, LSs 231-232
INPUTFILE=root://eoscms.cern.ch//eos/cms/store/user/cmsbuild//store/hidata/HIRun2022A/HITestRaw0/RAW/v1/000/362/321/00000/f467ee64-fc64-47a6-9d8a-7ca73ebca2bd.root

HLTMENU=/dev/CMSSW_14_0_0/HIon/V141

rm -rf run362321*

# run on 100 events of LS 231, with 100 events per input file
convertToRaw -f 100 -l 100 -r 362321:231 -s rawDataRepacker -o . -- "${INPUTFILE}"

tmpfile=$(mktemp)
hltConfigFromDB --configName "${HLTMENU}" > "${tmpfile}"
sed -i 's|process = cms.Process( "HLT" )|from Configuration.Eras.Era_Run3_cff import Run3\nprocess = cms.Process( "HLT", Run3 )|g' "${tmpfile}"
cat <<@EOF >> "${tmpfile}"
process.load('run362321_cff')
process.hltOnlineBeamSpotESProducer.timeThreshold = int(1e6)

# override the GlobalTag, connection string and pfnPrefix
from Configuration.AlCa.GlobalTag import GlobalTag as customiseGlobalTag
process.GlobalTag = customiseGlobalTag(
    process.GlobalTag,
    globaltag = "140X_dataRun3_HLT_v3",
    conditions = "L1Menu_CollisionsHeavyIons2023_v1_1_5_xml,L1TUtmTriggerMenuRcd,frontier://FrontierProd/CMS_CONDITIONS,,9999-12-31 23:59:59.000"
)

# run the Full L1T emulator, then repack the data into a new RAW collection, to be used by the HLT
from HLTrigger.Configuration.CustomConfigs import L1REPACK
process = L1REPACK(process, "uGT")

# to run without any HLT prescales
del process.PrescaleService

# # to run using the same HLT prescales as used online in LS 231
# process.PrescaleService.forceDefault = True
@EOF
edmConfigDump "${tmpfile}" > hlt.py

cmsRun hlt.py &> hlt.log

# remove input files to save space
rm -f run362321/run362321_ls0*_index*.*

The streamer files for the streamDQMOnlineScouting were prepared using the scouting specific menu:

#!/bin/bash -ex

# cmsrel CMSSW_15_0_4_patch1 
# cd CMSSW_15_0_4_patch1/src
# cmsenv
# scram b

RUNNUMBER=392642 # 2025 EphemeralHLTPhysics
LUMISECTION=174

# LS 174 
INPUTFILES="\
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics0/RAW/v1/000/392/642/00000/06ac627c-f97b-40ed-a279-16fdcf990c2c.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics1/RAW/v1/000/392/642/00000/8ec096b9-1293-4ee7-8e27-369889a521ff.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics2/RAW/v1/000/392/642/00000/e39b2b26-9b68-40e7-8e2d-c259637ea5d3.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics3/RAW/v1/000/392/642/00000/5f25f30f-1372-4875-ac25-0f437f2590be.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics4/RAW/v1/000/392/642/00000/ddcb337e-3536-4361-b69a-02ba755b9eb0.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics5/RAW/v1/000/392/642/00000/e7512b2b-d00a-4d71-924d-9563c2b36c18.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics6/RAW/v1/000/392/642/00000/7dd68eaf-bb9b-456e-9318-961c3903c84a.root 
    root://eoscms.cern.ch//store/data/Run2025C/EphemeralHLTPhysics7/RAW/v1/000/392/642/00000/a7b5720c-aa47-4783-b838-dce20b7130ee.root"
rm -rf run${RUNNUMBER}*

# run on 5000 events of given LS, with 1000 event limits per input file
convertToRaw -l 5000 -f 1000 -r ${RUNNUMBER}:${LUMISECTION} -o . -- ${INPUTFILES}

tmpfile=$(mktemp)

hltConfigFromDB --configName /users/jprendi/ScoutingOnlineDQM/Test0/HLT/V4 > dump.py


cat <<@EOF >> dump.py
process.load("run${RUNNUMBER}_cff")
del process.PrescaleService
del process.MessageLogger
process.load('FWCore.MessageLogger.MessageLogger_cfi')

process.GlobalTag.globaltag = cms.string('150X_dataRun3_HLT_v1')


process.options.numberOfThreads = 32
process.options.numberOfStreams = 32

process.options.wantSummary = True
@EOF

edmConfigDump dump.py > hlt.py


bash -c 'echo $$ > cmsrun.pid; exec cmsRun hlt.py &> hlt.log'
job_pid=$(cat cmsrun.pid)
echo "cmsRun is running with PID: $job_pid"

# remove input files to save space
rm -f run392642/run392642_ls0*_index*.*

# prepare the files by concatenating the .ini and .dat files
mkdir -p prepared
cat run392642/run392642_ls0000_streamDQMOnlineScouting_pid${job_pid}.ini run392642/run392642_ls0174_streamDQMOnlineScouting_pid${job_pid}.dat > prepared/run392642_ls0174_streamDQMOnlineScouting_pid${job_pid}.dat

The streamer files for the streamDQMGPUVsCPU were prepared using the following script:

#!/bin/bash -ex
RUNNUMBER=397813
LUMISECTION=131

# cmsrel CMSSW_15_1_X_2025-10-22-2300
# cd CMSSW_15_1_X_2025-10-22-2300/src/
# cmsenv
# git cms-addpkg DataFormats/TrackSoA DataFormats/TrackingRecHitSoA DQM/Integration
# git remote add AdrianoDee [email protected]:AdrianoDee/cmssw.git; git fetch AdrianoDee
# git cherry-pick 1eb24e6536182945d6f7ac55fedc023a9f0d4410
# git cherry-pick 9a3ef9ae14f452aa15d3f868b97c63f6b996b1dc
# cmsenv
# scram b

INPUTFILE=root://eoscms.cern.ch//eos/cms/store/express/Run2025F/ExpressPhysics/FEVT/Express-v2/000/397/813/00000/f95a79f9-18ef-48f1-951a-2ae78c1a107f.root
rm -rf run${RUNNUMBER}*

# run on 100 events of LS 131, with 100 events per input file
convertToRaw -f 100 -l 100 -r ${RUNNUMBER}:${LUMISECTION} -o . -- "${INPUTFILE}"

tmpfile=$(mktemp)
hltConfigFromDB --configName /users/musich/tests/dev/CMSSW_15_1_0/CMSHLT-3147/GRun > "${tmpfile}"
cat <<@EOF >> "${tmpfile}"
process.load("run${RUNNUMBER}_cff")

# to run without any HLT prescales
del process.PrescaleService
del process.MessageLogger
process.load('FWCore.MessageLogger.MessageLogger_cfi')

process.options.numberOfThreads = 32
process.options.numberOfStreams = 32

process.options.wantSummary = True
process.GlobalTag.globaltag = cms.string( "150X_dataRun3_HLT_v1" )
# # to run using the same HLT prescales as used online in LS 1000
# process.PrescaleService.forceDefault = True

from HLTrigger.Configuration.common import *
def customizeHLTFor49149(process):
    ca_producers_pp = ['CAHitNtupletAlpakaPhase1@alpaka','alpaka_serial_sync::CAHitNtupletAlpakaPhase1']
    for ca_producer in ca_producers_pp:
        for prod in producers_by_type(process, ca_producer):
            if hasattr(prod, 'geometry'):
                g = getattr(prod, 'geometry')
                g.startingPairs = cms.vuint32( [i for i in range(8)] + [i for i in range(13,19)])
                setattr(prod, 'geometry', g)
    return process

process = customizeHLTFor49149(process)
@EOF
edmConfigDump "${tmpfile}" > hlt.py

cmsRun hlt.py &> hlt.log

bash -c 'echo $$ > cmsrun.pid; exec cmsRun hlt.py &> hlt.log'
job_pid=$(cat cmsrun.pid)
echo "cmsRun is running with PID: $job_pid"

# remove input files to save space
rm -f run397813/run397813_ls0*_index*.*

# prepare the files by concatenating the .ini and .dat files
mkdir -p prepared
cat run397813/run397813_ls0000_streamDQMGPUvsCPU_pid${job_pid}.ini run397813/run397813_ls0131_streamDQMGPUvsCPU_pid${job_pid}.dat > prepared/run397813_ls0131_streamDQMGPUvsCPU_pid${job_pid}.dat
cp run397813/run397813_ls0131_streamDQMGPUvsCPU_pid${job_pid}.jsn prepared/run397813_ls0131_streamDQMGPUvsCPU_pid${job_pid}.jsn
# manually edit to remove the extra 0 in the json file.

Possible extenstions

There are two more clients for which the unitTets could be activated in the future, namely:

 <!-- streamDQMCalibration is required -->
 <!-- <test name="TestDQMOnlineClient-ecalcalib_dqm_sourceclient" command="runtest.sh ecalcalib_dqm_sourceclient-live_cfg.py" /> -->
 <!-- streamDQMCalibration is required -->
 <!-- <test name="TestDQMOnlineClient-hcalcalib_dqm_sourceclient" command="runtest.sh hcalcalib_dqm_sourceclient-live_cfg.py" /> -->

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Repository files navigation

DQM-Integration

Streamer Files

Recipe to regenerate Streamer files (when streamer layout gets broken)

Possible extenstions

About

Uh oh!

Releases

Packages

Contributors 6

Uh oh!

Name		Name	Last commit message	Last commit date
Latest commit History 36 Commits
run362321		run362321
run381594		run381594
run392642		run392642
run397813		run397813
README.md		README.md

cms-data/DQM-Integration

Folders and files

Latest commit

History

Repository files navigation

DQM-Integration

Streamer Files

Recipe to regenerate Streamer files (when streamer layout gets broken)

Possible extenstions

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Contributors 6

Uh oh!

Packages