ConfigDB parsing/dumping using CMSSW_15_1_0_pre1 associated with a VPSet lead to incorrect python

[Dr15Jones]

          I strongly believe that ConfDb parsing gets confused by the new cms.PSetTemplate - there is no error message about Ignoring any unknown type, and somehow unlabelled PSets turn up as module parameters in ConfDb (when one adds new instances of affected plugins) and then when extracting the python code, eg:

process.RecoTauProducer = cms.EDProducer( "RecoTauProducer",
[...]
    buildNullTaus = cms.bool( False ),
    verbosity = cms.int32( 0 ),
    modifiers = cms.VPSet( 
      cms.PSet(  name = cms.string( "sipt" ),
        plugin = cms.string( "RecoTauImpactParameterSignificancePlugin" ),
         = cms.PSet( 
           = cms.PSet( 
            minTrackPt = cms.double( 0.5 ),
            maxTrackChi2 = cms.double( 100.0 ),
            maxTransverseImpactParameter = cms.double( 0.1 ),
            maxDeltaZ = cms.double( 0.4 ),
            maxDeltaZToLeadTrack = cms.double( -1.0 ),
            minTrackVertexWeight = cms.double( -1.0 ),
            minTrackPixelHits = cms.uint32( 0 ),
            minTrackHits = cms.uint32( 3 ),
            minGammaEt = cms.double( 1.0 ),
            minNeutralHadronEt = cms.double( 30.0 )
          ),
           = cms.PSet( 
            minTrackPt = cms.double( 1.0 ),
            maxTrackChi2 = cms.double( 100.0 ),
            maxTransverseImpactParameter = cms.double( 0.03 ),
            maxDeltaZ = cms.double( 0.2 ),
            maxDeltaZToLeadTrack = cms.double( -1.0 ),
            minTrackVertexWeight = cms.double( -1.0 ),
            minTrackPixelHits = cms.uint32( 0 ),
            minTrackHits = cms.uint32( 8 ),
            minGammaEt = cms.double( 1.5 )
          ),

which is not proper python. So the new feature of PSTemplate is problematic for ConfDB.

Originally posted by @Martin-Grunewald in #47630 (comment)

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @Dr15Jones.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

Further comments from the PR
#47630 (comment)

I suspect the problem is that for a VPSet vector, it is assumed that the elements are PSets and those PSets do not have their own label/name. But now the VPSet contains in addition the entry/component template = cms.PSetTemplate as a component, which screws up this assumption.

[Dr15Jones]

Further comments from the PR
#47630 (comment)

I did a parsing (CMSSW_15_1_0_pre1) of a template into ConfDb, migrated a menu to it (existing instances stay fine) and then added a new instance of the RecoTauProducer, which appeared in ConfDb and in the extracted python as the above,
with the new nested parameters.

[Dr15Jones]

@Martin-Grunewald

When you say you added a new instance of the RecoTauProducer could you post exactly how you did that? Did you add a new _cfi file? Did RecoTauProducer already exist in the DB and it was the addition that failed? I'm trying to determine the starting point as my attempt to perform the actions (I think) CondDB is doing on the configurations are not showing any issues with the PSetTemplate.

[makortel]

assign core, hlt

[cmsbuild]

New categories assigned: core,hlt

@Dr15Jones,@makortel,@Martin-Grunewald,@mmusich,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

Replying here to #47630 (comment) (@mmusich)

I suppose from here as the auto-generated cfi-s take precedence.

That isn't a cfi file and doesn't add anything to the Process if loaded into the Process (which AFAICT is what confdb is doing).

As far as I can tell, the only cfi file that uses the RecoTauProducer.py is
https://github.com/cms-sw/cmssw-cfipython/blob/CMSSW_15_1_0_pre1/RecoTauTag/RecoTau/combinatoricRecoTaus_cfi.py#L1-L5

Loading the combinatoricRecoTaus_cfi.py into python shows modifiers = cms.required.VPSet so a PSetTemplate in modifiers can not come through there. On the other hand, the builders parameter is set to a cms.VPSet() with a concrete PSet() and a template = cms.PSetTemplate()` (well, basically with the content of https://github.com/cms-sw/cmssw-cfipython/blob/CMSSW_15_1_0_pre1/RecoTauTag/RecoTau/RecoTauProducer.py)

[Martin-Grunewald]

I added the new plugin instance within the ConfDb GUI to an existing menu in ConfDb using the ConfDb template 15_1_0_pre1. The gui allows to add new module instances whose parameters get populated by what was found during parsing of that template. For the RecoTauProducer, ConfDb lists the following files as being used:

RecoTauProducer                                   RecoTauTag/RecoTau/combinatoricRecoTaus_cfi.py                                                                                                                                                          
RecoTauProducer                                   RecoTauTag/RecoTau/python/RecoTauCombinatoricProducer_cfi.py                                                  

The first one is in cfipython (so, generated), while the second one is an explicit cfi found in the standard ../python directory.

[makortel]

Curiously this part from the modifiers VPSet from the issue description

         = cms.PSet( 
           = cms.PSet( 
            minTrackPt = cms.double( 0.5 ),
            maxTrackChi2 = cms.double( 100.0 ),
            maxTransverseImpactParameter = cms.double( 0.1 ),
            maxDeltaZ = cms.double( 0.4 ),
            maxDeltaZToLeadTrack = cms.double( -1.0 ),
            minTrackVertexWeight = cms.double( -1.0 ),
            minTrackPixelHits = cms.uint32( 0 ),
            minTrackHits = cms.uint32( 3 ),
            minGammaEt = cms.double( 1.0 ),
            minNeutralHadronEt = cms.double( 30.0 )
          ),
           = cms.PSet( 
            minTrackPt = cms.double( 1.0 ),
            maxTrackChi2 = cms.double( 100.0 ),
            maxTransverseImpactParameter = cms.double( 0.03 ),
            maxDeltaZ = cms.double( 0.2 ),
            maxDeltaZToLeadTrack = cms.double( -1.0 ),
            minTrackVertexWeight = cms.double( -1.0 ),
            minTrackPixelHits = cms.uint32( 0 ),
            minTrackHits = cms.uint32( 8 ),
            minGammaEt = cms.double( 1.5 )
          ),

looks like the following part from builders.template https://github.com/cms-sw/cmssw-cfipython/blob/CMSSW_15_1_0_pre1/RecoTauTag/RecoTau/RecoTauProducer.py#L26-L51

        qualityCuts = cms.PSet(
          signalQualityCuts = cms.PSet(
            minTrackPt = cms.double(0.5),
            maxTrackChi2 = cms.double(100),
            maxTransverseImpactParameter = cms.double(0.1),
            maxDeltaZ = cms.double(0.4),
            maxDeltaZToLeadTrack = cms.double(-1),
            minTrackVertexWeight = cms.double(-1),
            minTrackPixelHits = cms.uint32(0),
            minTrackHits = cms.uint32(3),
            minGammaEt = cms.double(1),
            useTracksInsteadOfPFHadrons = cms.optional.bool,
            minNeutralHadronEt = cms.double(30)
          ),
          isolationQualityCuts = cms.PSet(
            minTrackPt = cms.double(1),
            maxTrackChi2 = cms.double(100),
            maxTransverseImpactParameter = cms.double(0.03),
            maxDeltaZ = cms.double(0.2),
            maxDeltaZToLeadTrack = cms.double(-1),
            minTrackVertexWeight = cms.double(-1),
            minTrackPixelHits = cms.uint32(0),
            minTrackHits = cms.uint32(8),
            minGammaEt = cms.double(1.5),
            useTracksInsteadOfPFHadrons = cms.optional.bool
          ),

with the cms.optional.bool being omitted, and the PSet labels being missing.