Pyroot crashes reading TClonesArray in a TTree

[AlexFinch]

Check duplicate issues.

• Checked for duplicates

Description

A pyroot python program crashes when reading a TFile containing a TTree containing a TClonesArray. It appears to crash the second time it accesses the TClonesArray. This has started happening since moving to ROOT 6.32 on AlmaLinux 9.4. It worked fine with 6.20.04 running on Centos7. The library for accessing the TTree information was created using MakeProject.

Reproducer

This is the python code that crashes

#!/usr/bin/env python3
import os
import ROOT
import argparse

parser = argparse.ArgumentParser(description='Analyse events.')
parser.add_argument("inputfile")
parser.add_argument("-o","--outputfile",default="Output.root",help="Name of output file")
args = parser.parse_args()

READEVENTANALYSISLIBRARY=ROOT.gSystem.Getenv("READEVENTANALYSISLIBRARY");
ROOT.gSystem.Load(READEVENTANALYSISLIBRARY);

globalRecon = ROOT.TChain("ReconDir/Global")
globalRecon.AddFile(args.inputfile)

OutputFile = ROOT.TFile(args.outputfile,"RECREATE","ND280 Analysis")
hTrackMomenta = ROOT.TH1F("TrackMomenta", "Charged Track Momentum", 100, 0, 1000.0)

entries =  globalRecon.GetEntries()
for entry in range(entries):
	print(" entry = "+str(entry))
	ientry = globalRecon.LoadTree(entry)
	if ientry < 0:
		break
	nb = globalRecon.GetEntry(entry)
	if nb<=0:
		continue
	
	for	pid in globalRecon.PIDs :
		if(pid.Charge != 0) :
			print(" charge is "+str(pid.Charge))
		#	hTrackMomenta.Fill(pid.FrontMomentum)
OutputFile.Write()
OutputFile.Close()

The crash traceback suggest this as the line causing the crash:

#7 0x00007fe963a68e9a in TBranchElement::ReadLeavesClones(TBuffer&) () from /home/aleph/ajf/t2k/root/lib/libTree.so
#8 0x00007fe963a60899 in TBranch::GetEntry(long long, int) () from /home/aleph/ajf/t2k/root/lib/libTree.so
#9 0x00007fe963a7326b in TBranchElement::GetEntry(long long, int) () from /home/aleph/ajf/t2k/root/lib/libTree.so
#10 0x00007fe963ad7300 in TTree::GetEntry(long long, int) () from /home/aleph/ajf/t2k/root/lib/libTree.so

ROOT version

6.32.00

Installation method

prebuilt binary root_v6.32.00.Linux-almalinux9.4-x86_64-gcc11.4.tar.gz

Operating system

AlmaLinux 9.4

Additional context

No response

[pcanal]

Can you provide the input file that lead to the crash? Alternatively can you rebuild ROOT in debug mode (-DCMAKE_BUILD_TYPE=Debug) and run with valgrind (valgrind --suppressions=$ROOTSYS/etc/valgrind-root.supp --suppressios=$ROOTSYS/etc/valgrind-root-python.supp python ... )

[AlexFinch]

I've put the file here:
hep.lancs.ac.uk/~ajf/pyrootCrash/example.root

[ferdymercury]

I've put the file here: hep.lancs.ac.uk/~ajf/pyrootCrash/example.root

To reproduce it, I thik it's also needed to upload the READEVENTANALYSISLIBRARY dll / so. Or was it just generated via: _file0->MakeProject("readeventanalysis"); ?

[AlexFinch]

As i said in the orginal post " The library for accessing the TTree information was created using MakeProject."

[ferdymercury]

Ok. The full stack trace in a debug build (with line numbers) is:

1 TBranchElement::ReadLeavesClones TBranchElement.cxx 4492 0x78be8e7a6cca 
2 TBranch::GetEntry                TBranch.cxx        1753 0x78be8e78b74b 
3 TBranchElement::GetEntry         TBranchElement.cxx 2707 0x78be8e79f51c 
4 operator()                       TTree.cxx          5661 0x78be8e82a667 
5 TTree::GetEntry                  TTree.cxx          5736 0x78be8e82ae59 
6 TChain::GetEntry                 TChain.cxx         1011 0x78be8e7c52b8 

so it crashes at:

clones->ExpandCreateFast(fNdata);

maybe because fObject is in a corrupt state:

TClonesArray* clones = (TClonesArray*) fObject;

[ferdymercury]

As crosscheck, I tried reading out from C++ and no crash there.

#include "TChain.h"
#include "TFile.h"
#include "TSystem.h"
#include "TTreeReader.h"
#include "TTreeReaderArray.h"
#include <iostream>
#include "/tmp/ReadEventAnalysis/ND__TGlobalReconModule__TGlobalPID.h"

using std::cout, std::endl;

void crash() {
    gSystem->Load("/tmp/ReadEventAnalysis/ReadEventAnalysis.so");
    TTreeReader     fReader("ReconDir/Global", new TFile("example.root"));
    TTreeReaderArray<Double_t> PIDs_Charge = {fReader, "PIDs.Charge"};
    TTreeReaderValue<Int_t> NPIDs = {fReader, "NPIDs"};

    while (fReader.Next()) {
        for (Int_t i = 0; i < *NPIDs ; ++i)
            cout << PIDs_Charge[i] << " " ;
        cout << endl;
    }
}

so in principle this is not related to #11769 or #8744

[dpiparo]

I can reproduce. This does not solve the issue with the "classic" iteration over TTrees/TChains in Python, that we need to fix anyway, however I would like to propose a different, more concise way to obtain the histogram using RDataFrame, the recommended interface for analysis:

#!/usr/bin/env python3
import ROOT
import argparse

parser = argparse.ArgumentParser(description='Analyse events.')
parser.add_argument("inputfile")
parser.add_argument("-o","--outputfile",default="Output.root",help="Name of output file")
args = parser.parse_args()

READEVENTANALYSISLIBRARY=ROOT.gSystem.Getenv("READEVENTANALYSISLIBRARY");
ROOT.gSystem.Load(READEVENTANALYSISLIBRARY);

df = ROOT.RDataFrame("ReconDir/Global", args.inputfile)
h = df.Define("frontMomenta", "PIDs.FrontMomentum[PIDs.Charge != 0]")\
      .Histo1D(("TrackMomenta", "Charged Track Momentum", 100, 0, 1000.0), 'frontMomenta')
h.Draw()

(for the syntax inside the Define, you can refer to the RVec documentation here)

[AlexFinch]

Thanks for the workaround. i will give it a try.

[AlexCai-Y]

I can reproduce. This does not solve the issue with the "classic" iteration over TTrees/TChains in Python, that we need to fix anyway, however I would like to propose a different, more concise way to obtain the histogram using RDataFrame, the recommended interface for analysis:

#!/usr/bin/env python3
import ROOT
import argparse

parser = argparse.ArgumentParser(description='Analyse events.')
parser.add_argument("inputfile")
parser.add_argument("-o","--outputfile",default="Output.root",help="Name of output file")
args = parser.parse_args()

READEVENTANALYSISLIBRARY=ROOT.gSystem.Getenv("READEVENTANALYSISLIBRARY");
ROOT.gSystem.Load(READEVENTANALYSISLIBRARY);

df = ROOT.RDataFrame("ReconDir/Global", args.inputfile)
h = df.Define("frontMomenta", "PIDs.FrontMomentum[PIDs.Charge != 0]")\
      .Histo1D(("TrackMomenta", "Charged Track Momentum", 100, 0, 1000.0), 'frontMomenta')
h.Draw()

(for the syntax inside the Define, you can refer to the RVec documentation here)

Hi dpiparo, may I ask will this bug be fixed in near future? My issue is a bit different from Alex's so the workaround does not really work, so if the "classic" iteration on TTree can run it would be great.

[vepadulano]

Hi, I have an update regarding this issue. I have created locally a TTree with one branch of type TClonesArray("MyPoint") where MyPoint is a custom TObject-derived class. If I iterate in Python over the entries of the tree like so

for event in tree:
    print(event.arr)

Then I also get a stacktrace similar to the one showed so far, e.g.

[/Users/vpadulan/Programs/rootproject/rootbuild/master-distrdf-debug/lib/libTree.so] TBranchElement::ReadLeavesClones(TBuffer&) /Users/vpadulan/Programs/rootproject/root/tree/tree/src/TBranchElement.cxx:4477
[/Users/vpadulan/Programs/rootproject/rootbuild/master-distrdf-debug/lib/libTree.so] TBranch::GetEntry(long long, int) /Users/vpadulan/Programs/rootproject/root/tree/tree/src/TBranch.cxx:1754
[/Users/vpadulan/Programs/rootproject/rootbuild/master-distrdf-debug/lib/libTree.so] TBranchElement::GetEntry(long long, int) /Users/vpadulan/Programs/rootproject/root/tree/tree/src/TBranchElement.cxx:2708
[/Users/vpadulan/Programs/rootproject/rootbuild/master-distrdf-debug/lib/libTree.so] TTree::GetEntry(long long, int)::$_1::operator()() const /Users/vpadulan/Programs/rootproject/root/tree/tree/src/TTree.cxx:5664
[/Users/vpadulan/Programs/rootproject/rootbuild/master-distrdf-debug/lib/libTree.so] TTree::GetEntry(long long, int) /Users/vpadulan/Programs/rootproject/root/tree/tree/src/TTree.cxx:5739
[<unknown binary>] (no debug info)

If instead I iterate over the tree like so

clonesarray = ROOT.TClonesArray("MyPoint")
tree.SetBranchAddress("arr", clonesarray)
for i in range(tree.GetEntriesFast()):
    clonesarray.Clear()
    tree.GetEntry(i)
    for cl in clonesarray:
        print(f"Point({cl.fX},{cl.fY},{cl.fZ})")

Then I can read the TClonesArray branch fine, with the correct values, without any crashes. Can anybody confirm with the original file using the logic I just described? Meanwhile, I'm working on debugging what seems to be the likeliest place where things are going wrong, i.e. in the Pythonization of TTree.getattr, which is what is triggered when calling globalRecon.PIDs in the original snippet.

[AlexFinch]

I checked and it turns out that adding the two lines

clonesarray = ROOT.TClonesArray("ND::TGlobalReconModule::TGlobalPID")
globalRecon.SetBranchAddress("PIDs", clonesarray)

to the original example is sufficient, one doesn't even need to loop over the clonesarray explicitly, so with that addition this works:

for entry in range(globalRecon.GetEntries()):
	globalRecon.GetEntry(entry)
	for pid in globalRecon.PIDs :
		if(pid.Charge != 0) :
			print(" charge is "+str(pid.Charge))

By the way, just to warn you that I retire at the end of the month so I will not be able to help with this after that.