[tmva][sofie] Add support for optimal memory allocation of dynamic tensors #20434

lmoneta · 2025-11-14T15:35:28Z

This Pull request provides support for optimal memory allocation of dynamic tensor.
A function to compute the total size and the optimal offset for each tensor given the dynamic input parameters (e.g. batch_size, number of input features, etc..) is added in SOFIE_Common.

github-actions · 2025-11-14T20:56:45Z

Test Results

22 files 22 suites 3d 23h 10m 4s ⏱️
3 792 tests 3 777 ✅ 4 💤 11 ❌
80 337 runs 80 184 ✅ 50 💤 103 ❌

For more details on these failures, see this check.

Results for commit 8db589a.

♻️ This comment has been updated with latest results.

sanjibansg

Looks good to me overall, just some questions:

sanjibansg · 2025-11-19T07:47:14Z

tmva/sofie/inc/TMVA/ROperator_BasicNary.hxx

-         for (size_t i = 0; i < fNBroadcastedInputs.size(); i++) {
-            inputs[i] = fNBroadcastedInputs[i] + "[id]";
+
+         // implement operator without broadcasting, but using loos on all indices


Suggested change

// implement operator without broadcasting, but using loos on all indices

// implement operator without broadcasting, but using loops on all indices

sanjibansg · 2025-11-19T15:49:17Z

tmva/sofie/inc/TMVA/ROperator_Concat.hxx

                     std::copy(inputData, inputData + inputLength, outputData.begin() + offset );
                     offset += inputLength;
-                     // data do not need to be written as a weight
+                     // data do not need to be written in teh generated code


Suggested change

// data do not need to be written in teh generated code

// data do not need to be written in the generated code

sanjibansg · 2025-11-20T10:59:25Z

tmva/sofie/src/RModel.cxx

+            //fGC += "std::vector<float> fTensor_" + i.first + ";\n";
            fGC += "float * tensor_" + i.first + " = nullptr;\n";
         } else if (i.second.type == ETensorType::DOUBLE) {
-            fGC += "std::vector<double> fTensor_" + i.first + ";\n";
+            //fGC += "std::vector<double> fTensor_" + i.first + ";\n";
            fGC += "double * tensor_" + i.first + " = nullptr;\n";
         } else if (i.second.type == ETensorType::INT64) {
-            fGC += "std::vector<int64_t> fTensor_" + i.first + ";\n";
+            //fGC += "std::vector<int64_t> fTensor_" + i.first + ";\n";
            fGC += "int64_t * tensor_" + i.first + " = nullptr;\n";
+         } else if (i.second.type == ETensorType::BOOL) {
+            //fGC += "std::vector<uint8_t> fTensor_" + i.first + ";\n";
+            fGC += "uint8_t * tensor_" + i.first + " = nullptr;\n";


maybe we remove the commented out code?

sanjibansg · 2025-11-20T11:01:36Z

tmva/sofie/src/RModel.cxx

      bool modelHasWeights = false;
      for (auto &i : fInitializedTensors) {
-         if (i.second.type() == ETensorType::FLOAT) {
+         if (i.second.IsWeightTensor()) {


Will it be an issue if we do not make type checks here?

sanjibansg · 2025-11-20T11:02:32Z

tmva/sofie/src/RModel.cxx

+   // for (auto &i : fDynamicTensorInfos) {
+   //    auto length = ConvertDynamicShapeToLength(i.second.shape);
+   //    out << SP << "if (" << length << " > 0) {\n";
+   //    out << SP << SP << "fTensor_" << i.first << ".resize(" << length << ");\n";
+   //    out << SP << SP << "tensor_" << i.first << " = fTensor_" << i.first << ".data();\n";
+   //    out << SP << "}\n";
+   // }


maybe we can remove this commented code?

sanjibansg · 2025-11-20T11:06:33Z

tmva/sofie/src/SOFIE_common.cxx

+
+struct MemoryEvent {
+  int t;      // time (i.e. operator index)
+  int type;   // 0 = END first, 1 = START


what does the tensor index signify here?

sanjibansg · 2025-11-20T11:09:08Z

tmva/sofie/inc/TMVA/ROperator_Gemm.hxx

+                  // /d to add a new intermediate tensor for broadcasted bias tensor
+                  // fNC2 = fNC + "bcast";
+                  // if (!fIsDynamic) {
+                  //    model.AddIntermed/ In case of session add broadcasting code in Session constructor and in GenerateInitCode
+                  // // we neeiateTensor(fNC2, model.GetTensorType(fNC), shapeY);
+                  // }
+                  // else
+                  //    model.AddDynamicTensor(fNC2,model.GetTensorType(fNC), fShapeY);
+                  // // do not add to lists of input/output tensors since broadcasted tensors are special
+                  // // and we manage their memory separatly
+                  // //fInputTensorNames.emplace_back(fNC2);
+                  // //fOutputTensorNames.emplace_back(fNC2);


if this else block is not needed anymore, maybe we can remove the if-else branching completely?

Add missing support for Dynamic tensors for some operators. With this commit a full support for dynamic tensor is available for ParticleNet model. Fix also a bug in Concat operator when the concat axis is not the first one

Since we use now for boolean tensors a std::vector<uint8_t> it is not needed to have a special treatment when the output ttype of the operator is a boolean (e.g. in Comparison)

…ensors Add a new function in SOFIE_common OrganizeMemory which computes the total memory and the offset for each tensor given tensor begin /end life and size. Fix also some small issue with dynamic tensor. One is for the bias of Gemm and Conv. The broadcasting of bias is done for dynamic tensor in the Session constructor only if needed. For the broadcasted tensor there is no need to create a new tensor, but the existing one is resized to the broadcasted needed size using vector::resize

… broadcasting The assert that was generated when broadcasting dynamic tensors was not correct

Apply also other fixes for the SOFIE tests and add a new test for StackMul

The order execution ws not set for tensor inputs to operators added using the GNN Sofie classes. This is now fixed and the correct memory mangement can be performed.

SOme fixes are needed for the test, since the session is not used for this tests. Need also to force using Session in case of Dynamic tensors Fix also a warning in Gemm operator and RModel

…on ctor Do an alphabetical order of Session shape parameters for dynamic tensors, otherwise they may get a random order. Observed different order on different platforms Add some small improvements in the generated code (add nunber and shape informations) when generating Gemm code

…on ctor Avoid creating a broadcasted bias tensor which uses lots of memory. Do broadcasting of the bias on the fly before computing Gemm by using the output tensor. This saves a large amount of memory on models using large Gemm calss like the atlas GNN model used for tracking

Add alias tensors to cope with identity operators. In this case just a pointer assignment is performed by the operator. Exclude this tensors in the allocation and take care of them in the dynamic memory pool Optimise Slice operator when slice is an identity

lmoneta requested a review from sanjibansg November 14, 2025 15:35

lmoneta self-assigned this Nov 14, 2025

sanjibansg reviewed Nov 20, 2025

View reviewed changes

lmoneta force-pushed the tmva_sofie_dynamic_tensor branch from a907978 to 72f683c Compare December 17, 2025 21:34

guitargeek added the in:TMVA label Dec 18, 2025

lmoneta force-pushed the tmva_sofie_dynamic_tensor branch 2 times, most recently from 2cc1f83 to e2275ff Compare January 6, 2026 13:46

lmoneta added 7 commits January 6, 2026 17:55

[tmva][sofie] Apply fixes for supporting Dynamic tensors

9edf0fa

Add missing support for Dynamic tensors for some operators. With this commit a full support for dynamic tensor is available for ParticleNet model. Fix also a bug in Concat operator when the concat axis is not the first one

[tmva][sofie] Remove special case handling bool outputs

aa4d008

Since we use now for boolean tensors a std::vector<uint8_t> it is not needed to have a special treatment when the output ttype of the operator is a boolean (e.g. in Comparison)

[tmva][sofie] Fix an issue in genereting code for dynamic tensor when…

556f0d7

… broadcasting The assert that was generated when broadcasting dynamic tensors was not correct

[tmva][sofie] Fix stacked MatMul and speedup LayerNorm

3e6691e

Apply also other fixes for the SOFIE tests and add a new test for StackMul

[tmva][sofie] Fix issue with order execution of tensors in GNN models

2fba9e5

The order execution ws not set for tensor inputs to operators added using the GNN Sofie classes. This is now fixed and the correct memory mangement can be performed.

[tmva][sofie] Apply fixes for the TestCustiomModelsFrom ROOT

55fb3e7

SOme fixes are needed for the test, since the session is not used for this tests. Need also to force using Session in case of Dynamic tensors Fix also a warning in Gemm operator and RModel

lmoneta force-pushed the tmva_sofie_dynamic_tensor branch from e2275ff to 55fb3e7 Compare January 7, 2026 10:29

lmoneta added 3 commits January 7, 2026 16:19

lmoneta force-pushed the tmva_sofie_dynamic_tensor branch from 1d0bd08 to 8db589a Compare January 8, 2026 18:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[tmva][sofie] Add support for optimal memory allocation of dynamic tensors #20434

[tmva][sofie] Add support for optimal memory allocation of dynamic tensors #20434

Uh oh!

lmoneta commented Nov 14, 2025

Uh oh!

github-actions bot commented Nov 14, 2025 •

edited

Loading

Uh oh!

sanjibansg left a comment

Uh oh!

sanjibansg Nov 19, 2025

Uh oh!

sanjibansg Nov 19, 2025

Uh oh!

sanjibansg Nov 20, 2025

Uh oh!

sanjibansg Nov 20, 2025

Uh oh!

sanjibansg Nov 20, 2025

Uh oh!

sanjibansg Nov 20, 2025

Uh oh!

sanjibansg Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	// implement operator without broadcasting, but using loos on all indices
	// implement operator without broadcasting, but using loops on all indices

	// data do not need to be written in teh generated code
	// data do not need to be written in the generated code

[tmva][sofie] Add support for optimal memory allocation of dynamic tensors #20434

Are you sure you want to change the base?

[tmva][sofie] Add support for optimal memory allocation of dynamic tensors #20434

Uh oh!

Conversation

lmoneta commented Nov 14, 2025

Uh oh!

github-actions bot commented Nov 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

Uh oh!

sanjibansg left a comment

Choose a reason for hiding this comment

Uh oh!

sanjibansg Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

sanjibansg Nov 19, 2025

Choose a reason for hiding this comment

Uh oh!

sanjibansg Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

sanjibansg Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

sanjibansg Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

sanjibansg Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

sanjibansg Nov 20, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

github-actions bot commented Nov 14, 2025 •

edited

Loading