Save TensorRT engine in weights file with onnx2leela and automate during runtime by saithepaithewhyyy · Pull Request #2428 · LeelaChessZero/lc0

saithepaithewhyyy · 2026-07-13T17:04:19Z

With respect to the following issue: #2185

This PR adds the following:-

Added a protobuf field for ONNX EPContext model marking (is_ep_context). Basically, so that a net can be marked as either a regular ONNX model or an EPContext model (embedded engine)
Onnx2leela can now determine if an ONNX model is an embedded engine model or not. If it is an embedded engine model, it packages the relevant _ctx.onnx file into an output weights file
The above packaging is automated verbatim in network_onnx during runtime. Given a .pb.gz file, it builds the embedded engine (_ctx.onnx) and packages it in a new -embedded.pb.gz file and marks the proto to true.

…ing runtime

Copilot

Pull request overview

This PR implements support for packaging TensorRT EPContext (embedded engine) ONNX models into lc0 weights files, enabling more stable TRT performance by reusing the embedded engine model instead of rebuilding engines at runtime.

Changes:

Adds is_ep_context to the OnnxModel protobuf to mark embedded EPContext models.
Extends onnx2leela to detect EPContext models and emit weights with the EPContext flag and model payload.
Updates the ONNX TRT runtime to dump/build an EPContext model and automatically write an -embedded.pb.gz weights variant.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
src/tools/onnx2leela.cc	Detects EPContext models and sets `is_ep_context` while packaging ONNX into weights.
src/neural/wrapper.cc	Makes backend network options inherit from parent options (affects how weights path is discovered).
src/neural/backends/onnx/network_onnx.cc	Enables TRT EPContext dumping/embedding and writes `-embedded.pb.gz` during runtime; adjusts TRT options.
proto/net.proto	Adds the `is_ep_context` field to the ONNX model proto.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

mooskagh · 2026-07-14T13:19:03Z

I'll take a look later, but just want to say that the suggestions for the Copilot are often too "defensive", and often extra checks that it suggests are not worth it (use your judgement).

saithepaithewhyyy · 2026-07-14T14:14:37Z

I'll take a look later, but just want to say that the suggestions for the Copilot are often too "defensive", and often extra checks that it suggests are not worth it (use your judgement).

Sure, thanks a lot! Yup seems like copilot suggestions are a bit defensive. Either ways, I have replied to all

Menkib64

Do we have to use file to store the context temporary everytime?
I haven't figured out how the loading code passes these to TensorRT but it would be useful if it could be loaded as string like we do for other models. If string cannot be passed, then I think it is better to use a temporary file/directory to pass the model and make sure it is deleted. The loading should also work if there is multiple threads running at the same time loading different engines.

We have a need to store many engines together. We use engines specifically optimized for a small range of batch sizes. For example 5090 can use optimized backend options with -o batch=7,steps=12. This requires loading 12 slightly different engines created from the same network. This would be an important feature to include when embedding the engine to a network file. There would be reason to add a couple of more engines to optimize a few key batch sizes even more. It could make sense to add special engines which handle only batch size 1 and exact minibatch size. These should offer a little faster evaluation for these special case. There might be need to add more small batch size kernels because TRT is bad at optimizing kernels for multiple batch sizes.

Menkib64 · 2026-07-18T13:43:35Z

+    }
+  }
+
+  if (provider_ == OnnxProvider::TRT && !is_ep_context_) {    


I don't think it is good idea to load a different file to what user specified. We should let user decide which network is used for a specific run.

This block of code only saves the embedded engine in a -embedded.pb.gz weights file, where the network would come from the --weights input if provided by the user or from the DiscoverWeightsFile if no input is given. The net_path is never passed on to the Ort::Session

The engine is loaded with only the net file the user provides.
Please do correct me if I am wrong somewhere

Right. I missed the point that it was about saving instead of load. There is options which are passed to the engine. We would need to define an option to give the desired embedded context file path. Backend should only generate the file when user requests it.

Right, I can do this, I was not sure if I can create an option for this and instead hardcode it, but I'll create that which would save it only when requested

Done i added the option dump_embedded_weights. When not used, the engine does not store the _ctx model in a new package weight file. If mentioned, it stores it at the path mentioned

also the _ctx model is now deleted at the end, so it does not get saved in trt_cache after run. I found no way to have this not save on runtime however (like model.onnx). Do you think it is possible?

Menkib64 · 2026-07-18T13:50:25Z

+  if (is_ctx) {
+    for (const auto& out : model.graph().output()) {
+      const auto& name = out.name();
+      if (onnx->has_output_policy() && name.find("policy") != std::string::npos) {


It is possible to get onnx models where output names don't follow naming convention. The names are provided as command line arguments to let user choose name maps to which output or input.

ah good point, ill use the cli arguments instead of the hardcoded strings

I removed this part altogether. Why set it again, when it was set before. I missed this previously

Menkib64 · 2026-07-18T13:53:44Z

    onnx->set_input_planes(in);
    data_type = GetDataType(model, in);
+    if(is_ctx){
+      data_type = GetDataType(model, "");


Why does empty string override the input data_type?

The reason for the empty-string fallback is that ctx models can have their input node renamed by TRT compilation

I didnt intend the overriding; it should have been an if else chunk

Shouldn't we save the renamed node names for inputs and outputs and pass the new name here?

Yes youre right, added that in

saithepaithewhyyy · 2026-07-20T13:13:41Z

Do we have to use file to store the context temporary everytime? I haven't figured out how the loading code passes these to TensorRT but it would be useful if it could be loaded as string like we do for other models. If string cannot be passed, then I think it is better to use a temporary file/directory to pass the model and make sure it is deleted. The loading should also work if there is multiple threads running at the same time loading different engines.

We have a need to store many engines together. We use engines specifically optimized for a small range of batch sizes. For example 5090 can use optimized backend options with -o batch=7,steps=12. This requires loading 12 slightly different engines created from the same network. This would be an important feature to include when embedding the engine to a network file. There would be reason to add a couple of more engines to optimize a few key batch sizes even more. It could make sense to add special engines which handle only batch size 1 and exact minibatch size. These should offer a little faster evaluation for these special case. There might be need to add more small batch size kernels because TRT is bad at optimizing kernels for multiple batch sizes.

Firstly, thanks a lot for having a look at this!

Yes this is a valid point, the context file does take up quite some space (171 MB on my system currently). I'll have to take a look if we can bypass the step of having to save this in the trt_cache folder. It should be possible, but in the case that this isnt found, it can always be manually deleted once the weights file is generated

EDIT: I have now added code that removes the _ctx.onnx when the run finished. However it still remains in trt_cache folder until the embedded weights are packaged in the file given in the dump_embedded_weights option

Menkib64 · 2026-07-26T08:00:43Z

      trt_options["trt_engine_cache_enable"] = "1";
+      trt_options["trt_dump_ep_context_model"] = "1";
+      trt_options["trt_ep_context_file_path"] = cache_dir;
+      trt_options["trt_ep_context_embed_mode"] = "1";


trt_ep_context_file_path needs similar prefix as cache file uses. This makes sure that concurrent exports work if building different engines for different batch sizes on multiple GPUs.

Added the prefix based naming

Menkib64 · 2026-07-26T08:03:03Z

-                          file.onnx_model().model().size(),
+  std::filesystem::path ctx_path = std::filesystem::path(cache_dir_) / "_ctx.onnx";
+  for (int step = 1; step <= steps_; step++) {
+    if (provider_ == OnnxProvider::TRT && is_ep_context_) {


This branch should be removed. We should be able to provide a context model just like an ordinary model. There is no need for special handling here.

It is open question if we have to disable cache uses when loading a context model.

Menkib64 · 2026-07-26T08:04:19Z

+      for (const auto& output : model.graph().output()) {
+        const auto& name = output.name();
+        if (md_out->has_output_policy() &&
+            name.find("policy") != std::string::npos) {


Should this use the name from original model?

Original model could have outputs named in any possible way. There is no requirement to include any specific string in the name.

Menkib64 · 2026-07-26T08:06:19Z

+
+  if (provider_ == OnnxProvider::TRT && !is_ep_context_ &&
+    opts.Exists<std::string>(SharedBackendParams::kDumpEmbeddedWeightsId)) {
+    std::string net_path = opts.Get<std::string>(SharedBackendParams::kDumpEmbeddedWeightsId);


Backend specific options don't have help messages currently. Backends just implement them using direct key string lookup. You can see how "gpu", "batch", "steps" are handled currently.

SharedBackendParams makes it look like option should be passed directly to Leela. But the option is parsed for --backend-options dump-embedded-weights=file_path.

Menkib64 · 2026-07-26T08:13:44Z

+      }
+
+      md_out->set_model(ctx);
+      md_out->set_is_ep_context(true);


We should store trt_profile_min_shapes, trt_profile_max_shapes and trt_profile_opt_shapes with the model. These parameters restrict when the engine is valid. For example 5090 could be configured using --backend-options batch=7,steps=12 which generates 12 different engines. Each engine is valid for only 7 batch sizes. Loading has to verify that the context file matches requested optimization state.

This also puts another requirement to context files. We want to embed many engines together to provide optimized engines for full batch range.

Yup, have added these. Also tested with various settings with backendbench

Menkib64 · 2026-07-26T08:18:42Z

-  for (int step = 1; step <= steps_; step++)
-    session_.emplace_back(onnx_env_, file.onnx_model().model().data(),
-                          file.onnx_model().model().size(),
+  std::filesystem::path ctx_path = std::filesystem::path(cache_dir_) / "_ctx.onnx";


This path would have had race condition when initializing many paths. All temporary files must use unique filename like tmpfile created file.

I had a quick look into the TensorRT provider code. It looks to me that we can just use the existing loading code as is. Context file is a drop in replacement for onnx models. No need for temporary files.

Menkib64 · 2026-07-26T08:20:25Z

      trt_options["trt_layer_norm_fp32_fallback"] = "1";
      trt_options["trt_force_sequential_engine_build"] = "1";
-      trt_options["trt_context_memory_sharing_enable"] = "1";
+      trt_options["trt_context_memory_sharing_enable"] = is_ep_context_ ? "0" : "1";


Why is context sharing disabled?
Did you see a bug when loading them?
Did you test a multi-engine configuration like batch=7,steps=4?

Yes, i recall that the build failed to load the model when this was set, so i had it disabled for ep context models. But after reading your other reviews, i'll make some changes to the model loading to see if that solves it. If it does, I shall remove this diff

This is the exact error i got while running the embedde engine weights file, with this setting set to true:

2026-08-02 12:01:10.7885568 [E:onnxruntime:lc0, tensorrt_execution_provider.h:90 onnxruntime::TensorrtLogger::log] [2026-08-02 06:31:10 ERROR] IExecutionContext::setDeviceMemory: Error Code 3: API Usage Error (Parameter check failed, condition: memory != nullptr || mEngine.getDeviceMemorySizeInternal(mOptimizationProfile, false) == 0. setDeviceMemory: Cannot set memory to nullptr.) 2026-08-02 12:01:10.7978102 [E:onnxruntime:lc0, tensorrt_execution_provider.h:90 onnxruntime::TensorrtLogger::log] [2026-08-02 06:31:10 ERROR] IExecutionContext::enqueueV3: Error Code 3: API Usage Error (Parameter check failed, condition: noDeviceMemory. The engine requires 6727168 device memory. The IExecutionContext is created with ExecutionContextAllocationStrategy::kUSER_MANAGED or ICudaEngine::createExecutionContextWithoutDeviceMemory. IExecutionContext::setDeviceMemoryV2 should be called before enqueue/execute.) 2026-08-02 12:01:10.8144406 [E:onnxruntime:, sequential_executor.cc:615 onnxruntime::ExecuteKernel] Non-zero status code returned while running TRTKernel_graph_TRTKernel_graph_org.lczero/converted_12010072553396873023_0_10041065298831557996_0 node. Name:'TensorrtExecutionProvider_TRTKernel_graph_TRTKernel_graph_org.lczero/converted_12010072553396873023_0_10041065298831557996_0_0' Status Message: TensorRT EP execution context enqueue failed. Unhandled exception in worker thread: Non-zero status code returned while running TRTKernel_graph_TRTKernel_graph_org.lczero/converted_12010072553396873023_0_10041065298831557996_0 node. Name:'TensorrtExecutionProvider_TRTKernel_graph_TRTKernel_graph_org.lczero/converted_12010072553396873023_0_10041065298831557996_0_0' Status Message: TensorRT EP execution context enqueue failed.

I have a feeling this is related to the setting itself. What do you think?

The error message looks like it might be a bug in onnxruntime side. It fails to allocated context memory if loading embedded model.

Menkib64 · 2026-07-26T08:21:43Z

  int min_batch_size_;
  int gpu_;
+  // trt cache directory and flag for trt EPcontext model
+  std::string cache_dir_;


We don't have to store the cache directory because it is only used in constructor. The code can use helper function to return the string when it is needed in different parts of code.

Menkib64 · 2026-07-26T08:23:05Z

  int gpu_;
+  // trt cache directory and flag for trt EPcontext model
+  std::string cache_dir_;
+  bool is_ep_context_;


Runtime doesn't need to know about context file. We should keep it as a stack variable in constructor.

Menkib64 · 2026-07-26T08:27:01Z

-    onnx->set_input_planes(in);
-    data_type = GetDataType(model, in);
+    if (is_ctx) {
+      bool found = false;


There shouldn't be any need for special case here. Caller should provide input and output names to converter.

Removed this case

saithepaithewhyyy · 2026-07-27T08:38:48Z

Thanks a lot @Menkib64 for having a look, ill have a relook on the code. Apologies if it was a bit bothersome to review this.

saithepaithewhyyy · 2026-08-02T07:17:18Z

Made a couple of changes. Here's a rundown:-

Added prefix naming convention for the ctx files.
Embedded package weights support multiple shapes configuration
Removed loading logic for embedded engines. Uses the same logic (except for multi step embedded engines)

Here is some testing I did with the generated embedded package weights. Since each embedded weight has the engine baked into it, a particular shape can only be run for an embedded weight file that has the same shape baked into it. I have added an exception for this case.

Default:-

size, mean nps, mean ms,   sdev,     cv, max nps,  median, min nps, first max, first mean
   1,       20,   50.63, 0.0379, 0.0007,      20,      20,      20,   51.6496,    51.1538
   2,       40,   50.63, 0.0209, 0.0004,      40,      40,      39,   51.2964,    50.9542
   3,       59,   50.63, 0.0240, 0.0005,      59,      59,      59,   51.3002,    50.9201
   4,       79,   50.63, 0.0153, 0.0003,      79,      79,      79,   51.3491,    50.9868
   5,       86,   58.11, 0.0178, 0.0003,      86,      86,      86,   58.9883,    58.5488
   6,       96,   62.59, 0.0142, 0.0002,      96,      96,      96,   63.5354,    63.0272
   7,       91,    76.5, 0.0467, 0.0006,      92,      92,      91,   77.5483,    77.0046
   8,       94,    85.4, 0.0150, 0.0002,      94,      94,      94,   86.6241,    85.9815
   9,       97,   92.41, 0.0376, 0.0004,      98,      97,      97,   93.4660,    92.9277
  10,      103,   96.71, 0.0156, 0.0002,     103,     103,     103,   97.6259,    97.1589
  11,      107,     103, 0.0179, 0.0002,     107,     107,     107,  103.9298,   103.4571
  12,      110,   109.5, 0.2333, 0.0021,     110,     110,     109,  111.0847,   110.6124
  13,      110,   117.9, 0.0802, 0.0007,     110,     110,     110,  118.8826,   118.3230
  14,      116,     121, 0.0180, 0.0001,     116,     116,     116,  121.9979,   121.5165
  15,      109,   137.9, 0.0214, 0.0002,     109,     109,     109,  138.7651,   138.4088
  16,      112,   142.3, 0.0288, 0.0002,     113,     112,     112,  143.5514,   142.9252
  17,      113,   149.8, 0.1647, 0.0011,     114,     113,     113,  150.9601,   150.2807
  18,      116,   155.5, 0.0158, 0.0001,     116,     116,     116,  156.4069,   155.9458
  19,      119,   159.8, 0.0192, 0.0001,     119,     119,     119,  160.7277,   160.2635
  20,      122,   164.2, 0.0216, 0.0001,     122,     122,     122,  165.3464,   164.7481
  21,      114,   183.9, 0.1302, 0.0007,     114,     114,     114,  185.0707,   184.6698
  22,      116,   190.3, 0.0913, 0.0005,     116,     116,     115,  191.1741,   190.7254
  23,      118,   195.7, 0.2512, 0.0013,     118,     118,     117,  197.0801,   196.2794
  24,      119,   201.7, 0.1748, 0.0009,     119,     119,     119,  202.5590,   202.1118
  25,      120,   207.9, 0.0312, 0.0002,     120,     120,     120,  208.8152,   208.3384
  26,      123,   212.2, 0.0365, 0.0002,     123,     123,     122,  213.3785,   212.7528
  27,      124,     217, 0.1300, 0.0006,     125,     124,     124,  218.0712,   217.4922
  28,      126,   222.5, 0.0943, 0.0004,     126,     126,     126,  223.5003,   222.8751

Batch=8, Steps=2

size, mean nps, mean ms,   sdev,     cv, max nps,  median, min nps, first max, first mean
   1,       60,   16.59, 0.0519, 0.0031,      61,      60,      60,   26.5929,    21.5859
   2,       83,   24.04, 0.0635, 0.0026,      84,      83,      82,   25.0528,    24.5386
   3,      100,   30.06, 0.0296, 0.0010,     100,     100,     100,   31.0184,    30.5089
   4,      105,   38.22, 0.0340, 0.0009,     105,     105,     104,   39.4662,    38.8522
   5,      113,   44.32, 0.0826, 0.0019,     113,     113,     112,   45.3204,    44.8599
   6,      116,    51.7, 0.0728, 0.0014,     116,     116,     115,   52.5185,    52.1559
   7,      121,   57.72, 0.0451, 0.0008,     121,     121,     121,   58.6213,    58.1818
   8,      121,   66.27, 0.0772, 0.0012,     121,     121,     120,   67.2074,    66.7385
   9,      116,   77.39, 0.0362, 0.0005,     116,     116,     116,   78.3962,    77.8936
  10,      122,   82.01, 0.0319, 0.0004,     122,     122,     122,   83.0998,    82.5115
  11,      122,   90.28, 0.0365, 0.0004,     122,     122,     122,   91.4701,    90.8238
  12,      126,   94.99, 0.0623, 0.0007,     127,     126,     126,   95.9134,    95.4385
  13,      123,   105.6, 0.2348, 0.0022,     124,     123,     123,  106.2901,   105.8451
  14,      127,   110.1, 0.0731, 0.0007,     127,     127,     127,  111.0326,   110.5533
  15,      125,   119.6, 0.0721, 0.0006,     126,     125,     125,  120.6484,   120.1364
  16,      126,   126.5, 0.0848, 0.0007,     127,     127,     126,  127.3285,   126.9024
  17,      119,   143.1, 0.0889, 0.0006,     119,     119,     119,  144.1346,   143.6372
  18,      120,   150.5, 0.0797, 0.0005,     120,     120,     119,  152.4475,   151.4872
  19,      121,   156.6, 0.0970, 0.0006,     121,     121,     121,  157.7536,   157.1516
  20,      121,   164.7, 0.0634, 0.0004,     122,     121,     121,  165.8092,   165.2976
  21,      123,   170.8, 0.0856, 0.0005,     123,     123,     123,  171.9790,   171.3635
  22,      123,   178.2, 0.0574, 0.0003,     124,     123,     123,  179.2062,   178.7313
  23,      125,   184.3, 0.0507, 0.0003,     125,     125,     125,  185.5579,   184.9609
  24,      124,   193.1, 0.0661, 0.0003,     124,     124,     124,  194.1754,   193.7434
  25,      122,   205.3, 0.1315, 0.0006,     122,     122,     122,  205.4495,   205.3043
  26,      124,   209.9, 0.2285, 0.0011,     124,     124,     123,  210.1727,   209.9657
  27,      124,   218.3, 0.0892, 0.0004,     124,     124,     123,  218.7345,   218.5696
  28,      126,     223, 0.2170, 0.0010,     126,     126,     125,  223.5206,   223.2151

Batch=14, Steps=1

size, mean nps, mean ms,   sdev,     cv, max nps,  median, min nps, first max, first mean
  1,       42,    23.6, 0.0136, 0.0006,      42,      42,      42,   24.7826,    24.1808
   2,       70,   28.44, 0.0548, 0.0019,      71,      70,      70,   29.3406,    28.8656
   3,       92,   32.63, 0.0331, 0.0010,      92,      92,      91,   33.4935,    33.1242
   4,       90,   44.62, 0.0718, 0.0016,      90,      90,      89,   45.5048,    45.0513
   5,       98,   50.99, 0.0889, 0.0017,      98,      98,      97,   51.8807,    51.4316
   6,      107,   56.05, 0.0125, 0.0002,     107,     107,     107,   57.3853,    56.7048
   7,      117,   59.61, 0.0254, 0.0004,     118,     117,     117,   60.4652,    60.0540
   8,      110,   72.52, 0.0196, 0.0003,     110,     110,     110,   73.5161,    73.0239
   9,      112,   80.13, 0.0198, 0.0002,     112,     112,     112,   81.2692,    80.6559
  10,      120,   83.18, 0.0212, 0.0003,     120,     120,     120,   84.3465,    83.7862
  11,      114,   96.32, 0.0407, 0.0004,     114,     114,     114,   97.4976,    96.8986
  12,      121,    99.4, 0.0620, 0.0006,     121,     121,     120,  100.3288,    99.8679
  13,      121,   107.6, 0.0376, 0.0003,     121,     121,     121,  108.4957,   108.0673
  14,      128,   109.8, 0.0435, 0.0004,     128,     128,     127,  110.6513,   110.2467
  15,      111,   134.6, 0.2697, 0.0020,     112,     111,     110,  135.0925,   134.7902
  16,      115,   139.4, 0.0977, 0.0007,     115,     115,     114,  139.7017,   139.4867
  17,      118,   143.6, 0.1053, 0.0007,     119,     118,     118,  144.1462,   143.9622
  18,      116,   155.6, 0.1669, 0.0011,     116,     116,     115,  156.0126,   155.7785
  19,      117,   162.1, 0.2642, 0.0016,     117,     117,     116,  165.2834,   164.0508
  20,      120,   166.9, 0.0737, 0.0004,     120,     120,     120,  167.2829,   167.1104
  21,      123,   170.5, 0.0786, 0.0005,     123,     123,     123,  170.9780,   170.7624
  22,      120,   183.4, 0.0793, 0.0004,     120,     120,     120,  183.6726,   183.5862
  23,      120,     191, 0.1272, 0.0007,     120,     120,     120,  191.5909,   191.3423
  24,      124,     194, 0.0574, 0.0003,     124,     124,     124,  194.3530,   194.2014
  25,      121,   207.1, 0.2031, 0.0010,     121,     121,     120,  208.0990,   207.6935
  26,      124,   209.7, 0.2301, 0.0011,     124,     124,     123,  210.0023,   209.8257
  27,      124,   217.7, 0.0859, 0.0004,     124,     124,     124,  218.0631,   217.8639
  28,      128,     219, 0.0653, 0.0003,     128,     128,     128,  219.8228,   219.3561

Menkib64 · 2026-08-07T15:49:30Z

      trt_options["trt_layer_norm_fp32_fallback"] = "1";
      trt_options["trt_force_sequential_engine_build"] = "1";
-      trt_options["trt_context_memory_sharing_enable"] = "1";
+      trt_options["trt_context_memory_sharing_enable"] = is_ep_context_ ? "0" : "1";


The error message looks like it might be a bug in onnxruntime side. It fails to allocated context memory if loading embedded model.

Menkib64 · 2026-08-07T15:56:13Z

-      trt_options["trt_engine_cache_prefix"] =
-          "Lc0_ONNX_TRT_ORT_" + Ort::GetVersionString() + "_batch_" +
+      std::string cache_prefix =
+          "Lc0_ONNX_TRT_ORT_" + Ort::GetVersionString() + "_gpu_" +


Different gpu are expected to use the same cache file. Current initialization is done in the main thread only. I have a branch where backends can be initialized concurrently. This branch has extra initialization locking for onnx-trt so each GPU loads different batch size range. All GPUs should share the same cache file for the same batch size range.

I'm thinking that the same locking rule would apply to saving context files. It would have minimal help to use many GPUs to build context file concurrently because onnxruntime has some unnecessary locking in tensorrt execution provider.

cool, ill use the existing prefix then

Menkib64 · 2026-08-07T16:08:06Z

+  std::string cache_dir =
+      (std::filesystem::path(CommandLine::BinaryDirectory()) / "trt_cache")
+          .string();
+  is_ep_context_ = md.is_ep_context();


is_ep_context_ could be a stack variable.

Menkib64 · 2026-08-07T16:11:26Z

+    session_.emplace_back(
+        onnx_env_, model.data(), model.size(),
+        GetOptions(threads, batch_size_ * step, hash, optimize,
+                  dump_weights && !multi_step_embedded ? &ctx_paths[step - 1] : nullptr));


Should dump_weights be always false when the input has an embedded context?
Now I think that an error check together with size checks would make sense. It would throw an error if trying to dump from an embedded context.

Yeah, fair.
Yes dump weights should be false when the input itself has an embedded context, since the engine is basically just one node right, the EPContext stub. Theres nothing to dump.

I added some error checks around this for all possible cases. Theres an additional case if trying to dump but the provider itself isnt onnx-trt but any of the other ones. Another case is if the weight file is ep context but provider is not trt.

I think this covers all cases?

Menkib64 · 2026-08-07T16:15:17Z

      trt_options["trt_max_partition_iterations"] = "1000";
      trt_options["trt_min_subgraph_size"] = "1";
      trt_options["trt_engine_cache_enable"] = "1";
+      trt_options["trt_dump_ep_context_model"] = "1";


We should only enable dump context conditionally when constructor is asking for the context path.

thanks yes youre right

Save TensorRT engine in weights file with onnx2leela and automate dur…

70e7de4

…ing runtime

mooskagh requested a review from Copilot July 14, 2026 12:47

Copilot started reviewing on behalf of mooskagh July 14, 2026 12:47 View session

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread src/neural/backends/onnx/network_onnx.cc Outdated

Comment thread src/neural/backends/onnx/network_onnx.cc Outdated

Comment thread src/neural/backends/onnx/network_onnx.cc Outdated

Comment thread src/tools/onnx2leela.cc

add validatenet check for EPcontext model

0ce94db

Menkib64 reviewed Jul 18, 2026

View reviewed changes

saithepaithewhyyy added 5 commits July 22, 2026 20:40

add renamed names for the model

50ae600

add option to save _ctx file. Delete onnx file once done

2f61596

small fix to edge case

e6784bd

remove overwritting code

a49153a

need to have a check to make sure net_path is not empty

19d8cb4

saithepaithewhyyy requested a review from Menkib64 July 26, 2026 07:27

Menkib64 reviewed Jul 26, 2026

View reviewed changes

saithepaithewhyyy added 10 commits July 30, 2026 22:58

remove backendopt

7edfe10

removed the extra if branch

65c7bd4

changes to ctx packaged weights loading

4a18d5e

ordering of output heads to corre4ctly match

e071da0

missed this

3e9b676

add to proto

b27b9b8

multi step saving and loading for ep context models

51dc7ba

fix typo

c2ac001

undo remove whitespace

b3dc78c

add conditional to trt_context_memory_sharing_enable

f0cf637

saithepaithewhyyy requested a review from Menkib64 August 3, 2026 04:12

Menkib64 reviewed Aug 7, 2026

View reviewed changes

saithepaithewhyyy added 2 commits August 9, 2026 14:50

add exception, change prefix path and change var

9130cff

added error cases for dump-weights, is_ep_context and TRT provider

e92bec2

saithepaithewhyyy requested a review from Menkib64 August 10, 2026 14:53

Uh oh!

Conversation

saithepaithewhyyy commented Jul 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mooskagh commented Jul 14, 2026

Uh oh!

saithepaithewhyyy commented Jul 14, 2026

Uh oh!

Menkib64 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saithepaithewhyyy Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

saithepaithewhyyy commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

saithepaithewhyyy Jul 20, 2026 •

edited

Loading

saithepaithewhyyy commented Jul 20, 2026 •

edited

Loading