Don't embed external initializers into the proto to avoid 2GB limit #817

intbf · 2025-09-19T20:37:23Z

Description

When calling subgraph.ToProto don't save initializers, so that we don't bloat the model proto. This is especially important for models with ext initializers in memory that in total contain more than 2GB of data.

Once we have the list of those external initializers, we can update ther metadata inside the proto, so that they leverage the special marker: */_ORT_MEM_ADDR_/*. (The support for this location was recently fixed inside OV with openvinotoolkit/openvino#31632)

Motivation and Context

CVS-173057, CVS-172710

Todo/questions

Should we have some condition before using this approach? Maybe when model has those ext weights?
The code in GetModelProtoFromFusedNode currenly supports only one case, the regular case, but similar logic have to be applied to other cases like qdq stripping, scale fix, etc.
Add unit tests

Copilot

Pull Request Overview

This PR modifies the model proto generation to avoid embedding external initializers directly into the proto, preventing potential issues when models exceed the 2GB protobuf limit. The changes introduce a mechanism to handle initializers with external data by updating their metadata to use a special memory address marker instead of including the actual data.

Key changes:

Add logic to exclude initializer data from proto generation when load_user_initializer_ is enabled
Implement metadata updating for external initializers using the */_ORT_MEM_ADDR_/* marker
Add comprehensive logging for debugging initializer processing

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

onnxruntime/core/providers/openvino/backend_manager.cc

Copilot

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

onnxruntime/core/providers/openvino/backend_manager.cc

Copilot

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 5 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

onnxruntime/core/providers/openvino/backend_manager.cc

Copilot · 2025-09-30T12:23:34Z

onnxruntime/core/providers/openvino/backend_manager.cc

+    // but then we have to update those initializers and set the external_data fields to mem_addr tag...
+    // proto is limited to 2GB, but let's use 512MB as threshold to be conservative and still gain some memory reductions.
+    constexpr size_t MAX_EMBEDDED_INITIALIZER_SIZE = 1024 * 1024 * 512;
+    const bool include_initializer_data_in_proto = !(session_context_.has_external_weights == true && 


The condition session_context_.has_external_weights == true is redundant. Simply use session_context_.has_external_weights since it's already a boolean.

Suggested change

const bool include_initializer_data_in_proto = !(session_context_.has_external_weights == true &&

const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&

Copilot · 2025-09-30T12:23:35Z

onnxruntime/core/providers/openvino/backend_manager.cc

-    subgraph.ToProto(*model_proto->mutable_graph(), true, true);
+    subgraph.ToProto(*model_proto->mutable_graph(), /*include_initializers*/true, 
+                     /*include_outer_scope_args*/true, /*execution_order*/0, /*include_initializer_data*/include_initializer_data_in_proto);
+


Remove the trailing whitespace on this empty line.

Suggested change

Copilot · 2025-09-30T12:23:35Z

onnxruntime/core/providers/openvino/backend_manager.cc

+
    print_model_proto_duration();
    DumpOpenVINOEPModel(onnx_model_path_name, model_proto.get(), fused_node);
+


Remove the trailing whitespace on this empty line.

Suggested change

… restores the metadata so OV can read them back Signed-off-by: bfilipek <[email protected]>

…re are external initializers in memory (more than one) Signed-off-by: bfilipek <[email protected]>

Signed-off-by: bfilipek <[email protected]>

…t initializers, comments, refactoring Signed-off-by: bfilipek <[email protected]>

ericcraw · 2025-10-01T17:13:20Z

onnxruntime/core/providers/openvino/backend_manager.cc

    print_model_proto_duration();
    DumpOpenVINOEPModel(onnx_model_path_name, model_proto.get(), fused_node);


we should probably dump the proto after we've finished transforming it in the case include_initializer_data_in_proto == false.

onnxruntime/core/providers/openvino/backend_manager.cc

Co-authored-by: Copilot <[email protected]>

Copilot

Pull Request Overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

MayureshV1 · 2025-10-02T00:43:06Z

openvinotoolkit/openvino#31632 was only added after OV in OV25.3, so we need to have a OV version check implemented.

…r the logic is executed, check for OV version Signed-off-by: bfilipek <[email protected]>

Signed-off-by: bfilipek <[email protected]>

…ver 2GB to show the proto limit (when the new logic for ext initializers is enabled, then the test passes) Signed-off-by: bfilipek <[email protected]>

onnxruntime/test/providers/openvino/openvino_ep_ext_init.cc

ericcraw · 2025-10-03T20:25:19Z

onnxruntime/test/providers/openvino/openvino_ep_ext_init.cc

+  for (size_t i = 0; i < std::min<size_t>(10, floats_per_initializer); ++i)
+    ASSERT_FLOAT_EQ(out_data[i], expected);


doesn't hurt to check them all.

Signed-off-by: bfilipek <[email protected]>

Copilot

Pull Request Overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

onnxruntime/test/providers/openvino/openvino_ep_ext_init.cc

Copilot · 2025-10-04T00:16:49Z

onnxruntime/core/providers/openvino/backend_manager.cc

+    const bool include_initializer_data_in_proto = !(session_context_.has_external_weights && 
+                                                     external_initializers_offset_and_length.size() > 1 && 


The condition external_initializers_offset_and_length.size() > 1 seems arbitrary. Consider using a named constant or documenting why specifically more than 1 external initializer is required to trigger this optimization.

Suggested change

const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&

external_initializers_offset_and_length.size() > 1 &&

// Optimization is only triggered if there is more than one external initializer,

// as the benefit of excluding data from the proto is only significant in that case.

constexpr size_t MIN_EXTERNAL_INITIALIZERS_FOR_OPTIMIZATION = 2;

const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&

external_initializers_offset_and_length.size() >= MIN_EXTERNAL_INITIALIZERS_FOR_OPTIMIZATION &&

Copilot · 2025-10-04T00:16:49Z

onnxruntime/core/providers/openvino/backend_manager.cc

+
+          LOGS(logger, VERBOSE) << "In-memory initializer EXT: " << src_init->name() << ", size: " << length;
+
+          SetExternalDataFields(proto_init, (const void*)offset, length);


Casting offset (which is a size_t representing a memory offset) directly to (const void*) is incorrect. The offset should be added to a base pointer to get the actual memory address, not used as a pointer itself.

Suggested change

SetExternalDataFields(proto_init, (const void*)offset, length);

SetExternalDataFields(proto_init, static_cast<const void*>(static_cast<const uint8_t*>(external_initializers_data) + offset), length);

Co-authored-by: Copilot <[email protected]>

javier-intel requested review from MayureshV1, ericcraw, javier-intel and Copilot September 23, 2025 13:59

Copilot AI reviewed Sep 23, 2025

View reviewed changes

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

preetha-intel self-requested a review September 24, 2025 04:41

intbf force-pushed the dont_write_ext_initializers branch from fd29454 to 2b34773 Compare September 26, 2025 13:22

intbf requested a review from Copilot September 26, 2025 19:23

Copilot AI reviewed Sep 26, 2025

View reviewed changes

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

intbf commented Sep 27, 2025

View reviewed changes

onnxruntime/core/providers/openvino/backend_manager.cc Outdated Show resolved Hide resolved

intbf requested a review from Copilot September 30, 2025 12:22

Copilot AI reviewed Sep 30, 2025

View reviewed changes

intbf marked this pull request as ready for review September 30, 2025 12:23

intbf added 4 commits October 1, 2025 00:23

early version, it doesn't embed initializers into the proto, but then…

30e9d5f

… restores the metadata so OV can read them back Signed-off-by: bfilipek <[email protected]>

improve code, refactor into smaller functions, run the logic when the…

70c2f37

…re are external initializers in memory (more than one) Signed-off-by: bfilipek <[email protected]>

revert the wrongly merged code

80de8ef

Signed-off-by: bfilipek <[email protected]>

Updated the condition for the new logic based on the total size of ex…

ef6f23d

…t initializers, comments, refactoring Signed-off-by: bfilipek <[email protected]>

intbf force-pushed the dont_write_ext_initializers branch from 7907cd7 to ef6f23d Compare October 1, 2025 07:23

ericcraw reviewed Oct 1, 2025

View reviewed changes

javier-intel approved these changes Oct 1, 2025

View reviewed changes

MayureshV1 requested a review from Copilot October 2, 2025 00:32

Update onnxruntime/core/providers/openvino/backend_manager.cc

165a661

Co-authored-by: Copilot <[email protected]>

Copilot AI reviewed Oct 2, 2025

View reviewed changes

intbf added 3 commits October 2, 2025 14:00

make the condition less strict - 32MB threshold, move debug dump afte…

fe2cf8c

…r the logic is executed, check for OV version Signed-off-by: bfilipek <[email protected]>

unit test that uses ext initializers, early version

e6727b1

Signed-off-by: bfilipek <[email protected]>

used kOrtSessionOptionsDisableCPUEPFallback, cleanups, model is now o…

033b6f9

…ver 2GB to show the proto limit (when the new logic for ext initializers is enabled, then the test passes) Signed-off-by: bfilipek <[email protected]>

ericcraw approved these changes Oct 3, 2025

View reviewed changes

address code review comments

d4e41c9

Signed-off-by: bfilipek <[email protected]>

MayureshV1 requested a review from Copilot October 4, 2025 00:16

Copilot AI reviewed Oct 4, 2025

View reviewed changes

Update onnxruntime/test/providers/openvino/openvino_ep_ext_init.cc

cba53b3

Co-authored-by: Copilot <[email protected]>

	const bool include_initializer_data_in_proto = !(session_context_.has_external_weights == true &&
	const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&


		print_model_proto_duration();
		DumpOpenVINOEPModel(onnx_model_path_name, model_proto.get(), fused_node);

		for (size_t i = 0; i < std::min<size_t>(10, floats_per_initializer); ++i)
		ASSERT_FLOAT_EQ(out_data[i], expected);

		const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&
		external_initializers_offset_and_length.size() > 1 &&

-    const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&
-                                                     external_initializers_offset_and_length.size() > 1 &&
+    // Optimization is only triggered if there is more than one external initializer,
+    // as the benefit of excluding data from the proto is only significant in that case.
+    constexpr size_t MIN_EXTERNAL_INITIALIZERS_FOR_OPTIMIZATION = 2;
+    const bool include_initializer_data_in_proto = !(session_context_.has_external_weights &&
+                                                     external_initializers_offset_and_length.size() >= MIN_EXTERNAL_INITIALIZERS_FOR_OPTIMIZATION &&


		LOGS(logger, VERBOSE) << "In-memory initializer EXT: " << src_init->name() << ", size: " << length;

		SetExternalDataFields(proto_init, (const void*)offset, length);

	SetExternalDataFields(proto_init, (const void*)offset, length);
	SetExternalDataFields(proto_init, static_cast<const void>(static_cast<const uint8_t>(external_initializers_data) + offset), length);

Don't embed external initializers into the proto to avoid 2GB limit #817

Are you sure you want to change the base?

Don't embed external initializers into the proto to avoid 2GB limit #817

Conversation

intbf commented Sep 19, 2025

Description

Motivation and Context

Todo/questions

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 30, 2025

Choose a reason for hiding this comment

Uh oh!

ericcraw Oct 1, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

MayureshV1 commented Oct 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ericcraw Oct 3, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Uh oh!

Copilot AI Oct 4, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MayureshV1 commented Oct 2, 2025 •

edited

Loading