Fix onnxruntime 0D tensor issue on openvino endpoint

[sgbihu]

Description

The OrtTensorTypeAndShapeInfo struct lost the scalar information when converted from proto. The existing structure can't distinguish the dynamic rank and scalar cases. That makes the serialized graph lost shape area and the OV can't distinguish the scalar type.

Original tensor in prototxt:

   type {
      tensor_type {
        elem_type: 1
        shape {
        }
      }
    }

After serialized prototxt:

    type {
      tensor_type {
        elem_type: 1
      }
    }

• Add has_shape for OrtTensorTypeAndShapeInfo

Original Model

graph {
  name: "OpenVINOExecutionProvider_11295571201636618024_0"
  node {
    input: "absInput_1"
    output: "absOutput_0"
    name: "_0"
    op_type: "Abs"
    domain: ""
  }
  input {
    name: "absInput_1"
    type {
      tensor_type {
        elem_type: 1
        shape {
        }
      }
    }
  }
  output {
    name: "absOutput_0"
    type {
      tensor_type {
        elem_type: 1
        shape {
        }
      }
    }
  }
}

The graph after OrtEpUtils::OrtGraphToProto

graph {
  name: "OpenVINOExecutionProvider_11295571201636618024_0"
  node {
    input: "absInput_1"
    output: "absOutput_0"
    name: "_0"
    op_type: "Abs"
    domain: ""
  }
  input {
    name: "absInput_1"
    type {
      tensor_type {
        elem_type: 1
      }
    }
  }
  output {
    name: "absOutput_0"
    type {
      tensor_type {
        elem_type: 1
      }
    }
  }
}

Motivation and Context

• CVS-173301

[adrianlizarraga]

Hi, thank you for looking into this. Could you please share more information on the kind of model that triggers this failure to distinguish between a scalar or a tensor with unspecified shape? I would have thought that onnxruntime would remove this ambiguity internally. A small repro model (or a description of one) would be very useful. Thanks!

[adrianlizarraga]

Being able to set has_shape to true when dim_params is not null seems like a convenient coincidence. Perhaps it would be good to pass that information to this function? I haven't fully thought it through but perhaps something like the following could work:

1. Add a has_shape parameter to this GetTensorShapeAndTypeHelper function.
2. Callers of GetTensorShapeAndTypeHelper can use onnx::TypeProto_Tensor::has_shape() to pass this information to GetTensorShapeAndTypeHelper.

[sgbihu]

abs_lostdim_case.zip
This is the test model, you can use the abs_0d_input.onnx as input. The abs_0d_lostdim.onnx is the dumped model after serialization.

[sgbihu]

For your suggestion, could you take a look at onnxruntime/core/framework/onnxruntime_typeinfo.cc:L269, L285:L310. Here already convert the HasShape to nullptr or a valid pointer. After you review that code, I can add it if you think a new parameter is necessary.

[adrianlizarraga]

Yes, I think a new parameter is better. Right now, using dim_params != nullptr seems like a coincidence. When you upstream to ORT, others may have other suggestions, but I do think this is preferable.

Note that the line utils::HasShape() internally calls onnx::Typeproto_Tensor::has_shape() (as I pointed out in my original message). This seems like the value we want to pass down to GetTensorShapeAndType().

One other thing to note: the GetTensorShapeAndTypeHelper function is also called in core/session/custom_ops.cc:L83. This location also has access to onnx::TypeProto_Tensor::has_shape().

[adrianlizarraga]

Thanks again for fixing this and providing the model.

[sfatimar]

@sgbihu Do we really need this in intel development branch, I think you can directly raise to MSFT ? Do you need our team help to do that

[yuslepukhin]

utils::HasShape

[yuslepukhin]

The decision to put true or false in this file is not clear to me. The type_protos here are not from the original protobuf, but rather from generic type registrations. How do we know that this particular tensor had real shape in the original protobuf since all we have here is the OrtValue which obviously has a real shape.

[sgbihu]

I set this value based on http://github.com/intel/onnxruntime/blob/ovep-develop/include/onnxruntime/core/framework/tensor_shape.h#L109 , because the tensor doesn't have an API to indicate this is a scalar. Do you know an API can do this?

And I think the purpose of adding the has_shape is to fix the information lost at model conversion stage. If we already have the shape, it is not a dynamic rank case. I think this can get the correct shapes when dumping models.

[yuslepukhin]

I think there is a mix of two different things here.

Tensor and TensorShape classes are runtime classes. They will always have a shape that will be inferred, otherwise we cannot run.

IsScalar() is the method on TensorShape that will tell you if it is a scalar or not based on the shape that was inferred or contained (not exposed via public API) since it can also be found out from the shape vector returned.

The public APIs that are you are dealing with have to do with specific concrete OrtValues that contain tensors that are a result of either creating a feeding input to the model or receiving an output. So they do not have anything to do with the protobuf of the original model. The same would apply to intermediate values, which is a new path that appeared with EP plug-ins.

They are not direct counterparts of what is found in protobuf which is a metadata and this is what you are probably trying to get.

It would be helpful to see an example of the model that you are trying to deal with and what exactly you are trying to achieve (serialize the model?)

As far as the model input/output goes, Session API can return ValueInfo that contains TypeInfo. That can tell you the dimension count but does not tell you if it has the shape, perhaps it is an omission.

[adrianlizarraga]

As far as I understand, an OrtValue is already known to have a shape, so passing true here makes sense to me.

It would be helpful to see an example of the model that you are trying to deal with and what exactly you are trying to achieve (serialize the model?)

The onnx model file has been provided in this comment: #816 (comment)

The plugin EP gets the model as a OrtGraph via the new APIs. The EP then converts the OrtGraph to protobuf so that it can be further processed by openvino compiler, which currently only supports protobuf models (not OrtGraph). Please correct me if I'm wrong @sgbihu

[sgbihu]

Yes, OV only handle the protobuf models for now. I also pasted a prototxt at the PR description, and the shape information is lost. That means a scalar to a dynamic rank tensor.

[yuslepukhin]

This PR is being deprecated in favor of microsoft#26164