CanonicalAAS: Define Ordering Rules and Canonical Representation for AAS

[mhrimaz]

Is your feature request related to a problem? Please describe.
In AAS only at two places we conceptually have ordering :

• SubmodelElementList when the orderRelevant flag is true
• The order of Keys within Reference .

In all other scenarios, ordering has no impact on the meaning of the asset. For example, elements inside a Submodel are conceptually unordered, meaning their sequence does not carry semantic significance.

However, the issue arises because the JSON representation of AAS uses arrays rather than dictionaries for serialization. While this approach is practical and widely supported, it unintentionally suggests that the order of elements in arrays is meaningful. This misunderstanding leads to users and tools, such as Package Explorer, implementing ordering-related features like "Move Up" and "Move Down" for elements inside a Submodel, which is conceptually incorrect.

This creates a situation where users and developers rely on ordering where it is not intended, causing confusion and potential interoperability issues. Addressing this misconception is critical to aligning implementations with the intended AAS concepts.

So if inside the ContactInformations Submodel we have this ordering:

Changing the order shouldn't be important for people. Which I am sure it is not the case:

But if your AAS is in circulation between diferent system, and for some reason they decide to mess around with the ordering of elements (at JSON serialization level), then it will be confusing.

Describe the solution you'd like

The Asset Administration Shell (AAS) doesn’t have a canonical form, which can lead to instability in how data is serialized. Similar to JSON canonicalization, having a standard format for AAS ensures consistent ordering and structure, making debugging easier. It also simplifies version control, such as using Git, by making it clear what has changed between versions. Additionally, a canonical form would help REST APIs provide data in a stable and predictable way, improving compatibility and reliability.

• Clients MUST NOT assume or imply any specific ordering of order-irrelevant elements, such as elements inside a Submodel. This is a common pitfall that can lead to errors when data ordering is not relevant from the perspective of metamodel.
• APIs MAY reorder elements to follow a canonical form where computationally feasible, unless the specific ordering of elements is critical to the meaning or functionality of the AAS.
• We need CanonicalAAS which describe how things should be normalized. Other canonicalization approach might be relevant and be reusable. See canonical XML , canonical JSON and canonical RDF.

Additional context
This is particularly relevant for the IDTA Ontology Workgroup, as ordering in RDF needs special consideration. If we only preserve the order for elements where it matters, round-tripping between JSON/XML and RDF (e.g., for Submodel elements) may result in changes to the order of other elements. If clients assume a specific order for purposes like display or processing, this can lead to interoperability issues. However, if we preserve the order for all elements to ensure a stable round-trip, we deviate from the correct conceptual model, where order should only matter for specific elements.

[BirgitBoss]

You are right, the specification is not defining any order on elements (only within classes and enumeration the order of attributes is relevant). Reason: although ordering is not relevant for JSON it is for XML.

I do not consider the "move up" and "move down" feature to be conceptually incorrect. The AASX Package Explorer is an editor for human beings and especially when designing Submodel Templates the ordering is extremely helpful to understand the model.

What exactly is your proposal for the CanonicalAAS? You assume we cannot just refer to the referenced canonical specifications for xml, json and rdf you added? What would be open?

[mhrimaz]

@BirgitBoss Thanks for your response.
Yes, this is not an inherent issue with Package Explorer. A viewer can choose to display the elements in any order, as long as the order does not affect the meaning.

However, in this case, Package Explorer reorders the elements even at the serialization level. This means that when the data is saved or transferred, the order of elements is changed. As a result, it may create the impression that the order is important. Users might then assume that they will always get the same order, even though that is not guaranteed.

To see a simple demonstration of this behavior, you can check out this Google Colab notebook: https://colab.research.google.com/drive/1Qz4vU06_h_fxlEb-SrZoNPveYcy-LEd_?usp=sharing

Regardless, having a proper way to canonicalize AAS would be useful for many scenarios. The referenced canonicalization methods are good to begin with, but they don't help much.

JSON canonicalization is the process of transforming JSON data into a consistent, standardized format. It ensures a consistent structure by sorting object keys alphabetically, removing unnecessary whitespace, and standardizing number and string representations. This ensures that the same JSON content always results in the exact same byte sequence, which is useful for digital signatures, hashing, and data comparison.

However, we need further consideration to canonicalize AAS. For example the submodelElements are now serialized as array in json. So applying JSON Canonicalization won't change the order of elements within this array. As a quick idea, elements can be sorted based on idShort (similar to sorting json keys).

Let's consider an example:
This is a submodel that we have (we assume everyone now using JSON and let's omit XML and RDF for now)

{
  "id":"something_48c66017",
  "modelType":"Submodel",
  "submodelElements":[
    {
      "idShort":"ManufacturerName",
      "modelType":"Property",
      "valueType":"xs:string",
      "value":"Y Manufacturer LLC"
    },
    {
      "idShort":"SupplierName",
      "modelType":"Property",
      "valueType":"xs:string",
      "value":"X GmbH"
    }
  ]
}

So now a software decide to use a different order and show SupplierName first (like PackageExplorer)

{
  "id":"something_48c66017",
  "modelType":"Submodel",
  "submodelElements":[
    {
      "idShort":"SupplierName",
      "modelType":"Property",
      "valueType":"xs:string",
      "value":"X GmbH"
    },
    {
      "idShort":"ManufacturerName",
      "modelType":"Property",
      "valueType":"xs:string",
      "value":"Y Manufacturer LLC"
    }
  ]
}

So now it is quite hard to tell if they are actually the same or not (the hash is not the same and the git diff will also tell there are changes).

Canonical AAS:

• JSON Canonicalization + sorted elements within containers that don't have ordering (SMC, SM, ...). So now both version can be canonicalized and yield the following:

{
    "id": "something_48c66017",
    "modelType": "Submodel",
    "submodelElements": [
        {
            "idShort": "ManufacturerName",
            "modelType": "Property",
            "value": "Y Manufacturer LLC",
            "valueType": "xs:string"
        },
        {
            "idShort": "SupplierName",
            "modelType": "Property",
            "value": "X GmbH",
            "valueType": "xs:string"
        }
    ]
}

So now for use cases like versioning, or signing the submodel, ... any changes in the order of elements won't cause any issue if we use CanonicalAAS.

Some further considerations:

• idShort is not always mandatory. So sorting only based on idShort might not be sufficient.
• Dangling References can also be related.
• Parameter in REST API to always have Canonical AAS as response
• Consistent canonicalization in different format like XML and RDF that yields stable round-trip

[mristin]

@mhrimaz I think there are two issues mixed in this discussion:

1. The order does matter for UI; this has been discussed many times in the workgroups -- and the current consensus, to the best of my knowledge, is that the order matters (*i.e., this is what @BirgitBoss wrote above), but
2. Depending on the assumptions of the system, the order might or might not matter for hashing and diff'ing.

It is very important to keep the two lines of argument in separation.

The round-trip over RDF might need to ignore the order if RDF does not support it. However, I see no need to break UIs for lack of order in RDF -- the UIs are more important in my book.

[mhrimaz]

@mristin My problem is not really related to RDF at this point. Let's assume we only have JSON serialization.

First:

So, how are people currently doing hashing and diffing? How can I track changes if my Submodel circulates between solutions from different vendors (FAAAST, BaSyx, Core-Works, Package Explorer, etc.)? If we have a canonical AAS form, wouldn't that be useful? Wouldn't it be beneficial to have this canonical form already provided by SDKs?
Do you see any added value here? if not, then let's close this feature request.

Second:

So to distinguish, we have order at two level (you know it better than me): One is at metamodel level for SubmodelElementList which is an ordered container. But in SubmodelElementCollection , or elements inside Submodel we don't have any ordering.

Even though the metamodel doesn’t define the order in SubmodelElementCollection, users expect an order when viewing these elements. The challenge is that the viewer should decide on the order based on external factors, not by the order of elements in the JSON array itself. Do you agree that this is an issue?
For instance, providing a way for users to reorder elements (like the "move up" and "move down" functionality in the Package Explorer) can mislead people into thinking that the order should be strictly preserved, when the metamodel doesn't require it.

So according to the metamodel I have the freedom to change the order of elements in a SubmodelElementCollection (for whatever reason). But now other people would say "Hey, in the Submodel Template ManufacturerName is the first element, why did you move it in the JSON? I want to keep the same order like Package Explorer". And now I need to explain to them, my hosting solution is in fact compliant to the metamodel, and I don't have the obligation. Talks to your AAS visualization vendor to find a solution to show the elements in the order that you want. Do we have any explicit statement like this in the specification? I don't think so.

graph TD;
    A["[Package Explorer] <br> Submodel Collection <br> with specific ordering"] --> B["[Hosting Solution] <br> Submodel Collection <br> with canonical ordering"];
    B --> C["[Another Viewer] <br> Submodel Collection <br> with different ordering"];

Loading

[mristin]

@mhrimaz wrote:

Even though the metamodel doesn’t define the order in SubmodelElementCollection, users expect an order when viewing these elements. The challenge is that the viewer should decide on the order based on external factors, not by the order of elements in the JSON array itself. Do you agree that this is an issue?

I respectfully disagree that there is an issue; or better said, I see the problem as the duality which I described above.

The order does matter for visualization, but not for semantic comparisons. So, if a system cares only about semantics, you can re-shuffle the elements. If your system visually shows the elements, the order does matter, and should be preserved.

There are thus two kinds of hashing & diffing: 1) with order considered, and 2) ignoring the order.

The two are not compatible and should be indicated separately.

[mhrimaz]

If your system visually shows the elements, the order does matter, and should be preserved.

A question: Eclipse BaSyx "MUST" preserve the order of elements within SubmodelElementCollection? or it "SHOULD"? or it "MAY"? (RFC wise)

These are just my thoughts, my main question is written above:

The order does matter for visualization, but not for semantic comparisons. So, if a system cares only about semantics, you can re-shuffle the elements.

Based on what logic should I reshuffle the elements? Should I reshuffle them based on idShort? What if I have a SubmodelElementList where the orderRelevant flag is false? This is what I am looking for. I don’t know if anyone is doing AAS versioning out there, but if they are, they must encounter the same issue. They need rules and logic to yield a canonical representation for comparisons.

I don’t mind if viewers display the elements in a different order. But if you ask them how they do it, the answer would likely be: "Change the order in the JSON array," right? At the implementation level, Package Explorer doesn’t change the order based on a config file—it simply reorders the elements in XML or JSON. I am not pointing fingers at Package Explorer; everyone else naturally does the same. IMO, relying on the ordering of elements in a JSON array is wrong and will backfire in the future. Implementers shouldn’t rely on the array order, as there’s no explicit guarantee it will remain the same when circulating through different systems.

[mristin]

@mhrimaz wrote:

Based on what logic should I reshuffle the elements?

It's not that you should re-shuffle, just that the order does not matter, again provided that your system cares only about the semantics and not about the visualisation. Your diff should abstract away the order, and the hash should be consistent regardless of the error.

I don’t know if anyone is doing AAS versioning out there, but if they are, they must encounter the same issue.

As a matter of fact, we (as in some colleagues and I) are working on it in form of a paper which should be submitted soon.

[mhrimaz]

@mristin That's great! I'm looking forward to reading your paper. I also appreciate your time and the discussions here.

What is the correct RFC term here.... that would be my final question:
An AAS hosting solution, like Eclipse BaSyx, ("MUST"/"SHOULD"/"MAY" ) preserve the order of elements within SubmodelElementCollection.

• MUST
• SHOULD
• MAY

Note:
Maybe when your work is ready, you can tell me if these two Submodel are different or not

{
  "id":"something_48c66017",
  "modelType":"Submodel",
  "submodelElements":[
    {
      "idShort":"ManufacturerName",
      "modelType":"Property",
      "valueType":"xs:string",
      "value":"\u00E9 Manufacturer LLC"
    }
  ]
}

{
  "id":"something_48c66017",
  "modelType":"Submodel",
  "submodelElements":[
    {
      "idShort":"ManufacturerName",
      "modelType":"Property",
      "valueType":"xs:string",
      "value":"e\u0301 Manufacturer LLC"
    }
  ]
}

[mhrimaz]

To further think loud:

in the specification for SubmodelElementCollection it is written:

Note: the elements within a submodel element collection are not ordered. Every element has a unique ID (its "idShort"). However, it is recommended to adhere to the order defined in the submodel template.

So here you are not saying I SHOULD/MUST keep the order of elements within a SMC. It is fine to use Submodel Template as a sort of config file to arrange the order of elements. I am absolutely sure no one implements a submodel instance viewer now based on the structure of submodel template.. they simply use the order of array in the submodel instance...

If the order of elements in a SubmodelElementCollection is not explicitly defined, different AAS hosting solutions may handle ordering inconsistently, leading to interoperability issues. Clients relying on implicit ordering may display data incorrectly when switching between AAS providers.

So still I believe we should have the following statement explicit in the spec:

An AAS hosting solution, such as Eclipse BaSyx, MAY preserve the order of elements within a SubmodelElementCollection, but this is not guaranteed. Clients MUST NOT assume that the order is retained and MUST use the Submodel Template ordering as a reference if they require a specific order for display purposes.

In contrast, for a SubmodelElementList, the order MUST be preserved.