Question: Does every message need a messageID?

[benfrancis]

It has been suggested by @RobWin and @hspaay that it would be useful for every message to have a messageID which uniquely identifies it. This is separate from the requestID/correlationID being discussed in #31.

We seem to be accumulating a lot of IDs in every message (!) and this isn't something I've ever found a need for, so I thought I'd ask for use cases for this feature.

Is this essential? Why is it important? What would be the result of not having one?

[hspaay]

The use of messageID should be optional but can be essential.

Duplicate message can and will happen in situations where the connection is intermittent. If duplicate messages are not allowed it is essential to provide the means to detect them. The use of messageID's are a common method to detect duplicates.

In a fleet management solution I've worked on in the past, cell connection from the vehicles was unreliable as you can imagine. When a connection is interrupted while sending a message it is quite possible that the server receives the message but the acknowledgement didn't make it back to the vehicle. After reconnecting the vehicle will retry sending the message. The server will receive it twice. I was suprised how often this happened.

[RobWin]

I hope this isn't too high-level, but I wanted to share my thoughts on this topic.

One of my all-time favorite books is Enterprise Integration Patterns. It has been a guiding resource for tackling real-world challenges in my projects.

When working with WebSockets, we are essentially building a point-to-point messaging system. This involves implementing Message Endpoints through protocol binding implementations and creating Messaging Gateways using WoT Servients. From a Thing’s perspective, we encounter various message types, including:

• Command Messages: Most of our request messages are actually command messages which result in an event message. A command is a request for the system to perform an action that changes the state of the system. Commands are named with a verb in the imperative mood. They contain all data necessary to perform the action. They are immutable. Examples are invokeAction, readProperty and so on.

• Event Messages: Events have the following chacharacteristics: They represent a state change that has happened in the past. Events are named in in past tense and in business/application language. They are immutable. They contain all data necessary for consumers of the event.
  Do not confuse them with Thing events. An event message can be a Thing Event, a Thing Property Reading message. I actually would prefer to call the messages types propertyChanged or propertyUpdated or eventEmitted. But it's just my personal taste. I'm also fine with propertyReading and event.

• Request-Reply Messages: Can be a command message and an event message which are correlated via a correlation identifier.

• Fire-and-Forget Messages: I actually think we should avoid Fire-and-Forget message in the design.

• Pub/Sub: Could be particularly relevant when multiple Thing Consumers connect to a single Thing Provider over a shared WebSocket connection. This is especially important in scenarios where a Gateway multiplexes multiple Things over a single connection to the Cloud using the Web Thing Protocol. In that scenario a subscriberId or subscriptionId might be needed as well.

WebSocket itself serves as a Point-to-Point Channel, which is a specific instance of the more general Message Channel.

Whenever I’ve worked with messaging systems, I’ve faced similar challenges at some point. For example:

• Correlation Identifiers: Essential for Request-Reply communication patterns.
• Guaranteed Delivery: Crucial in WebSocket systems where Messaging Gateways must buffer and retransmit key events if the WebSocket channel becomes temporarily unavailable.
• Message Expiration: Essential for defining the lifespan of messages, such as an event message (e.g., "smoke detected") or a command message (e.g., "turn lamp on"). This ensures that outdated messages are not transmitted, particularly when cached for guaranteed delivery in scenarios where the connection was temporarily unavailable.
• Message Resequencers: Required when messages are processed through different steps and might get out of order, requiring Message Sequence Identifiers.
• Recipient Lists: Crucial for event message delivery in the Provider Servient’s messaging gateway especially when multiple Consumers are subscribed to the same events.

When you take an end-to-end view of such a "messaging system," encompassing Messaging Gateways on both the Thing Consumer and Thing Provider sides, you realize the need to implement numerous patterns to ensure the system is robust and resilient.

In this context, Messages can be seen as having two core parts:

• Header: Information used by the messaging system that describes the data being transmitted and how it should be processed by a Messaging Gateway.
• Body: The actual payload, generally transmitted as-is without being processed by the messaging system.

In my view, we’re designing a WoT-specific messaging protocol for WebSockets. And I really believe that a key element of the message header is a unique message ID, which is a common feature in most messaging protocols. Currently, I use the message ID from a request message as the correlation ID in the corresponding response message.

Some messaging systems also allow both a unique message ID and a correlation ID in request messages to group multiple requests (e.g., in a "multiple request, single response" pattern). However, I don’t believe this complexity is necessary for the Web Thing Protocol at this stage.

[RobWin]

Here are some concrete use cases where a messageId is needed:

1. Request-Response Correlation

   • Use Case: A Consumer sends a request to a Thing Provider (e.g., invokeAction) and expects a response.
   • Why Needed: The message ID allows the Consumer to match the correlation id of the response to the corresponding request, particularly in asynchronous or multi-threaded environments where multiple requests are pending.
   • Smart Home Example: A smart home gateway sends a command to a thermostat to change the temperature and waits for confirmation.

2. Event Stream Deduplication

   • Use Case: A Provider sends event messages to multiple subscribed Consumers.
   • Why Needed: A unique message ID helps Consumers deduplicate events if the same event is received multiple times due to retries or network issues. Here is a blog post how RabbitMQ Streams uses a Message Sequece identifier to deduplicate messages.
   • Smart Home Example: A smoke sensor sends an event when smoke is detected. If the provider retries sending the event due to network issues after some minutes, the unique message ID allows the consumer to ignore duplicate messages. By leveraging deduplication, it prevents consumers from reacting to the same event multiple times, which could lead to unnecessary actions (e.g., multiple alarms in case of smoke detection).

3. Command Retry Logic and Deduplication

   • Use Case: A Consumer retries sending a command message if acknowledgment is not received (e.g., invokeAction) after a period of time.
   • Why Needed: A message ID allows the Thing provider to recognize and discard duplicate command messages caused by retries. For example, a toggleSwitch action could quickly turn a light off again if the initial turn light on action was delayed or took too long to complete, especially when the action is retried shortly thereafter.
   • Smart Home Example: In a smart home system, when a user presses a button in their app to turn on a light, the app sends a command to the Thing Provider to execute the action. The command includes a unique messageId. If the Thing Provider does not respond within a specified timeout period (due to issues in the device itself), the app assumes the request was not successfully processed and retries the command. The Thing Provider can use the messageId to track each request. When a new command is received, the Thing Provider stores the messageId along with its pending state in a queue or map, indicating that the request is being processed. If a retry command is received with the same messageId, the provider can check if the same request has already been processed or is still in progress. If so, it can safely discard the retry request to avoid toggling the light off immediately after turning it on due to duplicate messages.

4. Error Handling and Logging

   • Use Case: A message fails during processing, and debugging or auditing is required.
   • Why Needed: The message ID provides a unique reference to identify and trace the problematic message in logs or monitoring systems.
   • Smart Home Example: A smart lock fails to respond to a request to unlock the front door. The message ID allows developers to trace the specific request in the logs and identify if the issue was due to a network failure or an internal processing error.

5. Correlation in Server-Streaming or Long-Running Actions

   • Use Case: A Consumer sends a request (e.g., invokeAction) and receives a stream of actionStatus progress messages.
   • Why Needed: The message ID correlates the stream of responses (event messages) to the original request (command message).
   • Smart Home Example: A smart oven receives a command to start preheating. The oven sends progress updates (e.g., "Preheating started", "Preheating done"). The message ID ensures the actionStatus progress messages are tied to the specific preheat request, even if other commands (like setting a timer) are issued simultaneously. Modeling action status updates as events may not always be appropriate, particularly when the status of a long-running action is more directly tied to the request itself, rather than being an event that could be subscribed to by multiple consumers. This approach treats the action updates as responses to the original request, rather than broadcasting them as events, keeping the correlation more direct and meaningful in the context of the action lifecycle. But I struggle to find a better example.

6. Timeouts and Expirations

   • Use Case: A message should be processed within a certain time; if not, it is discarded or flagged as failed.
   • Why Needed: The message ID is used to track pending messages and enforce timeout rules.
   • Smart Home Example: If someone attempts to turn on a light but the command isn't processed within a few seconds due to connectivity issues, the command is flagged as expired. This ensures that the command is not retransmitted once the connectivity is restored, preventing the light from being switched on unnecessarily at a wrong time.

7. Multiplexed Connections

   • Use Case: A single WebSocket connection is shared by multiple Consumers to send messages to the same Provider.
   • Why Needed: The message ID ensures that responses are correctly routed to the originating Consumer in a multiplexed communication scenario. A messageId might eliminate the need for additional identifiers like a senderId.
   • Smart Home Example: In a smart home system, multiple consumers, such as mobile apps, may want to control and monitor a thermostat. Rather than each app establishing its own WebSocket connection to the thermostat, they all share a single WebSocket connection between the cloud and the Thing Provider. When a user in one of the apps updates the target temperature (e.g., setting the thermostat to 24°C), the mobile app sends a command message to the cloud, which then forwards it to the thermostat provider over the shared connection. The message ID ensures that the cloud can correctly identify and track the command, ensuring that the thermostat adjusts the temperature accordingly. Once the temperature is updated (e.g., from 22°C to 24°C), the thermostat sends an event message back through the cloud to the mobile app, confirming the new target temperature. The message ID helps the cloud route the response to the correct app and update the user interface.

8. Guaranteed Delivery

   • Use Case: A Provider ensures that critical messages are delivered to the Consumer, even if the connection is temporarily unavailable.
   • Why Needed: The message ID is essential for tracking undelivered messages and ensuring proper retransmission.
   • Smart Home Example: A water leak sensor detects a leak and sends an event. If the connection to the consumer is unavailable, the message is cached for later delivery. The message ID ensures the event is sent only once when the connection is restored.

[benfrancis]

@RobWin Thank you for this detailed list of use cases, this is extremely helpful and there are lots of use cases I hadn't considered.

Proposal: What if every message has a messageID member, but only response messages (e.g. propertyReading/actionStatus/event) have a correlationID member (whose value is the message ID of the request it relates to)? Would that be sufficient? That doesn't support correlating multiple request messages, but I don't think we currently have a use case for that and it means we wouldn't need three mandatory IDs in every message.

---

Use Case: A single WebSocket connection is shared by multiple Consumers to send messages to the same Provider.

Side Note: Whilst the Use Cases & Requirements document says that a single WebSocket should be shareable between multiple interaction affordances and multiple Things, it doesn't currently say it should be shareable between multiple Consumers. That's not currently a requirement and I don't think it would work well with the assumptions made around authentication and WebSocket re-use (a single set of credentials is provided when opening the WebSocket, using standard HTTP security schemes, so that credentials aren't needed inside every message). This is really a separate topic though so we should file a separate issue if this something people see as an important use case.

[RobWin]

Yes, I agree to both.
An optional correlationId or groupId in request messages could also be added later, if needed

[hspaay]

Proposal: What if every message has a messageID member, but only response messages (e.g. propertyReading/actionStatus/event) have a correlationID member (whose value is the message ID of

It probably could work but it gets convoluted. correlationID and messageID have two different purposes. Mixing them up like this could cause confusion. The idea of messageID is that it is unique to the message but now the response re-uses it instead. The logic for messageIDs is also unrelated to that of correlation ID.

This looks like premature optimization to me. What is the benefit? If payload size is important then there are better ways to address this, by changing encoding for example. I doubt it has a measurable impact.

My counter proposal is to keep them separate and make messageID optional. It is only useful for actions that are not idempotent anyways.

[hspaay]

Use Case: A single WebSocket connection is shared by multiple Consumers to send messages to the same Provider.

That's not currently a requirement and I don't think it would work well with the assumptions made around authentication and WebSocket re-use

Yes indeed. This will not allow identification of the consumer which is critical for security. I also don't see a practical use-case that needs this.

[RobWin]

I disagree and agree :)

The MessageID should be mandatory and unique for every message. This is critical for debugging and auditing purposes, ensuring that each message can be tracked. Beyond that there are other significant use cases I have described above.

As already mentioned. In many messaging systems, correlation data is used to link requests to responses. Typically, the request contains optional correlation data, which the responder copies, when available, into the response to allow correlation for the requestor. This is a common pattern and is well-documented in messaging systems and educational materials. And very easy to implement.

The nature of correlation data varies across systems:

• Some systems allow complex objects.

• Others restrict correlation data to simpler forms, such as UUIDs.

But I agree. By allowing optional correlation data in request and response messages, we enable flexibility without overloading the purpose of the mandatory MessageID field.

Web Thing Protocol Implications:

If the Web Thing Protocol wants to avoid optional correlation data in requests, the MessageID would need to be used as a substitute. However, this approach has limitations and may not align with how correlation data is handled in other systems.

Proposed Solution:
Introduce an optional CorrelationId in both request and response messages.

Retain the mandatory MessageID for its primary purpose (unique identification)

[RobWin]

I also don't see a practical use-case that needs this.

I could provide real live use case, if needed.
But I'm fine if we keep this use case out of scope.

[hspaay]

The MessageID should be mandatory and unique for every message. This is critical for tracing and auditing purposes,

I don't doubt that it is critical for this kind of use. However, these are application/policy requirements that are not neccesarily protocol binding requirements. The binding just has to be able to support these application requirements.

I do doubt that all applications require it. Would you force the use of messageID in the messaging for those applications?

[hspaay]

By allowing optional correlation data in request and response messages, we enable flexibility without overloading the purpose of the mandatory MessageID field.

Yep that keeps things clear. If no correlation data is provided then you won't get a response. As simple as that.

[RobWin]

I do doubt that all applications require it. Would you force the use of messageID in the messaging for those applications?

Yes, because let's say a Thing Provider is implemented by a different company directly in a device and the company is implementing the Web Thing Protocol, but without messageIds.
You implement a Thing Consumer and need the messageId. What do you do now?
Either its mandatory and you can count on it or it's worthless as an interoperability standard

[RobWin]

Yep that keeps things clear. If no correlation data is provided then you won't get a response. As simple as that

False, at least in my understanding.
If there is no correlation data in the request, the requestor doesn't need it in the response.
If its available, its necessary in the response. This is how it usually works. At least in the systems I know :/

[hspaay]

False, at least in my understanding.

I think we're trying to say the same thing :)

[RobWin]

Maybe :)
But there is a difference between "you won't get a response" and "you get a response without correlation data" 😶‍🌫️

[hspaay]

But there is a difference between "you won't get a response" and "you get a response without correlation data" 😶‍🌫️

You're right! Its funny as we have such a different perspective. From a hub/gateway point of view having a Thing send a response to the hub without correlation data is useless as the hub can't determine which consumer connection to forward the response to. Hence my comment on not sending it.
From a Thing point of view, a response can simply be sent to the consumer connection the request came in on. The consumer can still process the response.
Unfortunately the system level behavior will differ between these two environments.

[RobWin]

I mean if the requestor (hub) sends a request with correlation data, the Thing must add the correlation data to the response to comply with the Web Thing protocol specification.
If the requestor does not attach correlation data, then it's fine if the response does not contain correlation data.

This would be documented in the spec.
If a thing does not copy the correlation data, when available, from a request to the responses, then it does nor comply to the spec and it would be a bug in the implementation.

[hspaay]

If a thing does not copy the correlation data, when available, from a request to the responses, then it does nor comply to the spec and it would be a bug in the implementation.

Agreed.

What behavior is expected though if no correlationID is present in the request. Should the Thing still send a response? I interpret this situation as if no requestID is provided then there is no interest in a response, so no need to send one. But that is based on the (incorrect) assumption that there is no use for such a response. The Thing doesn't care whether its consumer is a hub/gateway or another client.

(btw, maybe this is its own separate issue as we are no longer talking about messageID)

[RobWin]

As described above and how it works in other messaging systems with request-response pattern, when no correlation data is needed.

If the requestor does not attach correlation data, then it's fine if the response does not contain correlation data. But still a response is returned.

[hspaay]

Okay, makes sense. Thanks for clarifying.

This does imply that if a consumer sends a request to the hub/gateway without correlation data, the hub would have to add it when forwarding the request to the Thing, otherwise it won't be able to correlate the response from the Thing to return to the original consumer. The hub must also remove the correlation from the response to the original consumer.
This is not a problem, just a realization that it is needed.

[benfrancis]

OK, the consensus so far appears to be that in addition to a mandatory thingID, all messages should have a mandatory messageID and an optional correlationID. If a Consumer -> Thing message includes a correlationID then Thing -> Consumer messages which correspond to the same operation should include a correlationID member with the same value.

This is definitely more IDs than I had anticipated would be necessary in messages, but I can see the use cases for them.

[benfrancis]

Answer: Yes.