Proposal: Move Forms out of the TD (TD2.0)

[hspaay]

24-Mar-2025 Update 1: Changed title and intro based on comments.

This is an outline of the proposal to solve problems that Forms bring by moving them out of TD2.0.

Why move Forms out of the TD

Forms are overkill, complex, unnecessary, and very very annoying.

All protocols bindings need to exchange the same information: operation, thingID, affordance name, correlationID, messageID, message type (request, response, notification), action status, and input or output data.

Thats it. It always boils down to this set of information. Describing the miriad ways of how protocols can do this using forms in every single affordance is unnecessary. There is no need to spread this out over the TD, especially since it depends on the protool used. Right now forms pretty much let you build a separate protocol binding for every single affordance. This makes the words 'utter overkill' look inadequate.

Having to specify a form in every single affordance can double the TD size, cost extra processing power and resources to handle, without any benefit. It adds unnecessary complexity, increases the risk of bugs, makes testing more difficult and hinders adoption. It is the wrong solution for a simple problem. Lets get past it in TD2.0 while we can.

What is described in forms can be moved to the protocol binding. The TD only needs to reference which protocol bindings are supported and what endpoint to use.

This is IoT, not a multi-media server chat, telephony, broadcasting system that interfaces with every known system known to mankind in the past and future.

Proposal

Leverage operations and let the protocol binding define how to transfer the information for an operation.

The TD only needs to specify the protocol binding used and how to establish a secure connection. A Thing level protocols section in the TD is all that is needed. For example:

{
  protocols: {
      "websocket": {
         "endpoint": "wss://address:port/path",
         "security": "stuff goes here",
         "encoding": "json",
      }
      "mqtt": {
         "endpoint": "mqtts://address:port/"
         "security": "stuff goes here",
         ...
      }
  }
}

The TD defines how to connect. The protocol defines how to exchange messages for operations.

All protocols exchange the same information: operation, thingID, affordance name, correlationID, messageID, message type (request, response, notification) and input or output data. Let the protocol specify how to exchange this information.

The latest websocket proposal (not strawman) is a good example. It works.

... but what about ... Yep, leave them in the comments. I'm sure there are valid concerns so lets take a look. The benefits are too great to ignore this.

Bonus oppertunity

As a bonus this is an oppertunity to standardize the messaging envelopes used to send requests, receive responses and notifications, instead of reinventing this wheel for each protocol binding that needs a message envelope. This would reduce SDK development time, simplify a standard test suite to validate interoperability, and also encourage adoption especially for consumers that need to support multiple protocols. A good candidate for standardized message envelopes are the ones defined in the latest websocket protocol currently in draft.

Of course, if a protocol doesn't benefit from using an envelope, for example because addressing and headers can carry this information, then don't use one.

An example:
The http-basic protocol v2 could specify the path to use. GET "/readproperty/{name}" for the operation to read a property. This is in the protocol specification instead of the forms. BaseURI can still be used as currently specified in the TD giving sufficient flexibility to different server environments.

Edge cases

Yes there are always edge cases why this (might) not work. Please leave these in the comments.

In case it is really necessary to change the encoding for a specific affordance then specify the different encoding in the data schema of that affordance.

What is lost with this change is the ability to use different protocols for different affordances of the same Thing. I have yet to see a valid use-case for this that can't be addressed with the proposed solution. But maybe you have a few to share?

ps: I reserve the right to modify this proposal to address some of the concerns raised and fix dyslexia issues. Thank you for reading this far.

[relu91]

I hope to get back to you with a longer comment, but I just wanted to mention that we are working to bake something that goes in this direction (although it is less groundbreaking). The PR is quite overwhelming to read, but basically you can have a look at this file. In particular, to the examples section.

[benfrancis]

@hspaay wrote:

This is an outline of the proposal to solve problems that Forms bring by removing them from TD2.0.
...
Forms are overkill, complex, unnecessary, and very very annoying.

I think you would like Mozilla's legacy Web Thing API. Mozilla spent a lot of time arguing for simple "links" instead of "forms" in Thing Description 1.x, but ultimately lost the argument. We proposed a much simpler Web of Things with a single specification that specified a JSON-based Thing Description format, a REST API and WebSocket API that everyone uses (to enable ad-hoc interoperability), and that's it.

The reason this didn't happen is due to a different fundamental philosophy held by other members of the Working Group about how the Web of Things should work, which is in describing what already exists (in a protocol agnostic way), rather than defining a new protocol that devices would have to implement in order to interoperate. i.e. declarative custom protocol bindings for each Thing, rather than shared prescriptive protocol bindings shared by all Things.

The reason that Forms exist is as a protocol agnostic mechanism which can theoretically declaratively define how to invoke any operation on any existing (brownfield) connected device.

Right now forms pretty much let you build a separate protocol binding for every single affordance.

Forms actually are protocol bindings. If you look at the definition of Protocol Binding in WoT Architecture you will notice it says "W3C WoT serializes Protocol Bindings as hypermedia controls." (i.e. Forms).

Although the terms "Protocol Binding" and "Binding Template" are often incorrectly used interchangeably, a "binding template" just defines a vocabulary. An actual protocol binding is specific to a Thing and is defined using JSON Forms in a Thing Description, using one of those vocabularies.

Leverage operations and let the protocol binding decide how to transfer the information for an operation.

That is exactly what protocol bindings do, but what I think you have misunderstood is that a binding template doesn't define a protocol binding. It just defines a vocabulary for defining a protocol binding in a Thing Description (which is specific to that Thing).

I think what you are asking for is that instead of having Thing-specific protocol bindings defined in Forms, there should be common protocol bindings which are defined out-of-band and shared between all Things using a given protocol.

The good news is, there are mechanisms that make this possible - profiles, and sub-protocols - which both offer alternatives to using custom declarative protocol bindings using a binding template vocabulary in Forms:

• A Profile constrains all the extension points of a Thing Description, including the protocol binding, and defines a prescriptive protocol binding that all conformant Things must use
• The subprotocol member of a Form can be used to reference an out-of-band shared protocol binding (e.g. the Web Thing Protocol), rather than describing the protocol binding separately in every Form

Both of these mechanisms still use Forms, but the actual protocol binding is defined outside of the Thing Description in a shared specification.

The relationship between profiles and binding templates is still subject to a lot of debate, but essentially today there are two alternative ways of using the Web of Things:

1. The descriptive approach of using vocabularies from binding templates, best suited to brownfield devices
2. The prescriptive approach of using profiles (or sub-protocols), best suited to greenfield devices

The prescriptive approach doesn't work if you want to try to directly consume an existing API for a Thing which was never intended to work with the Web of Things, e.g. the REST API used by Philips Hue devices. We can't tell Philips to just change the HTTP method or payloads of their API, so the idea of the descriptive approach is to try to describe their own specific way of using HTTP instead, using Forms*.

Whilst I think it is very unlikely that the former approach will go away in WoT 2.0, it is my hope that these two approaches will work better together and will be more clearly defined.

(The shared connections proposal @relu91 cris mentions above will also help cut down on the verbosity of Thing Descriptions).

---

* I personally believe that the descriptive approach of trying to declaratively define every possible interaction of every possible connected device using every possible protocol using a single JSON file format is flawed. It provides extensibility, but not interoperability. However, I respect that other members want to use the Web of Things that way, whilst I focus on using a more prescriptive approach (e.g. using the Web Thing Protocol), with gateway adapters which bridge other protocols to that common (sub)protocol.

[benfrancis]

@relu91 wrote:

The PR is quite overwhelming to read, but basically you can have a look at this file. In particular, to the examples section.

Now merged at https://github.com/w3c/wot-thing-description/blob/main/proposals/initial-connection/README.md

[hspaay]

@relu91 says: "... wanted to mention that we are working to bake something that goes in this direction (although it is less groundbreaking). " @benfrancis says "... now merged at..."

Very interesting. That link has a very thought-out and well described problem description, unlike my own rant which bleeds my frustration all over the place.

You're both right of course, there are multiple problems at work here, one being the lack of specification on how to re-use connections, the other the need to duplicate forms in affordance in the TD.

Using thing level defaults without the need to specify repeatedly in each affordance would go a long way to bring sanity back. I do wonder if there isn't a way to eliminate the need for forms as that would simplify things a lot.

The tendency in this industry is to solve problems by adding things. I find that problems are often better solved by removing things. Sometimes it helps to take a step back and see what can be removed. The experience gained in v1.0 often provides the insights and understanding that makes this possible, as long as there is a willingness to at least take a look at it.

[hspaay]

@lu-zero sorry you don't like the proposal. Its okay, I'm sure you are not alone. It would help to understand why though. What goes wrong when eliminating forms?

[hspaay]

@benfrancis wow a very thoughful and insightful response, as usual.

I think you would like Mozilla's legacy Web Thing API. ...

Yeah you're right I like it a lot.

The reason this didn't happen is due to a different fundamental philosophy held by other members of the Working Group about how the Web of Things should work, which is in describing what already exists...
...
The reason that Forms exist is as a protocol agnostic mechanism which can theoretically declaratively define how to invoke any operation on any existing (brownfield) connected device.

These are fair points. Having to support decades of legacy right from the start is never going to be pretty. I'm more hard-core on this. Do you build for the future or for the past? Legacy support can be compartimentalized without forcing it all over the TD though.

Forms actually are protocol bindings. If you look at the definition of Protocol Binding in WoT Architecture ...
Although the terms "Protocol Binding" and "Binding Template" are often w3c/wot-binding-templates#143, a "binding template" ...

Yes you are right of course. I'm playing fast and loose with the terminology. That is not helping to bring across my point.

I think what you are asking for is that instead of having Thing-specific protocol bindings defined in Forms, there should be common protocol bindings which are defined out-of-band and shared between all Things using a given protocol.

Yes that is exactly what I'm asking for.

The good news is, there are mechanisms that make this possible - profiles, and sub-protocols - which both offer alternatives to using custom declarative protocol bindings using a binding template vocabulary in Forms:

The way the websocket (sub)protocol works is IMHO all that is needed - forms are not needed to make this work. The same approach can be applied to http-basic, mqtt and others. Hence I see Forms as making things unnecessary complex. Having multiple approaches will hinder interoperability or adoption as consumers must work with all of them.

A Profile constrains all the extension points of a Thing Description, including the protocol binding, and defines a prescriptive protocol binding that all conformant Things must use
The subprotocol member of a Form can be used to reference an out-of-band shared protocol binding (e.g. the Web Thing Protocol), rather than describing the protocol binding separately in every Form
Both of these mechanisms still use Forms, but the actual protocol binding is defined outside of the Thing Description in a shared specification.

Oof, I need to read this a few more times to understand it. In the interim I'll use it wrong I'm afraid.

...essentially today there are two alternative ways of using the Web of Things:
The prescriptive approach doesn't work if you want to try to directly consume an existing API for a Thing which was never intended to work with the Web of Things, e.g. the REST API used by Philips Hue devices. ...

Yeah well put. This is likely the crux of the matter. To support brownfield or just greenfield. It would have been better to find a way to support brownfield through a separate protocol binding (sorry when using this term wrong) without forcing it all over the TD. Directly consuming an existing API might help a few on the short term, but hinders the rest for the future by its complexity. Its like adopting a hydra as a pet.

Whilst I think it is very unlikely that the former approach will go away in WoT 2.0, it is my hope that these two approaches will work better together and will be more clearly defined. (The shared connections proposal @relu91 cris mentions above will also help cut down on the verbosity of Thing Descriptions).

Good to see there is a push in the right direction. Is WoT 2.0 allowed to break compatibility with WoT 1.x? If so then I sincerly hope that forms are retired in favor of the other proposals in the works. It would be good to have a better overview of what would break when Forms are removed so an out-of-band protocol binding/adapter alternative can be provided to support legacy devices. Keeping TDs clean and simply is key to its adoption.

I'm using a service adapter to support zwave and 1-wire protocols in hiveot and convert them to TDs. It works well and doesn't need forms. If Philips already supports TDs then the (out of band) protocol binding only needs to map the messaging for the operations.

• I personally believe that the descriptive approach of trying to declaratively define every possible interaction of every possible connected device using every possible protocol using a single JSON file format is flawed. It provides extensibility, but not interoperability. However, I respect that other members want to use the Web of Things that way, whilst I focus on using a more prescriptive approach (e.g. using the Web Thing Protocol), with gateway adapters which bridge other protocols to that common (sub)protocol.

Yes, well put. It looks like the impediment is vested interest, not technical. It serves a few members but not the community. I hope they are open to out-of-band alternatives that benefit the community now TD 1.x has provided the experience.

This proposal of removing Forms still stands. Even if it is just as a reminder it is worth striving for simplicity over complexity.

[benfrancis]

@hspaay wrote:

It would be good to have a better overview of what would break when Forms are removed so an out-of-band protocol binding/adapter alternative can be provided to support legacy devices.

Everything. Everything would break. Currently a Thing Description with no Forms is:

1. Invalid
2. Useless

The only way I can see of getting rid of Forms at this point would be if a decision was made for WoT 2.0 to only support greenfield implementations and not try to describe existing IoT devices using "legacy" protocols.

I think that directly describing existing IoT devices using existing IoT protocols is currently considered the primary use case of the Web of Things by many (if not most) Working Group members, so I don't see that happening.

Probably the best we can do is to remove as much verbosity as possible for the Thing Descriptions of Things whose developers choose to use a prescriptive profile/sub-protocol.

[lu-zero]

@lu-zero sorry you don't like the proposal. Its okay, I'm sure you are not alone. It would help to understand why though. What goes wrong when eliminating forms?

They are needed to describe the current reality, the work on allowing terse/compact representation would make them less impacting for some use-cases, but they are needed nonetheless.

[hspaay]

@benfrancis @lu-zero Okay I don't mean just leaving them out and not doing anything else. I agree that would break everything.

The distinction I make is that the TD defines what the Thing does, while the protocol binding describes how to communicate with the Thing. Forms, if you like them that much, can be injected into the TD by the protocol binding. If you choose to use that specific protocol then you adopt the Forms approach. Other protocols like the websocket binding don't need them at all to interact with the Thing once connected.

This separation of concerns allows for (dynamic) TDs that are adapted depending on how you connect to the thing. It is also both past and future proof as only the protocol binding defines how to communicate with the thing. Right now we're stuck with Forms as a mechanism for identifying communication at an amazing granularity and any future innovation has to carry this legacy along. TD's are just fine without Forms.

You might wonder, what is the difference with what we have today? The difference becomes apparent as soon as you want to support anything else than basic http. A messaging based protocol like websocket or mqtt doesn't need forms to function (except to describe connection information). These protocols do just fine describing the messages for the operations.

This proposal doesn't stand on its own. It will need the connection based improvements that are in the works.

Thank you for the willingness to discuss this and the constructive feedback. I appreciate it immensly.

[danielpeintner]

The only way I can see of getting rid of Forms at this point would be if a decision was made for WoT 2.0 to only support greenfield implementations and not try to describe existing IoT devices using "legacy" protocols.

I think this statement is true and is exactly what we should try to avoid.
Hence, I don't think we can get rid of forms.

[egekorkan]

@hspaay first of all thank you for expressing your thoughts. No matter how "offensive" you think they are, we need to hear them and see where the frustration lies.

From my understanding, the main frustration is when an ecosystem is using TDs within well-defined boundaries, e.g. all using the same message format over websocket, TDs are over engineered. You can simplify a lot when you can make assumptions about the Things. As others have pointed out, this is not the goal of this work, i.e. we want to include every thing. However, with the initial connection proposal, the assumption about the Thing's communication style will be much less verbose.

Maybe we should document why we have forms in the first place as a design decision (that was taken many years ago).

[hspaay]

Thanks @egekorkan. That pretty much sums it up.

The only way I can see of getting rid of Forms at this point would be if a decision was made for WoT 2.0 to only support greenfield implementations and not try to describe existing IoT devices using "legacy" protocols.

I think this statement is true and is exactly what we should try to avoid.
Hence, I don't think we can get rid of forms.

@benfrancis @danielpeintner Yeah I was a bit black and white with my original title. I've amended it to saying 'move out of the TD'.

It would go a long way to not have them be required in all action/event/property affordances.

[egekorkan]

Alright cool :) In that case we can close this issue and create one for explaining why we have forms