Modify the codec description model to ease describing changes

[alvestrand]

Current discussions on codecs have uncovered several issues with our current model:

• No adequate representation of send-only / receive-only codecs (setCodecPreferences vs unidirectional codecs #2888)
• Hard to describe dynamic revision of the set of codecs as needed for encoded-transform (encoded-transform #172)
• Lack of clarity in how to get codecs enabled that are not negotiated by default (webrtc-extensions #100)

This note suggests some changes that may address those issues.

Current model

As of Jan 11, 2024, the codecs are implicitly described as part of the platform implementation. Their API representation is in the form of the RTCRtpCodec dictionary and its sub-dictionaries RTCRtpCodecCapability (no additional members) and RTCRtpCodecParameters (which adds payload type).

Codecs are manipulated by setCodecPreferences, and accessed through the final steps to create an offer - where it filters against the list of implemented send codecs and the list of implemented receive codecs (depending on transceiver direction). Codec preferences are stored on transceiver, in the [[PreferredCodecs]] slot.

The final steps to create an offer or answer is not influenced by the current negotiation state, but there is a NOTE in the setCodecPreferences section suggesting that answers should only have codecs that appear in the offer.

Suggested revised model

Codecs supported, and current preferences, are stored in the sender and receiver, as an ordered list of codecs. Each entry in the list has, in addition to what’s in RTCRtpCodec, two fields:

• A boolean “enabled”
• A payload type that may or may not be present

When creating a sender or receiver, its [[codecs]] slot is initialized to the list of implemented codecs for sending / receiving. Each “enabled” flag is set in a platform dependent manner, and the associated PT is left undefined.

When calling setCodecPreferences, checking is done against receiver.[[codecs]], not against sender/receiver.getCapabilities(). If setCodecPreferences() includes a codec with the “enabled” flag set to false in the receiver’s [[codecs]] slot, it is set to “true”.

When generating an offer or answer, all “enabled” codecs are included in the offer, and the receiver’s PT values are set to the offered values. For sendonly media sections, the initial codecs are taken from the sender; for recvonly media sections, the initial codecs are taken from the receiver; for sendrecv, the union of sender and receiver is used.

When applying a remote offer or answer, the sender’s PT values are set.

If a codec in a remote offer or answer is on the supported list, but “enabled” is false, “enabled” is set to true. This ensures that the codec is included in subsequent offers and answers. (Note: This is the only part of this reimagining that might include a behavior change, but we suspect that this is the way current implementations behave anyway.)

Possible extensions that are easier in the revised model

These changes need to go into relevant documents until implemented, not in webrtc-pc.

Adding to the set of supported codecs

This functionality is needed for encoded transform, which may generate payload types that are not natively supported by the platform. The transform acts as a codec. On a receiver transform, attempting to enqueue a frame results in an error.
This can be done by adding to the sender and receiver’s [[codecs]] list. Potentially this can be done by an API call, or by picking up on input/output codec info on a transform.

Enabling a disabled codec through API

An API call can be added on sender and on the receiver to get the current codec list including disabled codecs. This can then be put back into setCodecPreferences() which would enable it in offers for recvonly or sendrecv sections. This is nicer if “enabled” is exposed.

Preventing a codec from being negotiated

Still tough. setCodecPreferences does NOT do this. This is intentional, see this WPT

Supporting sendonly codecs

This is easily done by having the codec in the sender’s list only. It will then be offered on sendonly and sendrecv sections. Sendonly codecs are not available to setCodecPreferences().

Supporting recvonly codecs

As above, but offered only on recvonly and sendrecv sections.

[fippo]

The current situation is indeed not good. When adding codecs that are rarely used like H264 with 444 they get added to the SDP, consuming a payload type and increase the size of the SDP (and remember that we had a comparison between the WebRTC SDP and the Empire State building in one slide deck).

setCodecPreferences can be (somewhat) used to limit this but it requires action.

[alvestrand]

If we allow setParameters() to enable a disabled codec (https://w3c.github.io/webrtc-extensions/#dom-rtcrtpencodingparameters-codec), the only missing piece is discoverability.
And in simulcast, we deferred discoverability to MediaCapabilities (decodingInfo with "webrtc" - ask, and you shall learn). If we were to address the bug asking for CodecCapability, perhaps we have all the pieces?

[alvestrand]

I'll ask for this issue to be discussed at the February WG meeting, since I feel that doing this will make resolving our other issues easier, but want to have WG consensus on the approach before preparing the PR.

[henbos]

Moving the check from the checking a global state to checking a sender/receiver state, in order to unblock changing sender/receiver capabilities, makes sense to me.

If a codec in a remote offer or answer is on the supported list, but “enabled” is false, “enabled” is set to true. This ensures that the codec is included in subsequent offers and answers. (Note: This is the only part of this reimagining that might include a behavior change, but we suspect that this is the way current implementations behave anyway.)

I think we need to do this for unidirectional codecs anyway? Hopefully we already do it

The final steps to create an offer or answer is not influenced by the current negotiation state, but there is a NOTE in the setCodecPreferences section suggesting that answers should only have codecs that appear in the offer.

I think this NOTE is wrong and should be deleted (#2933). I also think there is some major confusion about a=rtpmap versus preferences (at least for me, but maybe it's crystal clear for everyone else), so I filed #2932. I also think that Fippo's PR needs to remove this sentence from createAnswer. Does that make sense to everyone or am I still confused?

[dontcallmedom-bot]

This issue was discussed in WebRTC February 2024 meeting – 20 February 2024 (Modify the codec description model to ease describing changes #2925 #2935)