Add SFrame packetization handling for SFrameTransform

youennf · youennf · commit f580cb7dde2b · 2025-04-30T12:31:40.000+02:00
Introduce RTCRtpScriptTransformType with "sframe" value.
diff --git a/index.bs b/index.bs
@@ -138,6 +138,7 @@ The <dfn abstract-op>readEncodedData</dfn> algorithm is given a |rtcObject| as p
 1. Increment |rtcObject|.`[[lastEnqueuedFrameCounter]]` by <code>1</code>.
 1. Let |frame| be the newly produced frame.
 1. Set |frame|.`[[owner]]` to |rtcObject|.
+1. Set |this|.`[[writable]]` to |this|.`[[transform]]`.`[[writable]]`.
 1. Set |frame|.`[[counter]]` to |rtcObject|.`[[lastEnqueuedFrameCounter]]`.
 1. If the frame has been produced by a {{RTCRtpReceiver}}:
     1. If the relevant RTP packet contains the
@@ -150,6 +151,7 @@ The <dfn abstract-op>readEncodedData</dfn> algorithm is given a |rtcObject| as p
         [[RTP-EXT-CAPTURE-TIME#timestamp-interpolation|timestamp interpolation]] and set |frame|.`[[senderCaptureTimeOffset]]`
         to the most recent value that was present.
     1. Otherwise, set |frame|.`[[captureTime]]` to undefined and set |frame|.`[[senderCaptureTimeOffset]]` to undefined.
+    1. If |frame| was produced by a [=SFrame depacketizer=], set |frame|.`[[useSFrame]]` to true.
 1. If the frame has been produced by a {{RTCRtpSender}}, set |frame|.`[[captureTime]]` to the capture timestamp
     using the methodology described in [[RTP-EXT-CAPTURE-TIME#absolute-capture-timestamp]] and set frame.`[[senderCaptureTimeOffset]]`
     to undefined.
@@ -162,8 +164,10 @@ The <dfn abstract-op>writeEncodedData</dfn> algorithm is given a |rtcObject| as
 1. Let |data| be |frame|.`[[data]]`.
 1. Let |serializedFrame| be [$StructuredSerializeWithTransfer$](|frame|, « |data| »).
 1. Let |frameCopy| be [$StructuredDeserializeWithTransfer$](|serializedFrame|, |frame|'s [=relevant realm=]).
+1. If |frame|.`[[useSFrame]]` is true, set |frameCopy|.`[[useSFrame]]` to true.
 1. Enqueue |frameCopy| for processing as if it came directly from the encoded data source, by running one of the following steps:
     * If |rtcObject| is a {{RTCRtpSender}}, enqueue |frameCopy| to |rtcObject|'s packetizer, to be processed [=in parallel=].
+        If |frameCopy|.`[[useSFrame]]` is true, |rtcObject|'s MUST use a [=SFrame packetizer=] or skip processing of |frameCopy|.
     * If |rtcObject| is a {{RTCRtpReceiver}}, enqueue |frameCopy| it to |rtcObject|'s decoder, to be processed [=in parallel=].
 1. Return [=a promise resolved with=] undefined.
 
@@ -177,26 +181,40 @@ As [$writeEncodedData$] ensures that the transform cannot reorder frames, this w
 
 ## Extension attribute ## {#attribute}
 
-A RTCRtpTransform has two private slots called `[[readable]]` and `[[writable]]`.
+A RTCRtpTransform has the following private slots:
+1. `[[readable]]` of type {{ReadableStream}}.
+1. `[[writable]]` of type {{WritableStream}}.
+1. `[[owner]]` of type {{RTCRtpSender}} or {{RTCRtpReceiver}}.
+1. `[[useSFrame]]` of type boolean.
 
 Each RTCRtpTransform has an <dfn abstract-op for=RTCRtpTransform>association steps</dfn> set, which is empty by default.
 
 The <dfn attribute for="RTCRtpSender,RTCRtpReceiver">transform</dfn> getter steps are:
 1. Return [=this=].`[[transform]]`.
 
 The `transform` setter steps are:
-2. Let |transform| be the argument to the setter.
-3. Let |checkedTransform| set to |transform| if it is not null or to an [=identity transform stream=] otherwise.
-3. Let |reader| be the result of [=ReadableStream/getting a reader=] for |checkedTransform|.`[[readable]]`.
-4. Let |writer| be the result of [=WritableStream/getting a writer=] for |checkedTransform|.`[[writable]]`.
-5. Initialize |newPipeToController| to a new {{AbortController}}.
-6. If [=this=].`[[pipeToController]]` is not null, run the following steps:
+1. Let |transform| be the argument to the setter.
+1. Let |checkedTransform| set to |transform| if it is not null or to an [=identity transform stream=] otherwise.
+1. Let |reader| be the result of [=ReadableStream/getting a reader=] for |checkedTransform|.`[[readable]]`.
+1. Let |writer| be the result of [=WritableStream/getting a writer=] for |checkedTransform|.`[[writable]]`.
+1. Initialize |newPipeToController| to a new {{AbortController}}.
+1. If [=this=].`[[pipeToController]]` is not null, run the following steps:
     1. [=AbortSignal/Add=] the [$chain transform algorithm$] to [=this=].`[[pipeToController]]`'s [=AbortController/signal=].
     2. [=AbortController/signal abort=] on [=this=].`[[pipeToController]]`.
-7. Else, run the [$chain transform algorithm$] steps.
-8. Set [=this=].`[[pipeToController]]` to |newPipeToController|.
-9. Set [=this=].`[[transform]]` to |transform|.
-10. Run the steps in the set of [$association steps$] of |transform| with [=this=].
+1. Else, run the [$chain transform algorithm$] steps.
+1. If [=this=] is a {{RTCRtpSender}}, run the following substeps:
+    1. Let |useSFrame| be true if [=this=]  is configured to use a [=SFrame packetizer=] and false otherwise.
+    1. If |useSFrame| is equal to |checkedTransform|.`[[useSFrame]]`, abort these substeps.
+    1. Configure [=this=]'s packetizer to use SFrame if |checkedTransform|.`[[useSFrame]]` is true and to not use SFrame if |checkedTransform|.`[[useSFrame]]` is false.
+    1. [=Update the negotiation-needed flag=] for [=this=]'s connection.
+1. Otherwise, run the following steps:
+    1. Let |useSFrame| be true if [=this=] is configured to use a [=SFrame depacketizer=] and false otherwise.
+    1. If |useSFrame| is equal to |checkedTransform|.`[[useSFrame]]`, abort these substeps.
+    1. Configure [=this=]'s depacketizer to use SFrame if |checkedTransform|.`[[useSFrame]]` is true and to not use SFrame if |checkedTransform|.`[[useSFrame]]` is false.
+    1. [=Update the negotiation-needed flag=] for [=this=]'s connection.
+1. Set [=this=].`[[pipeToController]]` to |newPipeToController|.
+1. Set [=this=].`[[transform]]` to |transform|.
+1. Run the steps in the set of [$association steps$] of |transform| with [=this=].
 
 The <dfn abstract-op>chain transform algorithm</dfn> steps are defined as:
 1. If |newPipeToController|'s [=AbortController/signal=] is [=AbortSignal/aborted=], abort these steps.
@@ -243,7 +261,8 @@ SFrameTransform includes GenericTransformStream;
 enum SFrameTransformErrorEventType {
     "authentication",
     "keyID",
-    "syntax"
+    "syntax",
+    "packetization"
 };
 
 [Exposed=(Window,DedicatedWorker)]
@@ -270,13 +289,19 @@ The <dfn constructor for="SFrameTransform" lt="SFrameTransform(options)"><code>n
 5. Set |this|.`[[role]]` to |options|["{{SFrameTransformOptions/role}}"].
 6. Set |this|.`[[readable]]` to |this|.`[[transform]]`.`[[readable]]`.
 7. Set |this|.`[[writable]]` to |this|.`[[transform]]`.`[[writable]]`.
+7. Set |this|.`[[useSFrame]]` to true.
 
 ## Algorithm ## {#sframe-transform-algorithm}
 
 The SFrame transform algorithm, given |sframe| as a SFrameTransform object and |frame|, runs these steps:
 1. Let |role| be |sframe|.`[[role]]`.
-1. If |frame|.`[[owner]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'.
-1. If |frame|.`[[owner]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'.
+1. If |sframe|.`[[owner]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'.
+1. If |sframe|.`[[owner]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'.
+1. If |sframe|.`[[owner]]` is a {{RTCRtpReceiver}} and |frame|.`[[useSFrame]]` is not true, [=queue a task=] to run the following steps:
+    1. [=fire an event=] named {{SFrameTransform/onerror|error}} at |sframe|,
+        using the {{SFrameTransformErrorEvent}} interface with its {{SFrameTransformErrorEvent/errorType}} attribute set to {{SFrameTransformErrorEventType/packetization}}
+        and its {{SFrameTransformErrorEvent/frame}} attribute set to |frame|.
+    1. Abort these steps.
 1. Let |data| be undefined.
 1. If |frame| is a {{BufferSource}}, set |data| to |frame|.
 1. If |frame| is a {{RTCEncodedAudioFrame}}, set |data| to |frame|.{{RTCEncodedAudioFrame/data}}
@@ -297,6 +322,7 @@ The SFrame transform algorithm, given |sframe| as a SFrameTransform object and |
 1. If |frame| is a {{BufferSource}}, set |frame| to |buffer|.
 1. If |frame| is a {{RTCEncodedAudioFrame}}, set |frame|.{{RTCEncodedAudioFrame/data}} to |buffer|.
 1. If |frame| is a {{RTCEncodedVideoFrame}}, set |frame|.{{RTCEncodedVideoFrame/data}} to |buffer|.
+1. Set |frame|.`[[useSFrame]]` to true.
 1. [=ReadableStream/Enqueue=] |frame| in |sframe|.`[[transform]]`.
 
 ## Methods ## {#sframe-transform-methods}
@@ -309,6 +335,27 @@ The <dfn method for="SFrameTransform">setEncryptionKey(|key|, |keyID|)</dfn> met
     3. [=Resolve=] |promise| with undefined.
 4. Return |promise|.
 
+## SFrame packetization integration ## {#sframe-packetization}
+
+A <dfn>SFrame packetizer</dfn> is responsible to generate SFrame packets from media content.
+In the context of this specification, the [=SFrame packetizer=] is not responsible for doing the actual encryption.
+Instead, the transform is responsible for doing so. The [=SFrame packetizer=] is responsible for splitting
+SFrame frames as needed so that they fit in RTP packets.
+
+Similarly, a <dfn>SFrame depacketizer</dfn> is responsible to assemble RTP packets into a complete SFrame frame.
+It is not responsible for doing the actual decryption, the transform is responsible for doing so.
+
+WebRTC encoded transform model is a per frame processing. SFrame can either be applied on each frame or on subframes.
+WebRTC encoded transform model is naturally aligned with aplying SFrame on a frame as a whole.
+To preserve WebRTC encoded transform model when applying SFrame on subframes, the following conceptual steps can be done:
+1. On sending side, the {{SFrameTransform}} may first split the media frame in subframes like would do a regular media packetizer,
+    and apply the SFrame encryption on each subframe. It then concatenates the encrypted subframes as a unique encrypted frame. 
+    The transform provides the encrypted frame and information of each subframe so that the [=SFrame packetizer=]
+    generates individual packets for each subframe.
+2. On receiving side, the [=SFrame depacketizer=] assembles all individual subframe RTP packets as a unique encrypted frame.
+    It is responsible to give the necessary subframe information to the transform so that the transform can apply the SFrame decryption
+    on each individual subframe contained in the unique encrypted frame and concatenate each decrypted subframe as a unique decrypted media frame.
+    If decryption of a single subframe fails, the whole encrypted frame is discarded.
 
 # RTCRtpScriptTransform # {#scriptTransform}
 
@@ -942,9 +989,13 @@ interface RTCRtpScriptTransformer : EventTarget {
     readonly attribute any options;
 };
 
+enum RTCRtpScriptTransformType {
+    "sframe"
+};
+
 [Exposed=Window]
 interface RTCRtpScriptTransform {
-    constructor(Worker worker, optional any options, optional sequence&lt;object&gt; transfer);
+    constructor(Worker worker, optional any options, optional sequence&lt;object&gt; transfer, optional RTCRtpScriptTransformType type);
 };
 
 [Exposed=DedicatedWorker]
@@ -956,15 +1007,16 @@ interface KeyFrameRequestEvent : Event {
 
 ## Operations ## {#RTCRtpScriptTransform-operations}
 
-The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worker, options)"><code>new RTCRtpScriptTransform(|worker|, |options|, |transfer|)</code></dfn> constructor steps are:
+The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worker, options)"><code>new RTCRtpScriptTransform(|worker|, |options|, |transfer|, |type|)</code></dfn> constructor steps are:
 1. Set |t1| to an [=identity transform stream=].
-2. Set |t2| to an [=identity transform stream=].
-3. Set |this|.`[[writable]]` to |t1|.`[[writable]]`.
-4. Set |this|.`[[readable]]` to |t2|.`[[readable]]`.
-5. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|).
-6. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »).
-7. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »).
-8. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
+1. Set |t2| to an [=identity transform stream=].
+1. Set |this|.`[[writable]]` to |t1|.`[[writable]]`.
+1. Set |this|.`[[readable]]` to |t2|.`[[readable]]`.
+1. If |type| is equal to {{RTCRtpScriptTransformType/"sframe"}}, set |this|.`[[useSFrame]]` to true.
+1. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|).
+1. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »).
+1. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »).
+1. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
     1. Let |transformerOptions| be the result of [$StructuredDeserializeWithTransfer$](|serializedOptions|, the current Realm).
     2. Let |readable| be the result of [$StructuredDeserializeWithTransfer$](|serializedReadable|, the current Realm).
     3. Let |writable| be the result of [$StructuredDeserializeWithTransfer$](|serializedWritable|, the current Realm).
