diff --git a/index.bs b/index.bs index 7d43920..4ab70f8 100644 --- a/index.bs +++ b/index.bs @@ -91,37 +91,44 @@ argument, ensure that the codec is disabled and produces no output. ### Stream creation ### {#stream-creation} +Each {{RTCRtpSender}} or {{RTCRtpReceiver}} has its own media thread on which media flows, +from a source on which to read encoded data to a sink on which to write encoded data. +The source is the packetizer for {{RTCRtpSender}} and the decoder for {{RTCRtpReceiver}}. +It operates in the [=media thread=] and is modeled as a {{ReadableStream}}. +The sink is the encoder for {{RTCRtpSender}} and the depacketizer for {{RTCRtpReceiver}}. +It operates in the [=media thread=] and is modeled as a {{WritableStream}}. + At construction of each {{RTCRtpSender}} or {{RTCRtpReceiver}}, run the following steps: -2. Initialize [=this=].`[[transform]]` to null. -3. Initialize [=this=].`[[readable]]` to a new {{ReadableStream}}. -4. Set up [=this=].`[[readable]]`. [=this=].`[[readable]]` is provided frames using the [=readEncodedData=] algorithm given |this| as parameter. -5. Set [=this=].`[[readable]]`.`[[owner]]` to |this|. -6. Initialize [=this=].`[[writable]]` to a new {{WritableStream}}. -7. Set up [=this=].`[[writable]]` with its [=WritableStream/set up/writeAlgorithm=] set to [=writeEncodedData=] given |this| as parameter. -8. Set [=this=].`[[writable]]`.`[[owner]]` to |this|. -9. Initialize [=this=].`[[pipeToController]]` to null. -10. Initialize [=this=].`[[lastReceivedFrameTimestamp]]` to zero. -11. [=Queue a task=] to run the following steps: - 1. If [=this=].`[[pipeToController]]` is not null, abort these steps. - 2. Set [=this=].`[[pipeToController]]` to a new {{AbortController}}. - - 3. Call pipeTo with [=this=].`[[readable]]`, [=this=].`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to true and [=this=].`[[pipeToController]]`.signal. +1. Initialize [=this=].`[[transform]]` to null. +1. Initialize [=this=].`[[lastReceivedFrameTimestamp]]` to zero. +1. Initialize [=this=].`[[pipeToController]]` to null. +1. Initialize [=this=].`[[source]]` to a new {{ReadableStream}}. +1. Set up [=this=].`[[source]]`. [=this=].`[[source]]` is provided frames using the [=readSourceData=] algorithm given |this| as parameter. +1. Initialize [=this=].`[[sink]]` to a new {{WritableStream}}. +1. Set up [=this=].`[[sink]]` with its [=WritableStream/set up/writeAlgorithm=] set to [=writeSinkData=] given |this| as parameter. +1. [=Queue a task=] to run the following step: + 1. [=Queue a task=] on [=this=]'s media thread to run the following steps: + 1. If [=this=].`[[pipeToController]]` is not null, abort these steps. + 1. Set [=this=].`[[pipeToController]]` to a new {{AbortController}}. + + 1. Call pipeTo with [=this=].`[[source]]`, + [=this=].`[[sink]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to true and [=this=].`[[pipeToController]]`.signal. ### Stream processing ### {#stream-processing} -The readEncodedData algorithm is given a |rtcObject| as parameter. It is defined by running the following steps: +The readSourceData algorithm given |rtcObject| runs the following steps in the [=media thread=]: 1. Wait for a frame to be produced by |rtcObject|'s encoder if it is a {{RTCRtpSender}} or |rtcObject|'s packetizer if it is a {{RTCRtpReceiver}}. -2. Let |frame| be the newly produced frame. -3. Set |frame|.`[[owner]]` to |rtcObject|. -4. [=ReadableStream/Enqueue=] |frame| in |rtcObject|.`[[readable]]`. +1. Let |frame| be the newly produced frame. +1. Set |frame|.`[[owner]]` to |rtcObject|. +1. [=ReadableStream/Enqueue=] |frame| in |rtcObject|.`[[source]]`. -The writeEncodedData algorithm is given a |rtcObject| as parameter and a |frame| as input. It is defined by running the following steps: +The writeSinkData algorithm given |rtcObject| and |frame| runs the following steps in the [=media thread=]: 1. If |frame|.`[[owner]]` is not equal to |rtcObject|, abort these steps and return [=a promise resolved with=] undefined. A processor cannot create frames, or move frames between streams. -2. If the |frame|'s {{RTCEncodedVideoFrame/timestamp}} is equal to or larger than |rtcObject|.`[[lastReceivedFrameTimestamp]]`, abort these steps and return [=a promise resolved with=] undefined. A processor cannot reorder frames, although it may delay them or drop them. -3. Set |rtcObject|.`[[lastReceivedFrameTimestamp]]` to the |frame|'s {{RTCEncodedVideoFrame/timestamp}}. -4. Enqueue the frame for processing as if it came directly from the encoded data source, by running one of the following steps: - * If |rtcObject| is a {{RTCRtpSender}}, enqueue it to |rtcObject|'s packetizer, to be processed [=in parallel=]. - * If |rtcObject| is a {{RTCRtpReceiver}}, enqueue it to |rtcObject|'s decoder, to be processed [=in parallel=]. +1. If the |frame|'s {{RTCEncodedVideoFrame/timestamp}} is equal to or larger than |rtcObject|.`[[lastReceivedFrameTimestamp]]`, abort these steps and return [=a promise resolved with=] undefined. A processor cannot reorder frames, although it may delay them or drop them. +1. Set |rtcObject|.`[[lastReceivedFrameTimestamp]]` to the |frame|'s {{RTCEncodedVideoFrame/timestamp}}. +1. Enqueue the frame for processing as if it came directly from the encoded data source, by running one of the following steps in the [=media thread=]: + * If |rtcObject| is a {{RTCRtpSender}}, enqueue it to |rtcObject|'s packetizer. + * If |rtcObject| is a {{RTCRtpReceiver}}, enqueue it to |rtcObject|'s decoder. 5. Return [=a promise resolved with=] undefined. ## Extension attribute ## {#attribute} @@ -132,29 +139,33 @@ The transform getter step 1. Return [=this=].`[[transform]]`. The `transform` setter steps are: -2. Let |transform| be the argument to the setter. -3. Let |checkedTransform| set to |transform| if it is not null or to an [=identity transform stream=] otherwise. -3. Let |reader| be the result of [=ReadableStream/getting a reader=] for |checkedTransform|.`[[readable]]`. -4. Let |writer| be the result of [=WritableStream/getting a writer=] for |checkedTransform|.`[[writable]]`. -5. Initialize |newPipeToController| to a new {{AbortController}}. -6. If [=this=].`[[pipeToController]]` is not null, run the following steps: - 1. [=AbortSignal/Add=] the [=chain transform algorithm=] to [=this=].`[[pipeToController]]`.signal. - 2. [=AbortSignal/signal abort=] [=this=].`[[pipeToController]]`.signal. -7. Else, run the [=chain transform algorithm=] steps. -8. Set [=this=].`[[pipeToController]]` to |newPipeToController|. -9. Set [=this=].`[[transform]]` to |transform|. - -The chain transform algorithm steps are defined as: -1. If |newPipeToController|'s [=AbortSignal/aborted flag=] is true, abort these steps. -2. [=ReadableStreamDefaultReader/Release=] |reader|. -3. [=WritableStreamDefaultWriter/Release=] |writer|. -4. Assert that |newPipeToController| is the same object as |rtcObject|.`[[pipeToController]]`. - -5. Call pipeTo with |rtcObject|.`[[readable]]`, |checkedTransform|.`[[writable]]`, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |newPipeToController|.signal. -6. Call pipeTo with |checkedTransform|.`[[readable]]`, |rtcObject|.`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |newPipeToController|.signal. +1. Let |transform| be the argument to the setter. +1. If |transform| is not null, run the following steps: + 1. If |transform|.`[[readable]]` is [=ReadableStream/locked=], [=throw=] a {{TypeError}}. + 1. [=WritableStream/getting a writer|Get a writer=] for |transform|.`[[writable]]`. + 1. [=ReadableStream/getting a reader|Get a reader=] for |transform|.`[[readable]]`. +1. [=Queue a task=] in [=this=]'s [=media thread=] to run the following steps: + 1. let |pipeToController| be a new {{AbortController}}. + 1. Let |internalTransform| be an [=identity transform stream=]. + 1. If |transform| is an {{SFrameTransform}}, set |internalTransform| to an [=SFrame transform stream=] given |transform|. + 1. If |transform| is an {{RTCRtpScriptTransform}}, set |internalTransform| to a [=script transform stream=] given |transform|. + 1. Run the [=chain transform algorithm=] with [=this=], |internalTransform|.`[[readable]]`, |internalTransform|.`[[writable]]` and |pipeToController|. +1. Set [=this=].`[[transform]]` to |transform|. + +The chain transform algorithm, given |rtcObject|, |readable|, |writable| and |pipeToController|, runs these steps in rtcObject's [=media thread=]: +1. If |pipeToController| [=AbortSignal/aborted flag=] is true, abort these steps. +1. If |rtcObject|.`[[pipeToController]]` is not null, run the following steps: + 1. [=AbortSignal/Add=] the [=chain transform algorithm=] with |rtcObject|, |readable|, |writable| and |pipeToController|, to |rtcObject|.`[[pipeToController]]`.signal. + 1. [=AbortSignal/signal abort=] |rtcObject|.`[[pipeToController]]`.signal. +1. Else run the following steps: + + 1. Call pipeTo with |rtcObject|.`[[source]]`, |writable|, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |pipeToController|.signal. + 1. Call pipeTo with |readable|, |rtcObject|.`[[sink]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |pipeToController|.signal. +1. Set |rtcObject|.`[[pipeToController]]` to |pipeToController|. This algorithm is defined so that transforms can be updated dynamically. There is no guarantee on which frame will happen the switch from the previous transform to the new transform. +If a new transform overwrites an old transform, all frames will go either through the old or the new transform from the source to the sink. If a web application sets the transform synchronously at creation of the {{RTCRtpSender}} (for instance when calling addTrack), the transform will receive the first frame generated by the {{RTCRtpSender}}'s encoder. Similarly, if a web application sets the transform synchronously at creation of the {{RTCRtpReceiver}} (for instance when calling addTrack, or at track event handler), the transform will receive the first full frame generated by the {{RTCRtpReceiver}}'s packetizer. @@ -183,40 +194,43 @@ SFrameTransform includes GenericTransformStream; The new SFrameTransform(options) constructor steps are: -1. Let |transformAlgorithm| be an algorithm which takes a |frame| as input and runs the SFrame transform algorithm with |this| and |frame|. -2. Set |this|.`[[transform]]` to a new {{TransformStream}}. -3. Set up [=this=].`[[transform]]` with [=TransformStream/set up/transformAlgorithm=] set to |transformAlgorithm|. -4. Let |options| be the method's first argument. -5. Set |this|.`[[role]]` to |options|["{{SFrameTransformOptions/role}}"]. -6. Set |this|.`[[readable]]` to |this|.`[[transform]]`.`[[readable]]`. -7. Set |this|.`[[writable]]` to |this|.`[[transform]]`.`[[writable]]`. - -## Algorithm ## {#sframe-transform-algorithm} - -The SFrame transform algorithm, given |sframe| as a SFrameTransform object and |frame|, runs these steps: -1. Let |role| be |sframe|.`[[role]]`. -2. If |frame|.`[[owner]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'. -3. If |frame|.`[[owner]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'. -4. Let |data| be undefined. -5. If |frame| is a {{BufferSource}}, set |data| to |frame|. -6. If |frame| is a {{RTCEncodedAudioFrame}}, set |data| to |frame|.{{RTCEncodedAudioFrame/data}} -7. If |frame| is a {{RTCEncodedVideoFrame}}, set |data| to |frame|.{{RTCEncodedVideoFrame/data}} -8. If |data| is undefined, abort these steps. -9. Let |buffer| be the result of running the SFrame algorithm with |data| and |role| as parameters. This algorithm is defined by the SFrame specification and returns an {{ArrayBuffer}}. -10. If |frame| is a {{BufferSource}}, set |frame| to |buffer|. -11. If |frame| is a {{RTCEncodedAudioFrame}}, set |frame|.{{RTCEncodedAudioFrame/data}} to |buffer|. -12. If |frame| is a {{RTCEncodedVideoFrame}}, set |frame|.{{RTCEncodedVideoFrame/data}} to |buffer|. -13. [=ReadableStream/Enqueue=] |frame| in |sframe|.`[[transform]]`. +1. Set |this|.`[[role]]` to |options|["{{SFrameTransformOptions/role}}"]. +1. Let |sframeTransform| be a [=SFrame transform stream=] given |this|. +1. Set |this|.`[[readable]]` to |sframeTransform|.`[[readable]]`. +1. Set |this|.`[[writable]]` to |sframeTransform|.`[[writable]]`. + +## Algorithms ## {#sframe-transform-algorithms} + +A SFrame transform stream given |transform| is created by running the following steps: +1. Let |transformAlgorithm| be an algorithm which takes a |frame| as input and runs the [=SFrame transform algorithm=] with |transform| and |frame|. +1. Set |transform|.`[[sframeTransform]]` to a new {{TransformStream}}. +1. [=TransformStream/Set up=] |transform|.`[[sframeTransform]]` with [=TransformStream/set up/transformAlgorithm=] set to |transformAlgorithm|. +1. Return |sframeTransform|. + +The SFrame transform algorithm, given |sframeTransform| and |frame|, runs these steps: +1. Let |role| be |sframeTransform|.`[[role]]`. +1. If |frame|.`[[owner]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'. +1. If |frame|.`[[owner]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'. +1. Let |data| be undefined. +1. If |frame| is a {{BufferSource}}, set |data| to |frame|. +1. If |frame| is a {{RTCEncodedAudioFrame}}, set |data| to |frame|.{{RTCEncodedAudioFrame/data}} +1. If |frame| is a {{RTCEncodedVideoFrame}}, set |data| to |frame|.{{RTCEncodedVideoFrame/data}} +1. If |data| is undefined, abort these steps. +1. Let |buffer| be the result of running the SFrame algorithm with |data| and |role| as parameters. This algorithm is defined by the SFrame specification and returns an {{ArrayBuffer}}. +1. If |frame| is a {{BufferSource}}, set |frame| to |buffer|. +1. If |frame| is a {{RTCEncodedAudioFrame}}, set |frame|.{{RTCEncodedAudioFrame/data}} to |buffer|. +1. If |frame| is a {{RTCEncodedVideoFrame}}, set |frame|.{{RTCEncodedVideoFrame/data}} to |buffer|. +1. [=ReadableStream/Enqueue=] |frame| in |sframeTransform|.`[[transform]]`. ## Methods ## {#sframe-transform-methods} The setEncryptionKey(|key|, |keyID|) method steps are: 1. Let |promise| be [=a new promise=]. -2. If |keyID| is a {{bigint}} which cannot be represented as a integer between 0 and 264-1 inclusive, [=reject=] |promise| with a {{RangeError}} exception. -3. Otherwise, [=in parallel=], run the following steps: - 1. Set |key| with its optional |keyID| as key material to use for the SFrame transform algorithm, as defined by the SFrame specification. - 2. If setting the key material fails, [=reject=] |promise| with an {{InvalidModificationError}} exception and abort these steps. - 3. [=Resolve=] |promise| with undefined. -4. Return |promise|. +1. If |keyID| is a {{bigint}} which cannot be represented as a integer between 0 and 264-1 inclusive, [=reject=] |promise| with a {{RangeError}} exception. +1. Otherwise, [=in parallel=], run the following steps: + 1. Set |key| with its optional |keyID| as key material to use for the [=SFrame transform algorithm=], as defined by the SFrame specification. + 1. If setting the key material fails, [=reject=] |promise| with an {{InvalidModificationError}} exception and abort these steps. + 1. [=Resolve=] |promise| with undefined. +1. Return |promise|. # RTCRtpScriptTransform # {#scriptTransform} @@ -289,30 +303,39 @@ interface RTCRtpScriptTransform { ## Operations ## {#RTCRtpScriptTransform-operations} - The new RTCRtpScriptTransform(|worker|, |options|, |transfer|) constructor steps are: -1. Set |t1| to an [=identity transform stream=]. -2. Set |t2| to an [=identity transform stream=]. -3. Set |this|.`[[writable]]` to |t1|.`[[writable]]`. -4. Set |this|.`[[readable]]` to |t2|.`[[readable]]`. -5. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|). -6. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »). -7. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »). -8. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps: +1. Set |this|.`[[worker]]` to |worker|. +1. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|). +1. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps: + 1. Let transformer be a new {{RTCRtpScriptTransformer}}. 1. Let |transformerOptions| be the result of [$StructuredDeserialize$](|serializedOptions|, the current Realm). - 2. Let |readable| be the result of [$StructuredDeserialize$](|serializedReadable|, the current Realm). - 3. Let |writable| be the result of [$StructuredDeserialize$](|serializedWritable|, the current Realm). - 4. Let |transformer| be a new {{RTCRtpScriptTransformer}}. - 5. Set |transformer|.`[[options]]` to |transformerOptions|. - 6. Set |transformer|.`[[readable]]` to |readable|. - 7. Set |transformer|.`[[writable]]` to |writable|. - 8. Let |event| be the result of [=creating an event=] with {{RTCTransformEvent}}. - 9. Set |event|.type attribute to "rtctransform". - 10. Set |event|.transformer to |transformer|. - 11. Dispatch |event| on |worker|’s global scope. + 1. Set |transformer|.`[[options]]` to |transformerOptions|. + 1. Let |transformer|.`[[t1]]` to an [=identity transform stream=]. + 1. Let |transformer|.`[[t2]]` to an [=identity transform stream=]. + 1. Set |transformer|.`[[readable]]` to |transformer|.`[[t1]]`.`[[readable]]`. + 1. Set |transformer|.`[[writable]]` to |transformer|.`[[t2]]`.`[[writable]]`. + 1. Let |event| be the result of [=creating an event=] with {{RTCTransformEvent}}. + 1. Set |event|.type attribute to "rtctransform". + 1. Set |event|.transformer to |transformer|. + 1. Set |this|.`[[transformer]]` to |transformer|. + 1. [=Dispatch=] |event| to |worker|’s global scope. // FIXME: Describe error handling (worker closing flag true at RTCRtpScriptTransform creation time. And worker being terminated while transform is processing data). +A script transform stream given |transform| is created by running the following steps: +1. Set |t1| to an [=identity transform stream=]. +1. Set |t2| to an [=identity transform stream=]. +1. Set |scriptTransform|.`[[writable]]` to |t1|.`[[writable]]`. +1. Set |scriptTransform|.`[[readable]]` to |t2|.`[[readable]]`. +1. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »). +1. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »). +1. [=Queue a task=] on the DOM manipulation [=task source=] |transform|.`[[worker]]`'s global scope to run the following steps: + 1. Let |readable| be the result of [$StructuredDeserialize$](|serializedReadable|, the current Realm). + 1. Let |writable| be the result of [$StructuredDeserialize$](|serializedWritable|, the current Realm). + 1. Call pipeTo with |readable|, |transform|.`[[transformer]]`.`[[t1]]`.`[[writable]]`. + 1. Call pipeTo with |transform|.`[[transformer]]`.`[[t2]]`.`[[readable]]` and |writable|. +1. Return |scriptTransform|. + ## Attributes ## {#RTCRtpScriptTransformer-attributes} A RTCRtpScriptTransformer has three private slots called `[[options]]`, `[[readable]]` and `[[writable]]`.