Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor spec to introduce media thread #107

Closed
wants to merge 1 commit into from
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
209 changes: 116 additions & 93 deletions index.bs
Original file line number Diff line number Diff line change
@@ -91,37 +91,44 @@ argument, ensure that the codec is disabled and produces no output.

### Stream creation ### {#stream-creation}

Each {{RTCRtpSender}} or {{RTCRtpReceiver}} has its own <dfn>media thread</dfn> on which media flows,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is not right; in the non-transform case, the media thread on which media flows is not the thread that is visible to JS.

I'll suggest alternate wording (after the meeting).

from a source on which to read encoded data to a sink on which to write encoded data.
The source is the packetizer for {{RTCRtpSender}} and the decoder for {{RTCRtpReceiver}}.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems backwards. For Sender, media comes from the encoder and goes to the packetizer; for the Receiver, media comes from the depacketizer and goes to the decoder.

Either I am misunderstanding line 95 or I'm misunderstanding lines 96+98.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it needs to be reversed.

It operates in the [=media thread=] and is modeled as a {{ReadableStream}}.
The sink is the encoder for {{RTCRtpSender}} and the depacketizer for {{RTCRtpReceiver}}.
It operates in the [=media thread=] and is modeled as a {{WritableStream}}.

At construction of each {{RTCRtpSender}} or {{RTCRtpReceiver}}, run the following steps:
2. Initialize [=this=].`[[transform]]` to null.
3. Initialize [=this=].`[[readable]]` to a new {{ReadableStream}}.
4. <a dfn for="ReadableStream">Set up</a> [=this=].`[[readable]]`. [=this=].`[[readable]]` is provided frames using the [=readEncodedData=] algorithm given |this| as parameter.
5. Set [=this=].`[[readable]]`.`[[owner]]` to |this|.
6. Initialize [=this=].`[[writable]]` to a new {{WritableStream}}.
7. <a dfn for="WritableStream">Set up</a> [=this=].`[[writable]]` with its [=WritableStream/set up/writeAlgorithm=] set to [=writeEncodedData=] given |this| as parameter.
8. Set [=this=].`[[writable]]`.`[[owner]]` to |this|.
9. Initialize [=this=].`[[pipeToController]]` to null.
10. Initialize [=this=].`[[lastReceivedFrameTimestamp]]` to zero.
11. [=Queue a task=] to run the following steps:
1. If [=this=].`[[pipeToController]]` is not null, abort these steps.
2. Set [=this=].`[[pipeToController]]` to a new {{AbortController}}.
<!-- FIXME: Use pipeTo algorithm when available. -->
3. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with [=this=].`[[readable]]`, [=this=].`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to true and [=this=].`[[pipeToController]]`.signal.
1. Initialize [=this=].`[[transform]]` to <code>null</code>.
1. Initialize [=this=].`[[lastReceivedFrameTimestamp]]` to zero.
1. Initialize [=this=].`[[pipeToController]]` to <code>null</code>.
1. Initialize [=this=].`[[source]]` to a new {{ReadableStream}}.
1. <a dfn for="ReadableStream">Set up</a> [=this=].`[[source]]`. [=this=].`[[source]]` is provided frames using the [=readSourceData=] algorithm given |this| as parameter.
1. Initialize [=this=].`[[sink]]` to a new {{WritableStream}}.
1. <a dfn for="WritableStream">Set up</a> [=this=].`[[sink]]` with its [=WritableStream/set up/writeAlgorithm=] set to [=writeSinkData=] given |this| as parameter.
1. [=Queue a task=] to run the following step:
1. [=Queue a task=] on [=this=]'s media thread to run the following steps:
1. If [=this=].`[[pipeToController]]` is not null, abort these steps.
1. Set [=this=].`[[pipeToController]]` to a new {{AbortController}}.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This creates a WebIDL interface on a background thread, which we shouldn't do.

<!-- FIXME: Use pipeTo algorithm when available. -->
1. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with [=this=].`[[source]]`,
[=this=].`[[sink]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to true and [=this=].`[[pipeToController]]`.signal.

### Stream processing ### {#stream-processing}

The <dfn>readEncodedData</dfn> algorithm is given a |rtcObject| as parameter. It is defined by running the following steps:
The <dfn>readSourceData</dfn> algorithm given |rtcObject| runs the following steps in the [=media thread=]:
1. Wait for a frame to be produced by |rtcObject|'s encoder if it is a {{RTCRtpSender}} or |rtcObject|'s packetizer if it is a {{RTCRtpReceiver}}.
2. Let |frame| be the newly produced frame.
3. Set |frame|.`[[owner]]` to |rtcObject|.
4. [=ReadableStream/Enqueue=] |frame| in |rtcObject|.`[[readable]]`.
1. Let |frame| be the newly produced frame.
1. Set |frame|.`[[owner]]` to |rtcObject|.
1. [=ReadableStream/Enqueue=] |frame| in |rtcObject|.`[[source]]`.

The <dfn>writeEncodedData</dfn> algorithm is given a |rtcObject| as parameter and a |frame| as input. It is defined by running the following steps:
The <dfn>writeSinkData</dfn> algorithm given |rtcObject| and |frame| runs the following steps in the [=media thread=]:
1. If |frame|.`[[owner]]` is not equal to |rtcObject|, abort these steps and return [=a promise resolved with=] undefined. A processor cannot create frames, or move frames between streams.
2. If the |frame|'s {{RTCEncodedVideoFrame/timestamp}} is equal to or larger than |rtcObject|.`[[lastReceivedFrameTimestamp]]`, abort these steps and return [=a promise resolved with=] undefined. A processor cannot reorder frames, although it may delay them or drop them.
3. Set |rtcObject|.`[[lastReceivedFrameTimestamp]]` to the |frame|'s {{RTCEncodedVideoFrame/timestamp}}.
4. Enqueue the frame for processing as if it came directly from the encoded data source, by running one of the following steps:
* If |rtcObject| is a {{RTCRtpSender}}, enqueue it to |rtcObject|'s packetizer, to be processed [=in parallel=].
* If |rtcObject| is a {{RTCRtpReceiver}}, enqueue it to |rtcObject|'s decoder, to be processed [=in parallel=].
1. If the |frame|'s {{RTCEncodedVideoFrame/timestamp}} is equal to or larger than |rtcObject|.`[[lastReceivedFrameTimestamp]]`, abort these steps and return [=a promise resolved with=] undefined. A processor cannot reorder frames, although it may delay them or drop them.
1. Set |rtcObject|.`[[lastReceivedFrameTimestamp]]` to the |frame|'s {{RTCEncodedVideoFrame/timestamp}}.
1. Enqueue the frame for processing as if it came directly from the encoded data source, by running one of the following steps in the [=media thread=]:
* If |rtcObject| is a {{RTCRtpSender}}, enqueue it to |rtcObject|'s packetizer.
* If |rtcObject| is a {{RTCRtpReceiver}}, enqueue it to |rtcObject|'s decoder.
5. Return [=a promise resolved with=] undefined.

## Extension attribute ## {#attribute}
@@ -132,29 +139,33 @@ The <dfn attribute for="RTCRtpSender,RTCRtpReceiver">transform</dfn> getter step
1. Return [=this=].`[[transform]]`.

The `transform` setter steps are:
2. Let |transform| be the argument to the setter.
3. Let |checkedTransform| set to |transform| if it is not null or to an [=identity transform stream=] otherwise.
3. Let |reader| be the result of [=ReadableStream/getting a reader=] for |checkedTransform|.`[[readable]]`.
4. Let |writer| be the result of [=WritableStream/getting a writer=] for |checkedTransform|.`[[writable]]`.
5. Initialize |newPipeToController| to a new {{AbortController}}.
6. If [=this=].`[[pipeToController]]` is not null, run the following steps:
1. [=AbortSignal/Add=] the [=chain transform algorithm=] to [=this=].`[[pipeToController]]`.signal.
2. [=AbortSignal/signal abort=] [=this=].`[[pipeToController]]`.signal.
7. Else, run the [=chain transform algorithm=] steps.
8. Set [=this=].`[[pipeToController]]` to |newPipeToController|.
9. Set [=this=].`[[transform]]` to |transform|.

The <dfn>chain transform algorithm</dfn> steps are defined as:
1. If |newPipeToController|'s [=AbortSignal/aborted flag=] is true, abort these steps.
2. [=ReadableStreamDefaultReader/Release=] |reader|.
3. [=WritableStreamDefaultWriter/Release=] |writer|.
4. Assert that |newPipeToController| is the same object as |rtcObject|.`[[pipeToController]]`.
<!-- FIXME: Use pipeTo algorithm when available. -->
5. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |rtcObject|.`[[readable]]`, |checkedTransform|.`[[writable]]`, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |newPipeToController|.signal.
6. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |checkedTransform|.`[[readable]]`, |rtcObject|.`[[writable]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |newPipeToController|.signal.
1. Let |transform| be the argument to the setter.
1. If |transform| is not <code>null</code>, run the following steps:
1. If |transform|.`[[readable]]` is [=ReadableStream/locked=], [=throw=] a {{TypeError}}.
1. [=WritableStream/getting a writer|Get a writer=] for |transform|.`[[writable]]`.
1. [=ReadableStream/getting a reader|Get a reader=] for |transform|.`[[readable]]`.
1. [=Queue a task=] in [=this=]'s [=media thread=] to run the following steps:
1. let |pipeToController| be a new {{AbortController}}.
1. Let |internalTransform| be an [=identity transform stream=].
1. If |transform| is an {{SFrameTransform}}, set |internalTransform| to an [=SFrame transform stream=] given |transform|.
1. If |transform| is an {{RTCRtpScriptTransform}}, set |internalTransform| to a [=script transform stream=] given |transform|.
1. Run the [=chain transform algorithm=] with [=this=], |internalTransform|.`[[readable]]`, |internalTransform|.`[[writable]]` and |pipeToController|.
1. Set [=this=].`[[transform]]` to |transform|.

The <dfn>chain transform algorithm</dfn>, given |rtcObject|, |readable|, |writable| and |pipeToController|, runs these steps in rtcObject's [=media thread=]:
1. If |pipeToController| [=AbortSignal/aborted flag=] is true, abort these steps.
1. If |rtcObject|.`[[pipeToController]]` is not <code>null</code>, run the following steps:
1. [=AbortSignal/Add=] the [=chain transform algorithm=] with |rtcObject|, |readable|, |writable| and |pipeToController|, to |rtcObject|.`[[pipeToController]]`.signal.
1. [=AbortSignal/signal abort=] |rtcObject|.`[[pipeToController]]`.signal.
1. Else run the following steps:
<!-- FIXME: Use pipeTo algorithm when available. -->
1. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |rtcObject|.`[[source]]`, |writable|, preventClose equal to false, preventAbort equal to false, preventCancel equal to true and |pipeToController|.signal.
1. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |readable|, |rtcObject|.`[[sink]]`, preventClose equal to true, preventAbort equal to true, preventCancel equal to false and |pipeToController|.signal.
1. Set |rtcObject|.`[[pipeToController]]` to |pipeToController|.

This algorithm is defined so that transforms can be updated dynamically.
There is no guarantee on which frame will happen the switch from the previous transform to the new transform.
If a new transform overwrites an old transform, all frames will go either through the old or the new transform from the source to the sink.

If a web application sets the transform synchronously at creation of the {{RTCRtpSender}} (for instance when calling addTrack), the transform will receive the first frame generated by the {{RTCRtpSender}}'s encoder.
Similarly, if a web application sets the transform synchronously at creation of the {{RTCRtpReceiver}} (for instance when calling addTrack, or at track event handler), the transform will receive the first full frame generated by the {{RTCRtpReceiver}}'s packetizer.
@@ -183,40 +194,43 @@ SFrameTransform includes GenericTransformStream;
</xmp>

The <dfn constructor for="SFrameTransform" lt="SFrameTransform(options)"><code>new SFrameTransform(<var>options</var>)</code></dfn> constructor steps are:
1. Let |transformAlgorithm| be an algorithm which takes a |frame| as input and runs the <a href="#sframe-transform-algorithm">SFrame transform algorithm</a> with |this| and |frame|.
2. Set |this|.`[[transform]]` to a new {{TransformStream}}.
3. <a dfn for="ReadableStream">Set up</a> [=this=].`[[transform]]` with [=TransformStream/set up/transformAlgorithm=] set to |transformAlgorithm|.
4. Let |options| be the method's first argument.
5. Set |this|.`[[role]]` to |options|["{{SFrameTransformOptions/role}}"].
6. Set |this|.`[[readable]]` to |this|.`[[transform]]`.`[[readable]]`.
7. Set |this|.`[[writable]]` to |this|.`[[transform]]`.`[[writable]]`.

## Algorithm ## {#sframe-transform-algorithm}

The SFrame transform algorithm, given |sframe| as a SFrameTransform object and |frame|, runs these steps:
1. Let |role| be |sframe|.`[[role]]`.
2. If |frame|.`[[owner]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'.
3. If |frame|.`[[owner]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'.
4. Let |data| be undefined.
5. If |frame| is a {{BufferSource}}, set |data| to |frame|.
6. If |frame| is a {{RTCEncodedAudioFrame}}, set |data| to |frame|.{{RTCEncodedAudioFrame/data}}
7. If |frame| is a {{RTCEncodedVideoFrame}}, set |data| to |frame|.{{RTCEncodedVideoFrame/data}}
8. If |data| is undefined, abort these steps.
9. Let |buffer| be the result of running the SFrame algorithm with |data| and |role| as parameters. This algorithm is defined by the <a href="https://datatracker.ietf.org/doc/draft-omara-sframe/">SFrame specification</a> and returns an {{ArrayBuffer}}.
10. If |frame| is a {{BufferSource}}, set |frame| to |buffer|.
11. If |frame| is a {{RTCEncodedAudioFrame}}, set |frame|.{{RTCEncodedAudioFrame/data}} to |buffer|.
12. If |frame| is a {{RTCEncodedVideoFrame}}, set |frame|.{{RTCEncodedVideoFrame/data}} to |buffer|.
13. [=ReadableStream/Enqueue=] |frame| in |sframe|.`[[transform]]`.
1. Set |this|.`[[role]]` to |options|["{{SFrameTransformOptions/role}}"].
1. Let |sframeTransform| be a [=SFrame transform stream=] given |this|.
1. Set |this|.`[[readable]]` to |sframeTransform|.`[[readable]]`.
1. Set |this|.`[[writable]]` to |sframeTransform|.`[[writable]]`.

## Algorithms ## {#sframe-transform-algorithms}

A <dfn>SFrame transform stream</dfn> given |transform| is created by running the following steps:
1. Let |transformAlgorithm| be an algorithm which takes a |frame| as input and runs the [=SFrame transform algorithm=] with |transform| and |frame|.
1. Set |transform|.`[[sframeTransform]]` to a new {{TransformStream}}.
1. [=TransformStream/Set up=] |transform|.`[[sframeTransform]]` with [=TransformStream/set up/transformAlgorithm=] set to |transformAlgorithm|.
1. Return |sframeTransform|.

The <dfn>SFrame transform algorithm</dfn>, given |sframeTransform| and |frame|, runs these steps:
1. Let |role| be |sframeTransform|.`[[role]]`.
1. If |frame|.`[[owner]]` is a {{RTCRtpSender}}, set |role| to 'encrypt'.
1. If |frame|.`[[owner]]` is a {{RTCRtpReceiver}}, set |role| to 'decrypt'.
1. Let |data| be undefined.
1. If |frame| is a {{BufferSource}}, set |data| to |frame|.
1. If |frame| is a {{RTCEncodedAudioFrame}}, set |data| to |frame|.{{RTCEncodedAudioFrame/data}}
1. If |frame| is a {{RTCEncodedVideoFrame}}, set |data| to |frame|.{{RTCEncodedVideoFrame/data}}
1. If |data| is undefined, abort these steps.
1. Let |buffer| be the result of running the SFrame algorithm with |data| and |role| as parameters. This algorithm is defined by the <a href="https://datatracker.ietf.org/doc/draft-omara-sframe/">SFrame specification</a> and returns an {{ArrayBuffer}}.
1. If |frame| is a {{BufferSource}}, set |frame| to |buffer|.
1. If |frame| is a {{RTCEncodedAudioFrame}}, set |frame|.{{RTCEncodedAudioFrame/data}} to |buffer|.
1. If |frame| is a {{RTCEncodedVideoFrame}}, set |frame|.{{RTCEncodedVideoFrame/data}} to |buffer|.
1. [=ReadableStream/Enqueue=] |frame| in |sframeTransform|.`[[transform]]`.

## Methods ## {#sframe-transform-methods}
The <dfn method for="SFrameTransform">setEncryptionKey(|key|, |keyID|)</dfn> method steps are:
1. Let |promise| be [=a new promise=].
2. If |keyID| is a {{bigint}} which cannot be represented as a integer between 0 and 2<sup>64</sup>-1 inclusive, [=reject=] |promise| with a {{RangeError}} exception.
3. Otherwise, [=in parallel=], run the following steps:
1. Set |key| with its optional |keyID| as key material to use for the SFrame transform algorithm, as defined by the <a href="https://datatracker.ietf.org/doc/draft-omara-sframe/">SFrame specification</a>.
2. If setting the key material fails, [=reject=] |promise| with an {{InvalidModificationError}} exception and abort these steps.
3. [=Resolve=] |promise| with undefined.
4. Return |promise|.
1. If |keyID| is a {{bigint}} which cannot be represented as a integer between 0 and 2<sup>64</sup>-1 inclusive, [=reject=] |promise| with a {{RangeError}} exception.
1. Otherwise, [=in parallel=], run the following steps:
1. Set |key| with its optional |keyID| as key material to use for the [=SFrame transform algorithm=], as defined by the <a href="https://datatracker.ietf.org/doc/draft-omara-sframe/">SFrame specification</a>.
1. If setting the key material fails, [=reject=] |promise| with an {{InvalidModificationError}} exception and abort these steps.
1. [=Resolve=] |promise| with undefined.
1. Return |promise|.


# RTCRtpScriptTransform # {#scriptTransform}
@@ -289,30 +303,39 @@ interface RTCRtpScriptTransform {
</pre>

## Operations ## {#RTCRtpScriptTransform-operations}

The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worker, options)"><code>new RTCRtpScriptTransform(|worker|, |options|, |transfer|)</code></dfn> constructor steps are:
1. Set |t1| to an [=identity transform stream=].
2. Set |t2| to an [=identity transform stream=].
3. Set |this|.`[[writable]]` to |t1|.`[[writable]]`.
4. Set |this|.`[[readable]]` to |t2|.`[[readable]]`.
5. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|).
6. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »).
7. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »).
8. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
1. Set |this|.`[[worker]]` to |worker|.
1. Let |serializedOptions| be the result of [$StructuredSerializeWithTransfer$](|options|, |transfer|).
1. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
1. Let transformer be a new {{RTCRtpScriptTransformer}}.
1. Let |transformerOptions| be the result of [$StructuredDeserialize$](|serializedOptions|, the current Realm).
2. Let |readable| be the result of [$StructuredDeserialize$](|serializedReadable|, the current Realm).
3. Let |writable| be the result of [$StructuredDeserialize$](|serializedWritable|, the current Realm).
4. Let |transformer| be a new {{RTCRtpScriptTransformer}}.
5. Set |transformer|.`[[options]]` to |transformerOptions|.
6. Set |transformer|.`[[readable]]` to |readable|.
7. Set |transformer|.`[[writable]]` to |writable|.
8. Let |event| be the result of [=creating an event=] with {{RTCTransformEvent}}.
9. Set |event|.type attribute to "rtctransform".
10. Set |event|.transformer to |transformer|.
11. Dispatch |event| on |worker|’s global scope.
1. Set |transformer|.`[[options]]` to |transformerOptions|.
1. Let |transformer|.`[[t1]]` to an [=identity transform stream=].
1. Let |transformer|.`[[t2]]` to an [=identity transform stream=].
1. Set |transformer|.`[[readable]]` to |transformer|.`[[t1]]`.`[[readable]]`.
1. Set |transformer|.`[[writable]]` to |transformer|.`[[t2]]`.`[[writable]]`.
1. Let |event| be the result of [=creating an event=] with {{RTCTransformEvent}}.
1. Set |event|.type attribute to "rtctransform".
1. Set |event|.transformer to |transformer|.
1. Set |this|.`[[transformer]]` to |transformer|.
1. [=Dispatch=] |event| to |worker|’s global scope.

// FIXME: Describe error handling (worker closing flag true at RTCRtpScriptTransform creation time. And worker being terminated while transform is processing data).

A <dfn>script transform stream</dfn> given |transform| is created by running the following steps:
1. Set |t1| to an [=identity transform stream=].
1. Set |t2| to an [=identity transform stream=].
1. Set |scriptTransform|.`[[writable]]` to |t1|.`[[writable]]`.
1. Set |scriptTransform|.`[[readable]]` to |t2|.`[[readable]]`.
1. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »).
1. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »).
1. [=Queue a task=] on the DOM manipulation [=task source=] |transform|.`[[worker]]`'s global scope to run the following steps:
1. Let |readable| be the result of [$StructuredDeserialize$](|serializedReadable|, the current Realm).
1. Let |writable| be the result of [$StructuredDeserialize$](|serializedWritable|, the current Realm).
1. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |readable|, |transform|.`[[transformer]]`.`[[t1]]`.`[[writable]]`.
1. Call <a href="https://streams.spec.whatwg.org/#readable-stream-pipe-to">pipeTo</a> with |transform|.`[[transformer]]`.`[[t2]]`.`[[readable]]` and |writable|.
1. Return |scriptTransform|.

## Attributes ## {#RTCRtpScriptTransformer-attributes}

A RTCRtpScriptTransformer has three private slots called `[[options]]`, `[[readable]]` and `[[writable]]`.