EncodedVideoFrame needs to be marked serializable

[alvestrand]

The algorithm specified seems to indicate that EncodedVideoFrame (and audio) is serializable, since it uses the serialization steps to move it around.

This means that "structuredClone(frame)" should give sensible results.

We should mark the IDL accordingly.

(Related to #161)

[youennf]

The spec is using serialise with transfer as a way to copy the detach the data ownership and ensure there is no TOCTOU issue. I do not think the intent was to enable serialisation of frames.
We can rewrite the spec to detach the array buffer and copy the necessary information to make things clearer.
We can discuss making frames serialisable in parallel.

[jan-ivar]

The spec is using serialise with transfer as a way to copy the detach the data ownership and ensure there is no TOCTOU issue. I do not think the intent was to enable serialisation of frames.
We can rewrite the spec to detach the array buffer and copy the necessary information to make things clearer.

Thanks for explaining. I agree that unless the WG decides frames should be serializable, we need to rewrite that since neither RTCEncodedAudioFrame nor RTCEncodeVideoFrame contain the prerequisite serialization or deserialization steps necessary to have "Let serializedFrame be StructuredSerializeWithTransfer(frame, « data »)." and "Let frameCopy be StructuredDeserialize(serializedFrame, frame’s relevant realm)." work.

We can discuss making frames serialisable in parallel.

To save time, can this be that issue?

• If we decide NO, we do a PR with the above rewrite
• If we decide YES, we do a PR with the missing (de)serialization steps, and [Serializable] in the WebIDL

Does that work?

[jan-ivar]

@alvestrand's request seems based on need in the #160 use case, and now that structuredClone(value[, { transfer }]) exists in the web platform, inventing custom frame.clone() methods like in #161 seems wrong to me.

This would also solve whether to clone the frame buffer or not, by leaving it up to JS:

const frameCopy = structuredClone(frame); // clones the data as well

const frameMove = structuredClone(frame, {transfer: [frame.data]}); // transfers the data

[dontcallmedom-bot]

This issue was mentioned in WEBRTCWG-2023-09-12 (Page 49)