Add picture timing requirement to Ultra Low latency Broadcast with Fanout use case

[murillo128]

In traditional live streaming it is common place to send per-frame picture timing information, typically using the H264 Picture timing SEI messages (ITU-T REC H.264 annex D.2.3).

This timing information is later used to synchronize out of band events like ad insertion using SCTE 35 splice events (SCTE 35 2023r1 section 6.3). Per frame picture timing is widely used in video editing tools.

Implementation/spec wise, we can use the abs-catpure-time header extension which is exposed in RTCRtpContributingSource captureTimestamp WebRTC Extension .

Retrieving the picture timing information of the rendered frame is quite clumbsy at the moment at it requires interpolating values using the rtp timestamps of the frame which is the only available value apis exposing frames.

An attempt to add the capture timestamp values is being doing at different apis:

• video-rvfc Use abs-capture-time header extension instead of RTCP SR for calculating captureTime on remote sources when available WICG/video-rvfc#86 (comment)
• encoded trasnform Should we add captureTimestamp senderCaptureTimeOffset to the RTCEncodedXXXFrameMetadata if abs-capture-time is used? webrtc-encoded-transform#225
• Insertable Streams for MediaStreamTrack What is the timestamp value of the VideoFrame/AudioData from a remote track? mediacapture-transform#87 although this would probably need to be raised to webcodecs

[maxlapshin]

Hi! This is a great idea and I see two questions here:

1. how to get original timestamp
2. how to keep it inside whole video pipeline

Right now we are developing end-to-end latency measuring system and looking for an approach that will allow me to enable it in webrtc-published environments and when OBS is publishing rtmp.

Later frame must pass transcoders, including third party, without losing timecodes so looks like the best way is to put capture timecode into SEI. But SEI timing seems to have only second resolution and we need at least millisecond.

So maybe unregistered user SEI (which I don't like).

About original timestamp: why not to take just RTP timecode? It is very precise.

[maxlapshin]

Btw, @murillo128 are you sure that pic timing SEI is a good place for keeping realtime timestamp of a frame? I'm looking at our redmine and it seems that this mechanism can be related to HRD buffer.

[alvestrand]

SEI messages are codec specific.
Note that there's a number of developments around timestamps being added in webrtc-extensions currently.