Video over the network

[josuah]

This is a thread to gather information and keep track of the progress of video over the network for anyone interested in implementing it.

Topics include:

• Control protocol implementation (i.e. ONVIF, WebRTC, SIP)
• Streaming protocol implementation (i.e. SRT, RTP)
• Video stream container formats (i.e. MPEG-TS)
• Video pipeline control framework (i.e. Gstreamer, FFmpeg)
• Video compression support (i.e. H.264)
• Other initiatives (OpenHD, OpenIPC)
• Collaboration with other RTOS and ecosystems.
• Integration with other areas (i.e. NPU for detecting events on the feed, displays)

[josuah]

General discussion

When a topic does not really fit other categories, or simply to say hi.

[kartben]

what's the status of https://docs.zephyrproject.org/latest/samples/drivers/video/tcpserversink/README.html -- has someone tested it recently?

[josuah]

I tried it with native_sim about a month ago, and am testing now again:

Full details here.

P.S. I should remember to subscribe to my own discussion threads so that I get notifications.

[josuah]

Maybe some day we can have a Zephyr <-> Zephyr video streaming served on a static webpage: the USB Video Webcam sample for the local camera, SRTP (WebRTC) for the remote camera.

[josuah]

Streaming/control protocol (SRT, RTP, WebRTC, SIP, ONVIF...)

• https://en.wikipedia.org/wiki/Real-time_Transport_Protocol

[sayednasermoravej]

These are three types of protocols.

1. Control protocols.
   Streaming control protocols are communication protocols that manage how media (audio, video, etc.) is delivered and controlled during streaming over a network (especially IP networks). These protocols do not necessarily carry the media itself, but rather handle setup, control, synchronization, and teardown of media streams.

Common Streaming Control Protocols: SIP, SDP, ONVIF, WebRTC signaling.

SIP (Session Initiation Protocol):
Initiates, modifies, and terminates multimedia sessions (calls). Used with SDP for session description.
Common Use: VoIP, video calls

SDP (Session Description Protocol):
Describes multimedia sessions (used within RTSP, SIP). Includes codec info, ports, etc.
Common Use: VoIP, conferencing, RTSP

ONVIF(Open Network Video Interface Forum):
A control standard over SOAP/HTTP used in IP-based security products. Includes stream control and PTZ camera control.
Common Use: CCTV, security systems

WebRTC signaling (custom via WebSocket, SIP, etc.):
Media exchange in WebRTC is P2P, but signaling (control) uses external protocols like SIP or custom WebSocket.
Common Use: Web video calls, real-time communications

2. Streaming protocols.
   Streaming protocols define how media data is transmitted over a network so that it can be played back in real time or near-real time. These protocols manage, Media packetization, Buffering, Adaptation to bandwidth, and Live or on-demand playback.

Common Streaming Protocols: SRT, RTMP, RTP, WebRTC.

SRT (Secure Reliable Transport):
Transport Layer: UDP
Use Case: Remote video contribution, broadcasting

RTMP (Real-Time Messaging Protocol):
Transport Layer: TCP
Use Case: OBS → YouTube/Facebook

RTP (Real-time Transport Protocol): Low-latency, used with SIP/RTSP
Transport Layer: UDP
Use Case: VoIP, WebRTC, IP cameras

WebRTC
Transport Layer: UDP +SRTP
Use Case: Video chat, conferencing

3. Hybrid or Control-Enabled Protocols (Control protocols + Streaming protocols)

RTSP (Real Time Streaming Protocol):
Client-server protocol to control media sessions (e.g., play, pause, teardown). Often used with RTP for transport.
Common Use: IP cameras, surveillance, VoD

HLS (HTTP Live Streaming) – control built into HTTP:
Apple's streaming protocol using chunked files and playlists. Control via HTTP playlist updates.
Common Use: Web, mobile, live events

DASH (MPEG-DASH):
Adaptive bitrate streaming over HTTP. Similar to HLS. Control via manifest files (MPD).
Common Use: YouTube, Netflix

[josuah]

Thanks for the investigation @sayednasermoravej! That's great to have a high-level overview.

Through this it seems like RTP/SRTP are building blocks for about everything. Sometimes with long chains like SIP -> SDP -> RTSP -> RTP. Sometimes as only one supported protocol like WebRTC (SRTP = RTP + AES).

That could be some interesting upgrade path to the tcpserversink sample so that the user does not have to specify the format/resolution etc.

So if I understand, no camera control (PTZ, exposure) and no format negotiation (which resolution/FPS/bandwidth is supported on each end) yet with only RTP (a streaming protocol). This would need an extra control protocol to be supported, which would become a lot easier to implement once RTP is there... and be helped by some framework to deal with media (audio/video/metadata) like Gstreamer.

[sayednasermoravej]

You're welcome!

You're right — RTP/SRTP are indeed the main building blocks.

RTP typically runs on top of UDP.

And yes, to support features like PTZ control, an additional control protocol is needed.

[josuah]

Video stream container formats (MPEG-TS...)

• https://en.wikipedia.org/wiki/Comparison_of_video_container_formats

[josuah]

Video pipeline control framework (Gstreamer, FFmpeg...)

[josuah]

Hardware compression support (MJPEG, H.264, H.265...)

[josuah]

H.264 decoding, test data

I ordered this camera and captured a raw stream with this command:

v4l2-ctl --stream-mmap=100 --device /dev/video6 --stream-to=h264_raw_data_from_arducam_b0261.bin

h264_raw_data_from_arducam_b0261.bin.zip

ffprobe /tmp/h264_raw_data_from_arducam_b0261.bin:

Input #0, h264, from '/tmp/h264_raw_data_from_arducam_b0261.bin':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1280x720, 25 fps, 1200k tbr, 1200k tbn

file /tmp/h264_raw_data_from_arducam_b0261.bin:

/tmp/h264_raw_data_from_arducam_b0261.bin: JVT NAL sequence, H.264 video, main @ L 41

hexdump -C /tmp/h264_raw_data_from_arducam_b0261.bin | head

00000000  00 00 00 01 67 4d 40 29  96 54 02 80 2d c8 00 00  |....gM@).T..-...|
00000010  00 01 68 ee 38 80 00 00  00 01 65 88 80 40 00 49  |[email protected]|
00000020  f5 00 f1 01 28 e0 06 e9  8a 07 7d f3 00 bc d9 f5  |....(.....}.....|
00000030  b7 19 ab e3 7b 16 16 ae  f3 95 7e db 27 d4 5b fd  |....{.....~.'.[.|
00000040  2f 0a 6d 09 5a e5 de 78  cb 6e ce b7 4a 50 98 65  |/.m.Z..x.n..JP.e|
00000050  a5 c4 74 0e 73 56 71 3f  d2 5a f4 19 61 a6 cf 94  |..t.sVq?.Z..a...|
00000060  43 6f 83 fb ba 13 68 3f  bb ca 15 f7 6a f8 35 a0  |Co....h?....j.5.|
00000070  42 fa 3f 8f e0 b2 f5 10  b2 40 b9 bc 47 e1 6e 66  |[email protected]|
00000080  39 34 d2 dd 0e f1 9f 89  2d 5d 6a 45 a0 0b bc 60  |94......-]jE...`|
00000090  e5 71 97 b3 61 f9 ae 11  6f be 7f 76 b1 ff b3 62  |.q..a...o..v...b|

h264_analyze -o /dev/stdout /tmp/h264_raw_data_from_arducam_b0261.bin | head

!! Found NAL at offset 4 (0x0004), size 10 (0x000A)
0.8: forbidden_zero_bit: 0
0.7: nal->nal_ref_idc: 3
0.5: nal->nal_unit_type: 7
1.8: sps->profile_idc: 77
2.8: sps->constraint_set0_flag: 0
2.7: sps->constraint_set1_flag: 1
2.6: sps->constraint_set2_flag: 0
2.5: sps->constraint_set3_flag: 0
2.4: sps->constraint_set4_flag: 0

• h264_analyze can be obtained by compiling the https://github.com/aizvorski/h264bitstream library (no dependency, compiles easily).

• Online software: https://mradionov.github.io/h264-bitstream-viewer/

• Guide for decoding H.264 using ffmpeg and other tools: https://developer.ridgerun.com/wiki/index.php/H264_Analysis_Tools

• The raw "H.264 bitstream" (#1) is documented in the ITU document annex B: https://www.itu.int/rec/dologin_pub.asp?lang=e&id=T-REC-H.264-200305-S!!PDF-E&type=items (PDF of annex B only)

[josuah]

Other initiatives (OpenHD, OpenIPC, other RTOS)

• https://github.com/OpenIPC
• https://github.com/OpenHD