Tracking issue for video decoding

[Dinnerbone]

We've had a number of people ask about this, and whilst we may not solve it soon - we can use this to let them be updated when we make progress.

There's 6 codecs (that I know of) that we'll need to support inside DefineVideoStream:

• Sorenson H.263 (codec 2)
• Screen Video (codec 3)
• VP6 (codec 4)
• VP6 with Alpha (codec 5)
• Screen Video V2 (codec 6)
• H.264 (codec 7)

The biggest limitations will be that we need decoders that satisfy the following requirements:

• Must be usable on desktop on Mac, Windows and Linux
• Must be usable on browser either as built-in to the browser, or through WebAssembly
• Must be distributable legally
  • Video codecs and patents or licensing terms unfortunately go hand-in-hand
  • Asking end-users to install something would be a no-go for web

Issue #590 contains a relevant discussion on using ffmpeg to solve this.

Issue will be updated with more info as we progress.

[ROBERT-MCDOWELL]

about h264 dec and enc, I think the best choice would be ffmpeg lib converted in wasm or
just https://github.com/cisco/openh264
already used by chrome in PPAPI and firefox in NPAPI

[EZWrighter]

Adding a note here, that RTMP streaming is still used often in the browser for close to real time video delivery and flash is the only solution. This would be a huge feature if we could still stream RTMP after flash is no longer officially supported in browsers.

[ROBERT-MCDOWELL]

it looks like ffmpeg in wasm is more and more efficient and workers ready, which can open the door to a vast encoding decoding, including all Flash codecs of course
https://github.com/ffmpegwasm/ffmpeg.wasm

[EZWrighter]

I actually looked at this problem and I believe socket security is a huge issue for the streaming libraries, so I am guessing RTMP Streaming will never be possible via wasm :-(

[ROBERT-MCDOWELL]

this is not correct, Rust has already an RTMP library.
https://users.rust-lang.org/t/rtmp-live-streaming-in-rust/16869
also, with QUIC, future HTTP3, TCP and UDP low level protocol is available.

[EZWrighter]

But that isn't in a wasm assembly. From looking at that, it looks like an RTMP server implementation. The problem is with opening up a socket to communicate with the RTMP server from within the assembly. I don't think that will be allowed.

[ROBERT-MCDOWELL]

rtmp socket is just a tcp socket,
https://github.com/robymus/rtmp.js-websocket
https://github.com/yurydelendik/rtmp.js
https://github.com/illuspas/Node-Media-Server/blob/master/node_rtmp_client.js
rtmfp needs an udp socket, available for now only with webrtc, in near future with HTTP3
https://hpbn.co/webrtc/

[EZWrighter]

Looking at those projects, they all require some one off browser configuration...none of them are general purpose solutions.

My understanding is that the communication has to come from a higher level and then be bridged down into the webassembly. ffmpegwasm/ffmpeg.wasm#6 Maybe I am just out of my league here with the communication paths.

[kmeisthax]

All of the links Robert posted are for accessing RTMP streams in non-browser environments. For security reasons, code running in the context of a browser is prohibited from making arbitrary TCP or UDP connections. It is only allowed to make HTTP, WebSocket, and WebRTC requests to it's own origin or a consenting third-party origin using Fetch, XMLHttpRequest, or by including the content in some other high-level tag. There is no way to enforce a same-origin policy with sockets (you need an HTTP Host header in order to validate origins), ergo you don't get sockets on the web.

As for ffmpeg.wasm, in order to get H.263 or VP6 support we need to build it ourselves, which requires everyone who wants to build Ruffle to also have a working Emscripten build environment and git clones. This would be harder to integrate with cargo or CI systems. The story around mixed-language projects is a bit murkier on web than on desktop, so I'd prefer having access to pure-Rust software decoders. If I'm completely wrong about this, let me know, but we may ultimately have to write our own decoders for at least some of these codecs.

[lordhugo7880]

What is the current status for FLV and F4V video container support? Have these been fully implemented yet?

[Herschel]

No video implementation at all currently.

[lordhugo7880]

Ok, thanks for the info. Btw, according to page 72 (page 78 of the PDF file) of this document: http://download.macromedia.com/f4v/video_file_format_spec_v10_1.pdf , 'Screen video version 2' is listed as a separate video codec that needs to be supported.