membrane_h26x_plugin stream_format issue using RootEncoder Android lib

[olegokunevych]

Hello, we are currently using membrane_rtmp_plugin together with membrane_h26x_plugin libs in order to receive RTMP stream from Android devices. Under the hood it uses Root Encoder library (https://github.com/pedroSG94/RootEncoder) which successfully works with other media servers, however it doesn't work stable with Membrane. In one of approximately 10 cases it works correctly, but in almost all the cases Membrane.H264.Parser receives payload, but doesn't produce payload at all.
The corresponding part of pipeline is:

child(
 :http_adaptive_streaming_parser,
 %Membrane.H264.Parser{
   output_stream_structure: :annexb
 },
  get_if_exists: true
)
|> child(:debug_streaming_parser_filter, %Membrane.Debug.Filter{
  handle_stream_format: &dbg(&1)
})

For some reason in almost all cases, we don't receive handle_stream_format callback in debug filter. In rare cases we receive it and pipeline is working as we expect:

%Membrane.H264{
  width: 1280,
  height: 720,
  profile: :constrained_baseline,
  alignment: :au,
  nalu_in_metadata?: true,
  framerate: nil,
  stream_structure: :annexb
}

I've created a test with h264 dump streamed from Android app:
membraneframework/membrane_h26x_plugin#69

PS Other streaming tools (OBS, FFMpeg) works well with our pipeline.

[olegokunevych]

The difference in successful and failing payload is that successful first buffer includes sps/pps metadata, while failing doesn't:

%Membrane.Buffer{
  payload: <<0, 0, 69, 227, 101, 184, 0, 1, 128, 113, 255, 255, 240, 244, 80, 0,
    16, 23, 247, 223, 125, 247, 223, 125, 247, 223, 125, 247, 223, 125, 247,
    223, 125, 247, 223, 125, 247, 223, 125, 247, 223, 125, 247, 223, 125, 247,
    223, 125, 247, ...>>,
  pts: 38709000000,
  dts: 38709000000,
  metadata: %{
    h264: %{
      key_frame?: true,
      nalus: [
        %{
          metadata: %{
            h264: %{
              type: :idr,
              end_access_unit: true,
              new_access_unit: %{key_frame?: true}
            }
          }
        }
      ]
    }
  }
}

%Membrane.Buffer{
  payload: <<0, 0, 0, 18, 103, 66, 192, 41, 141, 104, 5, 0, 91, 164, 32, 32, 32,
    32, 240, 136, 70, 160, 0, 0, 0, 6, 104, 206, 1, 168, 53, 200, 0, 0, 10, 154,
    101, 184, 0, 4, 5, 159, 255, 255, 15, 69, 0, 1, 4, ...>>,
  pts: 258000000,
  dts: 258000000,
  metadata: %{
    h264: %{
      key_frame?: true,
      nalus: [
        %{
          metadata: %{h264: %{type: :sps, new_access_unit: %{key_frame?: true}}}
        },
        %{metadata: %{h264: %{type: :pps}}},
        %{metadata: %{h264: %{type: :idr, end_access_unit: true}}}
      ]
    }
  }
}

[varsill]

Hi @olegokunevych ! Thanks for reporting that.

I've seen your PR and after a tiny adjustment (the profile of the fixture .h264 file is Constrained Baseline and the test expects High profile) I was able to run these newly added tests successfully. Are you sure that that fixture you provided was dumped from a failed stream?

I suspect that the problem is caused by loosing the initial part of the stream where the only SPS/PPS were put.

A word of an explanation - H264 parser requires SPS/PPS to generate the stream format (it reads information like resolution or profile out of them). In case SPS/PPS are not provided, parser cannot generate stream format and therefore it cannot send any buffers further - it basically waits until the next SPS/PPS appear and discards the incoming frames as the stream is basically useless without metadata from SPS/PPS (decoder would behave similarly).
The way in which the problem with potentially missing SPS/PPS is usually handled is by either providing SPS/PPS via some more reliable "side channel" (you can use spss and ppss Parser's options for that) or by making encoder repeat SPS/PPS every keyframe.
It would be good to figure out why the initial part of the stream is sometimes getting discarded but as a quick fix I would suggest trying to find an option in the RootEncoder to repeat the parameter sets every keyframe.

Concerning these snippets with %Membrane.Buffer{} - how did you generate them? The payload seems to be written in avc1 or avc3 stream structure rather than annexb and I also wonder how did you manage to generate that first output if SPS/PPS were not present in the stream.
Best wishes!

[olegokunevych]

Hey @varsill thanks for detailed answer.
Test ran successfully with constrained_baseline for me as well. I used MediaMTX to receive RTMP stream and ffmpeg command (ffmpeg -i rtmp://localhost:1935/603047a9-c939-4f10-a96c-d80fe59a4e39 -c copy -bsf:v h264_mp4toannexb -f h264 recordings/output_stream.h264) which gave me an .h264 dump. Is it possible that MediaMTX makes some additional adjustments so that stream without SPS/PPS becomes stream with SPS/PPS?

Regarding the %Membrane.Buffer{}, these snippets have been generated before the parser with avc3 stream structured payload to see the difference between working/non working streams.

...
|> child(:debug_before_streaming_parser_filter, %Membrane.Debug.Filter{
  handle_buffer: &dbg(&1),
  handle_stream_format: &dbg(&1)
})
|> get_child(:http_adaptive_streaming_parser)
|> child(:debug_streaming_parser_filter, %Membrane.Debug.Filter{
          handle_stream_format: &dbg(&1)
})
...

[varsill]

Is it possible that MediaMTX makes some additional adjustments so that stream without SPS/PPS becomes stream with SPS/PPS?

I doubt that it's possible, SPS/PPS need to be generated by the encoder and then they might either be transported in band of the stream, or out of the band (e.g. in the FLV video tag as a part of the AVCDecoderConfigurationRecord)

Regarding the %Membrane.Buffer{}, these snippets have been generated before the parser with avc3 stream structured payload to see the difference between working/non working streams.

Oh, I see now, thanks for the explanation!

Could you dump the RTMP stream with the following simple pipeline:

# it requires adding dependency to `{:membrane_stream_plugin, "~> 0.4.0"}`
child(Membrane.RTMP.Source{url: ...})
|> child(Membrane.Stream.Serializer)
|> child(%Membrane.File.Sink{location: "output.msr"})

and send us the output.msr? (it's important to use Membrane.RTMP.Source instead of Membrane.RTMP.SourceBin)
It will be easier for us to debug it if we have exactly the same stream that is received by your source.

[olegokunevych]

sure, check the attached file.
And thanks for advance for fast and useful advices!

output.msr.zip

[olegokunevych]

AFAIK, we may have users, who can send SPS/PPS once a minute, or even once per few minutes. Is there any recommendations and tips how to handle such streams?

[varsill]

Thanks! I've run the provided output.msr with the following script:

Mix.install([
  {:membrane_h26x_plugin, "~>0.10.0"},
  {:membrane_stream_plugin, "~> 0.4.0"},
  {:membrane_file_plugin, "~> 0.17.2"},
  {:membrane_flv_plugin, "~> 0.13.0"}
])


defmodule Runner do
  import Membrane.ChildrenSpec

  require Membrane.RCPipeline, as: RCPipeline
  require Membrane.Pad, as: Pad

  def run() do
    pid = RCPipeline.start!()
    RCPipeline.subscribe(pid, _any)

    spec = child(:file, %Membrane.File.Source{location: "output.msr"})
      |> child(Membrane.Stream.Deserializer)
      |> child(Membrane.FLV.Demuxer)
      |> via_out(Pad.ref(:video, 0))
      |> child(%Membrane.H264.Parser{output_stream_structure: :annexb})
      |> child(:sink, %Membrane.Debug.Sink{handle_stream_format: &IO.inspect(&1, label: :stream_format)})
    RCPipeline.exec_actions(pid, spec: spec)
    RCPipeline.await_end_of_stream(pid, :sink)
  end

end

Runner.run()

and it seems that it properly parses it. Do you think it's possible that we encountered that once in 10 times situation where the stream does not fail?

we may have users, who can send SPS/PPS once a minute, or even once per few minutes.

I guess it means that they also send a keyframe "once a minute or even once per few minutes"? If not, the goal would be to make sure that each key keyframe comes with SPS/PPS pair.

[olegokunevych]

Thank you for this script, it helped me a lot in debugging!
That's sound weird, but I am experiencing different behavior when using Membrane.RTMP.Source and Membrane.RTMP.SourceBin. Source approach now looks:

 [
      child(:source_bin_video_tee, Membrane.Tee.Parallel),
      child(:source_bin_audio_tee, Membrane.Tee.Parallel),
      child(:source, %Membrane.RTMP.Source{
        client_ref: state.client_ref
      })
      |> child(:source_demuxer, Membrane.FLV.Demuxer)
      |> via_out(Pad.ref(:video, 0))
      |> get_child(:source_bin_video_tee),
      get_child(:source_demuxer)
      |> via_out(Pad.ref(:audio, 0))
      |> get_child(:source_bin_audio_tee)
 ]

But the main thing is that I am getting stream format for each and every streaming attempt. Pipeline is working as it should, both HLS and WebRTC. But it doesn't work with SourceBin approach.
That's why, using Membrane.RTMP.Source produced working dump.
I am currently in the middle of debug session, but let me know if you need any other details.