Constant frame drops on TCL for AVC/mp2

[stari4ek]

[REQUIRED] Issue description

ExoPlayer have constant frame drops when playing mpeg-ts with AVC/mp2/ac3, while other players using same decoder (OMX.realtek.video.decoder) play video well.
Audio goes through ffmpeg, since device does not have mp2 decoder, but it plays well.

00:41:01.035 droppedFrames [eventTime=5.25, mediaPos=3.10, window=0, period=0, 50]
00:41:03.840 droppedFrames [eventTime=8.06, mediaPos=5.90, window=0, period=0, 50]
00:41:06.467 droppedFrames [eventTime=10.69, mediaPos=8.53, window=0, period=0, 50]
00:41:08.790 droppedFrames [eventTime=13.01, mediaPos=10.85, window=0, period=0, 50]
00:41:11.635 droppedFrames [eventTime=15.85, mediaPos=13.70, window=0, period=0, 50]
....
00:48:01.763 droppedFrames [eventTime=425.98, mediaPos=424.35, window=0, period=0, 50]
00:48:04.459 droppedFrames [eventTime=428.68, mediaPos=427.04, window=0, period=0, 50]
00:48:07.212 droppedFrames [eventTime=431.43, mediaPos=429.80, window=0, period=0, 50]

There're ACodec drops in logcat:

04-22 12:03:08.925 18642 19298 E ACodec  : dropped buffer - (apk discard)
04-22 12:03:08.926 18642 19298 E ACodec  : dropped buffer - (apk discard)
04-22 12:03:08.928 18642 19298 E ACodec  : dropped buffer - (apk discard)
04-22 12:03:09.019 18642 19298 E ACodec  : dropped buffer - (apk discard)
04-22 12:03:09.021 18642 19298 E ACodec  : dropped buffer - (apk discard)
04-22 12:03:09.022 18642 19298 E ACodec  : dropped buffer - (apk discard)

• Playing in ExoPlayer demo: https://drive.google.com/open?id=1-h6iFqE4TGhpixG44_LETUrqR5IC5Yp1
• Playing in MXPlayer: https://drive.google.com/open?id=1WU3Wy9vLtaK0zaOVXrSxE98jqpgMFqbI

Media comes from "Nimble" streaming software.

Logs:

• ExoPlayer (https://drive.google.com/open?id=1OWkjaq2RoeSzL8gDtPkazBMU8weBGxA5)
• MXPlayer (https://drive.google.com/open?id=1d-LW673SVn24cCNnH5k0Tp56El7W4d6X)

[REQUIRED] Reproduction steps

Play provided media in ExoPlayer demo

[REQUIRED] Link to test content

• https://drive.google.com/open?id=1JtcxL5aRVYn_mvd40sbzfbcEKarboZml

[REQUIRED] A full bug report captured from the device

Do not own device. Issue is reported by user. Here is some info:

Manufacturer: TCL
Model: UnionTV
Brand: TCL
Version: 9 (REL)
Build: TCL/UnionTV/UnionTV:9/PPR2.180905.006.A1/builder10191303:user/release-keys
Kernel: 
CPU: 4 core(s) 900 MHz (family:1 features:129023)
CPU Architecture: 7 (os.arch: armv7l)
CPU features: half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm aes pmull sha1 sha2 crc32
Instruction set: armeabi-v7a armeabi
Resolution: 1280 x 720

[REQUIRED] Version of ExoPlayer being used

• dev-v2 (7214ad2)

[REQUIRED] Device(s) and version(s) of Android being used

please check above

[christosts]

On the attached recoding, vfpo is very low (-200ms), which indicates the device is too slow on decoding the video frames. However, the attached logs also contain E ACodec : dropped buffer - (apk discard) which is not an ExoPlayer message but comes from the platform. This might also indicate there is something stressing the decoder on this TV.

To begin, can you verify from the TV logcat which decoder is MXPlayer using? Is it still using Android's MediaCodec with OMX.realtek.video.decoder or does MXPlayer bundle its own avc decoder?

I also noticed the TV specs are Resolution: 1280 x 720 with CPU: 4 core(s) 900 MHz, but the video's resolution is 1920x1080, which is higher than the TV resolution, is this correct? Can you try to play with ExoPlayer a video that matches the TV's resolution? The answer to this question, in combination with the question above (is MXPlayer using its own decoder) would help understand if the device's decoder cannot keep up with that resolution.

Some additional investigation I performed: I tested on a limited set of devices I have access to at the moment but none has a realtek chipset. All tests I did with the attached file worked fine, apart from a Pixel 1st gen, where the screen output has decoder artifacts (decoding errors) and makes me think something is wrong with this file/encoding on the specific realtek decoder. Out of curiosity, I checked the file withffprobe which outputs number of reference frames (0+5) exceeds max (4; probably corrupt input), discarding one, I wonder if that is what is making the realtek decoder on the TV slow.

Let's start with the two questions above and see if there's something clear out of that.

[stari4ek]

From the attached log:

04-25 23:05:55.928  6478  7073 I MediaCodec: [OMX.realtek.video.decoder] setting surface generation to 6633473
04-25 23:05:55.940  6478  7074 E ACodec  : [OMX.realtek.video.decoder] setPortMode on output to DynamicANWBuffer failed w/ err -2147483648
04-25 23:05:55.979  6478  7005 I +MX.MCVideoDecoder.0: Codec's output format is `color-range: int32(2), color-standard: int32(1), color-transfer: int32(3), mime: string(video/raw), stride: int32(1920), slice-height: int32(1080), color-format: int32(842094169), crop: Rect(0, 0, 1919, 1079), android._dataspace: int32(260), hdr-static-info: data, width: int32(1920), height: int32(1080)}`
04-25 23:05:56.316  6478  6478 V MX.Player.Joint: Video size: 1920 x 1080
04-25 23:05:57.217  6478  7074 W ACodec  : [OMX.realtek.video.decoder] BAD CODEC: Output format changed unexpectedly from (diff) AMessage(what = 0x00000000) = {
04-25 23:05:57.217  6478  7074 W ACodec  :       int32_t android._dataspace = 260
04-25 23:05:57.217  6478  7074 W ACodec  :     } to (diff) AMessage(what = 0x00000000) = {
04-25 23:05:57.217  6478  7074 W ACodec  :       int32_t android._dataspace = 12648448
04-25 23:05:57.217  6478  7074 W ACodec  :     }

So, I believe it uses OMX.realtek.video.decoder (It's hard to imaging that such low-spec device can handle AVC software decoding. Also it uses ffmpeg for audio decoding (as in ExoPlayer)

BTW, there is no E ACodec : dropped buffer - (apk discard) when playing using MXPlayer

[morpig]

Hi @christosts,

Device owner here. For the second question, indeed it is running in 1280x720. However, I've no issues playing back 1080 content from YouTube and in this case, some IPTV channels. The stream above is captured from a UDP stream, which is used by telco providers for IPTV. I believe this is more a interlaced/progressive issue, because I am able to play some 1080 channels.

This is the codec info from the problematic stream above (Anne-Marie MTV),

ID                                       : 1 (0x1)
Format                                   : MPEG-TS
File size                                : 38.1 MiB
Duration                                 : 1 min 5 s
Overall bit rate mode                    : Variable
Overall bit rate                         : 4 895 kb/s

Video
ID                                       : 256 (0x100)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Format settings, GOP                     : M=2, N=24
Codec ID                                 : 27
Duration                                 : 1 min 5 s
Bit rate                                 : 4 460 kb/s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 25.000 FPS
Standard                                 : Component
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan type, store method                  : Separated fields
Scan order                               : Top Field First
Bits/(Pixel*Frame)                       : 0.086
Stream size                              : 34.7 MiB (91%)
Color range                              : Limited
Color primaries                          : BT.709
Transfer characteristics                 : BT.709
Matrix coefficients                      : BT.709

Audio
ID                                       : 257 (0x101)
Menu ID                                  : 1 (0x1)
Format                                   : MPEG Audio
Format version                           : Version 1
Format profile                           : Layer 2
Codec ID                                 : 4
Duration                                 : 1 min 5 s
Bit rate mode                            : Constant
Bit rate                                 : 192 kb/s
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 kHz
Frame rate                               : 41.667 FPS (1152 SPF)
Compression mode                         : Lossy
Delay relative to video                  : -807 ms
Stream size                              : 1.49 MiB (4%)

This is the codec info for the non-problematic channel:

General
ID                                       : 0 (0x0)
Format                                   : MPEG-TS
File size                                : 38.0 MiB
Duration                                 : 1 min 0 s
Overall bit rate mode                    : Constant
Overall bit rate                         : 5 302 kb/s

Video
ID                                       : 1000 (0x3E8)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : Main@L4.1
Format settings                          : CABAC / 4 Ref Frames
Format settings, CABAC                   : Yes
Format settings, Reference frames        : 4 frames
Format settings, GOP                     : M=8, N=48
Codec ID                                 : 27
Duration                                 : 1 min 0 s
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 25.000 FPS
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive

Audio
ID                                       : 2000 (0x7D0)
Menu ID                                  : 1 (0x1)
Format                                   : AAC LC SBR
Format/Info                              : Advanced Audio Codec Low Complexity with Spectral Band Replication
Commercial name                          : HE-AAC
Format version                           : Version 4
Format settings                          : Implicit
Muxing mode                              : ADTS
Codec ID                                 : 15-2
Duration                                 : 59 s 946 ms
Bit rate mode                            : Variable
Channel(s)                               : 2 channels
Channel layout                           : L R
Sampling rate                            : 48.0 kHz
Frame rate                               : 23.438 FPS (2048 SPF)
Compression mode                         : Lossy
Delay relative to video                  : 20 ms
Language                                 : English

Test content (non-problematic channel): https://drive.google.com/file/d/1J18vdwoUNzvn8CrKS4GsCoKtBlC0pLh9/view?usp=sharing

[christosts]

The first stream is using the AVC High Profile in interlaced mode, it is possible that this is causing issues. I don't understand from the descriptions if MXPlayer manages to play the same content (AVC High profile/interlaced) correctly, but I suggest you use AVC main profile in progressive.

https://developer.android.com/guide/topics/media/media-formats

[morpig]

I tried re-encode it (the problematic stream) to main profile in progressive mode in ffmpeg with -vcodec libx264 -profile:v main, and it played back properly in both Exoplayer and MX Player. But as an end-user who just consumes IPTV content, is there anything you can do without me re-encoding it in ffmpeg?

This is live TV, so it is impossible to re-encode all the channels that runs in interlaced mode. I can give logs from both Exoplayer and MX player if needed.

[christosts]

I will try to inspect what mxplayer might be doing with the content to play it while ExoPlayer cannot, but I cannot offer a solution that will work instantly.

An alternative, pull request #7132 is implementing video decoding with ffmpeg, and it's currently on its way to the dev-v2 branch. May I suggest that you try using ffmpeg decoder with the interlaced content (either build with the pull request or wait until it is pushed in its final form on the dev-v2 branch)?