[Bug]: bgr0 -> nv12 conversion looks bad #529
Labels
Comments
|
Few things:
|
My command is getting "a little" complicated... export LD_LIBRARY_PATH=/home/mark/pkgbuilds/ffmpeg/upstream/build/lib
./ffmpeg -hwaccel amf -hwaccel_device /dev/dri/renderD128 -init_hw_device amf=/dev/dri/renderD128 \
-f x11grab -framerate 60 -max_delay 0 -max_probe_packets 1 -thread_queue_size 1024 -video_size 2560x1440 -r 60 -i :0.0+0,0 \
-vf 'hwupload' \
-c:v hevc_amf \
-log_to_dbg 1 \
-usage lowlatency \
-rc vbr_peak \
-minrate 2M \
-maxrate 15M \
-quality quality \
-preset quality \
-vbaq 0 -enforce_hrd 0 -aud 0 -latency 1 -filler_data 0 \
-profile:v main10 \
-profile_tier high \
-b:v 8M \
-f nut -muxdelay 0 udp://10.0.3.55:5555It took some fiddling to get that Some logging that might be helpful: I don't see any message from ffmpeg that it's converting formats so i assume it's passed on to AMF (or VCN). |
|
try to use -color_range full |
|
With hwupload and color_range full:
And this is how i expect it to be (provided pix_format p010 and removed hwupload)
|
|
If you pass pix_fmt P010 to AMF HEVC encoder, it will initialize with HEVC 10-bit instead of default 8-bit. Yes, colors will be better. |
I'm doubtful if that will work. |
|
Two different issues:
If you want to check bit-ness of your output for these two cases, just write few frames to a file: -frames 100 -f out.mkv. You can also check color_range in the output HEVC stream. |
|
Ahh, i see what you mean! It indeed has I patched ffmpeg, re-added BGR0 to the formats for HEVC and I unconditionally enabled AMF_COLOR_BIT_DEPTH_10. The output now does have 10 bit (yuv420p10le) but it's completely solid green! |
|
Well... Something is still wrong. Could be your changes, command line, FFmpeg integration, general AMF, Linux specific things. It would help if you get some simple FFmpeg RGBA input like -mandelbrot filter or similar, make full command line with your parameters writing to a MP4 or MKV file and share the line. Also you can enable detailed FFmpeg log which in turn will enable detailed AMF log - part of these patches and share the log. |
|
Ok, will do! It's not being used 😒 This wasn't the case before. (same for |
|
Right, nope. This is a little more involved then just enabling one little thing.. We're now in AMF vtable and initialization magic. I'll happily apply a patch if you have one for me that enables log output else this is probably where i draw the line and rather wait for updated patches. |
|
We would welcome you as dev :) |
That doesn't show me much AMF logging at all.. Here's an output: As an aside, my ffmpeg is compiled with |
|
Compile option doesn't matter. AMF logs from installed SO. |
|
Alright, i have output for you. It's a lot... :) I have given it a png as image. I'll also use this in the future for banding tests.
It gives me an green screen as output (1 frame mkv). I'm not allowed to upload mkv to github.. Well, it's 1 frame and completely green, not much to see. Here are the things that caught my eye in the output: That is edit |
|
OK. I reproduced the problem with generated input on Windows, but it shouldn't matter. Some color conversion parameters for input and for output are not set and defaults don't work in this case. Below is what I added to hard-code all of them and force 10-bit. This fixes the green frame problem for me, but we will need to update the AMF encoder plugin for better RGBA input handling, 10-bit support and defaults. //input For your reference command line from Visual Code I used: BTW: We are very interested in your feedback such this one as it helps to improve AMF. |
|
Ha! You use god mode? That's not documented! AMF_ASSIGN_PROPERTY_INT64(res, ctx->encoder, AMF_VIDEO_ENCODER_HEVC_OUTPUT_COLOR_PROFILE, AMF_COLOR_PRIMARIES_BT2020); Note the primaries in the color profile 😉 Anyhow, confirmed locally! Thank you for these values. Is there anything i can do to help the AMF progress for that 8 bit -> 10 bit conversion? |
|
Yes, my bad. The value is outside the parameter range, so the call has no effect. Should be AMF_VIDEO_CONVERTER_COLOR_PROFILE_FULL_2020. AMF has deficiency in conversion from and to 8-10 bit between YUV formats only. Your input is RGBA so it works, 8, 10, 16-bit input. Do you have a different use case? There is no reason to convert RGBA8->NV12->P010, just loss of data. Please note that AMF has own Capture component for Linux. The advantage of using it is no transfer to and from system memory. All data stays in video memory. Though, it doesns't have FFmpeg integration yet. |
This is the desktop streaming usecase, shouldn't that be sufficient enough? I'm not a streamer and am not using any of this in a professional setting, it's all just internal in one home and hobby-based. However, desktop streaming - say on twitch - is a very real usecase. Right now it probably all works by converting it on the CPU to go from BGR0 to nv12 first before it's fed to AMF. My experiments also show that AMF can (mostly) do what i want. If i input it as p010 it looks mostly good.
Gimme gimme gimme :) You'd still have the 8 vs 10 bit issue, no? Could this work on X11 too? I'm asking because on x11 you can grab your input including the mouse, with kernel modesetting kmsgrab you can't get the mouse (or i don't know how). It's not fun to have a desktop stream when there's no mouse, i tried that.. You then need additional logic to grab the current mouse pixmap, stream that and do client side magic to draw the mouse in the same style as the server. Ugh.. The protocols and helper options might just be a little too limited for that on the KMS side at the moment so i prefer x11(grab) for now that just works. This being said, X11 and AMF Capture has some very interesting features when combined! In X11 you can track modified regions of content which could work perfectly on the AMF Capture side with |
|
AMF DirectCapture captures BGRA0 and it may be feed directly to encoder. With parameters I hardcoded the result will be 10-bit HEVC. You can check AMF DVR sample. |
|
Hmm, yeah, i get it from a zero-copy point of view. I might eventually also go the no-mouse route and make a solution that streams mouse pixmap changes out of band and render it client side. The downside is it gives a perceived feeling if being more responsive then it actually is, which can be mediated by introducing moude render latency to match video streaming latency.. pff, that is getting complex! The upside, to me, isn't just the zero copy but is the reduced bandwidth as mouse movements over - say - a background are essentially free. It's as soon as a mouse has any visual interaction with a ui element that it becomes part of the encoding process. Perhaps i'll go that route at some point, an out of stream mouse update. I'll keep it in-stream for the moment :) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Linux desktop streaming. I'm using x11grab which is a pixel format of bgr0.
AMF has a mapping for that:
AV_PIX_FMT_BGR0maps toAMF_SURFACE_BGRATo be fair, it's just a mapping. It's not wired up to HEVC. So i did add that one myself. In
amfenc.ci just addedAV_PIX_FMT_BGR0in theff_amf_pix_fmtsarray.Compiling ffmpeg with this makes it work and lets
bgr0be handled by AMF. Sweet! The more that can be offloaded to AMF in the encoding pipeline the better it is for latency purposes.The following table shows the conversion as i think they happen. I do have to put a big fat disclaimer here. I'm assuming the cpu does a pix format conversion when
pix_formatis supplied and that it does that conversion to a format the driver (hevc_amf) supports. I'm also assuming that no cpu conversion happens whenpix_formatis not supplied.Also note that no pix format in the table means i didn't supply
pix_formatas argument thus it's the source format. Which is bgr0.By far the best one in that table is that last one,
p010. The color representation is good and 10 bit (irrelevant for this issue) means less banding too, which you can see in the grey gradient behind the AMD logo if you look carefully!You can easily see that the first row looks very off. In theory this would be the best one as AMF would do all the work.
In practice it looks like the AMF conversion from bgr0 -> nv12 is not doing the same as the cpu based one when
nv12is provided aspix_format.At this point i'm not entirely convinced the issue is in ffmpeg (x11grab specifically) or in AMF. To try and rule out x11grab, i did modify it's code to output
BGRAinstead ofBGR0but the results were the same.Is this a hardware bug?
Setup (please complete the following information):
The text was updated successfully, but these errors were encountered: