[Bug]: VBAQ quality problem H264

[lllibs]

Describe the bug
VBAQ Quality loss in static dark areas with using OBS

Recording scenes with a lot of dark colors will result a low quality. (Screenshot 1)
In high motion, option give a lot cleaner image. (Screenshot 2)
VBAQ seems to work only in obs, ffmpeg visually gives the same result with off and on, quality in dark static stays good as intended.
Setting Filler_data to On helps a bit with quality in CBR.
Quality drops only after I-frame.

To Reproduce
Steps to reproduce the behavior:

1. OBS any version with any plugin below
   https://github.com/e00E/obs-amf
   https://github.com/obsproject/obs-amd-encoder
   obs-ffmpeg: Add texture-based hardware AMD encoder obsproject/obs-studio#6508
2. Set CBR/VBR control method with VBAQ enabled with any other options
3. Record dark area will result huge quality loss

Setup (please complete the following information):

• OS: Windows 10 21H2
• Driver Version: 22.3.1
• GPU: Reference Vega56
• Which component has the issue: Encoder H264

Screenshots

1. https://i.imgur.com/lx0bivd.png
2. https://i.imgur.com/dRNcqGy.png
3. https://i.imgur.com/cvJb5I6.png

Additional context
In H265 VBAQ visually gives nothing.
Drivers tested 21.12.1-22.5.2, 31.0.12000.20010.
All windows updates with clean installation.
Screenshot 3 is a sample.

[rhutsAMD]

In order to help reproduce the same issue, could you please provide the recorded streams as well as all of the encoder settings, including the bitrate that you are using?
Please also enable the AMF debug log and include it as well.

[lllibs]

Recorded with built-in OBS plugin, not sure how to provide AMF log, used debug mode in plugin
Screenshot of settings https://i.imgur.com/5VgAVL4.png
Recording.zip
log.txt

[lllibs]

22.7.1 and new 1.4.26 issue still present

[Johl7]

Some way to tweek VBAQs 'strenght' would be good, its tuned quite high... or so it seems.

[rhutsAMD]

At this time, we do not have plans to provide a VBAQ strength parameter.

Also, it is suggested to increase the target bitrate to 6000000 bit/s for 1080p 60fps video recording. The main cause of the quality issue you are seeing is due to the low bitrate being set.

[Johl7]

At this time, we do not have plans to provide a VBAQ strength parameter.

Also, it is suggested to increase the target bitrate to 6000000 bit/s for 1080p 60fps video recording. The main cause of the quality issue you are seeing is due to the low bitrate being set.

Aint the OP of this post using 20000000 bit/s? tis a much higher value than 6000000

[rhutsAMD]

The stream that was attached is about 118.36kbps which is too low for 1080p 60fps.

This issue was originally opened back in July. At that time, the AMF plugin for OBS still displayed the bitrate setting in bps. This was changed to kbps in August.
obsproject/obs-studio#7148

[Johl7]

The stream that was attached is about 118.36kbps which is too low for 1080p 60fps.

This issue was originally opened back in July. At that time, the AMF plugin for OBS still displayed the bitrate setting in bps. This was changed to kbps in August. obsproject/obs-studio#7148

Ohhh, thats right... i have also downloaded the clip to check, thanks for the clarification!

[llibs]

Hello! I'm the guy who originally opened the Issue. Unfortunately after almost a year I lost access to my old GitHub account, I see some misunderstanding here, the problem appears only when Adaptive Quantization is enabled, perhaps the problem is relevant on video cards of a certain generation such as Vega. Because i didn't find similar problems in the web.

The bitrate set with VBAQ enabled does NOT affect the final quality in any way, with a given 10000000 bits or with 100000000. The final output quality will be terrible with a low bitrate.

And this is a problem, because Discord seems to enabled VBAQ for amd-default. For example, streaming Tarkov has the same problems in dark areas. Quality is just really BAD, like 160p.

The problem appeared around AMF 1.4.18. There is also a problem with the h264 VBR_LAT mode. Each ( i ) frame flickers and have a low bitrate as if the VBV is set very low or absent, while in the settings VBVBufferSize=100000000.

If you need any more information, I can provide it, just tell me how.

And yeah obs latest version 29.1.1 with 23.4.3 still has is, however with a driver like 20.5.1 everything seems okay except VBAQ not working at all, but at least streaming in Discord is fine.

[rhutsAMD]

Could you please provide the following updated additional information at the time of reproducing this issue on the latest OBS and driver:

1. Screenshot of your settings
2. Coded stream
3. Are you still using a Vega56?

[llibs]

Hello again! First of all I must say that there was a little mistake, driver 20.5.1 IS the driver that starts to have the problem with VBAQ. 20.4.2 is the last version that works without a problem. Issue with the VBR_LAT persists on both 23.4.3 and 20.4.2. Apparently this is a completely separate.

Videos will be uploaded in google drive because github limit is 25mb, they are about 30-40mb.

1. • All of the key settings will be at the start of the recordings below.
2. • First of all. This time I will not be using a static picture. I will use a game, so that is more clearly.

(VBAQ Part)

2.1 - 23.4.3. bitrate is 20000kbs. (VBAQ on by OBS default. VBV OBS default)
https://drive.google.com/file/d/1avujYy6jn0cFBM3iLqxGkhlVjLXuoBE6

As you can see, when there is a lot dark/grey colors, QP ramps up to 49 (guess that's limit because if set MaxQP=40, its stops at this value)

2.2 - 23.4.3. bitrate is 20000kbps. vbaq=false
https://drive.google.com/file/d/1Y0sURzP3DPzvr-PXy1Y1L_XG15SYhMXE

No weird behavior like with VBAQ enabled, but when there is fast movement, picture starts to hardly macroblocking.

2.3 - 23.4.3. bitrate is 20000kbps. vbaq=false. VBV=40000kbps
https://drive.google.com/file/d/1gWIjZDyeRCrMQTTY5XpWWAJnj4ntsplA

Less macroblocking, but still is. Doesn't look like 20k bitrate

2.4 - 20.4.2. bitrate is 20000kbps. (VBAQ on by OBS default. VBV OBS default)
https://drive.google.com/file/d/1lPJd2CGg9_bVPDpaxU5fy4eGSXYXe7ah

Works as it has to be? Except VBAQ doesn't seem to do anything.

(VBR_LAT Part)

2.5 - 23.4.3. bitrate 20000kbps. All obs default. VBR_LAT
https://drive.google.com/file/d/1zFYkCFQcgQnaPxMYUP4VUVUmN7_r5hxB

As i wrote before, every (i) frame QP jumps up like VBV is to small.

2.6 - 20.4.2. bitrate 20000kbps. All obs default. VBR_LAT
https://drive.google.com/file/d/1KRsYed5VD07EbYgujqe48guIXzT0WCYg

Same behavior

3. Yes, Vega56 reference design with a blower, but made by msi.
   Screenshot from amd software -

4. And yeah. VBAQ in H265 still doesn't seem to do anything. OBS and FFMPEG. All PSNR SSIM VMAF scores are the same when its on or off. At least on Vega.

5. While I was doing all this, i noticed a strange behavior. My monitor is 2560x1080 (21:9). And when i set resolution in OBS like native or 1920x1080. It's all bad in dark areas. But if set something like 1920x1078. Sometimes it works? But after a while it still breaks. But at least its not looking 160p with a 300kbps. Maybe this helps somehow.

(edited: OBS downloaded from github in zip format. Running --portable mode with all default settings, apart than those indicated. All drivers with clean installation. Windows 10 22H2. All updates at the time)

[llibs]

Is this has been addressed? Is the issue has been able to reproduced? Anything?

[MikhailAMD]

Thanks for the detailed report. I confirm reproduction. We are working to fix this issue in one of upcoming drivers. We will notify when the driver is released.

[Johl7]

Just to add unto this, tested VBR_LAT on my Rx 470 with OBS, exactly the same behavior seems to happen.

Update: i had someone with an Rx 6950xt test this aswell, it also bumped the QP value on each keyframe(or so i would assume, since there was a burst o artifacting happening), using OBS and VBR_LAT ratecontrol.

[rhutsAMD]

Regarding the quality loss with VBR_LAT and the QP jump on every I-frame, this is by design. VBR_LAT as a rate control method is specifically designed for low latency use cases and thus every frame is forcefully made to be of similar size. Thus an I-frame, which is supposed to be 6-8 times large than P-frames, is compressed in size to approximately match that of other frames in order to fit this particular rate control method's paradigm. So there will be such flickering. VBR_LAT is ideal for streaming in low bandwidth scenarios and not for recording. For recording, it is better to use other RC methods.

[rhutsAMD]

A fix has been made internally regarding the VBAQ issue and will be available in a future public driver release.

[llibs]

Any updates on this? Is the fix available?

[Johl7]

i guess not, the fix is supposedly to already be set... by driver side, tho this issue post here in github still aint closed, so who knows

[rhutsAMD]

The corruption issue has been fixed in the public driver version 23.11.1.

[llibs]

Did a couple quick tests, i don't think that it is really fixed. Encoding trough FFmpeg still has no AQ in h264 despite in turned on or off, but in h265 it actually start make the difference. Using obs with h264 shows that "fix" is more like low strength of VBAQ, some macroblocks have a difference in the quantization, but it is too small. On the 20.4.2 AQ is clearly stronger. Does it make sense to continue writing here about problems if fixing them takes more than a year? Or there is no one to tinker with old video cards anymore and it's easier to bury them...

Couple of screenshots of what i'm talking about. Slightly different keyframes and all that (because ffmpeg doesn't work) but it doesn't change the point.

20.4.2 vbaq on

and the AQ

20.4.2 vbaq off

23.11.1 vbaq on

AQ

23.11.1 vbaq off

And the ada lovelace with temporal-spatial for example

Encoder seems to be really broken, just look at the size of i and p frames

20.4.2

23.11.1

And the best - the first frame of the recording, yum
20.4.2

23.11.1

[llibs]

And a couple more screenshots just to show how broken it is right now.

20.4.2

23.11.1

Ada

And the settings for the ffmpeg are.
Vega -g 100 -usage 0 -profile:v 257 -level 51 -quality 2 -rc 2 -enforce_hrd false -vbaq true -preanalysis true -coder cabac
and
Ada -g 100 -preset 18 -tune 1 -profile:v 2 -level 51 -rc 1 -rc-lookahead 25 -spatial-aq true -temporal-aq true -coder 1 -multipass 1 -bf:v 2

bitrate 20mbps, vbv x2

[DimkaTsv]

Vega -g 100 -usage 0 -profile:v 257 -level 51 -quality 2 -rc 2 -enforce_hrd false -vbaq true -preanalysis true -coder cabac
and
Ada -g 100 -preset 18 -tune 1 -profile:v 2 -level 51 -rc 1 -rc-lookahead 25 -spatial-aq true -temporal-aq true -coder 1 -multipass 1 -bf:v 2

Is there any additional issue that i didn't understand? Your complaint is on "low-res" effect in dark scenes caused by compression artifacts, and vbaq makes it worse or does VBAQ causes these artifacts to appear in the first place?. I cannot open some pictures from few comments above, sadly, github doesn't allow it for some reason.

One thing i can see is that in comparison, you use -spatial-aq and -temporal-aq for Ada, but only -vbaq for Vega... Nvidia had analog for -vbaq which was called Psycho Visual Tuning and referred in OBS code as psycho_aq. Now that option is... deprecated it seems? [i wonder what happened to it]. That means that commands used in latest example are not equivalent to each other, as different types of processing applied to both.

On other note, correct me if i am wrong, but isn't lookahead also available for AMF [yes, i checked ffmpeg AMF documentation, there is -pa_lookahead_buffer_depth]?
I am also not sure, but max-qp parameters can probably work for you if particular scenes are getting cut to such state. (But issue is that then bitrate becomes much less controllable in scenes with lots of details, so not recommended for streaming). Interesting option to just test options out.
Another parameter you can try is -pa_paq_mode = caq [or taq if you have performance to spare which i don't think so]

Hmmm... Is there an possibility for you to make some "raw" recording with high bitrate and high quality as sample for bug and then use transcoding with different variables on same exact sample to get more consistent results for bug reproduction? Then link source, transcoding commands and outputs with hint at what to look for and where/when. Also expected behaviour.

[rhutsAMD]

To help investigate further, could you please provide the latest source clips and FFmpeg verbose log? As well any encoder settings if they have been changed.