VCEENC (or AMF defaults?) forcing color primaries based on bit depth and not source.

[DimkaTsv]

Issue:
VCEENC (or AMF defaults?) forcing color primaries based on bit depth and not source.
As result video will look either oversaturated or bland, depending on source and mismatch.

Encoded 10bit with source being in BT.709 (or undefined) = oversaturated
Encoded 10bit -> 8bit... Well, this one depends on source

Issue affects both HEVC and AV1 encodes.

Test scenario
Used commands:
10 bit encode: %~dp0\VCEEncC64.exe -i %1 -o "%~n1_processed%~x1" --avsw --codec hevc --vbr 5000 --preset slow --profile main --output-depth 10 --audio-copy

8 bit encode: %~dp0\VCEEncC64.exe -i %1 -o "%~n1_processed%~x1" --avsw --codec hevc --vbr 5000 --preset slow --profile main --output-depth 8 --audio-copy

Tested sample - Bunny 10 sec sample video (as well as fragment of own screen capture (made in HEVC))
10 bit - Source: BT.709 + BT.709 --> Output: BT.2020 + PQ [oversaturated]
8 bit - Source BT.709 + BT.709 -- Output: BT.709 + BT.709

Tested sample - fragment of 4k+HDR BT.2020+PQ (+more HDR mastering)
10 bit - Source: BT.2020 + PQ -> Output: BT.2020 + PQ (lost mastering, but i had not set up encode with HDR transfer flags)
8 bit - Source: BT.2020 + PQ -> Output: BT.709 + BT.709 [bland]

Doesn't matter if i do Main10 or Main with --output-depth 10, (or even by using AVC on BT.2020 video, as it forces it down to BT.709, but at least this one i can kinda see why)

It seems like now output depth somehow sets default colorspace and enforces it ignoring source.
By encoding HEVC 10 bit BT.2020+PQ video with forced 8 bit flag, it will downgrade it to BT.709. But by encoding it as 10 bit, it will keep BT.2020+PQ state (even though did not transfer other parameters).

Expected behaviour
Well, i assume it is reasonable to expect that BT.709 sources should not encode to BT.2020+PQ just because 10 bit encode was requested...

Additional information:
Seems like it started since 8.24? That update, when many parameters had been defaulted to follow AMF spec. But in AMF defaults for this are undefined... Unless they did something and did not update documentation?
Prior to 8.23 (tested on 8.22) encoded video was stripped of color primaries whatsoever, which is... I guess different sort of nuance. But at least that was predictable.

[DimkaTsv]

Also... While i am at this. This one i know about for a while, but just remembered (stumbled upon yet again)

--qvbr <bitrate> requires <bitrate> to be passed. Without it encoder errors out, like with CQP, as it expects value to be passed. But bitrate does not matter for qvbr rate control method, and as result it still uses default QVBR_QP=23.

Moreover --qvbr-quality <QVBR_QP level> forces --qvbr mode by itself, while also providing actual method to control quality/bitrate.

Shouldn't --qvbr-quality just become default behaviour and turn into --qvbr <QVBR_QP>?

I shall probably move this text to more relatable thread...

Anyways, more info.
Checked Transcode HW... It also breaks colorspace, but in different way. When i use 10 bit enforce flag, instead of making videos oversaturated, it makes them look bleak.
And if i force 8 bit on BT.2020 HDR video, it preserved colors somehow, even though forced it to BT.709. But output video brightness became too low.
Probably should report this issue to AMF devs, then?

UPD: Reported to AMF repo. Will see for response.

[DimkaTsv]

Update: GPUOpen-LibrariesAndSDKs/AMF#527
Asked AMF devs. Got quick answer.

Well, besides secondary issue with colorspace conversion 8->10bit and in reverse, described oversaturation problem seem to stem from not passing color matrix/primary characheristic based on source and relying on AMF fallback option, which determines it purely based on encode bit depth.

I guess in list of changes to VCEEncC defaults something affected color matrix/primary transfer behaviour?

UPD: tested bit more. Seems like both --colorprim auto and --transfer auto are needed to fix issue as AMF defaults set both.
Only --colorprim auto still produces oversaturated video.
Only --transfer auto fixes oversaturation but colors are slightly distorted.

Can it be made into default (aka if unspecified) behaviour? (maybe even with --colormatrix auto, just in case?)

[rigaya]

The default for colormatrix/colorprim/transfer options is and will be "undefined". The confusion came out as AMF seems to set their default values and I had failed to override them in the application side. There was problem in overriding colormatrix/colorprim/transfer options in previous versions, VCEEnc 8.28 will fix this problem.

If you want to copy information from the input file, then you may want to use --colormatrix auto --colorprim auto --transfer auto. This was also not working fine in some cases in previous versions, and was also fixed in VCEEnc 8.28.

[DimkaTsv]

If you want to copy information from the input file, then you may want to use --colormatrix auto --colorprim auto --transfer auto.

That's how i escaped issue when it arised. Just added these 3 parameters

Can confirm change. Everything now encodes in BT.709 unless specified, which is same behaviour as prior to VCEEncC 8.24. Now it even extends to AV1. Thanks.

But... Just in case... Is AV1 output supposed to still report BT.2020+PQ in mediainfo while playing back with BT.709+BT.709+BT.709? It was transcoded from BT.2020 video, sure, but it isn't anymore after transcode.

UPD: Forced to 8-bit AV1 output reports BT.709 properly. Only 10-bit BT.2020 -> 10-bit encode does that it seems.
And issue happens even with BT.709 source encoded to 10-bit AV1.

So it looks like to be remains of issue. AMF still writes own matrix and transfer in AV1 mediainfo, even though it plays as intended.