VCEEnc HW Decoding not working

[SuperMitic]

I tried using StaxRip and AMD VCEEnc to encode a video. Encoding using VCEEnc HW Encoding and CPU Decoding is working, but if in StaxRip i select VCEEnc as Decoder (in order to use GPU HW Decoding), encoding is not working and it crash with this error:

AMFDEC: ERROR: Resolution changed during decoding.
AMFDEC: Failed to load input frame: unknown error..
Break in task AMFDEC: unknown error..
encoded 0 frames, 0.00 fps, 0.00 kbps, 0.00 MB
encode time 0:00:00, CPULoad: 0.0
Finished with error in VCEEncC.

These are the parameter i'm using: --avhw --codec hevc --output-depth 10 --quality slow --max-bitrate 21000 --colorprim bt709 --vbr 18000 -i. This issue appear when using --avhw.

GPU: RX 6600 with latest 23.3.2 driver
I don't if this problem is caused by latest 23.3.2 driver.

[DimkaTsv]

Can we at least see what source are you trying to transcode?

[91rinb18]

rx 6xxx显卡似乎并不能解码10位HEVC，如果你的源是HDR之类，只能用avsw。而且StaxRip这些软件有点古怪，差不多的参数俺用VCEENC输入命令行可以执行，但用StaxRip却不能工作。
The rx 6xxx card doesn't seem to be able to decode 10-bit HEVC, so if your source is HDR or something, you'll have to use avsw. And StaxRip is a bit odd for these programs, pretty much the same parameters I can execute with VCEENC by typing them into the command line, but it doesn't work with StaxRip.

[DimkaTsv]

The rx 6xxx card doesn't seem to be able to decode 10-bit HEVC
Now that is not true...

Source:

Video
ID                             : 1
Format                         : HEVC
Format/Info                    : High Efficiency Video Coding
Format profile                 : Main [email protected]@High
HDR format                     : SMPTE ST 2086, HDR10 compatible
Codec ID                       : V_MPEGH/ISO/HEVC
Duration                       : 2 min 41 s
Bit rate                       : 78.3 Mb/s
Width                          : 3 840 pixels
Height                         : 2 160 pixels
Display aspect ratio           : 16:9
Frame rate mode                : Constant
Frame rate                     : 23.976 (24000/1001) FPS
Color space                    : YUV
Chroma subsampling             : 4:2:0 (Type 2)
Bit depth                      : 10 bits
Bits/(Pixel*Frame)             : 0.394
Stream size                    : 1.47 GiB (100%)
Writing library                : ATEME Titan File 3.9.0 (4.9.0.0)        
Default                        : Yes
Forced                         : No
Color range                    : Limited
Color primaries                : BT.2020
Transfer characteristics       : PQ
Matrix coefficients            : BT.2020 non-constant
Mastering display color primar : Display P3
Mastering display luminance    : min: 0.0050 cd/m2, max: 1000 cd/m2
Maximum Content Light Level    : 845 cd/m2
Maximum Frame-Average Light Le : 196 cd/m2

Definitely was HW decoded. Even with HDR. And i had transcoded it via VCEEncC without issues.

Command: C:\Users-----\Videos\test sample>C:\Users-----\Desktop\Utilites\VCE_Encoder\VCEEncC_x64\VCEEncC64.exe -i "C:\Users-----\Videos\test sample\Test_softer.mkv" -o "Test_softer_processed.mkv" --avhw --codec hevc --profile main10 --tier high --level 6.2 --preset slow --ref 4 --ltr 3 --vbr 10000 --max-bitrate 15000 --qp-max 36 --vbaq --colorrange auto --colormatrix auto --colorprim auto --transfer auto --chromaloc auto --max-cll copy --master-display copy --atc-sei auto --audio-copy --chapter-copy

Result:

--------------------------------------------------------------------------------
Test_softer_processed.mkv
--------------------------------------------------------------------------------
VCEEnc (x64) 8.16 (r1491) by rigaya, Jun 24 2023 02:32:23 (VC 1936/Win)
OS:            Windows 11 x64 (22621) [UTF-8]
CPU:           AMD Ryzen 7 5800X3D 8-Core Processor [4.47GHz] (8C/16T)
GPU:           AMD Radeon RX 6750 XT [31.0.14057.5006]
AMF:           Runtime 1.4.29 / SDK 1.4.29
Input Info:    avvce: H.265/HEVC, 3840x2160, 24000/1001 fps
Output:        H.265/HEVC  main10 @ Level 6.2 (high tier)
               3840x2160p 1:1 23.976fps (24000/1001fps)
               avwriter: hevc => matroska
Quality:       slow
VBR:           10000 kbps
Max bitrate:   15000 kbps
QP:            Min: 0:0, Max: 36:36
VBV Bufsize:   10000 kbps
Bframes:       0 frames
Pre Analysis:  off
Ref frames:    4 frames
LTR frames:    3 frames
Motion Est:    Q-pel
Slices:        1
GOP Len:       240 frames
VUI:              matrix:bt2020nc,colorprim:bt2020,transfer:smpte2084,range:limited,chromaloc:topleft
MasteringDisp: G(0.265000 0.690000) B(0.150000 0.060000) R(0.680000 0.320000)
               WP(0.312700 0.329000) L(1000.000000 0.005000)
MaxCLL/MaxFALL:845/196
Others:        skipframe:on deblock vbaq

encoded 3861 frames, 53.56 fps, 10043.49 kbps, 192.80 MB
encode time 0:01:12, CPU: 0.0%, GPU: 4.1%, VE: 98.7%
frame type IDR   17
frame type I     17,  avgQP  17.71,  total size    1.74 MB
frame type P   3844,  avgQP  16.20,  total size  191.06 MB

That is why i asked person for his source file, maybe something is wrong with it. Decoding error for him was "resolution changed during decoding"

[91rinb18]

WIN10内置播放器确实可以硬解SONY的汤加蓝鲸HDR-265视频，但我直接用VCEEenc就是这个结果，换成avsw才可以工作。
是我一直不肯升级驱动导致的？
VCEEncC64 --avhw -i WhaleTongaHDR.mp4 --quality slow -u slow --cqp 15:17:19 --bframes 2 --ref 4 --profile high --level 4.1 --audio-codec aac:aac_coder=twoloop --audio-bitrate 192 -o lanj.mp4

Windows 10 built-in player can indeed hard decode SONY's Tongan Blue Whale HDR-265 video, but I directly use VCEEenc is the result, replaced by avsw can work. Is it because I haven't been able to upgrade the drive?

[DimkaTsv]

"Windows 10 built-in player can indeed hard decode SONY's Tongan Blue Whale HDR-265 video, but"

Can you upload source somewhere linking it there and/or share it's mediadata. And/or enable logging for VCEEncC if you think that error it's on level of encoder application/API. You can do that by using --log-level debug

"VCEEncC64 --avhw -i WhaleTongaHDR.mp4 --quality slow -u slow --cqp 15:17:19 --bframes 2 --ref 4 --profile high --level 4.1 --audio-codec aac:aac_coder=twoloop --audio-bitrate 192 -o lanj.mp4"

Also... did you just used --profile high parameter? And you didn't specify that you want H.264/AVC output? HDR isn't naturally supported by H.264, so i place my bet that this source is H.265/HEVC or AV1, and it tries to use HEVC on auto mode with high preset which is impossible.
You can get H.264 encoded HDR video... technically, but barely ANY hw decoder will support it.

By quote from internet:
" Hi10p, Hi422p, etc. are very uncommon in hardware decoders (no iPhone or Core i7 processors supports these)."
So if you source is somehow HDR H.264 video, then SW decoder only!

Moreover to encode HDR with H.264 video you would be needed to use --profile main10 or high10 or high422 high444 profiles... And looks like AMD encoder only supports high444 from this. You may also need to specify --output-depth 10 (and use HDR transfering parameters... A bunch of those) . But i won't guarantee success on H.264 anyways.

[91rinb18]

You are right. 俺知道它是10bit的HDR，俺只是测试下VCE的能力。默认而不指定编码器的话，它就是264编码，当时确实忘掉了。因为俺DELL显示器只是8bit的老家伙，俺记得10bit是HDR的先决条件，所以并没有转码成HDR。俺记得好像是下载——https://4kmedia.org/sony-whale-in-tonga-hdr-uhd-4k-demo

log-level debug就算了吧，俺看不懂，哈哈。俺只是个普通用户，不能和你们专家比……
File size : 815 MiB
Duration : 1 min 31 s
Overall bit rate : 74.6 Mb/s
Encoded date : UTC 2017-02-27 03:31:16
Tagged date : UTC 2017-02-27 03:31:16

Video
ID : 1
Format : HEVC
Format/Info : High Efficiency Video Coding
Format profile : Main 10@L5.1@High
HDR format : SMPTE ST 2086, HDR10 compatible
Codec ID : hvc1
Codec ID/Info : High Efficiency Video Coding
Duration : 1 min 31 s
Bit rate : 74.5 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Frame rate mode : Constant
Frame rate : 59.940 (60000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0 (Type 2)
Bit depth : 10 bits
Bits/(Pixel*Frame) : 0.150
Stream size : 813 MiB (100%)
Encoded date : UTC 2017-02-27 03:31:16
Tagged date : UTC 2017-02-27 03:31:16
Color range : Limited
Color primaries : BT.2020
Transfer characteristics : PQ
Matrix coefficients : BT.2020 non-constant
Mastering display color primaries : R: x=1.000000 y=1.000000, G: x=1.000000 y=1.000000, B: x=1.000000 y=1.000000, White point: x=1.000000 y=1.000000
Mastering display luminance : min: 0.1000 cd/m2, max: 0.5000 cd/m2
Codec configuration box : hvcC

Audio
ID : 2
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Codec ID : mp4a-40-2
Duration : 1 min 31 s
Bit rate mode : Constant
Bit rate : 192 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Frame rate : 46.875 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 2.10 MiB (0%)
Language : English
Encoded date : UTC 2017-02-27 03:31:16

You are right. I know it is 10bit HDR, I just test the ability of VCE. By default, without specifying the encoder, it is 264 encoding, which was really forgotten at that time. Since my DELL display is only an 8bit old guy, I remember that 10bit is a prerequisite for HDR, so I did not transcode to HDR. I think it was download -- https://4kmedia.org/sony-whale-in-tonga-hdr-uhd-4k-demo

log-level debug Forget it, I don't understand, haha. I am just an ordinary user, can not compare with you experts......