x86: decoding of mandatory prefixes is too strict

[jxors]

Work environment

Questions	Answers
System Capstone runs on OS/arch/bits	Ubuntu 24.04 (x86 64-bit)
Capstone module affected	x86
Source of Capstone	git clone
Version/git commit	8872be6

Instruction bytes giving faulty results

f33e66670fa484170000000000

Expected results

It should decode successfully, but fails to decode instead.

Steps to get the wrong result

With cstool:

cstool -d x64 f33e66670fa484170000000000

Additional Logs, screenshots, source code, configuration dump, ...

This issue appears to be caused by the logic in setPrefixPresent. It assumes that instruction prefixes appear in the order of their group (i.e., as if only 3ef366670fa484170000000000 should be valid), but as far as I can tell this is not the case.

The AMD reference manual Volume 1, Section 3.5.1 says:

The legacy prefixes can appear in any order in the instruction, but only one prefix from each of the five groups can be used in a single instruction.

Similarly, the Intel manual Volume 2, Section 2.1.1 says:

For each instruction, it is only useful to include up to one prefix code from each of the four groups (Groups 1, 2, 3, 4). Groups 1 through 4 may be placed in any order relative to each other.

Additionally, I have confirmed that my CPU executes this instruction normally.

I would be happy to work on a patch to fix this.

[jxors]

I have looked into this some more, and it seems that there may be two closely related bugs:

• The logic in setPrefixPresent does not appear to be correct. For example f2660fa484170000000000 should decode to shldw $0, %ax, (%rdi, %rdx) but decodes to shldl $0, %ax, (%rdi, %rdx) instead because of the mandatoryPrefix set by setPrefixPresent.
• On top of that, there is a bug somewhere that causes f33e66670fa484170000000000 to fail to decode correctly. I suspect this is related to the operand size override prefix 66. The ATTR_OPSIZE attribute is not set if f3 is selected as mandatory prefix, which would hide this bug.

From what I can tell, the LLVM TableGen tables do not generate (some?) entries for instructions with both ATTR_OPSIZE (from the 66 data size override prefix) and ATTR_XS or ATTR_XD (from the F2/F3 REP/REPNE prefixes).

The problem is that it is hard to determine which attribute should take priority: if the prefixes are a mandatory prefix, the REP prefix should take priority and the data size override should be discarded*(1). If the prefixes are not mandatory prefixes, the REP/REPNE prefix should be discarded since the data size prefix may affect the meaning of the instruction*(2).

*(1): except for instructions in 0FB.., where the data size override functions as normal but a REP mandatory prefix is also required so both should be preserved.
*(2): except for string instructions, where both the data size override and the REP/REPNZ prefixes can be present.