9. Brute Force Deobfuscation of Encoded Shellcode

While SHAREM’s emulation feature can deobfuscate encoded shellcode by emulating a decoder stub, SHAREM also possesses the capability to perform brute-force deobfuscation without emulation. This feature is useful when encountering obfuscated shellcode that is not able to be decoded via emulation and can use both parallel and distributed computing techniques to improve decoding efficiency. This can also be used to successfully deobfuscate, even if there is no decoder stub, although that is not likely. This feature can decode shellcodes obfuscated by using one or several keys to apply a bitwise operation to each byte. For example, each byte may be XORed by 0x62, incremented by 0x123, then XORed again by 0x99. To check whether a shellcode was successfully decoded, SHAREM searches for the PEB walking sequence in the potentially decoded bytes.

The brute-forcing menu can be accessed by typing b at the main menu. If a decoder stub is available in the shellcode, the s option will automatically detect it and select likely values and operations for fast decoding. The user simply needs to enter in the entry point of the shellcode, then confirm. For shellcodes with no decoder stub, the operation types to use during brute-forcing can be selected via the o option. Although SHAREM supports up to 5 operations, increasing the amount above 3 will begin to significantly increase potential combinations and decoding times.

The d option brings up a submenu of advanced brute-forcing options:

• fm – Fast mode decreases brute-forcing time by only applying operations to the first 40 bytes. While this option improves speed, a successful decryption may be missed if the PEB walking sequence comes later in the shellcode.
• fa – If desired, this option can be toggled to find every potential key or combination of keys rather than halting the brute-forcing procedure once a match is found. Note that we have found that in multiple rounds of encoding, e.g. adding 5, subtracting 2, xoring 3, that there can be multiple alternative combinations that can successfully deobfuscate shellcode in many cases! So while it may not find the one the author used first, it may find another one.
• d – Enables distributed mode. More information about this feature can be seen below.
• c – Allows the user to specify how many CPU cores will be used to brute-force the shellcode. By selecting “auto”, SHAREM will detect and use the maximum number of available cores. Choosing “1” will disable parallel computing and only use a single CPU, which can greatly increase the time it takes, if there are multiple encoding operations applied.
• n – Specify the nodes file for distributed mode. This file should contain a list of all of the IPs that will be used for distributed brute-forcing.
• o – Toggles whether a report file containing decryption results will be created.

SHAREM’s brute force deobfuscation is highly effective, and we have not noticed it ever finding false positives, nor have we noticed it missing any. Our algorithm at determine success is extremely effective.

If a valid set of decryption keys is found, SHAREM will display the key values as well as the disassembly of the decoded shellcode. Just like with decryption via the emulation feature, SHAREM will automatically switch to the newly decrypted shellcode for further analysis actions unless the user toggles back to encoded shellcode via the U option at the main menu.

The brute-force deobfuscation can also scan a binary and extra all possible operations and numerical constants, trying each. Thus, if present, it can do this in seconds, and one operation (e.g. XOR) could be down also in seconds, even if there is no decoder stub.

The only limitation with brute-force deobfuscation – and a significant one – is that it is restricted to just the operations selected, of which there are several possibilities. For instance, XORing each byte by a set key value, and then adding to each byte by a set value, etc. More elaborate or advanced encoding methods cannot be handled by brute-force deobfuscation at this time, and they would require the emulation to decode the shellcode, which it does automatically in little time.