I recently came across a new anti-emulation tactic for unpackers that I thought might be worth sharing. This one is a new angle on a previous technique, to use the error code returned from a Windows API call as part of the decryption key material -- previous posts are here and updated here. I even recall reading through these posts before joining on with Sophos (as part of my interview preparation!) and thinking "Well, isn't that clever."
Slightly less clever is to omit the call to GetLastError and use the straight-up return value from the API call -- especially when this return value is always zero. Last time I checked, the number zero does not exactly add too many bits of entropy to your key generation.
However, I will give the malware writer some credit -- I don't believe this was their only strategy for using the API call (even though the code explicitly mixes the zero return value in with the hard-coded key seed). Check out the disassembly below.
The stack construction seems particularly deliberate here. There's no real need to save the ecx, ebx or edx registers explicitly -- they should be preserved by API call. But SetMenuItemInfoW takes four arguments, and with the standard calling convention used by Windows APIs, it clears them from the stack, not the caller of the function -- hence the immediate restoration of the edx, ebx, and ecx registers after the call. But assuming an emulator is going to cheat and just skip over the API call without clearing the stack appropriately, then edx, ebx, ecx and the return address from dummy_api_call used by retn all become zero -- wiping out the important values held by those registers and returning the flow of execution to never-never-land.
In the end, this technique is less than successful at deterring analysis. The narrow focus on correctly emulating API behavior misses a vast array of other avenues available to an analyst -- there are lots of tools out there that let you manipulate the stack. Analysis for this packer proved reasonably trivial, and Sophos detects this with Mal/EncPk-FW.