When to judge a book by its cover

If you were to take a random exectuable file from our (or any other vendor’s) malware collection, the chances are it would be packed with something. These tools, often just referred to as packers (but sometimes split up into packers, protectors, encryptors and obfuscators) are applications that change executable files in several ways while still letting them run normally. The reasons for doing this can vary from wanting to compress the file to save space to making it harder for reverse engineers to find out how the program works.

These packing tools often change the structure of the program file so much that, to an anti-virus scanner, the file can appear to be completely different to the original despite the fact that, for the end user, the program behaves identically. This is why packers are very popular with malware authors — and in the last couple of years we’ve seen a trend of packers written just for malware, updated daily (or even hourly) so that the function of the original program and its malicious behaviour stays hidden from anti-virus scanners.

For us, there are a few way we can deal with these packers:

• We can detect them outright, causing any program packed with them to be blocked as malware, whether it’s actually harmful or not.
• We can write unpackers for them, which attempt to reverse the changes made by the packer and let our virus engine scan the program in its original state.
• We can use the presence of a certain packer when writing heuristic rules for judging whether a file is malicious or not. We wouldn’t be detecting the packer on its own, but if we see other suspicious properties of the file then together they might trigger detection.
• We can write specific detection for that sample only. If we do this, it’s likely that we’ll only ever detect that exact file. If the program is packed again with a different packer, or sometimes even with the same packer, we probably won’t be able to detect it with the same virus definition.
• We have HIPS technology in our endpoint product that will detect and block malicious behaviour of the program once it’s running regardless of the packer used. The downside here is that the user has already executed the malware so we’d rather detect it before that happens.

The main question we need to ask when deciding which of these approaches to take with a packer is “How dodgy is this?” Unfortunately, this isn’t an easily measurable thing. Here are some things we’d consider when deciding whether to detect a certain packer:

• Is it only ever seen with malware? If so, is it always seen used on the same families of malware?
• How is the packer distributed? Is it commercially sold or open source, or is it only made available on hacking forums and through malware-related sites?
• What is its primary function? Is it designed only to compress the input file or is there also anti-reversing functionality?
• If it has anti-reversing features, are they designed to thwart human analysts only or also to stop virus engines from analysing the file?
• Are there any commercial or popular freeware applications using it or is it only used by illegal programs such as cracks and keyloggers?

We need to consider the trade-offs when deciding what to do with a packer. If we block a packer outright then we’ll be guaranteed to detect all the new malware that’s packed with it. Unfortunately we’ll also be generating false positives on any legitimate programs packed with it. We also have to consider whether it’s fair to the author of the packer — who frequently has no control over who uses it — to block it just because the vast majority of files seen packed with it are malware.

For packers that are seen used in a considerable amount of malware but aren’t considered “bad” enough on their own for us to detect them outright, we need to think about writing an unpacker so that we can differentiate between clean and malicious files packed with it. This is an expensive process; it takes time to analyse the packer and understand it, and more time to write and test our own implementation of its unpacking algorithm. We need to weigh up the time we think it’ll take to unpack the packer against the improved detection we can expect as a result.

In the end, most vendors tend to do pretty much the same thing when it comes to packers. Most AV products are going to try to detect malware packers such as Tibs and Swizzor regardless of what’s underneath them. All of them will unpack the most widespread legitimate packers like UPX and ASPack. Where they tend to differ is in their approach to the less popular “grey area” packers and whether they unpack them, detect them or just issue specific detection when they’re used on malware. There’s no straightforward solution to this problem but most of those packers are so rare nowadays that they have a fairly minor effect on protection.