Proactive Detection – The devil’s in the detail

Last week saw the publication of the latest report from AV-Comparatives.org on proactive detection rates. The process followed is to take a product that is effectively out of date (i.e. no updates applied) and scan new malware samples to see what percentage would be caught. Depending on the product, this detection rate can range from over 70% to as little as 5%.

Quite rightly though the other part of this test concentrates on false positive rates, that is detection of “˜good’ files as malicious, or potentially malicious. The point being that there is a cost to blocking files that have legitimate use.

So although Sophos received the highest proactive detection rate (74%), we didn’t receive certification because our “˜false positive’ rate was far too high. On face value this doesn’t look encouraging. However, as the report highlights there are a number of caveats to this data.

Firstly, the report was created using the Suspicious File Detection option within Sophos Anti-Virus. Sophos marks files as “suspicious” if they display characteristics that are common to, but not exclusively found in malware. The setting is optional and quarantined files are marked as “suspicious”. It is easy to authorise files that are stopped.

For most consumers (that this report is aimed at) the differentiation between a file that is definitely “BAD” and something that “˜might be bad’ is very hard. Because the enterprise is our focus, consumer applications, especially, freeware and shareware applications are not the focus of our false positive testing. Companies may find that distinction easier, but also accept that most “suspicious” programs are not suitable for a corporate environment.

To give you an example, one of the applications incorrectly detected as “˜Mal/Packer’ was “˜PodTools’ a series of utilities for copying music on and off an iPod. This is a free application, whose author tries to protect the product by compressing it with a packing tool. However, the packing tool chosen is also very widely used by malware to hide the contents from scanners. By detecting the fact that the files is packed with this particular tool, we prevent large amounts of new malware. To our customers, the fact that we inadvertently block a tool for copying music as well is not seen as an issue, in fact in some ways it can seen as an extension of the application control functionality.

The second point made in the report regards the settings used. There is an option available within the Sophos products called “˜Extensive Scan Mode’. This turns off a number of checks so that all files are scanned from end to end and whilst it can occasionally find instances of malware normal scanning mode would otherwise miss, the overhead in terms of performance (up to 70% slower) and the increase in false positives means it is very rarely recommended. In fact the function was introduced many years ago specifically to look at “˜fragments’ of viruses within files. Disabling this option, reduces the number of “˜false positives’ significantly.

Without using the Suspicious or Extensive options, Sophos would have still detected 39% of the malware proactively. This is comfortably ahead of any of our major competitors (e.g. McAfee and Symantec) and “false positives” would have been drastically reduced by over 90%.

The final point made in the report is that this is the first time we have participated in this test, so it’s the first time we’ve been exposed to the particular false positive collection. At the end of each test, AV-Comparatives sends any false positives to the vendor so that they can verify that they agree they are “˜clean’, but obviously this means the vendor has an opportunity to correct the FP’s in future tests. So although the false positive test set changes over time, the changes are incremental, and vendors that have been part of the tests for a number of years have better visibility on the test set.

That said, we do take false positives very seriously, and reducing the number of “˜unwanted’ suspicious detections ensures that the effort by system administrators is a minimum. We are working hard on removing all the detections identified by this test, whilst continuing to maintain the highest possible proactive detection