I recently wrote about a comparative test of Linux products and how such a limited test set was not representative. In the past few days AV-Test.org have released their latest set of results. The methodology used is a far better way of testing on demand detection rates.
- A large collection is used (in this case over 800,000 samples)
- All samples are verified that they execute and perform "˜some sort of' malicious behavior
- Only samples from the last 6 months are used.
- All products tested use the "˜maximum' possible settings
However, detection rates of large collections like this are only part of the process that should be used when evaluating security products. False positive tests are just as important, incorrectly identifying clean files can take an administrator as long to resolve as a "˜real' infection.
Performance, memory footprint, detection with "˜default' settings (or at least the setting you are most likely to use) are all important factors.
On demand scans are typically run periodically, whereas "˜on access' scanning offers protection all the time so the detection rate, behavior and performance of this component is even more important.
Many security products contain additional protection layers, and as I have described previously the latest version of Sophos Endpoint Security and Control not only includes static scanning of files but also buffer overrun protection and runtime behavior blocking, which according to our own internal tests, catch approximately 50% of new malware without the need to produce and distribute a signature.
Having a security product on your machine is obviously vital, but just as important is the ability to manage that system, especially for our customers who are managing anywhere between 10 and 100,000 desktops. Ensuring every workstation is running up to date and correctly configured security is just as important as detection rate. We have often had cases where, once deployed, a large number of "˜old' malware is discovered on a network, not because their old solution couldn't detect the malware but because they hadn't been kept up to date or managed prior to the installation of Sophos.
Another important aspect to take into account is support. Not just the level of product support offered by the vendor, but how much resource an organization needs in order to manage and maintain the product.
There is also the question of the frequency and size of updates and of course analysis capabilities. If updates are only sent out once a day during the working week, less protection is available than a full 24 by 7 operation but conversely, having updates every few minutes or even every hour that are not well tested introduces unnecessary overhead in management and infrastructure.
So when evaluating a security product, all of these factors need to be taken into account.
- Detection tests like those carried out by reputable organisations such as AV-Test.org.
- Tests of on access and on demand default configuration like VB100 from Virus Bulletin magazine.
- Certifications on current threats such as those done by West Coast Labs.
- Independent product reviews such as those carried out by Cascadia Labs and reputable magazines.
More details of the independent tests carried out on Sophos products can be found here.