I’m currently at the International Anti-virus testing workshop hosted by Frisk (makers of F-Prot) in Reykjavik, Iceland. This may seem a strange topic for a two day conference but its actually a very complex subject.
To test an anti-virus scanner you need to obtain a collection of malicious files, then you need to verify that they are actually malicious, which is certainly non-trivial.
With the changing threat landscape, a simple on demand scan of a collection is not necessarily an accurate representation of the performance of a product, many products have combinations of protection mechanisms. The next release of Sophos endpoint security includes buffer overrun protection as well as runtime resource shielding, none of this functionality would be tested with the current testing techniques.
There have been a number of heated debates, the conclusions so far are :
– On demand scans of large collections of malware are not an accurate representation of the level of protection a product provides.
– On demand scans of large collections of malware are what customers and publications want to see.
– Any alternatives are very difficult
So it looks like traditional tests are here to stay, and the focus of the debate and presentations have shifted to how to ensure that the samples used in such tests are really malicious and not corrupt.
Coffee break is over so time to get back. More later