Who’s good at counting?

I’ve reported on a wide variety of tests on this blog. Some have been very good whilst others have been very questionable. Today, my attention has been drawn to another testing site that claims to have a new take on testing the effectiveness of anti-virus products. OITC have an Antivirus Systems’ Performance Analysis Center where they claim to analyse different products ability at detecting near zero day threats.

The methodology is to obtain malware samples that are supposedly brand new, check that they really are malware and then send them to Virustotal to see what the results are. Unfortunately OITC look at the results and only include the files if less than 25% of vendors detect them.

Does anyone actually fall for this? How can a reputable testing house make such a naive decision? They are effectively saying that if more than 25% of vendors detect a sample then it must be an old file that every vendor must have already seen. OITC have happily ignored the possibility that vendors have generic detection, in Sophos case known as Behavioral Genotypes. They have also ignored the possibilities of Suspicious detection and HIPs detection. Need I go on?

This type of test actually bears no resemblance to what a user might actually experience and certainly doesn’t compare to any other major comparative test. There are numerous testing houses out there whose figures are much more credible – av-test, av-comparatives and Virus Bulletin are 3 key players who all get very different detection results compared to these guys. I trust these tests even when I don’t like the actual results.

Choosing a test site to use is very much a matter of personal taste and what you are trying to demonstrate. If I wanted a quick and easy real time comparative site then I would probably go to SRI where happily Sophos are top of the league 🙂

Sophos are part of AMTSO and, along with all the major testing houses, are committed to improving the quality of testing to make tests relevant to the readers and users of them. Clearly here is a test to get our teeth into. AMTSO has its next meeting in 2 weeks time in Budapest and I know this test will be a topic of conversation over a few beers.

What surprised me most of all about this saga was the fact that a major bank is now using the OITC results as justification for pushing their latest offering…