RBS, Rapport and OITC anti-virus test results

An email from a customer today brought my attention to some anti-virus test results that have been published on the website of RBS (Royal Bank of Scotland).

At first glance, the test results look quite bad for Sophos (and even worse for Symantec and McAfee).

OITC test results promoted on RBS website

But if you dig a little deeper into the methodology used by OITC to come up with the results – published by RBS on their page promoting a security add-on called Rapport – then you actually find that the methodology is flawed, and that these test scores are about as useful as a chocolate teapot.

As Stuart Taylor describes in a post on the SophosLabs blog, OITC’s methodology actually penalises the likes of Sophos for their ability to proactively detect brand new malware using (in our case) behavioral genotype protection. That’s because they exclude from their tests any piece of malware which they find 25% or more of security products already detect.

That’s bonkers. (They did this I presume in the mistaken hope of determining if a piece of malware was new or not, but in the process penalised products which proactively detected it).

Furthermore, these results don’t give any allowance for layers of protection such as run-time suspicious activity or buffer overflow detection, both of which would be defending customers in the real world.

My advice? Check out the independent comparative tests from the likes of AV-Test, AV-Comparatives and Virus Bulletin. They may not always put Sophos top of the class for virus detection, but I sure trust their testing methodology more than OITC.

I hope in the future RBS might link to some of those tests for a more helpful indicator of the performance of anti-virus products in the future.