You may remember a couple of months ago, I attended a conference on testing of anti-virus products in Reykjavik, Iceland. One person that was obviously missing from the event was the author of a comparative review of Linux products. The ‘results’ have been published here and discussed by a number of different people including the guys at McAfee and Eddy Willems of EICAR.
Whilst I’m sure the tests were well intentioned, they are not particularly scientific and as the author admits, Sophos results were significantly improved if the tester turned on the relevant options, suggesting the product documentation was not consulted.
Having retrieved the samples (the author having posted the malware samples on a public website!!!) it appears a few extra settings were required, in particular, one of the samples was a email stored as MIME but the ‘decode MIME’ option wasn’t turned on. Another sample was in fact a potentially unwanted application and again, the option to enable detection for PUAs wasn’t used.
While Sophos performed better than many of its competitors, the sample set was far too small, the methodology was confused and the author obviously isn’t well versed in handling malware. All of this goes to show that testing anti-virus products is a lot more complex than grabbing a few samples and scanning them.