Comparative anti-malware tests: the RIGHT way to do them

Note: In the interest of open disclosure, I am senior malware researcher at SophosLabs with a keen interest in improving testing procedures and a founding member of the Anti-Malware Testing Standards Organisation (AMTSO).

The latest anti-malware tests, performed by Simon Edwards for Dennis Technology Labs, show that comparative testing can actually be a strong indicator of how well a security solution can protect a user.

Conducting these types of tests is not easy or even straight-forward, but Dennis Technology showed that it is indeed possible.

The internet is full of so-called testing bodies who scan their malware collections with a handful of on-demand scanners and publish the rankings.

Measuring on-demand detection rates provides a performance indicator, but it doesn’t actually tell users what they want to know: are their systems as safe as possible from the glut of malicious attacks out there?

Full product testing introduces what is known as a ‘malicious attack scenario’ to a victim computer.

Good tests try to emulate a known attack in a real-life setting, with the aim of recording the protective measures employed by the security software to stop the attack at the earliest stage. The ultimate goal is to assess how well the solution protected the victim system from attack.

These types of tests need to take into account system setup, user behaviour, installation configurations, etc. It is a complicated set of parameters to get right.

To illustrate this point, let’s look at web attacks using the most successful exploit kit we have seen in the last few years, the Blackhole exploit kit. There are a number of stages in the attack, and today’s multi-layered security solutions should attempt to prevent infection at each stage.

Stages of a web attack using the Blackhole exploit kit

First, this type of malicious attack is usually initiated via a simple web link. It can be delivered via email, drive-by exploit or in browser search results (a result of search engine poisoning).

In the case of email distribution, spam filters are present to filter out unwanted and malicious messages. With drive-by or search engine poisoning attacks, web filtering is there to catch the iFrames pointing to known URL patterns and should raise an alert.

Second, the attack usually employs simple redirector scripts. These are often encrypted, which alerts malicious script detection. Should the redirector scripts not be encrypted, URL filtering and reputation defences are designed to alert the user or administrator to suspicious activities.

These redirections point to a hosting server, which, in the case of the Blackhole exploit kit, features a “traffic direction system” (TDS). Collecting browser and OS version information, the TDS returns a collection of exploits tailored to the specific vulnerabilities present in the victim environment.

Third, exploits are delivered to the system under attack. Delivered content may contain simple VBScript downloaders, PDF, Flash or Java exploits, along with some rarely used Windows vulnerabilities. This malicious content is reliably detected by the up-to-date exploit prevention modules, on-access scanners, or content filtering components.

Fourth, should the exploits be successful, the victim machine reconnects to the hosting server for the binary payload, which is subsequently downloaded and executed.

And it is at this fourth stage where most real-time protection tests start their assessment; at the point where the URL points to the executable content. Attack stages one through three are rarely, if ever, considered by most real-time protection tests.

This shortcoming in tests was highlighted at a 2007 International Antivirus Testing Workshop. Discussion on this topic conceived the birth of the Anti-Malware Testing Standards Organization (AMTSO) the following year.

To improve the quality of testing – and to test in a way that truly mimicked a real-time user experience – seemed to be mission impossible at that time. Fortunately, the combined forces of testers and anti-virus experts created some very useful guidelines and best practices.

Dennis Labs participated actively from the very early days of AMTSO, working on documents like whole product testing guidelines. It is good to see that they were not interested only in theory, but adopted these guidelines in practice as well.

You can read Dennis Labs’s self-imposed guidelines to testing web threats, but here are the highlights:

• They do not take sample feeds from vendors
• They try to always use URLs containing exploits
• They may include social-engineering attacks
• They use complex samples.

Of course, for a multi-layered, anti-malware solution developer, the number one priority is the safety of its users’ systems.

People in the market for new security approaches look to independent tests as a guide.

And, yes, all vendors want to do well on tests, but they are so much more valuable if tests are properly executed and look at the whole security solution, rather than just some of the components.

_{Tested image and Thumbs up image courtesy of ShutterStock.}