In my day job as a tester of anti-malware solutions, for some time now I’ve been getting asked the same question by all sorts of people. They want to know how I plan to test against Advanced Persistent Threats, aka APTs.
It may seem like an obvious question – APTs are considered a major issue affecting larger organisations, they generally involve malware of some sort, so we should be able to compare how well different products handle them.
But targeted attacks are very different from your everyday malware, and testing protection against them turns out to be a very different kind of task.
Advanced Persistent Terminology
The phrase “Advanced Persistent Threat” has been around for some time.
The most commonly cited origin story dates back to 2006, when US Air Force researchers are said to have coined the term.
It really came into common usage in early 2010 though, when the “Operation Aurora” campaign penetrated networks at Google, Adobe and several other big name firms.
The term was immediately and enthusiastically taken up by media and marketing types, and as such was of course roundly dismissed by more technical folk as a mere publicity buzzword, quite unnecessary when the standard term “targeted attack” sufficiently covered the threats involved.
It has since been fairly generally adopted to refer to crafted, long-term attacks on larger organisations.
When Stuxnet was discovered, it became an archetypal APT.
It was aimed at a specific well-protected target, generally agreed to be systems controlling Iranian nuclear processing; it was tailored to penetrate that target, by using USB thumb drives to jump air gaps; and it had a payload designed to have a particular effect on that target, apparently to skew the running of fuel-processing centrifuges and potentially destroy them.
Perhaps it is this association which has confused the issue. The Stuxnet seen by the world was a specific item of malware, and it seems in many minds the idea of an APT has remained that simple – a single executable which can break in somewhere, find its way around and carry out its mission.
The reality is that most APTs, Stuxnet included, are more like long-term team operations than one-off, one-man attacks. They will involve many different people doing different things at different stages, with all their actions and the data they gather interweaving to produce the end result.
Some of the stages will involve actual items of malware, some will require more hands-on, real-time hacking and exploration of backdoored networks, and some will be much more human, such as physically poking around a building that houses systems which may be targeted.
Attacks and Prevention Techniques
The threats “targeting” home users and smaller businesses are not really targeting you at all – they don’t care who you are or what you do, they just want to break into your computer and harvest your bank login details, sniff for payment transactions, lock up your files and demand a ransom, or simply rent out time on your machine to spammers.
So, you can choose a general-purpose anti-malware solution, and can base your choice on standard factors like price, support quality, and performance in comparative tests. A product which provides a better range of protection against all sorts of attacks offers you a better chance of avoiding being hit by anything.
Larger business and institutions are in very different positions though.
Everyone from retailers to software houses to defense contractors to stock exchange operators will be running much bigger, more complex and more valuable networks, across multiple sites, with data and capabilities which could be hugely valuable if they fell into the wrong hands.
Of course, they still need to protect against the everyday broad-spectrum malware, and will probably be relying on high-quality anti-malware to help them do so.
They’ll probably have multi-layer protection at the desktop and at gateways. They’ll have used independent tests to help choose which products to use too, perhaps weighing up both comparative and certification data.
Of course, for them, factors like support and scalable management facilities will have considerably more weight than they would for small firms or individuals.
But they’ll also have a wealth of other security measures too. Sophisticated firewall rules, demilitarized zones, intrusion detection systems, data leak prevention, careful assignment of access rights, vetting of staff, contractors and visitors, and all manner of physical security measures all come into play when it comes to blocking targeted attacks.
The point about carefully targeted attacks, including APTs, is that they will take all these factors into account when designing their attack strategy.
If they know you have bulletproof glass, they’re not going to try throwing a brick through a window; if they know your AV’s behaviour monitor alerts on certain registry changes, they’re not going to make those changes until they’ve disabled or subverted the AV.
Actual Protection Testing
So how can we test how well anti-malware solutions perform against targeted attacks?
Sure, we can take a bunch of files known to have been used in targeted attacks in the past, and compare detection of them by various solutions, but this will have little value.
We will never be able to say “Product X would have protected you against Attack Z”, simply because if you had been using Product X rather than Product Y, Attack Z would have taken a different form. It would have been tweaked to ensure it got around any barriers Product X put up.
One thing we could do is to try to figure out how easily products can be bypassed or subverted, by creating new malware or attacking components of the solution.
The morality of the first is dubious (anti-malware firms hate it when you create new samples for them to deal with, given the amount of real-world stuff turning up every second), and the second is, one would hope, performed all the time as part of in-house QA and hardening.
The efficacy of both as a test metric is also suspect. We could put together a list of a dozen ways of taking down a solution, and rate products on how many they proved immune to, but there’s no saying that the guys behind Attack Z won’t come up with another method we never thought of, use it as their first attempt and successfully beat the product we rated top.
The whole point of these holes is that no-one thinks of them until someone thinks of them.
The only real way of measuring resilience to APTs is penetration testing of a fully-secured site or system, checking every potential hole you can think of to see if it’s covered, but again this relies on the pentester being at least as knowledgeable, well-resourced and creative as any possible attacker.
This sort of testing is also not really useful as a guide for others on what to do. Even if you could sum up all the measures in place in a site found to be fairly secure, it would be near-impossible to ensure you could reproduce that exact setup in a different place with different people, needs and behaviours.
Avoiding Potential Tragedies
There are all manner of solutions offering to protect from APT attacks, and for all of them it is similarly difficult to validate their performance thanks to the unknowable nature of potential attacks.
No matter how tough a set of locks you put in place, if you have something someone wants badly enough, they’re going to keep on picking away until they find a way through.
Even if your technological protections do prove too tough to crack there are likely to be other vectors – there aren’t many products promising to keep your sysadmins safe from blackmail or hypnosis, for example.
In the end, the best way to make sure your defences are as strong as they can be is to follow established best practices, using the most comprehensive set of protections and making sure they are properly implemented and maintained.
Each time a new penetration takes place, at least one which doesn’t rely on some sort of error or sloppiness, it’s vital that information on what went wrong is shared so that those best practices can be expanded and updated.
When fighting an invisible enemy, it’s hard to guard every angle, perhaps even to know all the angles which need guarding, but learning from the mistakes of everyone around us makes us the toughest possible nut to crack.
Images of target and padlocks courtesy of Shutterstock.
5 comments on “Can we test protection against targeted attacks?”
Great article. The only issue I had was with…
“If they know you have bulletproof glass, they’re not going to try throwing a brick through a window; if they know your AV’s behaviour monitor alerts on certain registry changes, they’re not going to make those changes until they’ve disabled or subverted the AV.”
Not so true. In a multifaceted attack, pounding on your bulletproof glass or hammering you AV could provide enough of a distraction for me to get into your systems thru another avenue and go unnoticed for a long time.
Point, take notice in all attacks and try to see patterns where others see noise.
Keep on fighting the good fight, thanks for the article.
Thanks for the comments – you’re right of course, what I meant was “they’re not going to expect throwing a brick to get them through the glass.” As you say, they might well include throwing bricks, or any similar distraction technique, as part of a bigger attack plan.
In the old days, the best you could do to keep the bad guys out was to stay off their radar.
Unfortunately, that’s no longer feasible; the bad guys are now interested in any PC they can own, in addition to targeted attacks. And, the ones who do targeted attacks also own PCs they commandeered via mass-market attacks.
BUT, an organization can try to assess what kind of threats they’re likely to see against them, and focus their “extra” efforts on that kind of threat. Always do the basics, but look at things which could affect YOU specifically.
For examples, a bank has iron-solid protections up (meeting or exceeding regulators’ standards), but they are vulnerable to human engineering. Educating their employees is a top priority, especially as to why their protections are in place.
A defense contractor also has iron-solid protections (also regulated and certified), but they are vulnerable to compromised-insider actions. Division of powers is paramount, along with investigation of employees and sub-contractors.
A mom-and-pop bakery is vulnerable to more run-of-the-mill mass-market hack attempts. They should stay on top of patching, and keep read-up on more general security threats. Follow basic safe-computing practices.
What do you think about paralell test? I mean that the same network traffic would be monitorized via SPAN ports by four different solutions against APTs during the same period of time. Comparing the results would help you to evaluate the products.
I would complete this test method by classifying the recognized actions, a recognized virus/malware based on pattern database would gain less points than a complex attack recognized by its behaviour.
John, what is your opinion about this test method?
This sort of parallel approach could be a good means of comparing network filtering products (note that there are others layers where protection against APTs is claimed), and rating different types of detection/protection according to the detection type or layer is a useful metric, although one which is difficult to properly weight (not sure why an exact detection would be worse than a behavioural one for example).
You still come down to the sample problem though. Even if you are throwing in samples (ideally full attack models from working live URLs) known to have been used in the past in an APT-type attack, you’re not really measuring protection against that specific APT – as soon as you know about it, it’s already done its job and is no longer really an APT, just another malware sample.
This is the main problem I was trying to highlight here, that because (by most definitions at least) APTs are specifically targeted at a given environment, and because that environment includes all the security infrastructure in use in that environment, seeing how other products fare against it in other settings once the threat has become a known quantity can’t show you how well those products would have fared had they been in place in the original environment – had they been there, the threat would have been different.
A popular way around this is trying to create your own attacks in what you think is the style APT people might use, but there you’re really only testing your own skills rather than measuring handling of all possible real-world attack techniques.