Thanks to Ross McKerchar, our CISO at Sophos, and Luke Groves, one of our Senior Penetration Testers,
for their help with this article.
The past week has seen the beginning of a saga that feels as though it could end up like Homer’s Odyssey or Virgil’s Aeneid…
…a fascinating, entertaining, confusing, politically charged and unpredictable tale, littered with lyrical allusions and based on mysterious sources; a supposedly factual tale that the tellers nevertheless describe in mythological terms as “like witnessing a unicorn jumping over a rainbow” and as “a feat akin to throwing a stick in the Yangtze River upstream from Shanghai and ensuring that it washes ashore in Seattle.”
(Actually, transporting a stick from the Yangtze and dumping it on a beach in Lake Washington isn’t a particularly difficult feat these days, thanks to long-haul air travel.)
This saga was years in the making and will probably end up as prescribed reading in years to come for any number of students who’d really rather be trying to fathom something altogether more straightforward, such as programming elliptic curve cryptography from scratch – or, for that matter, translating Homer from the original Greek.
We’re talking, of course, about the astonishing claims published by US technology publishers Bloomberg that Chinese military spies successfully infiltrated at least 30 major US companies, starting about three years ago, by covertly implanting ultra-tiny “zombie chips” onto server motherboards sold by a US server vendor called Supermicro.
According to Bloomberg, these chips could do two main things: call home, like any software bot or zombie, to fetch unauthorised software code; and inject this code into the system at a level below the operating system kernel, thereby subverting the kernel itself.
Bloomberg’s suggestion of how this might work is a rather simplistic example of patching the operating system so that “the server won’t check for a password—and presto! A secure machine is open to any and all users.”
In practice, access control to servers typically doesn’t work quite like that these days, with a single door that’s swung open by a function programmed into the operating system itself. But Bloomberg’s example is admittedly suggestive of the obvious danger of a kernel-level rogue helper, whether it’s hardware or software based, on any computer, whether it’s a server, a laptop or a phone.
Bloomberg seems to be saying that some of these rogue chips – allegedly added to selected motherboard builds only for specific customers, with the help of co-opted subcontractors – were surface mounted, yet small enough to evade even careful examination.
Apparently, the rogue items looked like other tiny parts named by Bloomberg as signal conditioning couplers, small components that are supposed to control electrical interference between parts of a circuit rather than to process and manipulate data in the system.
In one case, says Bloomberg, the zombie components were “thin enough that they’d been embedded between the layers of fiberglass onto which the other components were attached”, though the article almost sheepishly admits that this particular claim depends on “one person who saw pictures of the chips.”
Apple and Amazon accused
Anyway, hearsay and pictures aside, Bloomberg explicitly outs both Apple and Amazon not only as having been affected by this attack some three years ago, but also as having spotted the zombie chips, investigated the attack, and reported it to the relevant authorities in the US.
Only now, if Bloomberg has it right, is the full story starting to emerge, following several years of investigation.
But here’s the thing.
Apple and Amazon say exactly the opposite.
Indeed, Apple, in a firm but well-reasoned response, points out that numerous, more easily verified details claimed in Bloomberg’s story don’t add up, such as the number of servers it bought from Supermicro and how its server software was deployed, and therefore that the entire story might essentially be a comedy of errors.
As Apple puts it, in the right-of-reply afforded by Bloomberg, “We are deeply disappointed that in their dealings with us, Bloomberg’s reporters have not been open to the possibility that they or their sources might be wrong or misinformed.”
Apple’s Vice President of information Security, George Stathakopoulos, even wrote to the US Congress to clarify Apple’s unequivocal position on the issue, namely that its own “internal investigations directly contradict every consequential assertion made in the [Bloomberg] article.”
Intriguingly, Bloomberg itself admits to having purchased server hardware from Supermicro, but insists that “[Bloomberg has] found no evidence to suggest that it has been affected by the hardware issues raised in the article.”
Of course, given the deep mystery surrounding the story, and the possibility of zombie chips hidden where they can’t even be seen, buried in the material of the motherboard itself, you might wonder how Bloomberg feels confident to insist that Apple servers definitely were affected, while asserting that its own servers were not.
So far, the saga really is little more than a case of “he said, she said,” with anonymous sources and hearsay making up Bloomberg’s evidence, and official company statements to the contrary making up Apple and Amazon’s counterclaims. (How, indeed, could either Amazon or Apple prove a negative at this point?)
What to do?
And that leaves us with the $64,000 question, namely, “What to do?”
We put that question to our own security experts inside Sophos, and their answers all followed a similar theme, namely that all the things that a zombie chip of this sort – real or imaginary, it doesn’t matter – could be made to do…
…well, all those things can and are already being done by cybercriminals of all shades, in a wide variety of ways that can’t be fixed simply by switching your motherboard supplier or poring over your server hardware with magnifying glasses.
So, here are our top three tips for keeping the bad stuff out, and the good stuff in, even in a world where determined cybercriminals are using a range of tricks for getting in and stealing anything from computing power to customer data.
TIP 1. PARTITION YOUR NETWORKS
The divide-and-conquer approach worked well for Julius Caesar, and it can work well for you in making life harder for cybercrooks, whatever their motivation or ability.
Your marketing team’s online social media activities don’t need to take place on the same network that hosts your legal team’s database of documents; your cash registers don’t need to be directly connected to your payroll servers; and your ATMs don’t need to be visible to the wireless network in the canteen.
Imagine that Bloomberg’s allegations turn out to be true, and that Chinese spies have had a hardware foothold inside many of our networks for years – why make things even easier for them?
Don’t stick to the 1990s cybersecurity approach of having a hard exterior network shell of gateways and firewalls but a soft, gooey interior where any rogues in your midst can roam at will.
TIP 2. USE TWO-FACTOR AUTHENTICATION
Bloomberg’s example of how the “zombie implants” in the story might have worked talks about modifying kernel code to ensure that all password checks succeed, whether or not the right password is entered.
(Other kernel hacks already used in malware include modifying core kernel code at boot time; forcing all access control list checks to succeed, thus essentially turning every user into a full-blown administrator; and modifying security controls on allocated memory blocks to make exploits easier to launch.)
Adding external security validation checks to your network – for example, by requiring some sort of additional out-of-band mobile phone-based authentication when negotiating access from one part of the network to another – has two benefits.
Firstly, you reduce your reliance on internal devices that might already be compromised; secondly, you acquire an external audit trail that is harder for crooks to delete or modify to hide their tracks.
TIP 3. KEEP LOGS AND USE THEM
Lots of businesses keep logs, whether they realise it or not, for example via their operating system, anti-virus and firewall.
These logs, which provide corroboration of what happened, and when, and where, can be incredibly valuable both for prevention and cure.
Many users we speak to, however, look at their logs only rarely, and sometimes not at all – in which case, you might as well not bother wasting time collecting them.
In the Bloomberg story, for example, the “zombie chips” are said to have been capable – like most modern botnet malware – of calling home across the internet to pull down instructions on what to do next and the machine code to do it.
Zombie command-and-control traffic of that sort may be hard to spot, and you may not know what to check for at first, but network traffic is never totally invisible – unless you don’t bother to look out for it at all.
The last word
Has Bloomberg really uncovered what it thinks it’s found? Or has Bloomberg simply put two and two together and made seven?
Right now, we don’t think anyone knows, so we’re advising against taking any specific steps derived directly from the Bloomberg story to “remedy” this situation.
After all, if Bloomberg has the details wrong in this case, there are nevertheless embers of truth throughout the story, because we know that cybercrooks of all stripes are frequently discovered wandering around where they shouldn’t be, apparently at will – as any and every data breach story reminds us.
And even if Bloomberg had everything spot-on, and had provided specific details instead of relying on witnesses who claim to have seen pictures of chips added to motherboards, supply-chain hacks like this are only one of the many ways that modern criminals make off with your trophy data.
In three simple words: defence in depth.