Webhosting management company cPanel recently announced a worrying sort of compromise.
A break-in to one of the company's technical support servers put customers at risk by exposing Personally Identifiable Information (PII).
This time, the PII was of the most intimate sort: root (i.e. administrative) passwords.
As cPanel explained in a support forum:
You are receiving this email because you have opened a ticket with our support staff in the last 6 months. cPanel, Inc. has discovered that one of the servers we utilize in the technical support department has been compromised. While we do not know if your machine is affected, you should change your root level password if you are not already using ssh keys. If you are using an unprivileged account with "sudo" or "su" for root logins, we recommend you change the account password. Even if you are using ssh keys we still recommend rotating keys on a regular basis.
Fast forward to yesterday and cPanel had, thankfully, come up with a reponse that is a modest silver lining.
Giving remote support staff administrative access to your servers is always fraught with danger.
It's a bit like handing the keys to your fancy new car to the valet parking guy.
You assume that he's insured, and that he's a competent driver, and even that he's a lot more experienced than you at reversing in and out of the tricky spaces in the parking garage.
But he has, after all, got hold of your keys, and they're probably hanging on a peg in a rack where, for all you know, sneaky passers-by could make off with them.
(Or, worse still, make off with a copy of them, so you don't even know there's been a compromise.)
That's what happened here, to an unknown subset of customers over an extended period.
The good news is that cPanel has quicky moved towards a system based on disposable SSH keys that will now authorise cPanel support staff only temporarily to dig into your servers.
By automatically retiring the relevant cryptographic material after each session, cPanel hopes that this sort of problem can largely be avoided in the future.
Of course, there are still some ironies in what happened at cPanel.
If you were already using SSH keys on your servers, rather than typed-in passwords, you weren't affected by this hack.
→ SSH offers two main forms of authentication. One relies on you typing in a password every time you want to login. An attacker who sniffs or steals the password anywhere between you and the server can get in later at will. The second relies on a public-private key pair. You upload the public key to the server at the outset, and use your locally-stored private key to authenticate each time you access the server. You still have to keep the private key secure, of course, but you can store it off-line and encrypted, thus minimising your exposure to danger.
One wonders, therefore, why cPanel continued to allow the use of password-based logins for remote support purposes, and why it didn't make more effort to discourage its customers from working that way.
This is always a question you need to ask yourself after a data breach: did anyone actually need to collect the stolen data, or was it collected merely because it was operationally convenient to do so?
Another irony is that this is already being excused with the S-word.
This has been labelled "a sophisticated attack", in the same document in which cPanel admits that it does "not know the exact nature of this compromise."
That's another habit we need to break: that of adding dramatic adjectives (such as the A in Advanced Persistent Threat) even when we aren't sure that they actually fit the situation.
That's a bit like crying "fire" when what you really meant to say was "smoke", or "wolf" when you mean "moving object."
It also makes those who have suffered the misfortune of what they themselves consider to be an unsophisticated attack feel as though they're deserving mugus who had it coming to them, rather than the victims of cybercrime.