Sophos Cloud Optix
You should assume that your Drupal 7 website has been compromised if you didn’t patch it within just 7 hours of the release of Drupal 7.32 on 15 October 2014.
That’s the shocking warning from Drupal’s own maintainers in an extremely unusual public service announcement, marked Highly Critical:
Automated attacks began compromising Drupal 7 websites that were not patched or updated to Drupal 7.32 within hours of the announcement of SA-CORE-2014-005 - Drupal core - SQL injection. You should proceed under the assumption that every Drupal 7 website was compromised unless updated or patched before Oct 15th, 11pm UTC, that is 7 hours after the announcement.
Drupal is one of the world’s most popular website Content Management Systems (CMS) which puts the number of sites under scrutiny into the tens of millions.
The warning is a follow-up to an advisory (DRUPAL-SA-CORE-2014-005) issued on 15 October 2014 at 16:02 (UTC) that warned of a vulnerability that allowed attackers to completely compromise a Drupal 7 installation with a very simple SQL injection attack.
Ironically the vulnerability existed within Drupal’s own protection against SQL injection.
Drupal 7 includes a database abstraction API to ensure that queries executed against the database are sanitized to prevent SQL injection attacks.
A vulnerability in this API allows an attacker to send specially crafted requests resulting in arbitrary SQL execution.
Remember that a CMS is responsible for storing the content of your site in a series of databases, where it can be edited, backed up, tested, reviewed, subjected to change control and, of course, ultimately published in its public-facing form.
So once a Drupal site has been compromised, attackers can “manage” it to suit their own criminal agenda, including changing content, deleting content, and plundering its data.
Owning a site so completely also gives attackers a bridgehead from which they can attempt any number of privilege escalation attacks to gain control of the server it’s running on.
And, of course, once attackers control the content of your website, they can use it as a delivery point for their next wave of malware, inviting unsuspecting victims to visit what they assume is your trustworthy site, only to find it loaded with malicious content you didn’t put there yourself.
According to W3Techs between 1.9% and 5.1% of all websites use Drupal and about 65% of those use Drupal 7 (Drupal’s own usage statistics have it slightly higher at 84%).
At the time the advisory was issued there were roughly 1 billion websites on the internet, so at least 12 million sites needed patching.
With such a large number of targets it was highly likely that this simple and effective exploit would be automated. It always looked like a race against time and it seems that the first attacks followed within a few hours of the patch becoming available.
Today’s announcement warns users that patching now, so long after the event, is not enough. If you haven’t patched your site already then it’s been exposed long enough to have been scanned and turned over many times.
That doesn’t mean your site has definitely been compromised but, as the announcement says, you should proceed under that assumption.
So just how practical was it to get sites patched within 7 hours of the announcement?
Nik Roberts, owner of Drupal web development specialists Versantus, describes just how narrow the margins were on 15 October.
Once you're on a machine it only takes about 30 seconds to apply the patch but we have a mixture of sites hosted by us and others and it took about five or six hours to get 80 websites updated.
Many site owners will never have received the announcement and many that did will have been asleep.
What Drupal badly needs but doesn’t have is an automatic updater that rolls out security updates by default.
Drupal can automatically update modules automatically but it can only warn administrators that new core updates are available – it won’t install them automatically.
There are lots of good reasons for not forcing updates on people but the reality is that without them there will be many millions of site owners who either update too late or who never update at all.
Every Drupal 7 site that was unpatched after 23:00 (UTC) on 15 October 2014 is now a potential “sleeper agent” for cybercrooks.
WordPress, the most popular content management system in the world, took the plunge and rolled out automatic updates a year ago.
It’s time for Drupal to follow suit.
Automatic updates are a bad idea, especially on many enterprise platforms that Drupal runs on where they wouldn’t even work.
Currently, millions of those enterprise platforms are being used by hackers to mine bitcoins. Automatic updates would have prevented it. Meanwhile, WordPress has used it since Oct. of last year with great success.
For the special forked or modified Drupal sites that need manual updates, it would be an easy step to turn off auto updates.
I think how good the idea is depends on scale.
For enterprises automatic updates are an inferior alternative to their own processes, but for most people automatic updates are a superior alternative to doing nothing and then getting owned.
Automatic updates that are on by default but easy to turn off would work for everyone.
Automatic security updates for web applications is an extremely stupid idea. It means that the web application itself requires permission to write to the disk and replace the code that is running. This is something that we taught administrators and developers for years to avoid for secure applications.
No it doesn’t. You can hand over the updating process to (one or more) other tasks that each have no more than the privilege needed to do their bit.
The 12M number of affected sites is grossly over-estimated. Drupal reports less than a million Drupal 7 sites. Given that some of those are private intranet sites which would not be affected by wide-spread attacks (not publicly accessible), and that some percentage of the rest would be hosted on platforms which had protections in place, and that some other fraction would have patched in less than 7 hours, claiming that 12 Million sites are affected is absolutely inaccurate.
If there are 1,000,000,000 websites on the internet (which seems a reasonable estimate), and if 1.9% *or more* run Drupal (we give our source), and 65% *or more* of those run Drupal 7 (we give our source and note that Drupal quotes a yet higher figure), then, by the process of multiplication, you do indeed get a quantity of:
1,000,000,000 x 0.65 x 0.019 = 12,350,000
websites that probably “needed patching” (our words) with this fix.
So there is nothing “absolutely inaccurate” about what we said.
FWIW, your words suggesting that we “[claimed] 12 million sites are affected” seem carefully chosen to give the impression that we meant to suggest that 12 million sites were attacked and compromised.
As you say, some were patched within the seven hour window – good! But by Drupal’s own urgent admission, those “needed patching,” just as we said. You can’t exclude sites that were patched from our count of those that “needed patching.”
As for the concept of not patching because of other protections in place, or because your vulnerable web server is a “private intranet sites,” tell that to the victims of point-of-sale malware infections on retailers’ “private intranet systems,” eh?
These things are very difficult to estimate and my workings are there for people to see and evaluate for that reason – I also linked to both W3Tech and Drupal usage statistics for anyone who wants to dig deeper.
I note that whilst you mention things that would reduce your number, a total that Drupal themselves say is incomplete, you omit to mention any of the things that will inflate it.
I made my best effort to create a balanced article because there is a serious problem to report and I chose the low end of my range in an effort to be conservative.
All of which is a sideshow.
The nature and severity of the problem and the issues raised by the article are unchanged even if you change the number of sites at risk by more than an order of magnitude and there is nothing in my article that’s as shocking as the words from Drupal’s own security team.
Criminals don’t get opportunities like this very often.
If, say, 100,000 sites were compromised as a result of this bug they are going to be very difficult to clean up and as long as they’re infected they can, amongst other things, deliver another 500 Trillion spam emails a week.
What Drupal needs to allow the distros to work properly, so that admins can rely on the security support provided by distros. That lowers the cost of maintaining a site, integrates it with the usual security workflow, and also makes it far easier for one-man operations like I’m currently myself.
IIRC doesn’t WordPress check for automatic updates every 12 hours? Given that automatic exploits were being seen within 3 hours, and the advice to assume sites were compromised within 7 hours, doesn’t that mean an automatic update process (just) like WordPress’ would have been 5-9 hours too late?!
Remember that this is all about scale. The economic benefit for criminals comes with being able to compromise a lot of sites the same way.
If all WordPress installs work on the same start time and interval (very unlikely) then they’re either all on time or all late but they are, at worst, 5 – 9 hours late which is actually very good.
Criminals didn’t infect every machine at 7 hours and 1 minute and they’ll likely get better at it as time goes on, as they recruit more compromised servers.
Assuming they aren’t asleep when the alert comes out they’d have, at most, a 12 hour window to write and deploy some software to exploit this bug.
Then they’d have, at most, a 9 hour window to infect millions of machines instead of a window that takes weeks, months or even years to close (search our site for Conficker.)
Not only might that be enough to save millions of machines it might be enough to put organised criminals off even trying to exploit the bug.
Much more likely is that the time WordPress checks for updates is randomised so 25% of installs would be patched within the first 3 hours and 58% would be patched within 7 hours. The target for criminals has been cut in two before they’ve finished lacing their boots and they now have just 5 hours to infect the remaining 42% and vulnerable machines are getting harder and harder to find throughout that period.
> If all WordPress installs work on the same start time and interval (very
> unlikely) then they’re either all on time or all late but they are, at worst, 5 – 9
> hours late which is actually very good.
https://github.com/WordPress/WordPress/blob/master/wp-includes/update.php#L636
If I’m reading that correctly, you’re looking at checks around 7am and 7pm within the site’s time-zone, which should make it relatively easy to target sites most likely to be un-patched for longest.
> Assuming they aren’t asleep when the alert comes out they’d have, at most,
> a 12 hour window to write and deploy some software to exploit this bug.
> Then they’d have, at most, a 9 hour window to infect millions of machines
Except the whole scary thing about this story is that that is about what happened! You’re talking about running an exploit over a pre-known list of compromisable sites (alphabetically by some accounts!). Therefore, while I understand the auto-update suggestion, and it might mitigate the issue, I think WordPress users would be in exactly the same situation right now.
As for being asleep, it’s always useful when your security updates are pre-announced and scheduled! 😉