Sophos Cloud Optix
Google has revealed its emergency patching efforts to fix a widespread and “pernicious” software vulnerability that affected thousands of open source projects in 2015.
Referred to as “Mad Gadget” by Google (aka the Java “Apache Commons Collections Deserialization Vulnerability” CVE 2015-6420), the flaw was first highlighted by FoxGlove Security in November of that year, months after the first proof-of-concept code garnered almost zero attention.
That was despite it eventually affecting software shipped by Oracle, Cisco, Red Hat, VMware, IBM, Intel, Adobe, HP, OpenNMS , Jenkins and SolarWinds.
It was serious enough to figure as part of the 2016 ransom attacks on Baltimore’s Union Memorial Hospital in March 2016 and the infamous San Francisco Municipal Transportation Agency (MUNI) attack in November.
We should, then, view FoxGlove’s sarcasm in its 2015 alert as prescient: “No one gave it a fancy name, there were no press releases, nobody called Mandiant to come put out the fires.”
We now know the problem was that in 2015 everyone sat back and assumed the affected open source projects would get busy – except that many didn’t.
In March 2016, a Google researcher noticed the lack of activity and decided to start updating the code herself on GitHub. Enlisting help from colleagues to speed the effort, the team soon realised they had bitten off more than they could chew.
Says Google in its blog:
We were furthermore patching all the projects that depended on those projects and so forth. But even once those users upgraded, they could still be impacted by other dependencies introducing the vulnerable version of Collections.
Even projects that had been upgraded could later be undermined by vulnerable older versions of the code that were likely to hang around.
“We were alarmed when we discovered 2,600 unique open source projects that still directly referenced insecure versions of [Apache] Collections.”
In response, Google’s engineers initiated Operation Rosehub, a volunteer effort by 50 engineers dedicating 20% of their working time to the huge patching effort.
Going forward, we believe the best thing to do is to build awareness. We want to draw attention to the fact that the tools now exist for fixing software on a massive scale, and that it works best when that software is open.
Guerrilla patching on this scale is extremely unusual and might even be viable beyond open source. Last week, Acros Security coined the idea of the “0patch” by posting a patch for the Windows gdi32.dll memory disclosure zero day (CVE-2017-0038) that Microsoft has yet to fix.
You read that correctly: a third party has released a temporary patch affecting Microsoft. Goodness knows who might apply such a thing before Microsoft ships its own native fix in mid-March but a “zero patch” now exists for a zero day. Say its authors, ominously: “Now onward to writing the next 0-day 0patch.”
It a small insurgency but an interesting one. Hitherto, flaws were fixed when companies or developers got around to fixing them. With Rosehub and 0patch we’ve been given a window into an alternative future where not fixing something might become an open invitation for others to do it for you.