Researchers: Google gamed browser report that dissed Firefox

Security researchers at NSS Labs have charged Google with gaming the methodology and timing of a recent, Google-funded analysis of browser security — one that placed Mozilla Firefox lowest on the totem pole when compared with security in Google Chrome and Microsoft Internet Explorer.

NSS on Tuesday released a report on the browser evaluation, which was produced by product reseller Accuvant at Google’s behest.

Titled The Browser Wars Just Got Ugly, NSS’s report points out a myriad of methodology deficiencies in Accuvant’s analysis, such as the omission of frame poisoning: a Firefox feature that blocks exploits of layout code crashes.

Here are a few more of what NSS deems Accuvant’s methodology shortcomings:

The JIT hardening analysis failed to give ample credit to the more proactive technologies employed by IE9, which happened to not be present in Chrome.

Accuvant disabled highly relevant portions of non-Google browsers' protection without noting the impact on the overall results. This error in testing resulted in an erroneously negative assessment of the browsers' protection capabilities, since some browsers will only block malware during or after download and before execution.

By utilizing malware sites garnered exclusively from free public lists, the malware sample set was highly skewed in Google's favor. Justifying not using high-quality, professional malware feeds because Microsoft and/or Google may or may not subscribe to them is highly suspect.

NSS researchers also cast a hairy eyeball at the timing of the Accuvant study’s release, which came right on the heels of the quiet lapse of the Google-Firefox funding deal.

That deal, which ended in November, saw Google chipping in a whopping 84% of Mozilla’s $123 million in revenue in 2010, according to Ed Bott, who wrote this excellent analysis of Firefox’s uncertain future for ZDNet.

NSS isn’t necessarily laying blame on Accuvant. The problem is either that Accuvant became lax in defining methodology, NSS mused, or Google asserted “undue influence” on that methodology to its own advantage.

In an interview with Computerworld’s Gregg Keizer, NSS Chief Technology Officer Vikram Phatak put it this way: “This is a vendor-funded paper, and in these cases, the vendor is going to drive the methodology [of the testing], which appears to be the case here.”

At this point, Mr. Keizer reminded Mr. Phatak that NSS Labs itself has conducted vendor-funded browser security research in the past, including several Microsoft-sponsored NSS tests on anti-malware blocking technologies.

“There’s a reason why we don’t do that anymore,” Mr. Phatak told him.

I’m glad that NSS is back to independent research if it means we get good, objective analysis.

I’m glad to see a defender stand up for Firefox, which is getting pretty bruised lately, between its dipping market share and headlines pondering whether it might, in fact, be approaching the realm of toast.

What’s the takeaway? Should we completely ignore vendor-sponsored security research and never write about such?

It’s hard to see how we can, given that corporate clients certainly don’t. At any rate, there’s no reason why we, or they, should automatically dismiss research from the likes of Accuvant, which counts the estimable Charlie Miller as a Principal Research Consultant, for one.

NSS’s corporate clients were justifiably concerned about the Accuvant study and wanted to know whether Firefox could be trusted—that’s actually what got NSS researchers to poke at the Accuvant study.

I guess the takeaway is to take vendor-sponsored studies with a big, big grain of salt.

Even more importantly, to beat the update drum once again, let’s go for the low-hanging security fruit: make sure your browser, whether it’s Chrome or IE or Firefox or Opera or Safari, is up to date.