Researchers: Google gamed browser report that dissed Firefox

Filed Under: Featured, Firefox, Internet Explorer, Malware, Privacy, Vulnerability

FirefoxSecurity researchers at NSS Labs have charged Google with gaming the methodology and timing of a recent, Google-funded analysis of browser security — one that placed Mozilla Firefox lowest on the totem pole when compared with security in Google Chrome and Microsoft Internet Explorer.

NSS on Tuesday released a report on the browser evaluation, which was produced by product reseller Accuvant at Google's behest.

Titled The Browser Wars Just Got Ugly, NSS's report points out a myriad of methodology deficiencies in Accuvant's analysis, such as the omission of frame poisoning: a Firefox feature that blocks exploits of layout code crashes.

Here are a few more of what NSS deems Accuvant's methodology shortcomings:

The JIT hardening analysis failed to give ample credit to the more proactive technologies employed by IE9, which happened to not be present in Chrome.

Accuvant disabled highly relevant portions of non-Google browsers' protection without noting the impact on the overall results. This error in testing resulted in an erroneously negative assessment of the browsers' protection capabilities, since some browsers will only block malware during or after download and before execution.

By utilizing malware sites garnered exclusively from free public lists, the malware sample set was highly skewed in Google's favor. Justifying not using high-quality, professional malware feeds because Microsoft and/or Google may or may not subscribe to them is highly suspect.

NSS researchers also cast a hairy eyeball at the timing of the Accuvant study's release, which came right on the heels of the quiet lapse of the Google-Firefox funding deal.

That deal, which ended in November, saw Google chipping in a whopping 84% of Mozilla’s $123 million in revenue in 2010, according to Ed Bott, who wrote this excellent analysis of Firefox's uncertain future for ZDNet.

NSS isn't necessarily laying blame on Accuvant. The problem is either that Accuvant became lax in defining methodology, NSS mused, or Google asserted "undue influence" on that methodology to its own advantage.

In an interview with Computerworld's Gregg Keizer, NSS Chief Technology Officer Vikram Phatak put it this way: "This is a vendor-funded paper, and in these cases, the vendor is going to drive the methodology [of the testing], which appears to be the case here."

At this point, Mr. Keizer reminded Mr. Phatak that NSS Labs itself has conducted vendor-funded browser security research in the past, including several Microsoft-sponsored NSS tests on anti-malware blocking technologies.

"There's a reason why we don't do that anymore," Mr. Phatak told him.

I'm glad that NSS is back to independent research if it means we get good, objective analysis.

I'm glad to see a defender stand up for Firefox, which is getting pretty bruised lately, between its dipping market share and headlines pondering whether it might, in fact, be approaching the realm of toast.

What's the takeaway? Should we completely ignore vendor-sponsored security research and never write about such?

It's hard to see how we can, given that corporate clients certainly don't. At any rate, there's no reason why we, or they, should automatically dismiss research from the likes of Accuvant, which counts the estimable Charlie Miller as a Principal Research Consultant, for one.

NSS's corporate clients were justifiably concerned about the Accuvant study and wanted to know whether Firefox could be trusted—that's actually what got NSS researchers to poke at the Accuvant study.

I guess the takeaway is to take vendor-sponsored studies with a big, big grain of salt.

Even more importantly, to beat the update drum once again, let's go for the low-hanging security fruit: make sure your browser, whether it's Chrome or IE or Firefox or Opera or Safari, is up to date.

, , , , , , , , , ,

You might like

16 Responses to Researchers: Google gamed browser report that dissed Firefox

  1. You can always rely on SJVN for a good FOSS-bashing headline. I wouldn't read much into that.

  2. Shutterbug Doug · 1393 days ago

    Why did you use a picture of a Red Panda for something about Firefox? lol...just sayin'!

    • Chester Wisniewski · 1392 days ago

      Ok, Ok, Ok... It's my fault. I picked the picture, which adorably looks like a Firefox... Yet you are correct, it is a red panda.

      Can we just enjoy the cuteness and vulnerability?


      • Guido Fox · 1392 days ago

        Actually, the red panda is also known as a firefox.

      • bjacov · 1392 days ago

        Actually, Firefox == Red Panda, this is the same species :) So the choice was highly appropriate :)

  3. Pat · 1393 days ago

    Just FYI that's not a picture of a fox in the article. It's a Red Panda.

    • bjacob · 1392 days ago

      Which is correct :) Red Panda == Firefox. What's wrong actually is the firefox icon, which looks more like a fox, but now it's history and can't be changed anymore :)

  4. Osama S. · 1392 days ago

    Not sure why Google needed to get this report done in the first place. Chrome is doing pretty well and didn't need this report.
    Besides IE market share will remain high for quite sometime until corporations open up and allow other browser besides IE to be installed. But for sake to standardization, support, "security" etc. corporations are limited to IE.

    And now we are moving from a browser war to the war between research companies ;-)

    • There should be a war between research companies. Ones who do real research and stand by their results and ones who take the money, deliver "insights" that are exactly what their client says they are going to be and publish s**t. Category Two get rich and famous. But they're still s**t.

  5. Jay Nancarrow · 1392 days ago

    The analysis from NSS Labs is inaccurate. Google played no role in deciding the points of comparison, choosing methodologies or data sets, or influencing the conclusions of the Accuvant paper. The advantage of the Accuvant research is that its methodology and findings can be openly scrutinized, which isn't the case with studies performed by NSS Labs, nor the claims they made here.

    Jay Nancarrow
    Google Communications

    • Jason · 1392 days ago

      If Accuvant knew that Google funded the study, then Google played a role in deciding what happened even if Google didn't do it on purpose.
      They knew who was paying for it, therefore, that entity would be given special attention. It is human nature.
      Maybe there could be blind studies that are funded by Microsoft, Firefox, Opera, or Google where the group doing the testing doesn't know where the money came from. That would eliminate outside bias.

  6. Kevin G · 1392 days ago

    Not sure why anyone would believe a browser security report paid for by Google that listed Chrome as the most secure browser, or one paid for by Microsoft (or simply an infographic produced by Microsoft) stating Internet Explorer is the most secure, which it does a lot (NSS Labs...) Or any other company, for that matter. It's an obvious conflict of interest.

  7. Richard · 1392 days ago

    "the Google sponsored study that painted Firefox in a poor light might have been influenced by the sponsor"

    And in related news, ursine mammals eliminate biological waste in wooded areas! :o)

  8. Mike M. · 1392 days ago

    Google Chrome doesn't play fair. I have in the past installed new software on my PC only to find that without my knowledge the software came with an installation of Google Chrome, and it changed my default browser. I found this out only when I went online to register the new software. I will stick with Firefox and only use Google's search engine. I also will not, in the future, buy software from the company I bought from.

  9. Fubar · 1392 days ago

    re: google's "DON’T BE EVIL" philosophy explained:

    (found via bing)

  10. CynicalOne · 1391 days ago

    No artcile about about Chrome should neglect to mention the massive fraud they perpetrate on their advertisers. Whenever someone type inthe the url of an advertiser in the address bar it's counted as a click on an AdWords ad and the company is charged. Even though it wasn't a click on an ad. Yet another reason why I don't recommend PPC advertising for most of my internet marketing clients.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

About the author

I've been writing about technology, careers, science and health since 1995. I rose to the lofty heights of Executive Editor for eWEEK, popped out with the 2008 crash, joined the freelancer economy, and am still writing for my beloved peeps at places like Sophos's Naked Security, CIO Mag, ComputerWorld, PC Mag, IT Expert Voice, Software Quality Connection, Time, and the US and British editions of HP's Input/Output. I respond to cash and spicy sites, so don't be shy.