AMTSO Progress

It’s the day after the latest AMTSO meeting and I can report that we had a very successful meeting.

Symantec's Headquarters
Symantec's Headquarters
Around 50 of us met at Symantec’s headquarters in Cupertino and spent two days discussing various topics that are pertinent to the testing of security products.

Let me first remind you what AMTSO is about. You can find the charter on the AMTSO website but essentially the organisation is about promoting standards and guidelines to improve the quality of any tests that are carried out on security products. The organisation is not about telling testing institutions what to to, rather it’s about promoting tests that are fair and reasonable and provide quality information so that informed comparisons and decisions can be made.

To this aim AMTSO is currently involved in writing core documents that can be used by all people. The first document – AMTSO Fundamental Principles of Testing – was ratified in October 2008 and is exactly what it says, a few core principles for testers.

At the meeting in Cupertino several other documents were discussed with a view to them being ratified by, or at, the next meeting in Budapest in May 2009.

  • How to obtain malware samples – this is not as obvious as it first seems so this document aims to help promote understanding on best practice for obtaining samples and how to build the trust relationships necessary in the industry
  • Reviewing reviews – AMTSO expects to be asked to comment on reviews so it is defining a process that can be used to carry out this task that is both transparent and fair
  • Validating malware samples – This document aims to promote the use of samples in tests that are proven to actually be malware
  • Differences between on-demand testing and whole product testing – One of the many issues raised is just how narrow is the simple test of how many files does a product detect. This document explores the pros and cons of testing all the features in a product
  • Issues around creating malware – this is a fascinating topic. Historically the AV industry has always been against writing malware but testing houses have consistently ‘created’ malware to measure the effectiveness of a product’s proactive detection. This document will explore the benefits and the risks of different approaches of ‘creating’ malware
  • The glossary – every organisation needs one and AMTSO is going to define the terms used in it’s documentation in order to try and reduce the confusion that may exist as people interpret different words in their own way

As I hope is clear, the two days were very full and it was great to see the active participation of so many vendors and testers. There is always scope for others to get involved and the proposed documents will be posted on the AMTSO website over the coming weeks for comment.

Unfortunately we didn’t get to see the Babbage engine working and now I will be hoping the next batch of snow at Heathrow doesn’t leave me stuck in the Californian sunshine…