Webroot's Response to AVTest's November 2015 Mac OSX Test Results

In light of recent test results that were published by AVTest this week, we’d like to bring some very serious issues to light with their testing methodologies when it comes to testing Mac Antivirus products.
 
While AV testing can be a valuable benchmark for consumers when choosing an AV vendor, testing companies such as AVTest have been extremely reluctant to implement valid tests in order to accurately identify the efficacy of Mac AV products such as Webroot SecureAnywhere for Mac. AVTest’s recent test results were not only inaccurate, but have done a great disservice not only to our product, but to the industry as a whole.
 
As we discussed in our recent blog post “Are antivirus testing scores a true reflection of our Mac product efficacy? You decide…”, AVTest does not actually install any infections on their machine when conducting AV vendor tests. The reality of the situation is that AVTest is not using “real-world” simulations of malware executions and installs when testing Mac AV products. Instead, they perform their tests using a method called “zoo testing,” which involves putting hundreds of “malicious” file samples into a folder and then putting that folder on the testing Mac before running the test. Most importantly however, is that these files do not reside inside actual Mac bundles necessary for execution, they do not have the proper file extensions for execution/mounting, and have often had their executable bit stripped from the binary, in effect making them benign to any system. The issue here is that none of the “malware” is actually executed or installed on the testing machine, nor does it constitute any kind of real threat to the system. In addition, this type of approach is often easily “gamed” by AV companies that are simply looking to do well on the test and avoid negative publicity.
 
Another issue with their test was the initial inclusion of both malware and PUA (potentially unwanted applications) components to the test, however once the final tally of detections was calculated and sent to us for evaluation, we were told that the entire PUA directory would be removed before calculating the final results. Their original test included 514 files, however by removing the PUA directory they actually removed the majority of files from their initial test, reducing the overall number of files on the final test to no more than 65. In addition, it is important to note that over the past year there has been a major increase in the number of active PUA/Adware programs for Mac OSX, making them the most common type of “infections” seen in the wild by Mac users, however by removing the PUA directory from the test, it was not deemed important to test for those types of “infections”. Another common approach to compromising user’s machines is the use of phishing attacks and browser highjacks, however the ability to detect and stop these types of attacks was also not considered by AVTest.
 
In this most recent test there were many file samples that AVTest dinged us for not detecting, even though once the correct file extension was appended to the file, and the file was either executed or mounted, Webroot SecureAnywhere detected the “malware” samples nearly across the board. Using “correct” testing methods that require installing and mounting samples (as would be the case for an actual end user in the “real world”) we did our own in-house testing using the AVTest sample set and compiled data on the correct detection/remediation percentage that we should have received. According to our analysis, if the test was done correctly and the original 514 files were left on the test, we should have scored approximately a 98% overall detection rate. We shared that data with AVTest, along with an explanation of our methodology, and how it is pertinent to Mac OSX architecture. Unfortunately, AVTest made the choice to ignore our concerns, and continue to test us, and other AV vendors, using antiquated and inaccurate methodologies.
 
After the last round of AV testing in April of this year, we addressed these same concerns with AVTest about the inaccuracy of their testing methodologies, but our concerns were not taken seriously and addressed for future testing, therefore we opted not to participate in this latest November test. AVTest however went ahead and tested us anyway using the same testing methodologies that provide nothing but useless data to consumers about the efficacy of the products being tested. Now against our wishes, they have gone ahead and published this skewed data.
 
Rather than painting an accurate picture of the effectiveness of various Mac antivirus products on the market, AVTest’s methods simply incent AV vendors to game the test to avoid negative publicity. Here at Webroot we believe in participating in these types of tests in an ethical manner. We do not game the test, but in turn we also expect that the tests they conduct will be an accurate reflection of the types and methods of infection that our customers may encounter in the wild. Anything other than that is an invalid test, and an inaccurate measurement of the performance and true efficacy of our product.
 
The true test of our Mac product is it’s real-world performance in protecting our customers from unwanted malware, PUAs/Adware, malicious URLs, and phishing attacks. We are very confident that we do an excellent job of doing just that for the people that put their faith in our product, and our approach as a company. Until AVTest implements correct methods for testing Mac AV products, we will continue to opt out of these types of tests, and will be looking to have another independent testing house evaluate us using current day techniques for fairly and ethically evaluating security products.