[Discussion] - Antimalware testing is hard, disputing a flawed test is even harder

Hi Randy.
 
I woud have thought that it was not just the complexities of the antimalware products which are bad enough but once you throw in the all complexities of some of the new malware as well, it must be a nightmare to keep on top of.

Hi Randy,
 
I agree and never agreed with most testing firms over the years as I look at most tests and some testing is outright wrong but they have to defend there testing as it's there means of making money right?
 
Thanks,
 
Daniel 😉

Hi Randy, 

I agree and to add further to Jasper's point, with IOT becomming massive thay have so many more attack vectors and the complexities will sky rocket.

As soon as you add human checking there will be mistakes not by incompetance but because our brain fills in the gaps and malware is designed to make us miss the smallest things

Hi Jasper!
 
You are rght. It is a nightmare for vendors and testers alike to keep up with. As new malware emergerges atimalware products sometimes require massive updates to the engines themselves. As a webroot user you may see an update to the small user agent, but the technology powering the cloud that the agent talks to has and will continue to go through massive changes to keep up with innovative new malware attack techniques. The testers in turn have to adjust to be able to fully test the capabilties of the protections the vendors add. To pour gasoline on the fire, they have to find ways to account for different approaches different vendors use to provide test results that are fair to all of the vendors. I do not envy the testers... I once worked for a test lab 😉

Hi Shrubs,
 
Oh man, the IoT is a nightmare. There are so many IoT manufacturers that do not even know they need security. Then may who do know they need security don’t understand security well enough to know how to implement it. It will get a lot worse before it gets better.
 
Honestly it is the privacy implications that scare me more than the security problems though.
 
Recently I was asked for my thoughts about how to secure a smart home. My reply was “make it dumb again”

Hi Dean,
 
I particularly applaud you for pointing out that it is not due to incompetence. As long as humans make mistakes other umans will exploit that vulnerability. This is especially true for phishing attacks and other social engineering attacks

Thanks Daniel,
You put me in a difficult position, now I have to defend the testers. The testers at AV-Test, AV Comparatives, NSS, Virus Bulletin, and a couple of lesser known test labs are my friends. Good friends. They are driven by the same passion to help consumers that we are. And yes, they do have to earn a living too so there is money involved.
 
The problem I am attempting to address is the general perception (not yours) that the testers are always right. As a result there is no "appeals process" for the vendors. All of us vendors test our own products too. It's a bear.
 
There was a day when the testers would not even consider what the vendors had to say. I can't blame them, they were not treated with any respect. Over the years, primarily due to AMTSO, vendors and testers are working collaboratively to make the quality of testing better.
The testers don't always get it wrong. They also get it horribly wrong sometimes too. What matters then is that mistakes are admitted and that the results of the test are corrected post-publication.
It is perfectly fair for you to be skeptical of the test results, but please don't extrapolate that to questioning these people's integrity. We are actually on the same side – even at the times when we know we weren’t given a fair shake.
 
I appreciate you taking the time to comment and I look forward to lots more participation and discussions with you and the community!
 

@lets take it one step further as WSA is not tested per say by any of these testing places maybe accept MRG, so do you see that changing anytime soon if it's within your scope to say or find out?
 
MRG: https://community.webroot.com/t5/Announcements/Webroot-SecureAnywhere-pases-MRG-Effitas-2015-16-Banking/m-p/250631#M5672
 
This is from way back: https://community.webroot.com/t5/Announcements/Joint-message-from-AV-Comparatives-and-Webroot/m-p/17708
 
So is this one: https://community.webroot.com/t5/Security-Industry-News/Webroot-response-to-Mac-AV-Test-Results/m-p/145645/highlight/true#M8283
 
There are others but you get the picture.
 
Thanks,
 
Daniel

Yes, I do see the testing to be extremely difficult, but also agreeing with Daniel. Sometimes It's difficult to promote a product that has not been adequately compared to the other vendors out there. We see clients that sometimes do their own research and come up with incorrect assumptions based on the lack of information (in this case no proper comparisons between products) . 

It would seem that testing can only be as good as the understanding of what malware MAY do, and as soon as your product supposedly protects against current threats, and threats like them, something new will come along that will have you saying, “I never thought that could do that.”
 
i also worked in security, most recently application hardening. The real trick to malware is a product that does not just look for set patterns, but looks for and stops behavior that is suspicious. Then you have to walk that fine line between annoying your customer all the time, and not catching a new threat. 
 
There is simply no way to be able to test for all that completely. I can understand the frustration. 

People devote their lives to creating malware, its always changing and I guess determining how something actually is interacting with a system compared to how it is supposed to is hard.

People demand simple answers, don't they?
(including me, sometimes!)
 
"What's the best/cheapest/fastest/lowest impact AV?"
"I'll pick the one with shiny No.1 badge from Malwaretesters 'r us"
 
testers should explain methods better wrt how users read their reports
vendors should explain, upfront, to testers how their solution works and interacts with the testers protocol
buyers should stop asking for simple answers (but good luck with that!)
 
 

There is nothing that can be done to fully secure anything. Everything is (currently) built by humans which are themselves flawed. There will always a way in, always some piece that wasn't coded properly to protect against attacks and people that know the inner workings of how to bypass what was put in place.
 
With other comments mentioning IoT, absolutely, companies are churning and burning to get these products out as quick as possible without a concern for security. I've personally setup an IoT network that is unable to talk to the rest of my network, with a Pi-Hole DNS server that shows me all of the DNS queries on the network. It's surprising (but yet again, not really) to see where everything is calling out to these days. Smart TVs are constantly pinging back home, reaching out to google analytics or other crytic domains. These are the items we need to worry about. It's not just the age of virus / malware, but information gathering for 'quality purposes'. 
 
Trust no device and go into things expecting it can be compromised and do your best to protect that in any means possible (Router, Firewall, DNS, Agents, Isolation, etc). Hire the right people that know what to look for and work with products that you can rely on. Nothing is perfect, but make a portfolio that works for you. 

As I read some of the other comments I can't help but think that they are proving your point - people often take a portion of a statement you made and read into it based on their own biases and what they believe your conclusion should be (as a vendor).
You are generally right on target; testing as a for-profit business model is always going to [at the very least perceived to] be skewed to maximize their profitability. Testers are going to design test suites that fit their pre-concieved notions of how a)real-world users act; b)how anti-malware is done "right"; and c) what the results "really mean". Vendors will always push back at results that don't show them as #1 because they believe in their product and their approach to the issue.
The real test suite?  Ones own production environment. A long time in the network consulting/managed services arena has made me eschew potentially cozy relationships with other vendors (including one who handed me a 5-figure check for attending and speaking at their top reseller/top client events) for the superior stopping power that I've gotten from Webroot. My tests care about two things - stopping the bad guy and not messing up the client endpoint. Webroot has done both for me.

@wrote:
As I read some of the other comments I can't help but think that they are proving your point - people often take a portion of a statement you made and read into it based on their own biases and what they believe your conclusion should be (as a vendor).
You are generally right on target; testing as a for-profit business model is always going to [at the very least perceived to] be skewed to maximize their profitability. Testers are going to design test suites that fit their pre-concieved notions of how a)real-world users act; b)how anti-malware is done "right"; and c) what the results "really mean". Vendors will always push back at results that don't show them as #1 because they believe in their product and their approach to the issue.
The real test suite?  Ones own production environment. A long time in the network consulting/managed services arena has made me eschew potentially cozy relationships with other vendors (including one who handed me a 5-figure check for attending and speaking at their top reseller/top client events) for the superior stopping power that I've gotten from Webroot. My tests care about two things - stopping the bad guy and not messing up the client endpoint. Webroot has done both for me.

Thanks for the comment. You are exactly right. Jimmy Kuo, a longtime industry veteran, once told me that the best antivirus product is the one that just protected you from a threat.
 
One other comment. Antimalware testing as a non-profit is a non-starter. We deal with what we have to.

@wrote:
There is nothing that can be done to fully secure anything. Everything is (currently) built by humans which are themselves flawed. There will always a way in, always some piece that wasn't coded properly to protect against attacks and people that know the inner workings of how to bypass what was put in place.
 
With other comments mentioning IoT, absolutely, companies are churning and burning to get these products out as quick as possible without a concern for security. I've personally setup an IoT network that is unable to talk to the rest of my network, with a Pi-Hole DNS server that shows me all of the DNS queries on the network. It's surprising (but yet again, not really) to see where everything is calling out to these days. Smart TVs are constantly pinging back home, reaching out to google analytics or other crytic domains. These are the items we need to worry about. It's not just the age of virus / malware, but information gathering for 'quality purposes'. 
 
Trust no device and go into things expecting it can be compromised and do your best to protect that in any means possible (Router, Firewall, DNS, Agents, Isolation, etc). Hire the right people that know what to look for and work with products that you can rely on. Nothing is perfect, but make a portfolio that works for you. 

If I were to sum up what you said (which I am doing) Security is risk management. There is no perfect security.

@wrote:
@lets take it one step further as WSA is not tested per say by any of these testing places maybe accept MRG, so do you see that changing anytime soon if it's within your scope to say or find out?
 
MRG: https://community.webroot.com/t5/Announcements/Webroot-SecureAnywhere-pases-MRG-Effitas-2015-16-Banking/m-p/250631#M5672
 
This is from way back: https://community.webroot.com/t5/Announcements/Joint-message-from-AV-Comparatives-and-Webroot/m-p/17708
 
So is this one: https://community.webroot.com/t5/Security-Industry-News/Webroot-response-to-Mac-AV-Test-Results/m-p/145645/highlight/true#M8283
 
There are others but you get the picture.
 
Thanks,
 
Daniel

Hi Daniel. We are chomping at the bit to go toe-to-toe with our competitors. We are full participant in AMTSO (Antimalware Testing Standards Organization) because we, along with many other vendors and testers are working very hard to help bring testing to the place where all vendors can demonstrate the effectiveness of their technologies. That the testers are participants in AMTSO speaks to their desire to improve too. We all make mistakes. The whole point of the article is that it isn’t always sour grapes when a vendor disputes a test. Sometimes we are right about errors that materially affect the results of a test, we just want our reasons for disputing the results of a test to be considered impartially. I’m probably preaching to the choir here though!

@wrote:

@wrote:
@lets take it one step further as WSA is not tested per say by any of these testing places maybe accept MRG, so do you see that changing anytime soon if it's within your scope to say or find out?
 
MRG: https://community.webroot.com/t5/Announcements/Webroot-SecureAnywhere-pases-MRG-Effitas-2015-16-Banking/m-p/250631#M5672
 
This is from way back: https://community.webroot.com/t5/Announcements/Joint-message-from-AV-Comparatives-and-Webroot/m-p/17708
 
So is this one: https://community.webroot.com/t5/Security-Industry-News/Webroot-response-to-Mac-AV-Test-Results/m-p/145645/highlight/true#M8283
 
There are others but you get the picture.
 
Thanks,
 
Daniel

Hi Daniel. We are chomping at the bit to go toe-to-toe with our competitors. We are full participant in AMTSO (Antimalware Testing Standards Organization) because we, along with many other vendors and testers are working very hard to help bring testing to the place where all vendors can demonstrate the effectiveness of their technologies. That the testers are participants in AMTSO speaks to their desire to improve too. We all make mistakes. The whole point of the article is that it isn’t always sour grapes when a vendor disputes a test. Sometimes we are right about errors that materially affect the results of a test, we just want our reasons for disputing the results of a test to be considered impartially. I’m probably preaching to the choir here though!

Yes I understand AMTSO and also know that Webroot is a Member: https://www.amtso.org/members/ I use the Phishing page for a test of the Web Threat Shield and it works great: https://www.amtso.org/feature-settings-check-phishing-page/
 
Thanks again,
 
Daniel
 

I'm glad to hear that you and the testers are working together to make ALL of us have a better product. I would imagine with the ongoing arms race that is malware, it's good to have someone reality-check your solution.

Hi Randy,
 Firstly Welcome!
Your post is very informational and provides good views, ideas and concerns in areas that need to be addressed.
Depending on the type and level of testing people want to achieve users will have to take into consideration attacks mainly come from creativity patterned designs.  
Malware, viruses, crypto have all managed to transform into templates to provide a rich dense attack. It's not always about getting test right but good practise would be to run multiple tests and view the median results. The more information on the table better insight achieved. 
Look forward to future posts.
 Cheers,
James  
 

Yes I understand AMTSO and also know that Webroot is a Member: https://www.amtso.org/members/ I use the Phishing page for a test of the Web Threat Shield and it works great: https://www.amtso.org/feature-settings-check-phishing-page/

@,
Interesting that this works for you. I tried this and was able to access the page without a warning.  Double checked my policy and it is configured correctly (see below). Results are the same whether IE or Edge browser.  Any ideas what I'm missing here?
 

Hi Randy
 
Part of the problem is testing can be highly subjective and often lacks consistency between tests and vendors. This makes results very hard to read.

The best defences' in the world have lost a fight at times, lose the battle but win the war.  We as humans cannot take into consideration all of the morality or ethics codes in which we live in.  

Someone who designs malware/virus has "code" within them that guide them when they design.  They are making something "bad".  The Antimalware testers are doing a job they are on the opposite side of the "bad" they are doing things right - have a job, being a normal every-day person.
 
It's a lot easier to see the good side f you are on the bad side, humans that live in positive/good environment might not ever see the bad malware/ virus world. 
 
It is not incompetence, its a level of a good brain(s) trying to find the "bad".  All of which is how the world operates today.

I just got very deep right there - so I'll stop because sometimes what comes out of my head doesn't make sense to other people; as I cannot always articulate properly.  Which is why I work in IT! 😉

The problem we have is that no matter how hard you try to convince some clients that X product is good, they still want to see the scores, stats, and standardized testing scores. Doesn't help that many of those testing organizations will not test your product in the way that they were designed to work. Good article.