[Discussion] - Antimalware testing is hard, disputing a flawed test is even harder

Hi Randy.
 
I woud have thought that it was not just the complexities of the antimalware products which are bad enough but once you throw in the all complexities of some of the new malware as well, it must be a nightmare to keep on top of.

As soon as you add human checking there will be mistakes not by incompetance but because our brain fills in the gaps and malware is designed to make us miss the smallest things

Hi Randy, 

I agree and to add further to Jasper's point, with IOT becomming massive thay have so many more attack vectors and the complexities will sky rocket.

Hi Shrubs,
 
Oh man, the IoT is a nightmare. There are so many IoT manufacturers that do not even know they need security. Then may who do know they need security don’t understand security well enough to know how to implement it. It will get a lot worse before it gets better.
 
Honestly it is the privacy implications that scare me more than the security problems though.
 
Recently I was asked for my thoughts about how to secure a smart home. My reply was “make it dumb again”

@ wrote:
Thanks Daniel,
You put me in a difficult position, now I have to defend the testers. The testers at AV-Test, AV Comparatives, NSS, Virus Bulletin, and a couple of lesser known test labs are my friends. Good friends. They are driven by the same passion to help consumers that we are. And yes, they do have to earn a living too so there is money involved.
 
The problem I am attempting to address is the general perception (not yours) that the testers are always right. As a result there is no "appeals process" for the vendors. All of us vendors test our own products too. It's a bear.
 
There was a day when the testers would not even consider what the vendors had to say. I can't blame them, they were not treated with any respect. Over the years, primarily due to AMTSO, vendors and testers are working collaboratively to make the quality of testing better.
The testers don't always get it wrong. They also get it horribly wrong sometimes too. What matters then is that mistakes are admitted and that the results of the test are corrected post-publication.
It is perfectly fair for you to be skeptical of the test results, but please don't extrapolate that to questioning these people's integrity. We are actually on the same side – even at the times when we know we weren’t given a fair shake.
 
I appreciate you taking the time to comment and I look forward to lots more participation and discussions with you and the community!
 

Hi RAbrams
 
Have been following this thread with some interest and without the need to add my two pennies worth...until now. ;)
 
You mention testers and their integrity but as with any labeled 'group' whilst there the majority will be honest, diligent and do what they do with integrity, there will always be a minority who may 'stray' (to be kind to them) from that path. Even worse there may also be those that purport to be members of the 'group' or intimate that, and who are unscrupulously looking to profit, etc.
 
The perception that Daniel exposed is a very real one in my book because there is no clear & easy way to discern the credible from the disreputable, especially for users who may not be as well versed in the topic as the likes of Daniel and others. They get bitten, proclaim this and that casts dispersions on those of integrity, etc.
 
What would also help is if testing organisations explained their commercial links/how they make their money, i.e., some greater transparency would help cement the credibility of those organisations of integrity, more widely...which would be a good thing in my book.
 
Finally, in my book, both testing organisations & vendors should actively promote their collaboration more widely & openly...the case in point was when recently WRSA fared badly in a test, due to the testing organisation not having installed it correctly (if my failing memory does not fail me here) and the testing organisation did not overtly announce (as far as I am aware) that this was the reason.
 
We in the Community had to dig around and report that following the plethora of posts about WRSA's apparent 'bad' showing. Such episodes do nothing to (i) make those who know about these things less 'suspicious' (for want of a better word) & (ii) to give confidence to the average user who take the test results as 'gospel', that the testing organisations can be fully 'trusted'.
 
The same goes for the scant overt recognition that the way that WRSA works is different and until recently was not catered for by the testing organisations in terms of the way they test (I believe that some are working with Webroot to remedy that, and indeed some of that remedy may already be in place) which led to the widely held observation by the plain stupid, those with an axe to grind about WRSA's success, etc., that Webroot are 'scared to put WRSA up to independent testing...and then the conspiracy theories start about WRSA's capabilities, etc.
 
A simple declaration at the start  of a test result report, about why Webroot, or indeed any other main stream AM that might be in the same position, is not included, etc., would demonstrate partnership, etc., and go a long way to prevent the aforementioned 'dis-inforation' that can (i) blight an AM & (ii) cause users to doubt the integrity & usefulness of the testing organisations.  
 
This is a shame, as clearly the vast majority are driven by passion & integrity to that passion, etc.
 
Now, I may be in a minority view here and also may be misinformed as to the lay of the land here...but this is what I perceive (and I suspect that there are many more like me) but stand to be corrected if necessary.
 
Thank you for an excellent & informative discussion topic...I hope that there will be more like this.
 
Regards, Baldrick

@wrote:
There is nothing that can be done to fully secure anything. Everything is (currently) built by humans which are themselves flawed. There will always a way in, always some piece that wasn't coded properly to protect against attacks and people that know the inner workings of how to bypass what was put in place.
 
With other comments mentioning IoT, absolutely, companies are churning and burning to get these products out as quick as possible without a concern for security. I've personally setup an IoT network that is unable to talk to the rest of my network, with a Pi-Hole DNS server that shows me all of the DNS queries on the network. It's surprising (but yet again, not really) to see where everything is calling out to these days. Smart TVs are constantly pinging back home, reaching out to google analytics or other crytic domains. These are the items we need to worry about. It's not just the age of virus / malware, but information gathering for 'quality purposes'. 
 
Trust no device and go into things expecting it can be compromised and do your best to protect that in any means possible (Router, Firewall, DNS, Agents, Isolation, etc). Hire the right people that know what to look for and work with products that you can rely on. Nothing is perfect, but make a portfolio that works for you. 

If I were to sum up what you said (which I am doing) Security is risk management. There is no perfect security.

It would seem that testing can only be as good as the understanding of what malware MAY do, and as soon as your product supposedly protects against current threats, and threats like them, something new will come along that will have you saying, “I never thought that could do that.”
 
i also worked in security, most recently application hardening. The real trick to malware is a product that does not just look for set patterns, but looks for and stops behavior that is suspicious. Then you have to walk that fine line between annoying your customer all the time, and not catching a new threat. 
 
There is simply no way to be able to test for all that completely. I can understand the frustration. 

@wrote:
The problem we have is that no matter how hard you try to convince some clients that X product is good, they still want to see the scores, stats, and standardized testing scores. Doesn't help that many of those testing organizations will not test your product in the way that they were designed to work. Good article. 

So very true! I have been using WSA since the beginning even Beta Tested the first version for approx 8 months before it's release in  the fall of 2011 for the 2012 version and never had an infection at any time in  7 years!
 
Cheers,
 
Daniel 😉

Yes, I do see the testing to be extremely difficult, but also agreeing with Daniel. Sometimes It's difficult to promote a product that has not been adequately compared to the other vendors out there. We see clients that sometimes do their own research and come up with incorrect assumptions based on the lack of information (in this case no proper comparisons between products) . 

There is nothing that can be done to fully secure anything. Everything is (currently) built by humans which are themselves flawed. There will always a way in, always some piece that wasn't coded properly to protect against attacks and people that know the inner workings of how to bypass what was put in place.
 
With other comments mentioning IoT, absolutely, companies are churning and burning to get these products out as quick as possible without a concern for security. I've personally setup an IoT network that is unable to talk to the rest of my network, with a Pi-Hole DNS server that shows me all of the DNS queries on the network. It's surprising (but yet again, not really) to see where everything is calling out to these days. Smart TVs are constantly pinging back home, reaching out to google analytics or other crytic domains. These are the items we need to worry about. It's not just the age of virus / malware, but information gathering for 'quality purposes'. 
 
Trust no device and go into things expecting it can be compromised and do your best to protect that in any means possible (Router, Firewall, DNS, Agents, Isolation, etc). Hire the right people that know what to look for and work with products that you can rely on. Nothing is perfect, but make a portfolio that works for you. 

People devote their lives to creating malware, its always changing and I guess determining how something actually is interacting with a system compared to how it is supposed to is hard.

Hi Randy,
 Firstly Welcome!
Your post is very informational and provides good views, ideas and concerns in areas that need to be addressed.
Depending on the type and level of testing people want to achieve users will have to take into consideration attacks mainly come from creativity patterned designs.  
Malware, viruses, crypto have all managed to transform into templates to provide a rich dense attack. It's not always about getting test right but good practise would be to run multiple tests and view the median results. The more information on the table better insight achieved. 
Look forward to future posts.
 Cheers,
James  
 

Hi Baldrick, I'm sorry for the tardy reply. You do raise some interesting points. There are some willfully dishonest "testers out there, but (and this is my own opinion) how influential are they? How much business is a 0 percent score is any company going to lose when the tester reaches a very tiny percentage of the world? The perception is OK as long as it doesn't bend the perception of reality. I do not believe that ant of the influential test organizations are unscrupulously looking for profit.
 
I have a blog in review right now that addresses the economics of security product testing!
One of the biggest problems with test reports where the test was pretty good (none or perfect) is in the analysis. Few people think of that aspect. It isn't that they're stupid, it's not intuitive enough to many people to recognize that data is not the same thing as information. Mea Culpa. It wasn't until I was into my career in this field that I was made to understand that with respect to security product testing..
 
As for conspiracy theories... I am of the belief that conspiracy theorists conspire to create conspiracy theories :-)