MRG Effitas - three recent tests - reading carefully the results

I recently contributed my thoughts to another website regarding Webroot’s performance in a recent test by the MRG Effitas AV testing organisation (1) and I thought they might be worth repeating here..
 
The test concerned was the 360 Degree Assessment & Certification test (what MRG describe as their “Real World” test, also testing “the full spectrum of malware”) for Q3 2018. The test can be found here.
 
(1) Also: useful page to bookmark for future reference: here
 
--------------------------------------------------------------------------------------------------------------------------------

Chart 1
ITW Samples
 
I believe this chart is a resumé of all the samples tested by MRG in this test with the exception of the PUAs, Adware, Fileless Exploits and FPs: there are in all 329 of them. Looking just at the picture and the colours and ignoring everything else, the result may look alarming for Webroot. Indeed it had me alarmed…just for a moment: that is, until I properly read and studied the report. A disclaimer here: I am not an IT expert at all, just someone who knows the AV product I use (Webroot), and who tries to read the whole lab report and understand the results in the light of the nature of that product.

First of all, I discovered what appears to be a small mistake that MRG has made in this report (albeit the only mistake I have found so far). They say that the “table is sorted by smallest amount of failures”. But this, I assume, means that Webroot should not be in second to last place but next to Microsoft as the “Miss” rate is 0.30%, less than ESET and McAfee and equal to Microsoft. Also, please note, this means that Webroot missed just one of the 329 samples. And I will come back to that “miss” in a sentence or two.

What else do we discover? We find that Webroot is monitoring and journaling some of the (unknown) malware samples before making a determination, just as it is designed to do: in this case 88 of them—which it subsequently determined to be bad. But is it 88? Or is it 89? The one that it “missed”—would it have “missed” it had the period been 25 hours instead of 24? Or 36 hours? etc. I assume that Webroot was still monitoring that file at the end of that 24-hour period.

Most people on the website where I posted do not agree with Webroot’s methodology of monitoring before declaring as bad, those samples that have not yet been determined as bad, but as I pointed out, they have to agree that it is behaving precisely as it is designed to do. And that the “miss” of 1 out of 329 malicious files/processes is maybe, depending of course on your point of view, not a “miss” at all.

Also, how does it compare with Microsoft where, I assumed, it should be placed next to, according to MRG’s results? When I said that Webroot should have been placed next to Microsoft, I would imagine that you assumed that was to the right of Microsoft as Microsoft makes far more immediate determinations and far less determinations in the 24 hour period following insertion of these malicious files into the machine. But I would disagree. I do not know Windows Defender in depth but I assume that, unlike Webroot, it has two not three classifications: Good and Bad. I assume it does not have an Unknown classification that automatically triggers close monitoring and journaling—not to speak of imposing highly restricted privileges for that Unknown file. And if this is true, this presumably means that those malicious files on the machine protected by Windows Defender have free rein to do whatever harm they wish, up and until they are determined as bad. I would therefore put Webroot very much to the left of Microsoft.

Also please don’t forget: it is very possible that that “missed” file (1 out of 329) was maybe not really missed at all.
 
-----------------------------------------------------------------------------------------------------------------------------------------------

Chart 2
Ransomware Samples

Not much to speak of here.

The only thing I would express is my surprise that there are only TWO ransomware samples. I would have preferred rather more.

-----------------------------------------------------------------------------------------------------------------------------------------------

Chart 3
Financial Malware

Once again, Webroot is behaving exactly as it is designed to do. And once again, I would place Webroot to the left of Microsoft.

And please note that, according to Webroot’s paradigm, it has successfully detected 100% of the malicious samples.

-----------------------------------------------------------------------------------------------------------------------------------------------

Chart 4.
PUAs/Adware

Here I have an issue. So incidentally do most of the helpers at the Webroot Community Forum. Although Webroot has become somewhat more proactive regarding PUAs than hitherto, in our opinion not enough so!

Maybe it’s a question of priorities for Webroot (2). After all, it is true that PUAs are not malicious per se. But they can be a confounded nuisance. And Webroot’s ambivalent attitude to them could in the long term affect customers’ perception of their product.
 
(2) plus also the economic issue: i.e. the potential ruinously costly lawsuits that could be, and have been brought in the past, against AV organisations by those pesky PUA makers

-----------------------------------------------------------------------------------------------------------------------------------------------

Chart 5.
Fileless exploits.

This looks bad…at first blush. But wait a moment. What was the point of entry for these exploits? “Some URLs come from our regular honeypots” (p.7). Is this where the (three) exploits came from? And if so, are those honeypots visible to the ordinary online user? If not, are they visible to the Webroot BrightCloud bots? In real life, dodgy stuff comes from (visible) dodgy URLs. And the Webroot Web Threat Shield is particularly good at singling out those dodgy URLs. And I believe that the Webroot BrightCloud bots revisit each website every 24 hours (? Please correct me, admins, if this is wrong) to search for any negative change in a URL status. If I am correct that these websites are invisible to the normal user and therefore to the Webroot bots (and maybe I am not?), I doubt that, in a real life situation, where Webroot BrightCloud is constantly inspecting all visible websites, these exploits would have got through the Web Threat Shield.

Incidentally, as most people know, Webroot is currently developing an anti-exploit module that will even further strengthen protection against this threat.

-----------------------------------------------------------------------------------------------------------------------------------------------

Chart 6.
FPs

Webroot scored 0.10% false blocks. According to my calculations, that is one false block out of 997 samples. OK not perfect but also not, by any measure, bad.

-----------------------------------------------------------------------------------------------------------------------------------------------

Conclusion.

According to MRG’s criteria (p.5), AV products must make "initial" detection to make it to Level 1 Certification. That automatically rules Webroot out of Level 1 due to its particular methodology.

Given the results and Webroot’s way of working, I am very satisfied with this report for Webroot (bar the PUAs).

-----------------------------------------------------------------------------------------------------------------------------------------------

Very important: not to be forgotten are the other two MRG tests that were published in the last few days:
MRG Effitas Online Banking/Browser Security Certification Project – Q2 2018 Level 1
and
MRG Effitas Online Banking/Browser Security Certification Project – Q2 2018 Level 2
 
This is perhaps the most indisidious of all families of malware: malware that can empty your bank accounts. To be noted is that whilst five of the eleven products tested passed the Level 1 test, only three passed the Level 2 test (3). One of those is Webroot.
 
(3) Some of us would discount one of those products because of potential backdoor security threats highlighted by all major Western governments. For those of us who do, that leaves only two AV products on the list.