Latest AV-TEST results

I still assume they are not testing the rollback and remediation that WSA can do with the repair test as the system is already pre-infected with malware before installation of WSA. :@ But I would like to see an improvement in this area as many users do install an AV solution when it's already infected or as I have learned over the years it's to late to clean as it can't be trusted so do a reformat and reinstall the OS.
 
http://www.av-test.org/en/test-procedures/test-modules/
 
TH

I believe Webroot can,as with MRG,commission a test that would take into account WSA's core features.If Kaspersky can commission a test testing their Safe Money feature,Webroot probably can do the same with their journaling and rollback.I do not ever see them changing their testing parameters in their regular tests.It might be worth the financial investment on Webroot's part.I am assuming with the repair score,they aren't taking into account the technologies that make WSA what it is.AT the end of the day,it's about keeping customers happy and satisfied and Webroot does that better than anyone.You do not see these forums flooded with infection complaints like Zeroaccess etc,nor are the phone lines flooded either,so Webroot must be doing something right.I have been,and continue to be,happily satisfied.I am sure Webroot will continue to improve like it always has:D

I would like to see/read an offcial explanation/response from Webroot regarding this subject, as they did with AV-Comparatives.
 
 I'm not sure they didn't test the rollback feature. I'm assuming that since the protection rate is really good (which contradicts AV-comparatives results :D) they had to test the journaling feature and, as far as I know,  'rollback' feature is linked to that one in particular... I don't know... It would be nice that Webroot said something about this subject.
 
On the other hand... WSA's score history in AV-Test is nice (this contradicts AV-comparatives 😃 again) but surprinsingly WSA's score in June test was better than in the current one (both performed under Windows 7). What has happened in the meantime?
 
UVOX

I forgot to post the link to WSA's score history in AV-Test:

http://www.av-test.org/en/tests/home-user/producer/webroot/

I will leave it to someone better qualified to explain like Kit.Just a quick tidabit as i understand it.AV-TEST tests products at their default settings,so most likely what happened is that testing encountered files that were unknown to the cloud and had to be placed in monitor until they could be identified.Keep in mind the entire time the system is being monitored and the files access is restricted..like a sandbox..and when the threat is identified,all actions can be remediated quickly.Testing does not account for this.Once the file is seen to be running on a test system,it's a fail,even though it can't actually do anything per se.Will let someone better qualified later chime in to further fill you in.

The only thing is the Systems are pre-infected before they install WSA so I can see the Rollback feature wouldn't be able to do it's thing in that condition. IMO
 
TH

I gave up on all such AV tests a long time ago and I don't read them anymore because in fact they are just artifical tests rather than the real tests. Most of the state of the art security solutions, what WSA is and keeps primacy over the rest, are so smart and inteligent to recognize that test files are in reality no real threats posed. Moreover still I think the personal preferences prevail over results of AV tests. Let's see in case of WSA, its strongest is to catch up on execution, hence quite loosing in the conventional tests. However does it mean that WSA is weak, certainly not. We know where the strength of WSA is and will keep it regardless on test results. Just my two cents :D

OK, let's say the rollback feature didn't have any chance to work... so I wonder why WSA's repair score in June AV-Test was better than in the current one (both performed under Windows 7).... 5 out of 6 (June)  vs. 2.5 out of 6 (October)
 
If rollback feature didn't have any chance to work in this October... it didn't have any chance to work in June test either...
 
Any suggestion?
 

That would indicate there was less need for the use of journaling and rollback. I'm not personally privy to the details of that test to say for certain, but it would stand to reason that it already had more of the threats in question blacklisted at the time of that test than in this one.

How's come they didn't test windows 8?

Hi ,
 
Now AV-TEST release new test on windows 8 .
 
Best Regards ,
 

I just noticed they changed some criteria. This time they don't show the 'repair score' and moved the 'performance score' out of the usability criterion.
 
Any ideas?

The reason largely has to do with Windows 8 coming with Windows Defender and the difficulty in finding malicious samples not detected by WD which are needed to verify registry remediation.
 
From an AVTest.org notice regarding these changes:
With Windows 8 and a pre-installed AV program running on the system,
some tests are no longer useful or would have to be much more complex to
still cover realistic scenarios. This especially includes the "REPAIR"
testing, as we would need to be very selective with samples not
detectable by Microsoft Windows Defender to test pre-installation
repair. As we also want to cover rescue media (e.g. installation CDs)
and specific stand-alone rescue tools in future, we have decided to
perform stand-alone (opt-in) reviews in future, covering all these
aspects. The monthly results will still include some small-scale repair
tests, but the results won't be published.
 
Besides this, we will continue testing of all Microsoft AV offerings,
but the results will be presented in a slightly different way, to show
the 'baseline' (or 'out-of-box' protection) which is part of the
operating system. This is especially a reaction to the tight integration
of the different security mechanisms in Windows 8 (Windows Defender, IE
SmartScreen, Desktop SmartScreen). It becomes increasingly difficult to
say this is the protection offered by the security products and this is
what is blocked by the operating system, so we handle it separately and
will show what 'out-of-box' protection each Windows operating system
provides. For the older systems (e.g. Windows XP and 7) we will use
Security Essentials for that approach.

Grayson,
 
Thank you for the explanation!  I looked at the new stats, and noted that they did not appear the same, but I did not manage to find that info. 

To me and many in the Security Community can`t get over this Repair it's utter nonsense. 😠 And Microsoft Windows Defender 4.0 on Win 8 didn't even get certified at the bottom of the list.
 
TH

i think repair do not means on Security .
bc Security Softwares are for protect , no for repair .
 
any threat can change many setting ( files , registery and .. ) .
 
system clear is base for install security software .
 
 

There are a lot of comments in this thread which I would like to address to help clear things up and to inform everyone of what's in store for WSA's 3rd party testing in the future.
 
The first bit I want to touch on is AVTest.org's repair testing and WSA's historical performance in this test. First, this test doesn't expose WSA's journaling/rollback ability as the endpoint product is disabled so the infection can be installed. While this puts WSA at a disadvantage, it is also something we expect to happen to our users - many people buy/try our product as part of their attempt to remove a current infection. As a result, WSA is very smart with how it handles remediation of a system which is infected when WSA is installed. Remediation includes cleaning the registry, removing start-up pointers and restoring policies.
 
With this said, our analysis of AVTest 'Repair' reporting showed we were being docked for missing simple run keys or services - both times WSA can remove without needing journaling. As you would expect, this lead to a number of conversations between AVTest and Webroot as we knew something was impacting our remediation scan. Ultimately, what we discovered is that AVTest was not gracefully shutting WSA down. When WSA detects it was terminated in a possibly malicious fashion, it performs a silent memory scan on re-launch which in turn intentionally does not remediate the registry to avoid possible no-boot situations. This resulted in very poor scores around ~40% in some of these tests. To accommodate the testing environment, we added a registry flag to suppress the re-launch scan which in turn resulted in a normal scan where registry remediation takes place. After doing so our scores jumped into the 80% range. Here they remained until November 2012 and Jan 2013 tests where again we started seeing run keys in the misses - and again we've been in communication with AVTest. Due to these issues, I must say I'm happy to see the 'Repair' testing removed - though I would like to see a newer type of test which could expose repair through journaling and rollback.
 
This leads me to my next point which is to inform everyone that Webroot is actively working with a few companies who provide next generation testing for cloud based solutions like WSA. One of the key components to these tests is what I like to call a time-till-detection test. This type of test measures how long it takes a security solution to react to and identify a new threat. As importantly, it will expose the inability of traditional approaches to react quickly - or at all.
 
As we are in an early phase of these tests, I cannot disclose who we are engaged with at this time, though rest assured - more validation from more sources using the next generation of efficacy and performance testing is coming soon!

 
 
So they weren't doing a graceful shutdown of the app and wondered why WSA did bad? o_O 
 
That's saying something. Like really wanting an app to crash just so you can say it's unstable. Like they wanted to make WSA look bad or something... 
 
I really hope this is a lesson for other testing firms becuase you need to make sure you are doing things properly or this misleads people and also reflects bad on the testing firm was well, that people place a lot of trust on. 
 
 

Just to be clear - AVTest was very willing to work with us to understand why our performance differed from how we know our software works. I can understand their approach considering the difficulty in performing this type of test across 27+ AV's. 

I would imagine using any one test would be very difficult to be fully accurrate when comparing such a large sampling of different products, some of which work very differently from others.

Glad that they worked with you on it.

Testing antivirus is hard.

Glad to hear that they were willing to work with Webroot to a degree, and that WSA got good scores!

It's not so usual that a test laboratory would be so co-operative. Thanks to AV Test, other testers could have changed their mind about WSA, hopefully.

AV-TEST AWARD 2013 On 11th February 2014, the AV-TEST Institute will present the AV-TEST AWARD to the best security productsproviding protection against malware infections. This award honours the best home user and corporate products for Windows in the categories of PROTECTION and PERFORMANCE. This year, the AV-TEST AWARD will also be presented to an anti-virus solution providing the best protection for Android systems for the first time. The AV-TEST Institute uses its AV-TEST AWARD to reward the most effective anti-virus products tested in its different test categories.
The AV-TEST AWARD FOR BEST PROTECTION 2013 will be presented to the best product of the year in terms of its protective effect on the Windows und Android operating systems. This category evaluates products according to the protection that they provide against current threats such as malware infections caused by zero-day attacks such as those found in malicious websites and e-mails. 

Home Users (Windows):
Bitdefender Internet Security

Corporate Users (Windows):
F-Secure Client Security

Mobile Devices (Android):
Antiy AVL
 
 
The AV-TEST AWARD FOR BEST PERFORMANCE 2013 will be presented to thesecurity software that has the least influence upon a system once installed. The tests that are carried out involve typical activities such as loading websites, downloading software, installing and starting up programs and copying files.

Home Users (Windows):
Bitdefender Internet Security

Corporate Users (Windows):
Symantec Endpoint Protection
http://www.av-test.org/en/test-procedures/award/2013/