Why using VirusTotal for AV testing is a bad idea

By Marco Giuliani

As I am working in the malware research field, I usually read about antivirus testing comparatives done by known or - more often - unknown companies and websites. The most common trend lately looks like writing independent comparatives to show how security software badly react to fairly new threats. 

Indeed every day thousands of new malicious software is created and used for targeted attacks, which are becoming the primary threat for every security vendor. Here at Prevx, we are able to examine this tendency hourly.

 

Last article I've read about the effectiveness of antivirus solutions that has been written by FireEye on their official blog. Although the article is really interesting and well written, I would have been more careful when drawing conclusions.

 

The opinion of the author of this article is that security solutions are more effective against older threats than new and emerging ones, where the average detection rate is only 40% of products. Essentially, the author's words are: "AV is likely to detect a currently used bot binary "less than half" the time".

 

Although I agree that detecting new threats is a difficult challenge, because it needs powerful heuristic engines and proactive technologies, I've got doubts about the conclusions achieved.

 

Where I totally disagree, is the use of VirusTotal online scanner as the primary tool to check effectiveness of antivirus solutions. VirusTotal is a great and useful service and it can give users some statistics about detection rates, it can't be used as the tool that allow testers to write comparatives and judge antivirus's effectiveness.

 

Sadly, a number of so-called "independent" comparatives are relying upon VirusTotal results. This can't give a complete overview of security software's efficiency.

 

Detecting new and emerging threats without signatures is a hot topic that has been widely discussed even during international conferences. Everyone in the security field knows that, as today, plain signature detection is not sufficient. Developing new techniques to prevent and block targeted attacks and new malwares is one of the most important goals of every security vendor. It couldn't be different.

 

Here at Prevx it is our first goal. After Prevx Edge release, many happy users reported us that Edge has blocked new threats even if, after a VirusTotal scan, those suspected files were not detected by the Prevx engine included in VirusTotal online scanner.

 

For instance, a famous website has been attacked and malicious code has been injected inside the main page some days ago. Whilst lots of users tested the dropped malware on VirusTotal and drawing wrong conclusions, Prevx Edge has been able to heuristically block it since the beginning.

 

This is because many new heuristic techniques that we use can't be included inside the on-demand scanner, which will simply check if the plain file signature is present inside the community database.

 

I've exposed the situation as it is for Prevx, but this is common to other security software too. They often include new techniques - behavior blockers, heuristic behavior analyzers, dynamic heuristic engines and so on - used to mitigate (or override, most of times) the gap between malware creation and signature release.

 

If sometimes you find heuristic detections on VirusTotal, it doesn't necessary mean that the heuristic detection is totally implemented inside the on-demand scanner. Simply, there are some techniques that can't be easily implemented inside an on-demand version of the scanner. Anyway their role is crucial and they allow security software to detect 0day threats.

 

If you rely only upon VirusTotal results, then you could miss the real effectiveness of tested antivirus solutions.

 

This is why using VirusTotal for antivirus comparatives and testing is the wrong approach.