Nirsoft's Antivirus Hall of Shame

[mouser]

Martin over at ghacks writes today about a recent essay posted on the Nirsoft site, discussing the issue of false positives, and ranking antivirus tools.

False positives are when an antivirus tool flags a program as being a possible malware when it really isn't.   They can be a huge pain for small developers, creating unnecessary fear among users.  And some antivirus companies are outrageously irresponsible about these kinds of detections, not explaining to the user the difference between a known malware and a complete half-assed guess about something they don't understand.

We've talked a LOT about this issue on the DonationCoder forum over the last 10 years, and have been bit by these lazy antivirus coders on more than one occasion.

Anyway, the nirsoft post goes into some detail ranking antivirus tools according to their false positives.

Full Nirsoft essay: http://blog.nirsoft....es-of-nirsoft-tools/

(see also the ghacks summary: http://www.ghacks.ne...virus-list-of-shame/)



I do think it's worth repeating what I've said many times -- I don't expect the antivirus tools to be 100% right all the time -- I understand that sometimes they want to be better safe than sorry.  But the thing is, if you want to tell a user that you have found a file that you haven no experience with, and it has some patterns that remind you of something similar you may have seen before which might be a malware, but might not, fine, i have no problem with that -- TELL THE USER WHAT YOU KNOW AND TELL THEM HOW TO GET MORE INFORMATION AND TELL THEM HOW TO LET YOU KNOW IF YOU ARE WRONG.

Just do not go throwing up a siren telling the user that malware was found in some program if you aren't damn sure it has been.

[wraith808]

Just do not go throwing up a siren telling the user that malware was found in some program if you aren't damn sure it has been.

-mouser (October 19, 2015, 03:15 PM)

Let me preface this by saying that I totally agree with you... but to play Devil's Advocate, sometimes when new virii are released into the wild, there have been massive outbreaks because they just didn't know.  With this way, I'm sure that some have been caught that wouldn't have otherwise.  So how do you toe that line?

[mouser]

Just do not go throwing up a siren telling the user that malware was found in some program if you aren't damn sure it has been.

-mouser (October 19, 2015, 03:15 PM)

Let me preface this by saying that I totally agree with you... but to play Devil's Advocate, sometimes when new virii are released into the wild, there have been massive outbreaks because they just didn't know.  With this way, I'm sure that some have been caught that wouldn't have otherwise.  So how do you toe that line?

-wraith808 (October 19, 2015, 04:18 PM)

I really don't think it's that difficult to do.  I think it's less a matter of changing the functionality than the communication with the user.

Let the user choose whether to include these highly-sensitive heuristic checks or whether not too (almost all do this already).

When such a heuristic detection is encountered, EXPLAIN TO THE USER THE HIGHLY UNRELIABLE NATURE OF SUCH HUERISTIC CHECKS.
Explain that this could very well be a false positive, and that the file analyzed could in all likelyhood be completely safe.
But block access to it by default, and give them some links to help them figure out whether the file really is malicious.  Help them perform a multi-engine analysis easily (auto upload to virustotal, etc.).

And make it easier for software authors to get their non-malware false-positived software excluded if it ever does trigger a false alarm, by having staff that can verify and whitelist software quickly.

[rgdot]

Devil's Advocate part deux:

That is way too much for the typical user.

[Shades]

Or the typical end-user should up their game and actually grow some sense!

While that would be the best direction to go, it will never happen, because of 2 reasons:
1). More savvy end-users do not benefit the coffers from anti-virus vendors.
2). Typical end-users either have an inability to grow some sense or worse, they don't care.

Strike fear in the hearts of typical end-users will make anti-virus vendors (more) money and common laziness from the same end-users makes sure this situation won't change soon, if at all.

Nowadays I only scan with on-demand (online) anti-virus scanners on my own systems at my convenience. The heuristics use quite a lot of real-time computational resources to get to the wrong conclusion anyway. So I don't bother anymore. This I can do as I am a reasonably competent user and I (as a non-admin user) am the only one touching my systems. Also, I don't run illegal software or games and haven't visited "Russian bride sites" that show their "intimacy" skills, while supplying you with keygens and such, for quite some time now.

In any other use case, you shouldn't. But most people here on DC already know this.

[tomos]

I'll forgive an anti-virus for false-positives if they make it easy to report them (and they do something with that info).
That's why I stopped using Avira (Pro) a couple of years ago. I wasn't sure whether to mention them here, because at least you were able to report, and they did do something about it -- and they may have improved/streamlined the process since, but at the time, the {amount of false-positives} + {the difficulty reporting} was too much for me.

[Stoic Joker]

Or the typical end-user should up their game and actually grow some sense! While that would be the best direction to go, it will never happen, because of 2 reasons:1). More savvy end-users do not benefit the coffers from anti-virus vendors.2). Typical end-users either have an inability to grow some sense or worse, they don't care.

-Shades (October 19, 2015, 07:22 PM)

While my reflexive cynicism makes me inclined to agree... The positively abysmal performance of AV software over the last several years have caused many of the top security companies to come up with a rather new concept called the Human Firewall. Which I'll admit is a much catchier name than what I've been calling it - Defensive Driving on the Information Highway - for years.


So perhaps play time is over, and it really is time for people to knuckle down and learn how to drive.

[JavaJones]

Has anyone done any studies/surveys/etc. on how often these heuristics actually catch "0-day" vulnerabilities? You'd think this would be the kind of thing that the big AV companies crow about all the time, especially when a new vulnerability is discovered. "Dear Norton users, today you are safe from the massive BlarghNet virus outbreak because our advanced heuristics proactively detected the unsafe behavior before anyone even knew of its existence. Stick with Norton and be safe!", or whatever BS. Anyone heard/seen any press releases like that? I know I haven't. Maybe it's just not happening that often...

Even if heuristics are useful, I do think the behavior of AV products could really stand to be improved in the ways others are suggesting. 2 notable things, I think:

1: Try to explain to the user/give more info when a heuristic-based detection is made. One big improvement for at least many AV products would be clearly differentiating between something detected based on a clear, positive signature match (strong positive match) vs. a detection of "suspicious behavior" or code based on a heuristic. If a % match or degree of certainty could be estimated, all the better. Will the average user do much with the info? Maybe not. But that's not a good reason not to at least *try* to educate them.

2: Easily allow false positive reporting. Some AV apps do allow this, if I recall correctly, or perhaps just take an "allow" choice as a "vote" by their user community. But as we've seen from Nirsoft as well as some of DC's own software, the AV companies do not move fast enough on this, and are not responsive enough to the actual usage patterns of their customers.

- Oshyan

[Stoic Joker]

Anyone heard/seen any press releases like that? I know I haven't. Maybe it's just not happening that often...

-JavaJones (October 20, 2015, 01:57 PM)

Does bring to mind the old expression "Deafening Silence" doesn't it..

Your point 1 as stated would dovetail rather nicely with the Human Firewall (educational program basically) I mentioned earlier. If the AV companies toned down the jargon a bit, and just - 10 words or less - plainly stated we think this is trying to X the overall outcome would improve drastically virtually overnight. Hell some of these reports are so blatheringly unspecific that I can't even figure out what the heck they're trying to say half the time.

[Renegade]

ALYac     0     0     0     0     100

Woohoo for ALYac and my buddies over at ESTsoft~!

I really don't think it's that difficult to do.  I think it's less a matter of changing the functionality than the communication with the user.

-mouser (October 19, 2015, 04:48 PM)

I think you are overestimating how easy it is there. Communicating with the user is very, very far from easy for this kind of software.

1. They don't care.
2. Stop using big words.
3. like wtf heristick lip gloss huh lol?!!11!!1!\
4. wut wuz i doin
5. pron gam3z 4evah!!!!!1111!1

Or something along those lines.

You've got a very broad audience where computer literacy ranges from god-like to near zero. Addressing that range of people is tough, and the less computer literate they are, the more important the communications are.

And you've got the issue of not nagging them, but at the same time making sure that important notices aren't dismissed by reflex.

Can it be done? Sure. But I think the effort required for those communications (and the infrastructure to support the communications) is far more than most companies are willing to even entertain, even if they had the imagination for the task (which I doubt is there as it requires reimagining standard and common practices).

[Stoic Joker]

Can it be done? Sure. But I think the effort required for those communications (and the infrastructure to support the communications) is far more than most companies are willing to even entertain, even if they had the imagination for the task (which I doubt is there as it requires reimagining standard and common practices).

-Renegade (October 20, 2015, 11:15 PM)

It's actually not that hard to do if you get a bit of a buddy system going. If Email/system message X looks suspicious/odd/important., ask somebody, anybody...just as long as there is a second pair of eyes on it (it makes a difference). There will always be that one person in any group that is brighter than the rest, so use that person to your advantage and let the others go to them. Smaller groups of brighter people train the herd of others ... Leaving the IT staff to mop up the messes of those that don't catch on as fast.

Most importantly, force people to get in the habit of actually reading the messages that are presented to them. I've been informally training both our in-house staff, and the staff at our clients for years, and it's been quite successful. I do frequently get calls from client locations asking about strange messages/behavior from time to time ... But it's at the 'Just click no!' stage that I can get them out of on the phone now about 95% of the time.

[wraith808]

Most importantly, force people to get in the habit of actually reading the messages that are presented to them. I've been informally training both our in-house staff, and the staff at our clients for years, and it's been quite successful. I do frequently get calls from client locations asking about strange messages/behavior from time to time ... But it's at the 'Just click no!' stage that I can get them out of on the phone now about 95% of the time.

-Stoic Joker (October 21, 2015, 05:56 AM)

This.  100x this.  Winpatrol is great... if you read the messages.  It, more than anything else, has saved me from countless hours working on relatives' machines.  But it does take attentive computing.

[JavaJones]

Maybe people just click through because the current warning dialogs say little or nothing to them that they can understand or use to make any kind of rational choice. Maybe there are too many such dialogs (false positives, remember...). Both of these issues can be improved.

There are, of course, always going to be people you just can't reach or make understand what the software is saying. But that doesn't mean it's not worth trying to improve the current approach which uses obscure (even to me) references and terminology and provides minimal real, relevant information. It would be fairly easy to design improved dialogs for this sort of thing that provided a 2-3 sentence summary of the reason for the warning, e.g.:
"We've detected a program named 'myapp.exe' acting suspiciously on your computer, but it doesn't match any currently known virus. The unknown program appears to be trying to alter important files used by your operating system."
Or:
"The unknown program appears to be attempting to access the Internet in an unusual way", etc., e.g. on a non-standard part, or whatever, we're just dumbing it down here.

And then a prompt:
"If you're not sure whether this is legitimate behavior just choose to Quarantine the program and we'll suspend its activities. If you change your mind later you can always Restore it in your antivirus control panel, accessible from the system tray icon in the lower right of your screen."

And then you have one of those expandable dealies to get more info for advanced users, or an "Advanced Info" text button (don't make it look like something just anyone would want to click, i.e. not a shiny button). If a user clicks for advanced info they get a process name and path, and other info, maybe some buttons to open the process properties, or path, whatever.

That's just a simple idea off the top of my head. And I think it improves on almost every antivirus warning dialog I've ever seen. It would not be difficult or complicated to implement, every suspicious behavior heuristic maps pretty basically to a few simple categories like "unusual network activity", "trying to access or modify system files", etc. Just translate those into human readable dummy speak and put it in a friendly 2-3 sentence description.

Oh and yes, we need to make the messages mandatory to read, so use UAC prompts (why don't more antivirus apps do this when they detect problems!?).

- Oshyan

[wraith808]

Oh and yes, we need to make the messages mandatory to read, so use UAC prompts (why don't more antivirus apps do this when they detect problems!?).

-JavaJones (October 21, 2015, 01:56 PM)

UAC prompts don't make it mandatory to read.  Just to click!

[Renegade]

Most importantly, force people to get in the habit of actually reading the messages that are presented to them. I've been informally training both our in-house staff, and the staff at our clients for years, and it's been quite successful. I do frequently get calls from client locations asking about strange messages/behavior from time to time ... But it's at the 'Just click no!' stage that I can get them out of on the phone now about 95% of the time.

-Stoic Joker (October 21, 2015, 05:56 AM)

This.  100x this.  Winpatrol is great... if you read the messages.  It, more than anything else, has saved me from countless hours working on relatives' machines.  But it does take attentive computing.

-wraith808 (October 21, 2015, 11:28 AM)

For workplaces, it's one thing, but for the broader at-home audience, who will train them to read?

A while back MS put out some design guidelines for buttons where you had larger and smaller text on the buttons and they were anchored to the sides of the form that they were in so that you had very large, wide buttons.

As the buttons are the action items themselves, they promote actually reading the text more than when you have the text outside the buttons and simple yes/no/cancel text on the buttons.

That's one tactic to get people to read, but it could be improved. I think a wizard-like UI with buttons like that for more complex decisions could be used to get people to read more -- sort of like a "choose your own adventure" set of paths.

But getting people to read? Not all that easy.

I had one guy complaining about how my software didn't work after he bought it... he couldn't open any files, etc. etc. Turns out he never even installed it!!! You just can't compete with that kind of ignorance.

[wraith808]

I had one guy complaining about how my software didn't work after he bought it... he couldn't open any files, etc. etc. Turns out he never even installed it!!! You just can't compete with that kind of ignorance.

-Renegade (October 22, 2015, 12:39 PM)

You win.  We're doomed. 

[Stoic Joker]

For workplaces, it's one thing, but for the broader at-home audience, who will train them to read?

-Renegade (October 22, 2015, 12:39 PM)

You do realize they're the same people right?? Yeah sure, people have a tendency to turn off their brains when they get home...but some of it will still leak through. And we already know the current system isn't working for shit and never will.

But getting people to read? Not all that easy.

-Renegade (October 22, 2015, 12:39 PM)

Not that hard either, just be succinct and skip the pedantic jargon. 

I had one guy complaining about how my software didn't work after he bought it... he couldn't open any files, etc. etc. Turns out he never even installed it!!! You just can't compete with that kind of ignorance.

-Renegade (October 22, 2015, 12:39 PM)

You win.  We're doomed. 

-wraith808 (October 22, 2015, 02:38 PM)

Oh hell no he don't ... He's never been to the DMV.

[Steven Avery]

Remember, Nirsoft is the situation of using low-level tools, like reading the masked password. The company has a great reputation, but the only way I can think that could really work with a lot of companies would be if his programs were on the company white-list registry with a checksum approach. That is asking a lot.  Otherwise the programs do things that you want to be warned about.  However, you should be informed, not scared.