Author Topic: The DonationCoder "Superior Antivirus" Award/Certification (Read 50640 times)

mouser · « **on:** December 01, 2009, 10:55 AM »

In this thread, one of many on the DonationCoder forum where we are all screaming about the harm that lazy antivirus companies are doing with their false positives, I suggested that maybe we need to do something productive to encourage these companies to be more responsible about the alerts they show.

So today I want to begin that process by asking for your help in coming up with a short and clear list of requirements that would be worthy of our recognition for a new antivirus/anti-malware standard that is focused not on the number of virus detections, but on how users are told about alerts which may be false positives, and how well they deal with false positives.

Once we've got something I'd like to make an official web page about this, and then try to contact the antivirus companies and maybe get some other websites that want to join us in this movement. And hopefully one day in the near future we will be able to give this award out to a company and lavish them with praise, recommendations, reviews, etc.

Let me start out with my first draft of requirements for what i'll call the DonationCoder "Superior Antivirus" Award/Certification:

When a suspected malware is found, the user must be presented with a dialog that clearly describes:

The complete file path of the suspected file.
A description of the suspected malware (not just some cryptic name), with an easy link to search the web for more info about this virus and the file found.
A clear indication of the date that the antivirus signature matching the file was added, with a clear statement about the possibility that this may be a false positive, and telling the user some information about the confidence that the file is indeed a real malware vs a false positive. this should be a statement like "this is a generic rule of thumb pattern that was recently added, so the chance that this is in fact a false alarm and not a virus is quite high."
In the alert there should be a url to take the person to the antivirus company's forum where they can talk to others about whether the problem is real or not.
The user must be given an opportunity to not delete the file.
The user must be given an alternative to go to a page where they can report a suspected false positive
The user must be given the alternative to upload the file to an online site like virustotal for a second opinion.

Thoughts? What am i missing? Anything here that is asking too much?

scancode · « **Reply #1 on:** December 01, 2009, 11:03 AM »

Win32.Gen/DOCOVIR/PACK: A "detected threat"
[Heuristic] Application Packed with DoCoPack: Nothing to be worried about.

Which one do you think looks better in the stats?

mwb1100 · « **Reply #2 on:** December 01, 2009, 01:48 PM »

Let me start out with my first draft of requirements for what i'll call the DonationCoder "Superior Antivirus" Award/Certification:
-mouser (December 01, 2009, 10:55 AM)

That's a great list! I hope this initiative gains some traction.

Off the top of my head, what I'd like to add (I hope this isn't taking the discussion off-topic) is that firewalls should use a similar set of guidelines for when they detect something fishy. Often I get an IP address and port number and little else. I need the following information to have a hope of understanding whether I should be concerned or not. I understand that not all of this information might not be easily (or even possibly) determined, but to the extent possible, I'd like to see:

the process that's involved (including the full path of the .exe)
the DLL(s) involved (again, including path)
whether (and by whom) the binaries involved are signed
a reverse DNS on the IP address - if there is no reverse DNS name available, some "whois" information on the subnet block the IP is in might be nice
information on the port if it's a well-known (or commonly used for a particular purpose) port number
links to web information and/or a user forum that discusses the program and the communication (if appropriate)
a link to virustotal (or similar) for the binaries involved would also be nice

Of course, not all of this data needs to be in the initial notification, but it should be made available at the click of a 'more information...' button.

f0dder · « **Reply #3 on:** December 01, 2009, 06:22 PM »

Good idea, but it's never going to happen - for exactly the reason that scancode is hinting at (sorry to be a cynic, but that's the way the world works).

mouser · « **Reply #4 on:** December 01, 2009, 06:26 PM »

I'm as pessimistic about these things having any effect as anyone, if not more so. But I also view these things from a pragmatic cost-benefit analysis standpoint. It costs us little more than a few hours of our time to set something like this up, and a promise that any antivirus company that lives up to the standard will receive some publicity and praise from us. Surely worth a try.

f0dder · « **Reply #5 on:** December 01, 2009, 06:32 PM »

Well, it is worth a try - and after all, AnandTech got OCZ to do The Right Thing^TM and optimize their Vertex SSD firmware for random access at the cost of the (pretty irrelevant) sustained transfer rates... something marketing probably didn't like

mouser · « **Reply #6 on:** December 01, 2009, 06:33 PM »

anyone want to try to make a nice fancy professional looking graphic logo for the page?

JavaJones · « **Reply #7 on:** December 01, 2009, 06:53 PM »

I too like this idea a lot and would be glad to contribute anything I can. I think your list is pretty good as-is. The idea is to provide as many resources as possible for people to find more info, which of course they don't have to use, they can still just trust the program. I don't know how possible it would be, but some kind of "certainty rating", with a little graphic, on a 1-5 or 1-10 scale, or perhaps a percentage, would be great. E.g. "Win32.Gen/DOCOVIR/PACK has been detected as a threat. Avast is 35% certain that this is a genuine security threat." which would be based, at least in part, on similarity to predicted malicious behavior and code, for example.

- Oshyan

mouser · « **Reply #8 on:** December 01, 2009, 07:02 PM »

i think the rating idea is a good one -- but i'm trying not to "require" any details in this list that we don't feel are mandatory. so we might say that the program needs to report SOME estimate of certainty in some form. this is not going to be easy to put into a numerical form.

JavaJones · « **Reply #9 on:** December 01, 2009, 07:31 PM »

Fair point.

- Oshyan

scancode · « **Reply #10 on:** December 02, 2009, 01:49 AM »

"Win32.Gen/DOCOVIR/PACK has been detected as a threat. Avast is 35% certain that this is a genuine security threat."
-JavaJones (December 01, 2009, 06:53 PM)

Then "detects 100% of on-the-wild threats" becomes "detects 100% of what it thinks could be on-the-wild threats".
Doesn't look good on product pages

mouser · « **Reply #11 on:** December 02, 2009, 06:30 AM »

yeah how does this sound on the marketing page:

"We are the only antivirus program to meet the stringent requirements of the 'Superior/Honest Antivirus' certification -- which establishes the highest standards for reporting possible threats to users. read more.."

app103 · « **Reply #12 on:** December 02, 2009, 09:19 AM »

While false positives are frustrating to those that are having their work flagged, it can be more frustrating to the user when the messages they get from their antivirus leave them confused as to whether or not they should delete it or keep it. Anything that confuses the user more will raise the potential of a real threat slipping through, due to user ignorance.

Anything that could make a real threat seem not so threatening, potentially will lead to the user making the wrong choice.

So, while I understand and share your frustration, it has to be balanced with the need for the security product to effectively do it's job on both the computers of the more savvy power user, as well as the complete newbie, and everyone in between.

The more text, info, and options you give the user at the time of detection, the more they will be confused, the more will go wrong, and the more infected people there will be.

Any alert from an antivirus no matter how simple is likely to put the user in a state of panic, in which they potentially will not think clearly. Yes, this info may help them but more likely the panic will make them unable to mentally process that info at the time it is given.

Now I am not saying this info should not be available to the user. It would be great if it was, but not at the time of detection or bothering the average user. If it were included in an "advanced mode" then I could agree with it and would even welcome it.

mouser · « **Reply #13 on:** December 02, 2009, 09:27 AM »

one of the rare times i have to disagree with you app, about this being only for an advanced mode.
i could agree to an alert box with a very simple message and then a button to click for this more full info.

but my reasoning is based on agreeing with your statement:

Any alert from an antivirus no matter how simple is likely to put the user in a state of panic, in which they potentially will not think clearly.

these virus alerts scare the hell out of people.
and it is my impression that most of the time they are false alerts.

it is imperative that these bright red scare-the-death into you alert boxes tell people the concise information they need to know to make an intelligent decision.

the false positives are like the story of the boy who cried wolf -- you can't keep showing false positives and expect people to take you seriously when you really do find something wrong.

so the alert box, in default mode, has to help the user understand the real nature of the thread and HELP THEM make a decision about what to do.

the purpose of these guidelines are to establish a standard for information that needs to be available to users when an alert comes up.

Carol Haynes · « **Reply #14 on:** December 02, 2009, 09:45 AM »

I would like to add that any warning should be accompanied by the method of identification of the risk. For example if heuristic checking is responsible for the alert it should be made clear with a message like "No actual threat has been detected but some behaviours of this software suggest the possibility of unwanted activity" then all the other information you suggested.

Further if they are using pattern matching I think there should be a score on how many elements of the pattern match the found issue.

I have spent hours tracking down 'viruses' and 'trojans' only to discover that of all the possible indicators of malware presence there was only one possible marker - which turned out to be legitimate.

I am not saying that viruses and trojans exist but in the last few years I have not come across a single genuine attack on any of my computers (other than the odd spam with bad attachments which have almost entirely been removed by googlemail before they got to me). I have had plenty of false positives though and almost always involving dodgy heurisitics.

I have to say I haven't had any false positives with free AVAST!

Personally I think there is a bit of psychology going on here - if expensive secuity apps don't appear to be doing something useful customers will decamp to free solutions consequently almost all of the companies trying to cash in on cyberfear have to promote that fear in faulty heuristics. Maybe I am too cynical but the only people that gain anything from the general public by false alarms are the companies producing those alarms.

In recent months I have been to quite a few clients with viruses. Most are running one of the most popular solution (N or M) and can't understand why there was no warning and they got infected - generally these are the ones that are infected. A large number however have received warnings and been scared and generally these are the clients that have experienced false positive syndrome.

PS: Why don't we just set up a new website called starvethegreedybastards.com and have it is a gereral rant site for this sort of behaviour. Would fit in well with the donation ethos.

Carol Haynes · « **Reply #15 on:** December 02, 2009, 08:24 PM »

Oooo err ... its all my fault mother. A few hours after I posted the message above AVAST updated automatically (free and Pro versions) and promptly went into meltdown.

Since VPS/091203-0/3.12.2009 was automatically installed this evening apparently every user of Avast is now manically fighting false positives on their computers. So much so that the suipport section on Avast's website is impossible to get onto because of massive traffic.

The symptoms are that just about everything is classed as Win32:Delf-MZG [Trj] and the prompts come so thick and fast that you can't get to the icon to turn the AV off. The only solution I have found is to disable the software as soon as the icon appears in the system tray (not easy if it is usually hidden!).

I'm sure there will be a fix in the morning but it is a bit ironic that I had only just waxed lyrical about the reliability of Avast.

I suspect my phone will be ringing off the hook in the morning with customers complaining they have a mass of viruses - if I was unscrupulous I could make my fortune

biox · « **Reply #16 on:** December 03, 2009, 01:04 AM »

Oooo err ... its all my fault mother.
-Carol Haynes (December 02, 2009, 08:24 PM)

Thanks for the info....I knew it was your fault

Avast provided me with some much needed exercise this morning by having me jump between rooms. 'The-one-that-has-to-be-obeyed' sits in another room with XP pro and Avast pro only to scream every 2 seconds. Even the FW update is a virus.

When a suspected malware is found, the user must be presented with a dialog that clearly describes:
A clear indication of the date that the antivirus signature matching the file was added,
-mouser (December 01, 2009, 10:55 AM)

that applies for FWs as well, anyway please tell me when it got there NOT when you found it because I know when, after all I'm sitting right in front of the computer.[/list]

cmpm · « **Reply #17 on:** December 03, 2009, 02:46 AM »

So can we offer products up on this thread, and see/dissect if any of these expectations are met by any AV/security software?

mouser · « **Reply #18 on:** December 03, 2009, 08:39 AM »

A write-up about the avast freakout:
http://www.downloads...alse-positive-spree/

cmpm let's leave this post for working out the details of the award -- i dont think any antivirus company meets these requirements at this time.

cmpm · « **Reply #19 on:** December 03, 2009, 09:51 AM »

Ok, sounds good to me.
All these points are good ones if any 1 can achieve them all, I would be surprised.
I think it would have to a combination of more then one product.

nudone · « **Reply #20 on:** December 03, 2009, 10:10 AM »

would be nice of an open source project managed to complete the task.

barney · « **Reply #21 on:** December 03, 2009, 10:31 AM »

Just ran across this thread. Interesting concept.

'Fraid it would be useless to some of the folk I know, though, for purely physical reasons. They are touch typists.

Problem I've had with every AV I've used is that a warning window with a default action pops up and steals focus when the AV thinks it's found something. If I happen to be typing notes I've made into an editor, or perhaps copying a bit of PHP out of a book, I'm not looking at the screen. Even with an audible alert, I'll probably hit a hotkey or enter before I can stop typing & check the monitor to see what just beeped. Even if I am looking at the monitor, I may not be able to stop typing before I've dismissed the warning.

When that happens, I've just initiated whatever default action the AV uses upon I know not what file. [Sidebar. Have you ever quarantined/deleted a kernel file? It's fun.]

So, as long as the AV steals focus, any information it provides is frequently going to be useless. That said, the information would indeed be useful during, say, an intentional scan, or maybe during boot-up.

mouser · « **Reply #22 on:** December 03, 2009, 10:49 AM »

barney, i think thats a really good point -- maybe we can add to the list that the alert cannot be accidentally triggered by a keyboard press.
there should be no way to accidentally hit a key to have some action take place. maybe a time delay before it responds.

barney · « **Reply #23 on:** December 03, 2009, 11:11 AM »

Actually, that one's pretty simple.

Make certain no button has focus or can be triggered with a single key press, i.e., Quarantine would be Alt-Q, not just Q. Alternatively, make the default button perform no action, let it be a dummy.

The alternative is chancy, since the keyboard folk will wanna be able to tab to the button of their choice, so inadvertent action is still a possibility.

Don't think time delay would work unless there's some way to circumvent if the wrong action is taken.

There was something I used in Linux, don't remember what, that would pop up a window on top of all others, but did not steal focus from whatever was being done. It got your attention when you looked at the screen, but let you keep doing what you had been doing ... kinda weird, looking at a top-level window, but still typing into the window below it.

Maybe that could be done ... not that I'd expect the AV folk to buy it, but, still ...

mwb1100 · « **Reply #24 on:** December 03, 2009, 12:17 PM »

The focus-stealing problem can be solved by having the notification take the form of 'toast' popups (I'm not sure what the official name of this UI element is). The notification is visible, and I can interact with it if I like, but it doesn't steal focus from what I'm currently working on.

Outpost Firewall notifications do this for me today.