1
General Software Discussion / Have AV False Positives Improved? (Systemus 16 months later)
« on: April 11, 2022, 09:43 PM »
First, a quick thanks to BGM, Mouser, and many other authors who have created useful, or at least interesting, tools for us.
Systemus is a handy dandy system admin tool that BGM created for N.A.N.Y. 2020. Systemus had problems with false positives from antivirus products. Just out of curiosity, I ran Systemus through three meta-scanners, Jotti, VirusTotal, and MetaDefender, two days ago just to see what had changed. The results did improve (fewer presumably FPs), but are far from perfect.
At the end of 2020, I submitted Systemus to a half-dozen-ish vendors, plus a few more later on. Over the intervening couple of weeks, VirusTotal varied from 23 FPs initially, to 15 minimum, to 17. BGM reported 23/69 for VT in his response. I'm not sure why we had different totals, but I can imagine several possibilities (e.g., maybe BGM ran his PE Studio analysis a couple of weeks before his reply in the thread; or maybe he used .exe and I noted .zip, primarily); etc.). FPs also decreased on Jotti (no details) and MetaDefender (8 to 5) during those few weeks. Some other person also may have submitted FP requests during that few weeks or the intervening months. (See end of https://www.donationcoder.com/forum/index.php?topic=48696.msg442731#msg442731 for my comment and BGM's response.)
As several developers noted in that Systemus thread (and many other threads), this can be frustrating: each antivirus false positive is going to discourage users from experimenting with the software, but getting each AV vendor to evaluate each iteration (or even periodic stable releases) is like playing whack-a-mole at the bottom of the deep end of the pool using your nondominant foot with one eye closed and with some moles stuck in the up position. (I'm look at you, Webroot, who didn't actually evaluate/act on Systemus even after I submitted it twice; and you, McAfee in several guises, who has lots of fiddly restrictions; and a few others who make submitting arduous -- only webform with odd fields, only via forum, only via installed AV(!), only by author (not a user), etc. -- or even impossible as far as I can tell). A couple of reputation-based products aren't (or weren't a couple of years ago) willing to whitelist little known software even after their lab had reversed a FP in the main AV. Some authors recommend other authors just ignore FPs as too much trouble.
For comparison, these are April 2022 results for Systemus using the same files from my download folder. The dates here identify the last update to the engine/signatures.
Jotti
.zip (1/14):
Fortinet Apr 8, 2022 W32/PossibleThreat
VirusTotal
.zip (6/59 -- 59 excludes incompatible or nonreporting AVs):
AhnLab-V3 Malware/Win32.Generic.C3986407
Fortinet W32/PossibleThreat
MaxSecure Trojan.Malware.300983.susgen
McAfee Artemis!A5AC6681733F
McAfee-GW-Edition Artemis!Trojan
Panda Trj/CI.A
.exe (run today -- 8/68 -- excludes 4 incompatible and 1 nonreporting engine)
Same FPs as above except:
(different) McAfee-GW-Edition BehavesLike.Win32.Dropper.dh
(added) Palo Alto Networks Generic.ml
(added) Webroot W32.Malware.Gen
MetaDefender
.zip (0/35 but infected components flagged -- 35 includes 2 incompatible file type and 1 no result)
.exe (2/35 -- includes 1 no result)
Malware/Win32.Generic AhnLab Apr 9, 2022
Malware Webroot SMD Apr 8, 202
.dll (1/35 -- includes 1 no result)
Malware Webroot SMD Apr 8, 2022
I have no idea how much of the difference from two years ago may be due to FPs being fixed (either proactively or because someone submitted a request), or due to improved engines, or due to pruning because the miniscule installed base is no longer considered relevant (perhaps partially due to user whitelisting).
Note that due to different settings or other differences, "AhnLab" and "McAfee" reported "No Threat Detected" on MetaDefender while alerting on VT. This is not the first time I've noted a few differences between results for the same engine between these (and other) meta-scanning platforms. VT has a statement somewhere that mentions various reasons VT results may vary from the same vendor's installed product and/or web scanner. Some of these reasons (plus others) also apply between meta-AV platforms.
My personal impression is that, for the same vendor, VT tends to have more hits (almost always FPs for me) than MetaDefender and Jotti, but I have seen the reverse.
A couple of tools I used to automatically submit likely FPs to selected multiple sites via email were discontinued many years ago. I saw a few attempts to create something similar, but those weren't maintained (but I haven't looked recently). TechSupportAlert had a great list but it is not as useful anymore -- especially for FPs. MetaDefender's knowledge base has an updated list for its vendors (usually email, sometimes web or other) but doesn't explain the restrictions (i.e., address but not content or formatting restrictions). If jotti or VT has a similar list, I've missed it. VT really needs a list as some of their vendors/engines are obscure to this English speaker who only dabbles in security occasionally.
Systemus is a handy dandy system admin tool that BGM created for N.A.N.Y. 2020. Systemus had problems with false positives from antivirus products. Just out of curiosity, I ran Systemus through three meta-scanners, Jotti, VirusTotal, and MetaDefender, two days ago just to see what had changed. The results did improve (fewer presumably FPs), but are far from perfect.
At the end of 2020, I submitted Systemus to a half-dozen-ish vendors, plus a few more later on. Over the intervening couple of weeks, VirusTotal varied from 23 FPs initially, to 15 minimum, to 17. BGM reported 23/69 for VT in his response. I'm not sure why we had different totals, but I can imagine several possibilities (e.g., maybe BGM ran his PE Studio analysis a couple of weeks before his reply in the thread; or maybe he used .exe and I noted .zip, primarily); etc.). FPs also decreased on Jotti (no details) and MetaDefender (8 to 5) during those few weeks. Some other person also may have submitted FP requests during that few weeks or the intervening months. (See end of https://www.donationcoder.com/forum/index.php?topic=48696.msg442731#msg442731 for my comment and BGM's response.)
As several developers noted in that Systemus thread (and many other threads), this can be frustrating: each antivirus false positive is going to discourage users from experimenting with the software, but getting each AV vendor to evaluate each iteration (or even periodic stable releases) is like playing whack-a-mole at the bottom of the deep end of the pool using your nondominant foot with one eye closed and with some moles stuck in the up position. (I'm look at you, Webroot, who didn't actually evaluate/act on Systemus even after I submitted it twice; and you, McAfee in several guises, who has lots of fiddly restrictions; and a few others who make submitting arduous -- only webform with odd fields, only via forum, only via installed AV(!), only by author (not a user), etc. -- or even impossible as far as I can tell). A couple of reputation-based products aren't (or weren't a couple of years ago) willing to whitelist little known software even after their lab had reversed a FP in the main AV. Some authors recommend other authors just ignore FPs as too much trouble.
For comparison, these are April 2022 results for Systemus using the same files from my download folder. The dates here identify the last update to the engine/signatures.
Jotti
.zip (1/14):
Fortinet Apr 8, 2022 W32/PossibleThreat
VirusTotal
.zip (6/59 -- 59 excludes incompatible or nonreporting AVs):
AhnLab-V3 Malware/Win32.Generic.C3986407
Fortinet W32/PossibleThreat
MaxSecure Trojan.Malware.300983.susgen
McAfee Artemis!A5AC6681733F
McAfee-GW-Edition Artemis!Trojan
Panda Trj/CI.A
.exe (run today -- 8/68 -- excludes 4 incompatible and 1 nonreporting engine)
Same FPs as above except:
(different) McAfee-GW-Edition BehavesLike.Win32.Dropper.dh
(added) Palo Alto Networks Generic.ml
(added) Webroot W32.Malware.Gen
MetaDefender
.zip (0/35 but infected components flagged -- 35 includes 2 incompatible file type and 1 no result)
.exe (2/35 -- includes 1 no result)
Malware/Win32.Generic AhnLab Apr 9, 2022
Malware Webroot SMD Apr 8, 202
.dll (1/35 -- includes 1 no result)
Malware Webroot SMD Apr 8, 2022
I have no idea how much of the difference from two years ago may be due to FPs being fixed (either proactively or because someone submitted a request), or due to improved engines, or due to pruning because the miniscule installed base is no longer considered relevant (perhaps partially due to user whitelisting).
Note that due to different settings or other differences, "AhnLab" and "McAfee" reported "No Threat Detected" on MetaDefender while alerting on VT. This is not the first time I've noted a few differences between results for the same engine between these (and other) meta-scanning platforms. VT has a statement somewhere that mentions various reasons VT results may vary from the same vendor's installed product and/or web scanner. Some of these reasons (plus others) also apply between meta-AV platforms.
My personal impression is that, for the same vendor, VT tends to have more hits (almost always FPs for me) than MetaDefender and Jotti, but I have seen the reverse.
A couple of tools I used to automatically submit likely FPs to selected multiple sites via email were discontinued many years ago. I saw a few attempts to create something similar, but those weren't maintained (but I haven't looked recently). TechSupportAlert had a great list but it is not as useful anymore -- especially for FPs. MetaDefender's knowledge base has an updated list for its vendors (usually email, sometimes web or other) but doesn't explain the restrictions (i.e., address but not content or formatting restrictions). If jotti or VT has a similar list, I've missed it. VT really needs a list as some of their vendors/engines are obscure to this English speaker who only dabbles in security occasionally.