Author Topic: Free Windows firewalls found better than commercial ones (Read 40,772 times)

Rohit · « **on:** December 07, 2006, 12:25 PM »

I know many regulars here (including me) are looking for a good firewall solution, so this article might help

Quote

Free firewalls are better than their paid-for cousins. That is the surprising conclusion of a test of desktop firewalls by security researchers.

Researchers at David Matousec's matousec.com carried out tests on 21 leading products using 26 assessment programs known as "leak" testers. These simulated a total of 77 test attacks on firewalls, configured using both out-of-the-box and optimal security settings. Each firewall was then awarded points based on its ability to pass each leak test in both modes.

The only two products to achieve a rating of "excellent" turned out to be free-to-use software, the Comodo Personal Firewall v2.3, and the Jetico Personal Firewall v2.0 beta.

(snip) At the very bottom of the list in 21st place scoring a resounding zero, came Microsoft's own firewall ...

(snip) The researchers also hit the products with a "fake protection revealer" (FPR) designed to catch out software that had been optimised to pass some security tests without necessarily offering real-world protection. Only one product fell seriously foul of this test, Outpost Firewall Pro 4.0 ...

nudone · « **Reply #1 on:** December 07, 2006, 01:49 PM »

very interesting.

i was holding back about commenting futher on the comodo firewall (i'd recently installed it after reading a discussion elsewhere on this forum) but this appears to be good place to say my piece now.

after a few hours of my machine running, all of my programs that were connected to the net would crash. so that's AOL active virus shield, spyware terminator, comodo firewall, utorrent and emule. further, the fonts would vanish from windows/buttons and a very weird black shadow appears around the text of my desktop icons.

obviously, i thought my machine must be infected with something. all scans have so far not revealed anything to be worried about. the troubling thing is that this pattern of crashing behaviour is consistent when using the comodo firewall. since uninstalling it i've not had the problem return. i have been unable to reinstall zone alarm (which i was previously using) so i reverted back to using the windows inbuilt firewall - which i think i'll have to avoid after seeing Rohit's above post.

of course, maybe it's simply the combination of programs i have running that is causing the problem - what else can i assume. but for the moment, comodo firewall sounded so good to me but it simply became a disaster.

Carol Haynes · « **Reply #2 on:** December 07, 2006, 01:56 PM »

Seems a bit unfair to lump Windows Firewall in the list since it doesn't even pretend to block outgoing traffic !

Rohit · « **Reply #3 on:** December 07, 2006, 04:13 PM »

Quote from: nudone on December 07, 2006, 01:49 PM

... the troubling thing is that this pattern of crashing behaviour is consistent when using the comodo firewall. since uninstalling it i've not had the problem return. ...

Thanks for this feedback, nudone.

After reading the above article I promptly uninstalled my old Sygate Personal Firewall and installed Comodo. If I start experiencing crashes, I would know what to blame them on.

Quote from: Carol Haynes on December 07, 2006, 01:56 PM

Seems a bit unfair to lump Windows Firewall in the list since it doesn't even pretend to block outgoing traffic !

That's interesting, Carol. I never used the default Windows firewall, so I didn't know it blocked only incoming traffic. I guess Microsoft implemented this feature so as not to confuse casual (non computer-savvy) users.

mitzevo · « **Reply #4 on:** December 07, 2006, 04:24 PM »

Hmm, so what does that mean then..? Outpost is a lier?! I might go and take a look at Comodo..

hollowlife1987 · « **Reply #5 on:** December 07, 2006, 04:31 PM »

Quote from: Carol Haynes on December 07, 2006, 01:56 PM

Seems a bit unfair to lump Windows Firewall in the list since it doesn't even pretend to block outgoing traffic !

The new windows firewall in Vista does have the ability to block outgoing traffic.

mitzevo · « **Reply #6 on:** December 07, 2006, 04:40 PM »

Quote from: hollowlife1987 on December 07, 2006, 04:31 PM

Quote from: Carol Haynes on December 07, 2006, 01:56 PM
Seems a bit unfair to lump Windows Firewall in the list since it doesn't even pretend to block outgoing traffic !

The new windows firewall in Vista does have the ability to block outgoing traffic.

Windows is getting ahead of every one!

Rohit · « **Reply #7 on:** December 07, 2006, 05:12 PM »

Quote from: mitzevo on December 07, 2006, 04:24 PM

Hmm, so what does that mean then..? Outpost is a lier?!

It would seem so.

From the page about the leak test:

Quote

Another important result of our tests is firewall scoring against FPR. FPR stands for Fake Protection Revealer. This leak-test was implemented to reveal cheating on leak-tests. Outpost Firewall PRO 4.0 (971.584.079) was convicted of such cheating. It passes all leak-tests except FPR because of the implementation of user mode hooks (ring3) for security purposes. Our article Design of ideal personal firewall clearly says that ring3 hooks can not be used for security critical features. FPR does nothing but unhooks ring3 hooks which is always possible and thus bypasses such protection. This means that Outpost Firewall PRO cheats to be very strong against leak-tests but in fact it is very weak against real malware.

(The emphasised text is from the original article, not from me.)

Carol Haynes · « **Reply #8 on:** December 07, 2006, 05:52 PM »

Quote from: hollowlife1987 on December 07, 2006, 04:31 PM

Quote from: Carol Haynes on December 07, 2006, 01:56 PM
Seems a bit unfair to lump Windows Firewall in the list since it doesn't even pretend to block outgoing traffic !

The new windows firewall in Vista does have the ability to block outgoing traffic.

True but I presume they were testing Windows XP firewall

Carol Haynes · « **Reply #9 on:** December 07, 2006, 05:54 PM »

Quote from: Rohit on December 07, 2006, 05:12 PM

Quote from: mitzevo on December 07, 2006, 04:24 PM
Hmm, so what does that mean then..? Outpost is a lier?!

It would seem so.

From the page about the leak test:

Quote
Another important result of our tests is firewall scoring against FPR. FPR stands for Fake Protection Revealer. This leak-test was implemented to reveal cheating on leak-tests. Outpost Firewall PRO 4.0 (971.584.079) was convicted of such cheating. It passes all leak-tests except FPR because of the implementation of user mode hooks (ring3) for security purposes. Our article Design of ideal personal firewall clearly says that ring3 hooks can not be used for security critical features. FPR does nothing but unhooks ring3 hooks which is always possible and thus bypasses such protection. This means that Outpost Firewall PRO cheats to be very strong against leak-tests but in fact it is very weak against real malware.

(The emphasised text is from the original article, not from me.)

Not the only article accusing Outpost of cheating.

dk70 · « **Reply #10 on:** December 07, 2006, 06:41 PM »

Ive become tired of most freeware firewalls but did manage to install and set up Comodo Firewall. Was 100% happy but have no need for it. Im sure almost anyone can do it - with some interest for net setup of course. Take a look at their forum, tons of tips and tricks. Good documentation and the "right" attitude from developers is worth a lot since firewalls can be tricky to set up and made stable. Some fall in love, some get BSOD - typical for many freeware firewalls. Comodo so popular most incompatibilities should be mentioned somewhere.

Darwin · « **Reply #11 on:** December 07, 2006, 08:00 PM »

Like nudone, I've been holding off commenting on Comodo. I uninstalled the ZA Pro trial from my "backup" notebook (older machine that I intend to use in the event of hardware failure on my main notebook) and installed Comodo on it, which has been brilliant. Case in point: connecting the backup machine to our home network (with my wife's XP Home box and my XP Pro main box) worked well with no s/w firewall installed on the Win2k machine but proved impossible with ZA Free installed - I could find not way to configure ZA Free to connect. I have ZA Pro on my XP Pro machine so enabled the trial of ZA Pro on the Win2k machine and it connected immediately. I was trying to decide if connecting the backup machine to the home network was critical enough to justify purchasing another license for ZA Pro but decided to look for alternatives, at which point I discovered Comodo. After uninstalling ZA, I loaded Comodo and rebooted. On startup, the notebook connected to the home network without difficulty (I had to configure Comodo to do it, but there was nothing strenous involved - no heavy lifting!). Anyway, I'm going to leave my backup notebook running and connected to the home network and see if I can replicate nudone's problems. If I can't, I'll be seriously considering switching to Comodo when my ZA Pro subscription runs out.

lanux128 · « **Reply #12 on:** December 08, 2006, 02:18 AM »

this is interesting.. i had just opted for Windows Firewall about 2 mths ago & here it is ranked rock-bottom..

i'm not keen on looking for another firewall but all these talks of Comodo has been tempting. i hope it's not a case of "new broom sweeps well"..

f0dder · « **Reply #13 on:** December 08, 2006, 10:28 AM »

Quote

It's Free. Forever. No Catch. No Kidding

So, I wonder when they're removing the free version, or stuffing ads or spyware in it.

Carol Haynes · « **Reply #14 on:** December 08, 2006, 11:40 AM »

Quote from: lanux128 on December 08, 2006, 02:18 AM

this is interesting.. i had just opted for Windows Firewall about 2 mths ago & here it is ranked rock-bottom..

Not surprising it is ranked bottom though because it specifically doesn't do what they were testing.

Actually I think this whole article is a bit misleading because it is only testing for effectiveness against leaktests. It is in no way assessing the applications as firewalls. All of them fail at leat one leaktest so if that bothers you the only effective solution is to pull the network or telephone cable.

If you honestly think failing a leaktest is so bad then you need a firewall that can not only block all the known threats but also any potential new threats and no such beast exists.

mitzevo · « **Reply #15 on:** December 08, 2006, 02:18 PM »

Quote from: f0dder on December 08, 2006, 10:28 AM

Quote
It's Free. Forever. No Catch. No Kidding
So, I wonder when they're removing the free version, or stuffing ads or spyware in it.

There are many conspiracy theories on comodo's products.. but just read around their forums.. pretty much boills down to letting every one know who they are, so people will accept them for the paying products.. or some thing like that.. branding an excellent free firewall with their name seems logical for a better future in business I would think.

f0dder · « **Reply #16 on:** December 08, 2006, 02:37 PM »

Quote from: "Carol Haynes"

If you honestly think failing a leaktest is so bad then you need a firewall that can not only block all the known threats but also any potential new threats and no such beast exists.

Well, you can get 100% effective port-based filtering from hardware firewalls... but of course that doesn't detect "unauthorized applications" smuggling data on an allowed port. But then there's of course fancy-pants stateful packet inspection...

Quote from: "mitzevo"

There are many conspiracy theories on comodo's products.. but just read around their forums.. pretty much boills down to letting every one know who they are, so people will accept them for the paying products.. or some thing like that.. branding an excellent free firewall with their name seems logical for a better future in business I would think.

I've just seen too many free products that went shitware after they had a big enough user base

dk70 · « **Reply #17 on:** December 08, 2006, 06:33 PM »

Or die out like Fliseclab which was Comodo of last year http://www.filseclab.com/eng/products/firewall.htm

AdIyhc · « **Reply #18 on:** December 08, 2006, 07:33 PM »

Or get a HIPS(System Safety Monitor, etc) to go with Windows Firewall. There are free editions.
3rd party firewalls are better but not neccesary.

Carol Haynes · « **Reply #19 on:** December 08, 2006, 08:14 PM »

I have not come across "System Safety Monitor" - does anyone know of any comparisons of these types of tools - I have ProcessGuard, ApplicationGuard and RegDefend but I haven't used them in a while - maybe it is time to try them again but it would be good to know about useful alternatives too.

AdIyhc · « **Reply #20 on:** December 08, 2006, 08:29 PM »

Quote from: Carol Haynes on December 08, 2006, 08:14 PM

I have not come across "System Safety Monitor" - does anyone know of any comparisons of these types of tools - I have ProcessGuard, ApplicationGuard and RegDefend but I haven't used them in a while - maybe it is time to try them again but it would be good to know about useful alternatives too.

There's a 50% competitive upgrade for System Safety Monitor. More info here: https://www.syssafety.com/default.html
Active development, etc.

It's like PG+RegDefend together.

Carol Haynes · « **Reply #21 on:** December 08, 2006, 09:50 PM »

Yes I spotted that - but before I spend more money that I can't afford it would be good to see comparisons of such utilities if there are any ?

Darwin · « **Reply #22 on:** December 08, 2006, 09:56 PM »

Unfortunately, development (at least that reaches the public!) is moribund on both RegDefend and AppDefend (which comprise the GhostSecurity Suite). AppDefend has been in beta for over a year and is buggy. The developer appears to have dropped off the face of the earth; his complete absence from the forums associated with his applications is a frequent topic of discussion there, as is the lack of updates to GhostSecurity.

This is a pity because I really like both apps...

AdIyhc · « **Reply #23 on:** December 09, 2006, 01:51 AM »

Quote from: Carol Haynes on December 08, 2006, 09:50 PM

Yes I spotted that - but before I spend more money that I can't afford it would be good to see comparisons of such utilities if there are any ?

PG vs SSM : http://www.wilderssecurity.com/showthread.php?t=156448&highlight=system+safety+monitor
See the post by Paranoid2000

SSM or other HIPS : http://www.wilderssecurity.com/showthread.php?t=151692&highlight=system+safety+monitor
Some useful links in the first page

The free version of SSM is actually an old version.

Hope this helps.

Curt · « **Reply #24 on:** December 09, 2006, 11:17 AM »

Quote from: Rohit on December 07, 2006, 05:12 PM

Quote from: mitzevo on December 07, 2006, 04:24 PM
Hmm, so what does that mean then..? Outpost is a lier?!

It would seem so.

From the page about the leak test:

Quote
Another important result of our tests is firewall scoring against FPR. FPR stands for Fake Protection Revealer. This leak-test was implemented to reveal cheating on leak-tests. Outpost Firewall PRO 4.0 (971.584.079) was convicted of such cheating. It passes all leak-tests except FPR because of the implementation of user mode hooks (ring3) for security purposes. Our article Design of ideal personal firewall clearly says that ring3 hooks can not be used for security critical features. FPR does nothing but unhooks ring3 hooks which is always possible and thus bypasses such protection. This means that Outpost Firewall PRO cheats to be very strong against leak-tests but in fact it is very weak against real malware.

(The emphasised text is from the original article, not from me.)

As implied on others threads, I am an Outpost user and fan - so maybe I read this test with different eyes than you. Of course also Outpost should be made to stand real life attacks better, they all need that. But if you read all of the test, you will notice that ZoneAlarm and Outpost are both a lot better than the rest - and that these two are almost equal. The accuse on cheating is of course a problem, but do not forget that at least they are trying to stop leaking - many firewalls are not even trying! I am confident that the next major update from Agnitum will improve this even more.

One can quote: "Outpost Firewall PRO cheats to be very strong against leak-tests but in fact it is very weak against real malware" - but can one quote anyone saying Matousec's methods are accepted by anyone other that himself? Are his methods a true picture of "real malware"? Guess what: Agnitum thinks not!

And if one thinks not, on this tese, then the conclusion must be that Outpost did very well in the test. In fact, it did fine!

http://www.matousec.com/projects/windows-personal-firewall-analysis/results.php
(Smaller values of overall ratings mean better products.)