Need proxy software to capture outbound http requests

[wysocki]

I've searched a bit and can't find anything like this. I'm looking for a way to capture all the internet http url's that are requested from within my network. I'd like to capture and log the date,time,IP,and URL. This would seem to be a pretty easy thing to code (but it's beyond my skills) and I can't find anything that's simple to use that can do this. Any ideas?
Thanks in advance!

[Renegade]

Are you looking for a packet sniffer like Wireshark? It records everything.

[40hz]

+1 w/Renegade.

There are some easier to use products than Wireshark. But many are just Wireshark tricked out with a fancy UI and a 'pretty' report module. They all basically do the same thing. And most aren't free. So if it's not for business use, it can be hard to justify spending the money on them.

FWIW Wireshark's not that hard to set up or understand if you're willing to do a little reading and spend some time with it.

Just out of curiosity, what nefarious web use are you looking to check up on.  

[wysocki]

Thanks guys! I checked out Wireshark - holy mackeral! A bit more than I can handle unfortunately. Basically where this started was I'm looking for a way to see the browsing history in Firefox 3.x from a few computers on the network. They're being manned by volunteers working on a drug abuse hotline and so, even though it's technically a business, there's no money involved, or even available! Hence I'm looking for something cheap(free). Also, I don't want to actually put filtering in place because these guys occasionally really have to go to some "questionable" websites to do their job. I'd just like to be able to get an idea of what kind of resources are being used - or maybe misused - via the Internet so that I can weigh it against the quality of work achieved. I'd be happy seeing just the history logs on the computers, but some of the guys decide to change the history settings and wipe it out. Paranoia runs deep.

I've searched in vain for a way to lock the history down in Firefox, so I thought I'd somehow point the browser to a "proxy" that would log the url then pass it thru to the router undetected. I also thought about some way to monitor the database that stores the history and copy all new entries off to another table for archiving, but I'm stuck there too.

Any other ideas on how to do this out there?

[skwire]

First of all, this isn't quite as easy as it would seem.  Some questions:

1) Are you on a switched network?  Or are you using an old style hub to interconnect your computers?  If you're on a switched network, you're not going to be able to see other computer's network traffic from your PC...leading to the next question...

2) What kind of device are you using as a network gateway?  That's going to be your best bet at sniffing your network's internet traffic (since all of it has to go through this device).

[40hz]

The other thing you could do is set up a caching web proxy server and direct all network DNS requests through that. An examination of the logs would tell you what people were looking at.  But that would likely require it's own machine for best results.

As skwire mentioned it may be possible to do it on your gateway device too. Most have the ability to maintain a log of whatever network activity they handle. Usually you just tell it what you want to monitor along with the IPAddress of the admin machine on the network you want to send the data to.

But maybe it's easiest and best to just tell them all Internet use is monitored and not worry about productivity unless it obviously needs improving.

Also important - get legal advice before you start monitoring employees if they didn't have to sign a paper saying they were aware monitoring could be taking place when they started working for you. It's a legal grey area in some places. And in some states there's rules for what can be monitored, what notices need to be given, and how you're allowed to go about it. The rules vary from place to place, so do get a professional opinion before you start watching people.

Just my 2¢ 

[wysocki]

skwire:
I have a switched network, but I don't really need to see the traffic real-time - logging it would be fine on whatever pc. I have a low end Dlink router as the gateway currently and it doesn't do any logging of traffic. Guess I don't really understand what a proxy is well enough, but it's sounding like this is a dead end for me.

40hz:
Yeah, the standalone server would require a bit more tech and cash too. I'll check around to see if I can find some used router that could log the traffic though. These volunteers must know that they're being watched somewhat because we used to always go around to the pc's at night and re-enable their history tracking!

[skwire]

The other thing you could do is set up a caching web proxy server and direct all network DNS requests through that. An examination of the logs would tell you what people were looking at.  But that would likely require it's own machine for best results.

-40hz (May 18, 2011, 07:15 PM)

 I have a low end Dlink router as the gateway currently and it doesn't do any logging of traffic. Guess I don't really understand what a proxy is well enough, but it's sounding like this is a dead end for me.

-wysocki (May 18, 2011, 07:23 PM)

I've run a Linux-based router as my gateway device for over a decade now.  Something like this would replace (or supplement if you're using wireless) your Dlink device and they don't take much of a computer to run.  Almost any old box you have laying around would do; mine is a PII 350 from ages ago.  =]  I currently use Smoothwall Linux and it has a built-in proxy you could use.  Heck, it can even run that proxy in transparent mode so you wouldn't have to do any configuration on your employee computers.  In other words, they wouldn't even know they're going through a proxy. 

[Stoic Joker]

Sounds like it would be simpler to just lock the machines down via group policy. The IE clear history tab can be removed from internet settings. There is generally no good reason for letting users have the permissions necessary to install alternative browsers.

-and/or-

Go low tech ... Create, circulate and post a network AUP - costs what $6 for package of paper? First few times people get pulled aside to have to explain their actions...should deter the rest quite nicely.

[40hz]

+1 w/skwire on Smoothwall. It's a great solution. I have several clients using it. It's easy to set up and use. And it doesn't require too much hardware to run effectively. Highly recommended. And the price is right. 

[Ehtyar]

I used m0n0wall for quite a while at home, I found it had a nicer interface than Smoothwall and has lower requirements.

I'm currently using pfSense at work, and it works wonderfully. Tons of features and installable packages, based on FreeBSD.

Ehtyar.

[JavaJones]

pfsense on one of these babies is pretty sweet. We used them at 4 different locations at my last job. Only problem we had was with some VPN flakiness that came down to a particular security setting that we had to disable (just a minor security feature - the underlying encryption remained).

- Oshyan

[40hz]

pfsense on one of these babies is pretty sweet. We used them at 4 different locations at my last job. Only problem we had was with some VPN flakiness that came down to a particular security setting that we had to disable (just a minor security feature - the underlying encryption remained).

- Oshyan

-JavaJones (May 18, 2011, 09:35 PM)

pfsense is also an excellent option, and probably a better choice than Smoothwall at this point if you're willing to put a little effort into "grokking" what is a much more powerful security product. Time well spent IMO. We plan on moving our clients who have not subscribed to their ISP's managed router/firewall service over to it in the near future whenever possible.

There's an excellent 4-part tutorial/how-to for pfsense over at SmallNetBuilder's website. The fourth part gets into logging and may have eactly what you're looking for. You can find the first part of the series here.

Note: SmallNetBuilder    is an excellent resource, so spend a little time browsing it when you get a chance. It's perfect for someone in your line of work. Highly recommended.  


----

Another solution you might consider if network security is a concern  is something called Untangle. I've waxed poetic about this product before so I won't repeat my earlier comments. There is a very capable free version available, and non-profits can subscribe to the full version at the same discount rate offered educational institutions.