Welcome Guest.   Make a donation to an author on the site December 22, 2014, 10:27:55 PM  *

Please login or register.
Or did you miss your validation email?


Login with username and password (forgot your password?)
Why not become a lifetime supporting member of the site with a one-time donation of any amount? Your donation entitles you to a ton of additional benefits, including access to exclusive discounts and downloads, the ability to enter monthly free software drawings, and a single non-expiring license key for all of our programs.


You must sign up here before you can post and access some areas of the site. Registration is totally free and confidential.
 
Learn about the DonationCoder.com microdonation system (DonationCredits).
   
   Forum Home   Thread Marks Chat! Downloads Search Login Register  
Pages: [1]   Go Down
  Reply  |  New Topic  |  Print  
Author Topic: Web Log Filter  (Read 10125 times)
mouser
First Author
Administrator
*****
Posts: 33,796



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« on: October 21, 2006, 01:51:34 PM »

My Web Log Filter Tool is now available for download:


This is a super fast web log filter, which works off very simple scripts in order to take huge log files and make smaller ones with only the data you care about in them.

This is a tool designed for when your log files are so big that they cause your web log analysis tools to choke.  For example the DC logs are now over 4gb uncompressed, which is too big for almost every log file analyzer i've tried.  With Web Log Filter you can design scripts to extract just the log entries you care about and then load them into your favorite log analysis tool.

Features:
  • Designed for speed, can filter 1gb/min
  • Supports lots of conditional operations including regular expressions (or simpler operations for speed)
  • Supports modification during filtering (ie you can anonymize your logs, simplify urls, strip unwanted info)
  • Can redirect info into multiple output files (ie one script to create 10 output logs, 1 for each program you care about tracking)
  • Can parse multiple input files as if they were one big file
  • Can save and load projects for reuse

Imperfections:
  • No documentation yet
  • Very poor error reporting
  • Doesnt yet auto-decompress zip files, you must do that first

Examples:
  • You want to examine patterns of people downloading programs on your site, so you make a filter to extract only hits concerning .exe and .zip files.
  • You want to examine referrer patterns, so you create a filter script to discard all hits with blank referrers or referrers from within your own site.
  • You want to anonymize your logs so you can share them, so you create a script to hash randomize all ips and referrers.
  • You want to look at 404 errors and other nonstandard return codes, so you write a script to split a log files into different output files based on the return code.
  • You want to make a script to extract log files for specific pages and files so that you can give different users log files corresponding to their files/pages.


« Last Edit: October 24, 2006, 05:29:12 PM by mouser » Logged
mouser
First Author
Administrator
*****
Posts: 33,796



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #1 on: October 21, 2006, 01:57:04 PM »

ps.
here is a similar open source tool that looks quite nice and simpler to use if it will do what you need, though it doesn't have all the features of Web Log Filter:

http://www.logfilter.org/

Logged
mouser
First Author
Administrator
*****
Posts: 33,796



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #2 on: October 23, 2006, 02:21:14 PM »

http://www.donationcoder....r/WebLogFilter/index.html

Date filtering is not implemented yet; should be added this week.
« Last Edit: October 23, 2006, 02:25:09 PM by mouser » Logged
mouser
First Author
Administrator
*****
Posts: 33,796



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #3 on: October 24, 2006, 04:13:03 AM »

Version 1.09.01 is now up, with full date filtering.
Logged
mouser
First Author
Administrator
*****
Posts: 33,796



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #4 on: February 21, 2009, 04:58:41 AM »

I just want to say, even though i don't think anyone else on the planet uses this, that after employing it to analyze data in a 10gb log file, that it is a great tool, and i'd like to thank myself for coding it.  thumbs up
Logged
mouser
First Author
Administrator
*****
Posts: 33,796



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #5 on: February 21, 2009, 05:03:30 AM »

maybe i should update it a bit..
Logged
TucknDar
Charter Member
***
Posts: 1,094


Advanced coder of Nowt

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #6 on: February 21, 2009, 07:27:34 AM »

I just want to say, even though i don't think anyone else on the planet uses this, that after employing it to analyze data in a 10gb log file, that it is a great tool, and i'd like to thank myself for coding it.  thumbs up
You should donate something to the developer then Wink
Logged
cranioscopical
Friend of the Site
Supporting Member
**
Posts: 4,200



see users location on a map View Profile Read user's biography. Give some DonationCredits to this forum member
« Reply #7 on: February 21, 2009, 09:17:50 AM »

I just want to say, even though i don't think anyone else on the planet uses this, that after employing it to analyze data in a 10gb log file, that it is a great tool, and i'd like to thank myself for coding it.  thumbs up
You should donate something to the developer then Wink
And give yourself one of those mugs for carrying the thread single-handedly since 2006.
Logged

Chris
Dragon
Participant
*
Posts: 2


View Profile Give some DonationCredits to this forum member
« Reply #8 on: January 11, 2014, 03:42:16 PM »

I just want to say, even though i don't think anyone else on the planet uses this, that after employing it to analyze data in a 10gb log file, that it is a great tool, and i'd like to thank myself for coding it.  thumbs up

I'd like to thank you too.  It's very useful to extract out say all the PHP pages visited by a particular IP address so you can follow what they did to cause an error message or bug you're trying to fix.  Here's the little script I used:

//---------------------------------------------------------------------------
@RULE:
@CONDITION: notequal
@FIELD: host
@VAL: 12.34.56.78
@ACTION: discard
//---------------------------------------------------------------------------
//---------------------------------------------------------------------------
@RULE:
@CONDITION: contains
@FIELD: url
@VAL: .php
@ACTION: discard
@OPTIONS: not
//---------------------------------------------------------------------------
//---------------------------------------------------------------------------
@RULE:
@CONDITION: always
@ACTION: write
@ACTIONVAL: filtered-log.txt
//---------------------------------------------------------------------------

I have a feeling WebLogFilter doesn't get much notice only because there are so many other tools out there to do similar things, and WebLogFilter is Windows only.  I've noticed that it's almost always the first piece of software that comes along to perform a task that gets all the notice and word of mouth recommendations in the long run, even if something better comes along later.  But WebLogFilter is the only tool I've used that runs little scripts (which I find easier to deal with than moving the cursor around a long command line) that are easy to create with a wizard and can be saved.   http://www.weblogexpert.com/purchase.htm is a commercial tool that does something similar for $99 and up, although they also have a free "Lite" version that might be comparable.  I didn't try it as I found WebLogFilter first.  Even so, you obviously put a good amount of time into WebLogFilter and I find it useful, so thanks!
Logged
mouser
First Author
Administrator
*****
Posts: 33,796



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #9 on: January 11, 2014, 03:48:17 PM »

Thank you for taking the time to post, Dragon!
Since I posted this in 2006, you may be the first person to actually use it for real (other than me).
8 years to get my first user smiley
Glad that you find it useful.  thumbs up
Logged
Dragon
Participant
*
Posts: 2


View Profile Give some DonationCredits to this forum member
« Reply #10 on: January 11, 2014, 04:28:26 PM »

Oh, I bet others have found it useful and just didn't bother to register and post.  You should use WebLogFilter to pull out all the "downloaded WebLogFilter" entries from your last 8 years of logs!    cheesy
Logged
mouser
First Author
Administrator
*****
Posts: 33,796



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #11 on: January 11, 2014, 04:36:30 PM »

Good idea  Thmbsup
Logged
Pages: [1]   Go Up
  Reply  |  New Topic  |  Print  
 
Jump to:  
   Forum Home   Thread Marks Chat! Downloads Search Login Register  

DonationCoder.com | About Us
DonationCoder.com Forum | Powered by SMF
[ Page time: 0.041s | Server load: 0.13 ]