topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Friday March 29, 2024, 8:56 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: Web Log Filter  (Read 29356 times)

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Web Log Filter
« on: October 21, 2006, 01:51 PM »
My Web Log Filter Tool is now available for download:


This is a super fast web log filter, which works off very simple scripts in order to take huge log files and make smaller ones with only the data you care about in them.

This is a tool designed for when your log files are so big that they cause your web log analysis tools to choke.  For example the DC logs are now over 4gb uncompressed, which is too big for almost every log file analyzer i've tried.  With Web Log Filter you can design scripts to extract just the log entries you care about and then load them into your favorite log analysis tool.

Features:
  • Designed for speed, can filter 1gb/min
  • Supports lots of conditional operations including regular expressions (or simpler operations for speed)
  • Supports modification during filtering (ie you can anonymize your logs, simplify urls, strip unwanted info)
  • Can redirect info into multiple output files (ie one script to create 10 output logs, 1 for each program you care about tracking)
  • Can parse multiple input files as if they were one big file
  • Can save and load projects for reuse

Imperfections:
  • No documentation yet
  • Very poor error reporting
  • Doesnt yet auto-decompress zip files, you must do that first

Examples:
  • You want to examine patterns of people downloading programs on your site, so you make a filter to extract only hits concerning .exe and .zip files.
  • You want to examine referrer patterns, so you create a filter script to discard all hits with blank referrers or referrers from within your own site.
  • You want to anonymize your logs so you can share them, so you create a script to hash randomize all ips and referrers.
  • You want to look at 404 errors and other nonstandard return codes, so you write a script to split a log files into different output files based on the return code.
  • You want to make a script to extract log files for specific pages and files so that you can give different users log files corresponding to their files/pages.

Screenshot - 10_21_2006 , 1_51_06 PM.png
« Last Edit: October 24, 2006, 05:29 PM by mouser »

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Web Log Filter
« Reply #1 on: October 21, 2006, 01:57 PM »
ps.
here is a similar open source tool that looks quite nice and simpler to use if it will do what you need, though it doesn't have all the features of Web Log Filter:

http://www.logfilter.org/


mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Web Log Filter
« Reply #2 on: October 23, 2006, 02:21 PM »
https://www.donation...LogFilter/index.html

Date filtering is not implemented yet; should be added this week.
« Last Edit: October 23, 2006, 02:25 PM by mouser »

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Web Log Filter
« Reply #3 on: October 24, 2006, 04:13 AM »
Version 1.09.01 is now up, with full date filtering.

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Web Log Filter
« Reply #4 on: February 21, 2009, 04:58 AM »
I just want to say, even though i don't think anyone else on the planet uses this, that after employing it to analyze data in a 10gb log file, that it is a great tool, and i'd like to thank myself for coding it.  :up:

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Web Log Filter
« Reply #5 on: February 21, 2009, 05:03 AM »
maybe i should update it a bit..

TucknDar

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 1,133
    • View Profile
    • Donate to Member
Re: Web Log Filter
« Reply #6 on: February 21, 2009, 07:27 AM »
I just want to say, even though i don't think anyone else on the planet uses this, that after employing it to analyze data in a 10gb log file, that it is a great tool, and i'd like to thank myself for coding it.  :up:
You should donate something to the developer then ;)

cranioscopical

  • Friend of the Site
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,776
    • View Profile
    • Donate to Member
Re: Web Log Filter
« Reply #7 on: February 21, 2009, 09:17 AM »
I just want to say, even though i don't think anyone else on the planet uses this, that after employing it to analyze data in a 10gb log file, that it is a great tool, and i'd like to thank myself for coding it.  :up:
You should donate something to the developer then ;)
And give yourself one of those mugs for carrying the thread single-handedly since 2006.

Dragon

  • Participant
  • Joined in 2014
  • *
  • Posts: 2
    • View Profile
    • Donate to Member
Re: Web Log Filter
« Reply #8 on: January 11, 2014, 03:42 PM »
I just want to say, even though i don't think anyone else on the planet uses this, that after employing it to analyze data in a 10gb log file, that it is a great tool, and i'd like to thank myself for coding it.  :up:

I'd like to thank you too.  It's very useful to extract out say all the PHP pages visited by a particular IP address so you can follow what they did to cause an error message or bug you're trying to fix.  Here's the little script I used:

//---------------------------------------------------------------------------
@RULE:
@CONDITION: notequal
@FIELD: host
@VAL: 12.34.56.78
@ACTION: discard
//---------------------------------------------------------------------------
//---------------------------------------------------------------------------
@RULE:
@CONDITION: contains
@FIELD: url
@VAL: .php
@ACTION: discard
@OPTIONS: not
//---------------------------------------------------------------------------
//---------------------------------------------------------------------------
@RULE:
@CONDITION: always
@ACTION: write
@ACTIONVAL: filtered-log.txt
//---------------------------------------------------------------------------

I have a feeling WebLogFilter doesn't get much notice only because there are so many other tools out there to do similar things, and WebLogFilter is Windows only.  I've noticed that it's almost always the first piece of software that comes along to perform a task that gets all the notice and word of mouth recommendations in the long run, even if something better comes along later.  But WebLogFilter is the only tool I've used that runs little scripts (which I find easier to deal with than moving the cursor around a long command line) that are easy to create with a wizard and can be saved.   http://www.weblogexpert.com/purchase.htm is a commercial tool that does something similar for $99 and up, although they also have a free "Lite" version that might be comparable.  I didn't try it as I found WebLogFilter first.  Even so, you obviously put a good amount of time into WebLogFilter and I find it useful, so thanks!

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Web Log Filter
« Reply #9 on: January 11, 2014, 03:48 PM »
Thank you for taking the time to post, Dragon!
Since I posted this in 2006, you may be the first person to actually use it for real (other than me).
8 years to get my first user :)
Glad that you find it useful.  :up:

Dragon

  • Participant
  • Joined in 2014
  • *
  • Posts: 2
    • View Profile
    • Donate to Member
Re: Web Log Filter
« Reply #10 on: January 11, 2014, 04:28 PM »
Oh, I bet others have found it useful and just didn't bother to register and post.  You should use WebLogFilter to pull out all the "downloaded WebLogFilter" entries from your last 8 years of logs!    :D

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 40,896
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: Web Log Filter
« Reply #11 on: January 11, 2014, 04:36 PM »
Good idea  :Thmbsup: