DonationCoder.com Software > Web Log Filter

Web Log Filter

(1/3) > >>

mouser:
My Web Log Filter Tool is now available for download:


* https://www.donationcoder.com/Software/Mouser/WebLogFilter/index.html
This is a super fast web log filter, which works off very simple scripts in order to take huge log files and make smaller ones with only the data you care about in them.

This is a tool designed for when your log files are so big that they cause your web log analysis tools to choke.  For example the DC logs are now over 4gb uncompressed, which is too big for almost every log file analyzer i've tried.  With Web Log Filter you can design scripts to extract just the log entries you care about and then load them into your favorite log analysis tool.

Features:

* Designed for speed, can filter 1gb/min
* Supports lots of conditional operations including regular expressions (or simpler operations for speed)
* Supports modification during filtering (ie you can anonymize your logs, simplify urls, strip unwanted info)
* Can redirect info into multiple output files (ie one script to create 10 output logs, 1 for each program you care about tracking)
* Can parse multiple input files as if they were one big file
* Can save and load projects for reuse
Imperfections:

* No documentation yet
* Very poor error reporting
* Doesnt yet auto-decompress zip files, you must do that first
Examples:

* You want to examine patterns of people downloading programs on your site, so you make a filter to extract only hits concerning .exe and .zip files.
* You want to examine referrer patterns, so you create a filter script to discard all hits with blank referrers or referrers from within your own site.
* You want to anonymize your logs so you can share them, so you create a script to hash randomize all ips and referrers.
* You want to look at 404 errors and other nonstandard return codes, so you write a script to split a log files into different output files based on the return code.
* You want to make a script to extract log files for specific pages and files so that you can give different users log files corresponding to their files/pages.

mouser:
ps.
here is a similar open source tool that looks quite nice and simpler to use if it will do what you need, though it doesn't have all the features of Web Log Filter:

http://www.logfilter.org/

mouser:
https://www.donationcoder.com/Software/Mouser/WebLogFilter/index.html

Date filtering is not implemented yet; should be added this week.

mouser:
Version 1.09.01 is now up, with full date filtering.

mouser:
I just want to say, even though i don't think anyone else on the planet uses this, that after employing it to analyze data in a 10gb log file, that it is a great tool, and i'd like to thank myself for coding it.  :up:

Navigation

[0] Message Index

[#] Next page