This is a super fast web log filter, which works off very simple scripts in order to take huge log files and make smaller ones with only the data you care about in them.
This is a tool designed for when your log files are so big that they cause your web log analysis tools to choke. For example the DC logs are now over 4gb uncompressed, which is too big for almost every log file analyzer i've tried. With Web Log Filter you can design scripts to extract just the log entries you care about and then load them into your favorite log analysis tool.
- Designed for speed, can filter 1gb/min
- Supports lots of conditional operations including regular expressions (or simpler operations for speed)
- Supports modification during filtering (ie you can anonymize your logs, simplify urls, strip unwanted info)
- Can redirect info into multiple output files (ie one script to create 10 output logs, 1 for each program you care about tracking)
- Can parse multiple input files as if they were one big file
- Can save and load projects for reuse
- No documentation yet
- Very poor error reporting
- Doesnt yet auto-decompress zip files, you must do that first
- You want to examine patterns of people downloading programs on your site, so you make a filter to extract only hits concerning .exe and .zip files.
- You want to examine referrer patterns, so you create a filter script to discard all hits with blank referrers or referrers from within your own site.
- You want to anonymize your logs so you can share them, so you create a script to hash randomize all ips and referrers.
- You want to look at 404 errors and other nonstandard return codes, so you write a script to split a log files into different output files based on the return code.
- You want to make a script to extract log files for specific pages and files so that you can give different users log files corresponding to their files/pages.