Web Log Filter is a helper utility for people who have huge (Apache) web log files, over a gigabyte in size, and want to be able to analyze these logs in standard web log analysis tools.

If you've ever tried loading a 5gb log file into a standard desktop web log analysis program you know it can be an extremely painful process. Many web log analysis programs will simply crash if you try to load such a large web log file. Other will simply churn away for 60 minutes and use up your entire system memory and hard drive space, and then run so slowly as to make you want to slit your wrists.

Web Log Filter runs simple scripts that will rip through a log file at blazing speed and extract just the lines you care about (discarding the rest) and create one or more new log files, which are smaller and more manageable.

Some examples of uses for Web log Filter:

• You want to look at errors on your website; so you write a script to take a massive log file and create separate output log files for each different error code.
• You want to prepare a web log file for a user on your website, so you write a script to extract all references to his set of pages.
• You want to examine all outside referers to your website, so you write a script to discard all log lines with blank referrers or with referrers coming from within your own domain.
• You want to look at search engine patterns, so you write a script to extract only log lines coming from google.
• You want to combine a series of input log files while you filter them.
• You want to anonymize the host IP info in a log file (replace each unique IP with a random value).
• You want to correct changes in a log file.
• You want to split up a log file into separate files based on content (i.e. save each line to a file based on referrer domain).