Any webserver-desktop-logfile-analysis-Software recommendations?

[crono]

Hello everyone,

Im searching a programm for analyse my Apache-Access Logfiles. On the server Im using http://www.awstats.net/ but I want something which is not web-based. Any recommendations?

[mouser]

ive used a ton and will try to post some thoughts later - am also interested in other people's views.

[Gothi[c]]

please do post some,.. the only one i tried was a php thing and it was crap

[Renegade]

Analog - Web Storming - ClickTrax - Urchin - roll your own...

The best solution is to actually roll your own. That way you get to include real business logic.

Web log analysis is at best sketchy and doesn't provide you with "true" numbers. They are only a rough guess. (Another reason to roll your own.)

The best solution is to run things on the client for detection and store it on the server in a database. You also get the added advantage of only storing what you need.

All large scale web analysis must be done this way because logs get too large to process. Even on a relatively small site (or medium depending on what you consider large) you can have log files that are GBs for a day. Very small sites have log files in the 10's of MB per day.

Most software can't keep up with that kind of volume. That's where a dataserver can solve the problem. They're optimized to deal with data processing whereas your average log analysis software just can't compare. (A lot of resources go into optimizing RDBMSs like MS SQL Server, DB2, Informix, Oracle, etc.)

But for small sites this can often be overkill. A quick over view with Analog is good for most people with hobby sites or small web sites.

I'm not a fan of AwStats. I've found it just miserable to work with.

Another thing you can do is to write a program to dump your logs into a DB then just use pure SQL to ask questions. That's a cheap and easy way, though it is a bit time consuming as you'll still end up waiting for queries. But the quality of answers you can get is VERY high.

[crono]

Thanks you very much Renegade. What you write sounds really experienced. I think Im going to roll my own - which shouldnt be to hard... Parsing into a DB and running some SQL should be straight forward

[mouser]

i guess my view would be the opposite to renegade's.  the idea of rolling my own when sooo much work has been done on this seems a bit crazy.

i also tend to like local apps, and there are some good ones for the pc (Mach 5 Analyzer is pretty nice and fast).

though its true that if you start getting 1gb log files, these programs start to choke and you need to find alternative strategies.

[Gothi[c]]

I didn't like analog too much.
I'd much rather see something in between rolling your own and not, that is:
  a commandline utility that runs in a terminal that will sort through the logs as they come in, with colored output and statistics, as opposed to web-based solutions. grep'ing through the logs and using tail -f works too, but i'd like to see it better formatted, with a pretty ncurses interface or something. just like htop looks nicer than top And it'd be handy to have counters of pageviews (even though they don't mean anything), failed requests, intrusions, etc,..
Maybe with pluggable progressbar-like-meters as in htop.

[housetier]

Today I found Logfilter: An Apache Log Filter.

Logfilter is a tool for performing ad-hoc filtering of apache website log files.

This seems to be tightly coupled to the Apache Webserver.

[mouser]

house i've been looking for something like that for a while, nice find.

[Terry]

AlterWind Log Analyzer - http://www.alterwind.com

[kartal]

I like Alterwind, it is neat. I also am looking one for that also shows the cities in different countries rather than only countries.

AlterWind Log Analyzer - http://www.alterwind.com