DC member gothi[c] has been working on a really promising and very cool tool that does this kind of stuff and more -- he should post a progress report and some screenshots.
Yes, I'm continuously working on a scriptable server monitoring system.
Basically, you can write a local or remote sensor plug-in which returns a value.
Custom triggers can be set on the return value of these sensors. When a trigger is triggered, alerts can be sent out via email or sms. These email addresses can be added to a web interface, which also shows a graph over time of each of the sensor values.
Some more advanced scripting can be done with the triggers, for example, return debug information. (eg: return a top (list of processes sorted by cpu load) + running mysql queries + list of connected ip's when the server load is too high) This can be useful for debugging peak problems. (When server load peaks, and by the time you log in to diagnose the problem, all is back to normal).
All of this has a GUI (work in progress) to easily add/create/manage sensors, triggers, and multiple servers. All of it works except for the GUI which is still very much under development.
All of this will be released under GPL whenever I'm "done"
You may also want to have a look at Zenoss which is kind of similar to what I'm doing, except a lot more bloated, and seems to be a horrible cpu hog (it was when I tested it on my server anyway), it's not very easy to install unless you get the payware version. It has some really nice features though.
GUI:
Some shots of web interface:
