Mouser and I had a similar conversation over mail recently. He recommended phplist. I was looking at companies like aweber that provide their own servers to send mails.
phplist looks great. The problem with installing software like this on your server is that the server may be blacklisted, and then most of your messages will never reach your audience (without you knowing!).
Since I'm hosted in a off-the-mill dreamhost account, chances of having a neighbor spamming the hell out of people in my same server are high. Worse, it might be that if anyone at dreamhost has spammed ever, dreamhost is blacklisted.
I think that's the point the companies who sell services make. Of course, I don't have stats on how many messages would be lost (not sure if they do!).
In your case, ig you 'own' your own server, and can be pretty sure it's not blacklisted, they you can just install a program like these. If you are on a shared server, then all bets are off!
My 2c.