I have much experience with VPS. 80% of the time you can answer the question "should I get a VPS?" by answering the question "can you handle system administration of all aspects of a linux/windows/BSD that is sitting exposed on the internet 24x7?". If you can answer yes to that question, then you should get a VPS. If you can't, then I suggest you just get a feature rich web hosting account and start playing with linux/windows/BSD in your spare time until you can comfortable run your own server of your at home internet connection 24x7.
If you can safely and competently administer a VPS server, then there are some other things you should keep in mind.
With a VPS, you are paying for versatility, often at the expense of performance for popular tasks. For example, an inexpensive web hosting account will probably provide better performance for serving up static web pages and low-end PHP/CGI/Perl scripts. That is because the shared web hosting server is tuned for such work and probably has much better hardware.
Some VPS accounts are process/bandwidth sliced enough that you can't really run IRC daemons etc., even if allowed under licensing. You almost, without a doubt, cannot run a game server for more than two or three connections, if at all.
There are several other items, but in general:
1) If you are considering renting a dedicated server or collocating in the next 6-18 months, consider a VPS as a test run.
2) If you just want to play with a dedicated server without the cost of one, and you have the skills, consider a VPS.
3) If you just are an advanced user of hosting services, move up to a better web hosting account with more features, and don't deal with the hassles of updating, keeping secure, etc. that goes with VPSs.