I've used a PDA for years. I've always used Windows-based handheld units - stylus only, no keyboard. They are great for jotting quick notes - shopping list, that great idea that hits you out of the blue - but not so great for any kind of extended text entry. It can be done, it just isn't fun. I wouldn't want to take meeting minutes on a PDA, for example, unless I had an external keyboard.
I have never bought a NEW organizer. My rationale was always that I didn't want to spend $500+ on something that would be garbage the first time I dropped it on the sidewalk. Buying 2nd-hand PDAs on eBay is definitely the way to go. The things depreciate like mad, so you can get great deals. This seems to be an item that people use once or twice, then put in their closet for a year or two before they decide to sell it. Also, the market has shrunk of late, with Palm and Treo merging and a bunch of manufacturers abandoning Windows Mobile/CE. So you get a better selection if you shop among all the units that were ever available, instead of just what's being made now. And there have been some interesting devices made over the years. Some options to google include:
CASIO Cassiopeia - This is the model I have always used, and it works fine. Despite its small internal memory, my old E-115 can play movies and MP3s from a compact flash card, although doing so whacks the battery life. The battery is just now starting to flag after about 2-3 years of service. I think it cost me $60.
Bonus link: Play videos on any PDA with the Core Media Player (http://www.tcmp.org/)Psion 5MX - If you prefer a keyboard, this has one of the best PDA keyboards ever made. It's a grayscale unit, and good ones are hard to come by and still rather expensive (relatively - about $150 - $200.) It's also a NON-MICROSOFT piece of tech, if that matters. I think this unit probably provided the best compromise between a laptop and a PDA. Fits in a coat pocket or belt pack, instant on, etc. but has a full suite of productivity apps as well as the PDA basics like datebook, voice recorder, etc. Not sure if it plays MP3s or not, though.
Bonus link: Just the ones you'd want on eBay. Vadem Clio - This was one of a small number of PDA's based on Windows CE that ran at full 640x480 resolution, offering another angle on the laptop/PDA compromise. (The Psion 7 also went this route.) It combined a touchscreen and keyboard in a "flip-over" design that let you use the unit as a laptop, a tablet PC, or something like a digital picture frame.
Bonus link: Egregious Clio pictures. And yes, the first thing you do is load it up with freeware. That's the fun part! One of my personal favorites for this is PDAGold:
http://www.pdagold.com