As one of the foremost experts on unnecessarily complicating one's life I would like to offer a slightly different perspective.
Setting up such a PC has non-monetary costs. You'll have another PC to maintain, and the more complicated the system the more likely you are to make mistakes and the less time you'll have for other pursuits. If you use PayPal or banking fairly often, do you really want to have to go to another PC? When do the small risks justify extraordinary measures? For example, the consequences of having the websites you maintain hacked would be far greater than those of a fraudulent credit card transaction for which you wouldn't be liable.
Have you taken all the simpler steps first? Do you already use non-text, maximum-length, encrypted passwords? That doesn't require another PC.
If you're worried about banking and credit cards, have you taken all the steps you should take whether or not you set up a security PC, such as:
-setting up alerts on your accounts for transactions exceeding a certain amount
-monitoring your credit reports up to 3x a year for free
-setting up fraud alerts with the credit reporting agencies
-using virtual credit card numbers if you have any doubt about the vendor
If you're worried about the consequences of burglary, have you properly secured your house? Your data may be safe with FDE, but it's still a hassle to lose your stuff.
Have you thought about where the greatest risks actually lie? For example, I've used PayPal, eBay, and online banking extensively for years and have never had a problem, but I've had a couple of fraudulent transactions on a credit card I don't use online. Also, someone tried to open a Capital One account using my mother's information, and she's never touched a computer.
How often has your computer actually been infected with malware? I used to obsess about security software, etc. until I finally realized this just hasn't been a problem for me.
If you do decide to go ahead, I like the idea of a virtual machine (a cheap, simple, and convenient option). I dislike the idea of off-lease equipment (that is, equipment that was leased rather than bought, used for the lease period (often 3 years), and returned). Security and reliability are inextricably linked.
I don't mean to disparage your idea. Those of us who like to tinker with computers are inclined to think of adding new equipment as a solution. Setting up a security PC isn't a bad idea; it's just not the first idea one should consider.