I usually do what Eóin does, except I put my profile on my data partition (which is synchronized with my notebook and backed up regularly). With a good sync'er, I could leave out some files that I don't want them sync'ed. (E.g., I save passwords with my home desktop, but not with my notebook.)
Performance hit will be there if you put firefox profile on a flash drive, unless you move the cache folder to a HD. If you use Firefox3, more dynamic databases (those sqlite files) are in the profile folder.
I don't use portable Fox, though, so I don't know if it has any special design to minimize these potential performance penalties. And as Eóin said, if you want to use Firefox on computers you have no control of (with no Firefox installed), I guess the portable version is the only way to go.