The only 32bit program I've noticed running slower on 64bit Windows was Foxit Reader - complex PDFs rendered extremely slow, so slow that it was actually faster to render them running a 32bit OS under vmware. That was a while ago, so the problem might have been fixed, and it might have been specific to 64bit XP - I'm pretty sure it was related to GDI+. Don't have any complex PDFs lying around now, so can't test with recent Foxit + win7.
Other than that, you've already mentioned the 16-bit issue yourself. It's not only an issue with applications themselves, some 32bit apps have 16bit installshieldcrap installers. Luckily that's only really for old old versions of software.
There's drivers, which can be a problem with old hardware, but the situation is much better with vista and win7 than it was with 64bit xp.
The issue you'd be most likely to run into is shell extensions - 64bit explorer requires 64bit versions of shell extensions. Personally I stick with 32bit xplorer2 even though I run a 64bit OS, and that works like a charm.