I have both. I use IrfanView every day [Ctrl+Shift+I), and XnView twice a year or so. I must admit there are many features in IrfanView that I have not yet tried to use, but I believe IrfanView would be "anyones" choice (obviously not) of the two - at least if photography or converting were the two biggest issues. But of course XnView still is somewhat 'nicer' and easier. Less nerdish, you might say.
Besides, noutters, I remember that I couldn't figure out how to set transparent colours on GIFs and PNGs with XnView - can it do so?