I guess as a long term user of both I might put my 2c in
Is a big issue *everywhere* - I have had problems with my windows machine, usually not with drivers but with getting the right settings and connecting to networks. On most machines on windows the driver comes with its own wireless manager tool, and more often than not some network settings work on the native wireless tool but not the manufacturer's tool, and vice versa. People who dont travel much dont have that problem, but if you do, you have encountered it. Now we have perhaps forgotten how nasty it was on windows even 2 years or 1 year ago...
Most cards have decent driver solutions on linux - at least it is well documented what works and doesnt so you can avoid trying on a hopeless machine - but the "connect to a network" bit is the same mess as on windows - several different "supplicant" layers available, and lots of trial and error to figure out how to set up the different kinds of networks you encounter.
Too many cooks:
But the biggest obstacle to switching to linux on the desktop for me is: the mutually exclusive choices you have to make when it comes to desktop environmnent, sound and multimedia (and perhaps others). It is not always even clear what the choices mean you are giving up.
I have been using linux and bsd on the server side for ages and am totally convinced there - but there are no mutually exclusive choices on the server side linux. Installing Apache does not somehow prevent you from using entire sets of tools and software.
On the linux desktop, on the other hand, choosing to run one set of apps means you cannot run another set - your choice of desktop will suddenly limit what apps you can use, and sometimes not very clearly (better know which key library set your chosen set up uses, and not get gtk apps on the wrong desktop etc.). Yes, it is kind of possible to mix some but at a risk of weird problems, and certainly at a high performance cost.
Say you've got your distro installed and you read up on what apps might do to replace your photo management app, music player, screenshot editing tool, desktop recording, information manager, email clients etc. tools. It's very likely you might find apps that tick most boxes for your most wanted features. But what is also likely is that they will need different core libraries and desktop environments and you wont be able to run them all on your distro comfortably. Eg: you cannot have the best image/photo manager and the best screenshot tool at the same time - one's KDE and the other Gnome - and both need the full stack. Sorry. Got to use a half baked half finished app for one of the two tasks, make your choice.
Read about a great music app or video editing app? Sorry, can't use it, because they only work with a different audio or video layer than the one your distro installed. And you can try to install that layer, but then it is likely that your already installed audio solution will stop working and you will need a lot of trial and error to get all the layers happily hierarchized so everything works - and in some cases you just won't. Now I am close to an expert on the server so I can dig in the manpages and config files until I get something working, but it's a waste of time and I cannot be bothered, after a while.
The main problem with those sets of mutually exclusive choices is that they are not clearly documented - nobody spells it out for you, since it would imply admitting linux ain't the perfect solution, and would get you all the fanatics on attack - you cannot make an informed decision up front, and it is frustrating.
I think it would be the best thing that could happen if these projects could work towards a more compatible architecture on the audio, video and desktop layers - and in the meantime far better information on what works with what and what each choice you makes excludes you from
Frankly I ended up using neither gnome nor kde, and using a minimal setup with tools not much better than what I could get on a windows95/98 machine in 1998... and as many great text mode/console apps as I could.