even legit sites that you'd keep whitelisted might be compromised
As the manager of a web development team, this is something that I'm always paying attention to, much to the annoyance of some partners and even customers. You might be surprised how many third parties want us to directly reference js code that lives on somebody else's servers (for example, to show fancy interactive product info from the manufacturer).
My position is that my users have agreed to trust *me*, but they don't even know that they'd be implicitly trusting *you*. I don't have the authority to transfer my users' trust like that, so I simply will not allow your code to run in my site.
We've made a couple of sort-of exceptions. If they'll give us the code to verify ourselves, and host on our own servers, it's much less of an exposure. At least I can still have control over the stuff that I have responsibility for, rather than just abdicating that security consciousness.