I briefly tried deleting from the command line, can't remember if I tried wildcards, and got a message about file name being too long. But...
kudos to f0dder! chkdsk did the job - eventually. Basic "chkdsk" works, but as soon as I added a switch, "/f", it complained that I didn't have rights and it needed to be elevated. I'm using Vista Home Premium from a user account with UAC On... So I logged off, logged back on as administrator, tried again - only to find the same thing.
I can't remember exactly what I did then, but I copied chkdsk.exe to the root of the USB drive, fiddled about with its Properties to make it "Run as admin," and somewhere along the way it ran "chkdsk /f". There's a thread similar to this elsewhere on DC, where Carol Haynes notes (from memory) that "chkdsk /x" would be more appropriate in these cases. In fact my "chkdsk /f" instantly turned itself into "/x" because it said the drive had to be dismounted. At last it ran, I accepted the defaults about changing to a file etc., and was able to delete the mess. Windows didn't try very hard to make it easy, though.
Along the way I learnt a few things. I didn't know that USB devices have serial numbers, and can also have labels, and that
SyncBack Pro can recognise either or both as a way of uniquely identifying a particular USB device, so you only run the right profile against it.
USB Safely Remove, which is on Bits du Jour again on Wednesday 9th June, can conveniently give USB devices permanent letters, i.e., as I understand it, a particular USB device is always recognised as "g:\" or whatever. Presumably it too understands serial numbers.
I didn't try Unlocker because I thought that was mainly for deleting files that Windows thought were still in use, even where they weren't. It's very useful for that situation, but this situation seemed different.
Thanks, folks!