I figured out what is wrong, thanks to Sysinternals Filemon. The problem has to do with the quotation mark issue that I mentioned before in passing, though I didn't know this would happen.
When I have this as the path for the "Optional 3rd party tool":
C:\Program Files\Process Explorer\procexp.exe /p:n
And then click the browse button, ProcessTamerConfigurator.exe does a query on the full contents of the text box, and gets a "path not found" error. It's getting an error because it's trying to resolve the path and the command line parameter, not just the path. If I put this in the "Optional 3rd party tool" text box:
C:\Program Files\Process Explorer\procexp.exe
The browse button works fine (that is, the Open dialog appears correctly).