Here's what's happening:
When you do an active window capture, SC is trying to figure out the partial transparency behind the rounded corners. TO do this, it actually does something a bit involved, it paints the background of the window in black then white and compares them and uses the difference to remove the partial transparency around the borders outside the window.
However, when you try this on a child (modal only?) window, the parent prevents this trick from working.
So you have 3 options in such a case:
1. Move the child off the parent background. This will make it work.
2. Turn off the option to try to grab partial window backgound transparency in the Windows Capture options dialog.
3. Live with the background captured in the image or crop it yourself afterwards. You might try the new advanced option under the Edit menu called "Auto Adjust Selection to Fit" as an alternative way of shrinking the outside crop to the window borders.
