Get the simplest case working first before adapting it.
I did that; still failed. I asked over at the AHK forum; lots of people read my post, but nobody answered.
Eventually, I came to the comclusion that the reason it didn't work was because a fullscreen DOS app doesn't have the same attributes that are being tested for to detect a fullscreen Windows app, and racked my brains for a while to see if I could come up with an alternative approach.
I discovered that (under XP, anyway) Alt-Enter with the desktop as the foreground app brings up display properties. That led me to a plan...
; MyCode should be replaced with the appropriate window title -- can be set in a batchfile with Title "<whatever>"
; semi-disable Alt+Enter
click 2000,2000 ; click somewhere far, far away from any open windows
; Alt-enter on desktop brings up XP display properties, in a fullscreen DOS app it pushes it back to a window
click 275,430 ; the Cancel button on the XP Display Properties dialog, regardless of tab
So I don't actually test for the state, I do something that affects both states differently and deal with the outcome -- I click cancel, or I click in the window with the DOS app in it, which just selects it or some text in it depending on whether quickedit mode is on or not. Either way, I have a solution.
I felt quite clever.
Although I grant it's a lot less elegant than I'd like.