If you use the expression form of the function as in the example you can test "Name" for blank on failure.
If (! Name)
{
MsgBox Message Registration Failed
}
; do stuff if it worked
Edit: Also even though the docs say you do not need to have all the params in the function definition I have found stuff works better if I use all four.
If your registration code does return the expected function name then it may be something with Windows where the change of state is not broadcasting a message for some reason. But esp. when first trying to get code to work it is good to check the return codes for errors or wrap the code in try catch so that errors are reported.
Edit2: Skrommel has not been around in a while so I took the liberty of sticking in my $.02.

Edit3: Also try setting DetectHiddenWindows, On before calling OnMessage since you are not using a Gui.
Edit4: AHK uses an implied window handle called LastFound. See the examples for OnMessage and DetectHiddenWindows. A standard trick is to call WinExist() with a blank param to set LastFound to the handle of the current AHK program. That's the most likely cause.
A better resource is:
https://autohotkey.com/boards/index.phpsee the "ask for help" board. You will get a better explanation than the one I gave I am sure. That's where I go when I get stuck on something AHK.
