I can see why they designed GammaPanel that way, since it controls more than just brightness.
Could you use GammaPanel to create a series of hotkeys that increase/decrease brightness, but that are still easy to use and remember, such as Ctrl+Num0, Ctrl+Num1, Ctrl+Num2, and so on, or Ctrl+F1, Ctrl+F2, Ctrl+F3, and so on? Not ideal for you I guess; I'm just trying to help.