OK, I've taken a cursory look at the source. The crash happens on line 151 in MainForm.cs, upon a call to StartHook(this.Handle); - my guess is that this hook is in a 32bit native DLL, which obviously can't be loaded when dotNET jits CircleDock for a 64bit target CPU. By creating a build configuration for "x86" (instead of "any CPU") and copying some files around manually, I was able to get CircleDock running (although there's some other problems, but we'll focus on those later).
Now, it's showing that I'm not exactly super familiar with dotNET - I have trouble seeing how the various DLL files are generated; does Orbit.Hook.dll , for instance, have corresponding source files in the project? I only really see it referenced in the source via a DllImport statement, so I'm wondering
Yes, you appear to be right, StartHook(this.Handle);
is a 32-bit DLL. It was a component I took from the Orbit Dock project (a circular dock that was abandoned a few years ago before completion). The Orbit Dock project files I have are quite unstable and I don't think it was meant for 64-bit OS's. This function resides in the Orbit.Hook.dll file as you have mentioned. It is basically used for hooking the middle mouse button to toggle the visiblity of the dock. It is written is C++, which doesn't really flow with the rest of the project because I think it was written by someone else than the author of Orbit Dock.
However, I have a solution, we can just remove Orbit.Dock.dll altogether and all the hooks currently in Circle Dock and replace them with a C# hook for the mouse. In the source code you have, there is a project called "Global Hook" and a file called "UserActivityHook.cs". We can easily implement the mouse hook with it. I will work on an Alpha 7 with this.
For the DLL files in the Debug folder used to execute the program, I just manually copied the Orbit.XXXXX.dll ones and the AMS.Profile.dll because they were already built in other projects. The GlobalMouseKeyboardHook.dll is generated by the Circle Dock solution from the "Global Hook" project when you build the solution. The dll files you see in the main Circle Dock project folder is what the IDE references and it will complain if they are not there. They are not used for execution.
Can you tell me the steps you took in compiling the "x86" version of Circle Dock and which files you moved around?
P.S. If you look in the License.txt file in the main Circle Dock folder, you will find the links to the AMS and Orbit Dock projects.