Distributed compiling and clustering

[Gothi[c]]

Distributed compiling and clustering

Compiling large amounts of code takes time, sometimes even massive amounts of time. Some of us see the test-build as a well deserved coffee break, but after a while it gets old. Having to wait for a long time, in some cases hours, for a compile to finish, is not only counter-productive but makes debuging in many cases alot harder. (eg: you can't just make a quick change in your code, quickly run it, and see what it does.) 

The solution: Gather around those old computers you have lying around everywhere (and I'm sure many of us do), and set up a little compile farm to help your computer crunch some code.

The tools: Finding the right tool for the job can be a bit tricky, and reading up on clustering might make your head explode initially, but there are truely some great tools out there that make this all to easy, and that's what this review is about.

In my short adventure through the distributed compiling and clustering world, I have run in to quite a few options:


The first was building a beowulf cluster with tools such as heartbeat (http://www.linux-ha.org/).
From wikipedia(http://en.wikipedia.org/wiki/Beowulf_cluster):

A Beowulf cluster is a group of usually identical PC computers running a FOSS Unix-like operating system, such as GNU/Linux or BSD. They are networked into a small TCP/IP LAN, and have libraries and programs installed which allow processing to be shared among them.

Unfortionally not all my computers are identical so this was not an option.


Then there is openMosix, which is a kernel patch for linux that lets you share cpu power and memory over any number of machines, or as wikipedia (http://en.wikipedia.org/wiki/OpenMosix) describes:

openMosix is a free cluster management system that provides single-system image (SSI) capabilities, e.g. automatic work distribution among nodes. It allows program processes (not threads) to migrate to machines in the node's network that would be able to run that process faster. It is particularly useful for running parallel and intensive input/output (I/O) applications. It is released as a Linux kernel patch, but is also available on specialized LiveCDs and as a Gentoo Linux kernel choice.

And last but not least there is distcc. Distcc is actually the only one that will work with windows. Distcc is different from all of the above as it focuses on distributed compiling rather than regular clustering. It requires very little setup.  You can use it togeather with ccache, which makes it even faster.  From wikipedia(http://en.wikipedia.org/wiki/Distcc):

distcc works as an agent for the compiler. A distcc daemon has to run on each of the participating machines. The originating machine invokes a preprocessor to handle source files and sends the preprocessed source to other machines over the network via TCP. Remote machines compile those source files without any local dependencies (such as header files or macro definitions) to object files and send them back to the originator for further compilation.

Note that none of the above requires any tampering with makefiles or creating complex build scripts.


The results:

openMossix had a fairly easy setup ( just configure / install the kernel and run the daemon ) and did seem to do a good job with applications that are cpu-intensive (such as a compile job) however I ran into some problems now and then, getting segmentation faults. I assume it's my fault, but after playing with it for a day I was ready to try something new.

Distcc was VERY impressive. It seems like the perfect tool for the job. Setting it up was very easy (just install distcc, and set it as your default compiler, it has a configuration tool that sets the participating hosts, and you just have to start the distccd daemon specifying which ip's to allow) and it worked right away. Required though is that your build-environment has the same versions of things. (like same version of gcc, ld, etc,..) but that's quite easy to deal with. I must say the speedup was significant. distcc comes with a monitoring tool (openMosix does too) that shows the running jobs on the farm. Now I can finally compile things on my slow computer, taking advantage of the speed of my faster computer. I tested distcc with one machine running windows (distcc running in cygwin) and the other running gentoo linux (http://www.gentoo.org ). Because both platforms were different I had to set up a cross compiling envoronment (binutils come in handly) which worked out just fine. Later I tried it with one machine running gentoo, the other gentoo on vmware with windowsXP host. I must say this was the easyest of all, as there was no additional setup needed for cross-compilation.

I was also reading that you can set up distcc to run on openMosix, but i did not get into that. (I'm curious as to what the difference would be in benchmark results with just distcc and distcc+openMosix)


Conclusion:
Distcc seems to be the best tool for the job,  and to save yourself some cross-compiling trouble, the uber easyest is to set it up on the same platform.

Screenshots:

http://images.google.com/images?q=openmosix&svnum=10&hl=en&lr=&client=firefox-a&rls=org.mozilla:en-US:official&sa=N&imgsz=xxlarge

[f0dder]

Isn't there this issue with distcc, though, that you will be somewhat bottlenecked if some machines in the compile farm is noticably slower than the others? It's pretty nifty anyway, too bad I haven't found any native windows solutions for this (cygwin sucks and doesn't count) - well, except for those expensive ones.

Btw, you should check out http://ccache.samba.org/ and combine it with distcc...

[Gothi[c]]

Isn't there this issue with distcc, though, that you will be somewhat bottlenecked if some machines in the compile farm is noticably slower than the others?

Only when it is misconfigured,... there is a faq entry on the distcc site about this. When you set the hosts in the proper order and you set the correct -j option it should be fine.  The machines with most speed (or less load) you set at the beginning of the hosts list. If you'd have a huge farm that is also running lots of other things, you could write an easy little script that updates the order according to the load on the different machines. Running it on a home network with a few computers you shouldn't even ever run in to that problem.

cygwin sucks and doesn't count

I agree, but at least you can build a windows binary with it using distcc, which is better than nothing at all I guess. I also dislike cygwin, which is why I switched to gentoo on vmware.

Btw, you should check out http://ccache.samba.org/ and combine it with distcc...

Yeah, distcc works great with ccache

[f0dder]

Well, consider the scenario where you have a 1-ghz machine and a 2-ghz machine with distcc. The 2ghz machine compiles "small.c" while 1ghz compiles "templateheavy.cpp", and then the 2ghz has to wait for small.obj before it can link. This is of course an oversimplification, but the idea is that there's some "sync points" in makefiles/builds where you have to catch up

[Gothi[c]]

Well, consider the scenario where you have a 1-ghz machine and a 2-ghz machine with distcc. The 2ghz machine compiles "small.c" while 1ghz compiles "templateheavy.cpp", and then the 2ghz has to wait for small.obj before it can link. This is of course an oversimplification, but the idea is that there's some "sync points" in makefiles/builds where you have to catch up Sad

Well, if you have the priority set in a way that the 2GHz is before the 1GHz machine, wouldn't it compile it on the 2GHz machine?
-or- you could just take the 1 Ghz machine out of the loop. Tell it not to use the slower one. Like, if you want to compile something on a 1GHz machine, and there's 3 2GHz machines, it'd still be allot faster just not to compile on the 1GHz machine, and take localhost out of the hostlist.... dunno

[f0dder]

Well, if you have the priority set in a way that the 2GHz is before the 1GHz machine, wouldn't it compile it on the 2GHz machine?

Depends on how files are served - first-come-first-served then it wouldn't matter. Unless the master machine decides it's been waiting too long and redelegates job to the faster machine, but I dunno how it works

Like, if you want to compile something on a 1GHz machine, and there's 3 2GHz machines, it'd still be allot faster just not to compile on the 1GHz machine, and take localhost out of the hostlist.... dunno

Exactly - but it still sucks not being able to use that extra gigahertz .

[mouser]

wow - fantastic mini review for programmers!
gothic you rock.

[gjehle]

if you need to do cross compiling over different architectures try this

http://www.gentoo-wi...TIP_AMD64-x86-distcc

i'm going to try this in a minute with my 4400+ x2 and the old 1ghz box
gonna write a little bit about it if i succeed, looks easy enough

[Gothi[c]]

That's cool, gjehle! I wish I had a 64b cpu to fool around with.

I actually got around to compiling code::blocks on linux, seems to be stable enough sofar (unlike what i've read in reviews of the linux version on past versions), though time will tell. - but- here's the twist:
It was -very- easy to make code::blocks use distcc!

Just replace all the g++ / gcc with distcc in Settings->Compiler.

Everything still works normally, you still get all warnings in the log etc, just get a majour speed boost

Here's a screenshot of my 3ghz box doing the work for my slow 1ghz Dell disaster computer.

[mouser]

that is really really cool.