Topics

26

Developer's Corner / secure automated backup on *nix.

« on: May 26, 2007, 01:50 AM »

After having lost months of work on various projects(ironically one of them was a backup tool) due to HD crashes and such, I wrote a little Bash shell script that will automatically archive a directory (and all it's sub-folders) and upload it via secure copy (scp) to a remote shell account. I figured I would share this with my dc buddies in case any one can use it.

It reads a simple configuration file in which you can configure a few things, including files to EXCLUDE from the backup. (for example: object files and other garbage files generated while compiling a program).

I will post step-by-step instructions on how to use this, but first, here is the script:

Code: Text [Select]

1. #!/bin/bash
2.  
3. ###########################################################################################
4. #                                                                                         #
5. # Simple secure automatic backup script by Gothi[c]                                       #
6. # https://www.donationcoder.com                                                            #
7. #                                                                                         #
8. # Takes 1 argument: settings file                                                         #
9. #                                                                                         #
10. # Settings file can have the following directives:                                        #
11. #                                                                                         #
12. # - backup (which folder to back up)                                                      #
13. # - scphostname (scp remote machine host name)                                            #
14. # - scpusername (scp username on remote machine)                                          #
15. # - scpdestdir (scp destination folder on remote machine)(optional)                       #
16. # - except files (which files to exclude from the backup) (can have multiple directives)  #
17. # - logfile (where to log events)                                                         #
18. # - scpdestfile (optional filename on remote server, if not specified one will be         #
19. #                generated based on the current date)                                     #
20. # - enable scp (whether or not to enable scp upload of backups)                           #
21. # - enable email (whether or not to enable emailing backups)                              #
22. # - email (email address to send backups to, if enabled)                                  #
23. # - sendmail (optional)(path to the sendmail program, used to send email)                 #
24. #                                                                                         #
25. # Requires following GNU utilities to be installed (these are present on about any GNU    #
26. # system): cat,sed,grep,tar,bzip2                                                         #
27. #                                                                                         #
28. # Requires following GNU utilities to be installed for scp uploading: scp (openssh)       #
29. #                                                                                         #
30. # Requires following GNU utilities to be installed for emailing: sendmail, uuencode       #
31. #                                                                                         #
32. ###########################################################################################
33.  
34. #### first check the sanity of our user. ####
35.  
36. if [ ! -n "$1" ]; then
37.   echo "ERROR: Missing settings file argument."
38.   echo "Usage: $0 settingsfile".
39.   exit $E_BADARGS
40. fi  
41.  
42. #### check if settings file is readable ####
43.  
44. if [ ! -r "$1" ]; then
45.   echo "ERROR: Could not read provided settings file: $1."
46.   echo "Please ensure the file exists and that you have read permissions to it."
47.   exit $E_BADARGS
48. fi
49.  
50. #### init variables ####
51.  
52. SETTINGS_FILE=$1
53. LOCAL_SRC_FOLDER=`cat $SETTINGS_FILE | grep "backup:" | sed -e 's/backup:\( \)\?//g' | sed -e 's/\( \)\+//'`
54. ARCHIVE_FILENAME=`basename $LOCAL_SRC_FOLDER`-`date +%F`.tar.bz2
55. REMOTE_HOSTNAME=`cat $SETTINGS_FILE | grep "scphostname:" | sed -e 's/scphostname:\( \)\?//g' | sed -e 's/\( \)\+//'`
56. REMOTE_USERNAME=`cat $SETTINGS_FILE | grep "scpusername:" | sed -e 's/scpusername:\( \)\?//g' | sed -e 's/\( \)\+//'`
57. REMOTE_DIRECTORY=`cat $SETTINGS_FILE | grep "scpdestdir:" | sed -e 's/scpdestdir:\( \)\?//g' | sed -e 's/\( \)\+//'`
58. REMOTE_FILENAME=`cat $SETTINGS_FILE | grep "scpdestfile:" | sed -e 's/scpdestfile:\( \)\?//g' | sed -e 's/\( \)\+//'`
59. LOG_FILE=`cat $SETTINGS_FILE | grep "logfile:" | sed -e 's/logfile:\( \)\?//g' | sed -e 's/\( \)\+//'`
60. EXCEPTIONS=( `cat $SETTINGS_FILE | grep "except files:" | sed -e 's/except files:\( \)\?//g' | sed -e 's/\( \)\+//' | sed ':a;N;$!ba;$s/\(.*\)\n/\1 /'` )
61. EMAIL=`cat $SETTINGS_FILE | grep "email:" | sed -e 's/email:\( \)\?//g' | sed -e 's/\( \)\+//'`
62. SCP_ENABLED=`cat $SETTINGS_FILE | grep "enable scp:" | sed -e 's/enable scp:\( \)\?//g' | sed -e 's/\( \)\+//'`
63. EMAIL_ENABLED=`cat $SETTINGS_FILE | grep "enable email:" | sed -e 's/enable email:\( \)\?//g' | sed -e 's/\( \)\+//'`
64. PROG_SENDMAIL=`cat $SETTINGS_FILE | grep "sendmail:" | sed -e 's/sendmail:\( \)\?//g' | sed -e 's/\( \)\+//'`
65.  
66. #### create archive ####
67.  
68. TAR_COMMAND="-cvjf /tmp/$ARCHIVE_FILENAME $LOCAL_SRC_FOLDER"
69. for exception in $(seq 0 $((${#EXCEPTIONS[@]} - 1))); do
70.   TAR_COMMAND="$TAR_COMMAND --exclude ${EXCEPTIONS[$exception]}"
71. done
72. tar $TAR_COMMAND &> /dev/null
73.  
74. #### scp upload ####
75.  
76. if test "$SCP_ENABLED" != ""; then
77.   if test "$REMOTE_FILENAME" = ""; then
78.     SCP_REMOTE_FILE="."
79.   else
80.     SCP_REMOTE_FILE=$REMOTE_FILENAME
81.   fi
82.   SCP_COMMAND="/tmp/$ARCHIVE_FILENAME $REMOTE_USERNAME@$REMOTE_HOSTNAME:.$REMOTE_DIRECTORY/$SCP_REMOTE_FILE"
83.   scp $SCP_COMMAND &> /dev/null
84. fi
85.  
86. #### email upload ####
87. echo "email enabled = $EMAIL_ENABLED"
88. if test "$EMAIL_ENABLED" != ""; then
89.   echo "email enabled"
90.   if test "$PROG_SENDMAIL" = ""; then
91.     PROG_SENDMAIL="sendmail"
92.   fi
93.   if test "$EMAIL" != ""; then
94.     cat /tmp/$ARCHIVE_FILENAME | uuencode /tmp/$ARCHIVE_FILENAME | ${PROG_SENDMAIL} $EMAIL
95.     echo "email sent"
96.   fi
97. fi
98.  
99. #### delete temporary archive file ####
100.  
101. rm /tmp/$ARCHIVE_FILENAME
102.  
103. #### logging ####
104.  
105. if test "$LOG_FILE" != ""; then
106.   touch $LOG_FILE
107.   if [ -w $LOG_FILE ]; then
108.     THE_DATE=`date +%F`
109.     THE_TIME=`date +%T`
110.     echo "[$THE_DATE] ($THE_TIME): Completed backup for $SETTINGS_FILE." >> $LOG_FILE
111.   fi
112. fi
113.  
114. exit 0

(you can download it here)


Here is how I recommend you set it up:

•   Download the script above and copy it to your /usr/local/bin -or- /usr/bin -or- /bin directory
   
•   Make sure you can execute it, just type autobackup (i am assuming in this guide that you saved the script under that filename) and hit enter, you should get something like: "ERROR: Missing settings file argument." Which is good news, it means the script executes and is in our execute path. (if you get a command not found error, try moving the script to a different bin directory (eg: /bin)) (also note that you need to be root to write into these folders)
  
  You may or may not have to chmod it (chmod a+rx /usr/local/bin/autobackup) (replace /usr/local/bin with your bindir)
   
•   Decide where you want to store the configuration files, a good place would be /etc/backups.
    In this guide I will use as example, a project called bpmnotepad of which i want to keep a daily, weekly, and monthly backup.
  
    So I am going to create the configuration files bpmnotepad-daily, bpmnotepad-weekly and bpmnotepad-monthly.
  
    First I need to create the /etc/backups directory, so as root, "cd /etc" and then "mkdir backups" (without the quotes of course)
  
    I will want to run these backups as my regular user (eg: the user gothic) so i am changing the ownership of that folder with "chown gothic /etc/backups" so the user gothic can read and write that folder. (maybe also make sure to chmod it, just in case your default permissions are restrictive (rare though) using the command "chmod +rw /etc/backups"
   
•   I also want my backed up files to not just be dumped into the home folder on the remote machine where I have shell access, so I am going to ssh connect to the remote machine and create a backups folder in the home directory on the remote machine. (ssh [email protected],   then mkdir ~/backups) (the ~ tilde is a shortcut to your home directory)
   
•   We also don't want scp to prompt us for a password, for this to work securely, we'll have to upload our public ssh key to the remote server and tell it to trust us. If you don't have a public ssh key yet, you can create one by running the command "ssh-keygen -t dsa" on your LOCAL machine, then paste the contents of the generated PUBLIC key file on the remote machine into the file ~/.ssh/authorized_keys To test if you no longer need the password, disconnect from the remote machine, and re-establish an ssh connection (ssh user@hostname) and it should not prompt you for a password. If you are having problems with this step, there is an exellent tutorial here
   

Now here are the example configuration files for our bpmnotepad project, which will go in /etc/backups:

/etc/backups/bpmnotepad-daily

[Select]

backup:        /home/gothic/development/projects/bpmnotepad
logfile:       /var/log/backups/bpmnotepad

except files:  *.d
except files:  *.o

enable scp:   yes
scphostname:   linkerror.com
scpusername:   linkerr
scpdestdir:    /backups
scpdestfile:   bpmnotepad-daily.tar.bz2

/etc/backups/bpmnotepad-weekly

[Select]

backup:        /home/gothic/development/projects/bpmnotepad
logfile:       /var/log/backups/bpmnotepad

except files:  *.d
except files:  *.o

enable scp:   yes
scphostname:   linkerror.com
scpusername:   linkerr
scpdestdir:    /backups
scpdestfile:   bpmnotepad-weekly.tar.bz2

/etc/backups/bpmnotepad-monthly

[Select]

backup:        /home/gothic/development/projects/bpmnotepad
logfile:       /var/log/backups/bpmnotepad

except files:  *.d
except files:  *.o

enable scp:   yes
scphostname:   linkerror.com
scpusername:   linkerr
scpdestdir:    /backups
scpdestfile:   bpmnotepad-monthly.tar.bz2

As you can see, the only thing that changes in the different files is the remote filename, so we keep 3 copies at all times, which are overwritten with each scheduled backup. Also don't forget to edit the 'backup' line and have it point to the folder where your project or folder you want to have backed up is located. also notice the 'except files' lines, these tell us that i don't want any files with the .d or .o extention in my backup (c++ object files), these lines are optional, you can add as many exception file masks as you want.
The destdir and destfile lines are also optional. When destfile is not given, a filename will be generated from the name of your project/to_backup folder + the date of the backup. (also see information in the script header)

The logfile line is also optional, i forgot to mention this earlier, if you want to keep a logfile, you can create a directory /var/log/backups (follow the same steps as you did to create the /etc/backup directory, but apply to /var/log instead of /etc) -OR- just make it log to some file in your home directory, put whatever you want. or just remove the logfile line.

• Now we should have everything set up for our backups. Before we make them run automatically, we should test if it actually works. To do so, in the context of our example, we would issue the following command:
  
  autobackup /etc/backups/bpmnotepad-daily
  
  ( the script takes only 1 argument (the configuration file to use) )
  
  If everything went well, the script should pause for a while (depending on how much you have to backup, it can take a while), and then return back to the prompt. The script is NOT verbose and will not give you any output when successful (as is the GNU standard). There will only be output when errors have occured.
  
  After the command finishes, doublecheck on the remote server if the backup file was created.
  
• If the above test was successful, we can now do the actual scheduling part. For this we will use cron.
  create a new temporary file, and put something like this in it:
  
  0 1 * * * autobackup /etc/backups/bpmnotepad-dayly
  0 1 * * 1 autobackup /etc/backups/bpmnotepad-weekly
  0 1 1 * * autobackup /etc/backups/bpmnotepad-monthly
  
  The syntax of this is:
  minute_of_hour hour_of_day day_of_month month_of_year day_of_week command_to_run
  (the example backs up every day at 1 AM for the daily backup, every monday at 1am for the weekly backup and every 1st day of the month at 1am for the monthly backup)
  (more information on this here)
  
  After creating the temporary file run the command:
  
  crontab tmpfile
  
  (where tmpfile = the temporary file you just created)
  

That's it, now we are set up for automated backups over a secure connection. I know it seems like alot and the above guide may be intimidating to look at, but the only reason why it's so long is because I've tried to be very elaborate.

I hope someone else besides me can make some use of it
I'll put this script + guide up on my website at some point.


_{(ps: sorry it isn't an ahk script, i'll leave that to skrommel, but us *nix geeks are allowed some fun every now and then too )}

27

General Software Discussion / Amazing realtime 3D network traffic visualization

« on: April 20, 2007, 04:00 PM »

An amazing network monitoring research project from Netcosm (http://www.netqos.com)
Project page here: http://www.youtube.c.../watch?v=dtC6ZM0_m8U
Video of the thing in action: http://www.youtube.c.../watch?v=dtC6ZM0_m8U

 
 

28

Living Room / Software patents, and the crazy business side of software.

« on: April 08, 2007, 03:11 AM »

Here is an interesting essay on software patents, and the business side of software development:

The most common is to grant patents that shouldn't be granted. To be patentable, an invention has to be more than new. It also has to be non-obvious. And this, especially, is where the USPTO has been dropping the ball. Slashdot has an icon that expresses the problem vividly: a knife and fork with the words "patent pending" superimposed.

The scary thing is, this is the only icon they have for patent stories. Slashdot readers now take it for granted that a story about a patent will be about a bogus patent. That's how bad the problem has become.

The problem with Amazon's notorious one-click patent, for example, is not that it's a software patent, but that it's obvious. Any online store that kept people's shipping addresses would have implemented this. The reason Amazon did it first was not that they were especially smart, but because they were one of the earliest sites with enough clout to force customers to log in before they could buy something. [1]

We, as hackers, know the USPTO is letting people patent the knives and forks of our world. The problem is, the USPTO are not hackers. They're probably good at judging new inventions for casting steel or grinding lenses, but they don't understand software yet.

http://www.paulgraha...softwarepatents.html

29

General Software Discussion / GPS Software?

« on: March 25, 2007, 01:47 PM »

I've been playing with the idea of purchasing a cheap gps receiver from ebay that I can hook up to our laptop, since that would be a cheaper solution than having to buy a PDA with gps included,... But then I was wondering what software is out there that I can use, that will tell me where to go while driving? (like those announcer voice things that till you 'make a right here'); i was also wondering if there are any free/open source alternatives out there.

I know that I can use stuff like xastir or google earth to show my location on a map, but that won't include the announcing or the automatic route recalculations etc,... maybe there are google earth plugins for this?

I'd like to know about any GPS software you've used (especially the freeware/donationware/opensource alternatives).

30

Announce Your Software/Service/Product / Tolipo - todo list for programmers

« on: March 18, 2007, 07:43 AM »

Just a quick little todo app I wanted for myself.
I wanted a todo application that would generate todo.txt/changelog.txt files for my projects automagically, and keep track of a relation between changelog and finished todo items, starting from a plain hierarchical todo gui which i could use to just organise my thoughts. I know that there is a thousand and one todo applications out there already, which is why I was a bit hesitant to start this project, I searched around and tried various todo applications out there, but none did what I wanted them to do, or they did too much. The vast majority seemed to be suffering from heavy feature bloat. I just wanted something very simple and minimal that did what I want it to do. So I eventually broke down and made this, I thought I would share.

I wouldn't call it a finished product just yet, but it's usable.

Known issues:
  - When adding a new project or task, or when marking one as completed, the
    treeview will refresh and expand all nodes (instead of remembering which ones you had open or closed.)

  - On linux: when moving the main window to a different workspace/desktop new
    dialogs do not appear on the correct workspace.

  - When the changelog text field is left emtpy they do not show up in the
    changelog treeview when marked as completed. (they are however written to the txt files using the summary text)

I'll get around to fixing these some day

obligatory screenshot:


I made a silly little web page for it, which is where you can go download it:
http://www.linkerror.com/tolipo.cgi

31

Living Room / Geek squad / Best buy Busted!

« on: February 07, 2007, 03:43 AM »

What I have always suspected:

Inexperienced computer users will call upon a computer repair service such as Geek squad when their computer is malfunctioning. I have always suspected that there must be lots of fraud in that field, since either the 'experts' don't know wtf they are doing,- or they just want to make an extra buck. I've heard stories of people that have worked in the field,... One guy I know said he was called in for a computer repair job, where appearantly the client had forgotten to plug in the power cable. He said he plugged in the cable, then waited 2 hours, when the client came back, he charged $200 for repair. ($100 / hour).

Just today I stumbled upon a news broadcast on youtube in which they use a hidden camera to expose fraud in Geek squad, Best Buy, and some other places.

http://www.youtube.c.../watch?v=cBvUt2bIQFk

If you're not too knowledgeable in hardware, WATCH OUT!! Try to find a geek friend first.

32

Developer's Corner / Debugging gems: runtime code profiling in Linux - KCacheGrind

« on: January 30, 2007, 04:27 AM »

I recently stumbled upon the (so far) best FOSS runtime code profiling tool I ever tried: KCacheGrind

It is really a front-end for the common *nix debugging tool valgrind, but the way it visualizes the data makes the valgrind actually allot more usable.

You can navigate through the call graph tree, go into different levels and subtrees, view how much time was spent in each function, detect memory leaks, bugs, etc,...

A nice addition to my debugging arsenal, I thought I would share

33

Developer's Corner / Clash of the languages

« on: January 25, 2007, 03:06 AM »

Cool statistics on programming language popularity up to date with long term graphs:

http://www.tiobe.com/tpci.htm

Looks like C++ has been on a constant downhill while java remains popular. I don't like this trend. I don't like it at all.

[edited to attach image]

34

Living Room / Ladybug gets even

« on: January 05, 2007, 11:09 PM »

 

Check this out...

http://minuscule.tv

35

Living Room / Cody - The 3d model

« on: December 22, 2006, 11:23 PM »

Cody the 3d model

I quickly put together a 3d model of Cody in blender in case anyone wants to use it in a game or movie or whatever

What the model actually looks like


Quick cartoon-rendering.

Here's the model:
http://www.linkerror...com/stuff/cody.blend

36

General Software Discussion / Xastir: What GPS, weather and radio have in common.

« on: December 19, 2006, 11:47 AM »

Weather?
Most people, if they want to know what the weather is like, they'll look out the window, or turn on the television.

For us geeks this won't do.

If I remember correctly, there was a discussion on DC about weather programs or even a review, I can't remember, I may have mentioned xastir in there, but I'd really like to start a thread dedicated to this fine piece of software, because I have been able to use it in so many different ways, and it goes quite beyond just weather.

Radio? GPS?
Xastir is APRS software aimed towards ham radio operators, search and rescue teams, and weather spotters.
APRS (http://en.wikipedia.org/wiki/APRS) is a position reporting system that runs on top of the ax.25 protocol (http://en.wikipedia.org/wiki/AX.25), which is an amateur radio version of the x.25 (http://en.wikipedia.org/wiki/X.25) protocol. Initially, the way this worked, was a station (radio transmitter) is hooked up to a little modem which is in turn hooked up to a GPS receiver. The modem would translate the GPS signal to APRS AX.25 packets and send them to the radio as Packet Radio FSK (http://en.wikipedia....rg/wiki/Packet_radio). Later the modems were more commonly replaced with laptops or desktop computers.

Internet
Nowerdays you don't even need a radio anymore to use APRS. Ever since the Internet came about, radio amateurs have started to make 'bridges' between radio and the internet. There are many internet APRS servers out there that mirror all packets received from local radios, relays and other internet servers. Some of them are dedicated to emergency services or weather data.

Additional data
Over the years, lots of additional data has found it's way into APRS packets next to the position reporting. There is weather data, graphical icons and objects, polygons, bbs-style messaging, email and much more.

Space
There are a few satellites equipped with APRS relay systems. The International Space Station also has an APRS relay/bbs onboard. Check out http://www.amsat.org for more info about APRS in space.

Maps
Xastir can display tigermaps (http://tiger.census....i-bin/mapbrowse-tbl/) automatically downloaded from the internet and shows the aprs icons on top of it, it is also capable of layering different maps on top of eachother such as Live weather radar maps, satelite images like google earth, bitmaps, and much more... All in all, xastir supports 125 map formats.

Additional features
  - Can show a trail tracking a target.
  - Can show a halo/circle around an object with the diameter representing signal strength
  - Layers
  - It has special weather and satellite modes.
  - ...

Practical uses
- Obiously, you can use it to check the position of anyone with an APRS system connected to either the internet or a radio (if you have a ham license). And display it on a pretty map as a little car icon or a little walking guy, or whatever.
- You can also use it to check the weather, areas with weather warnings may have a line around them or some other object, while you can set all regular weather stations to show the temperatures on the map, etc...
- But you can get very creative with it too. I wrote a flight simulator 2004 plugin once that would translate my airplane's position to APRS packets and send them over LAN to my other computer, showing my plane position on a xastir map. If I would stand on a runway in the game, I could see it very accurately on a runway in Xastir on a satellite map. I had awesome screenshots of that, but they got lost in a hd format Anyway, here's a screenshot of the weather around here today



To use xastir you can just connect to an internet APRS server, you don't need anything else.

List of APRS servers: http://www.aprs-is.net/APRSServers.htm

To connect, just click Interface->Interface control->Add->Internet server
Make sure to enter the right port number, if you click on a server on that APRS server list, it will typically have it's own page showing the correct port number.

Xastir website: http://www.xastir.org/

37

Living Room / You better watch out when picking a screenname nowerdays.

« on: December 19, 2006, 08:52 AM »

http://www.finnegan....e=news&pubType=2

When corporations, lawyers and technology meet, shit hits fans.

38

Living Room / Pirates of Sillicon Valley

« on: December 08, 2006, 02:12 PM »

Interesting movie for you guys to watch:
  Part 1: http://www.youtube.c.../watch?v=3BJu2GAkf2k
  Part 2: http://www.youtube.c.../watch?v=7jjepLRzH8o
  Part 3: http://www.youtube.c.../watch?v=m-5i4IJeZdM
  Part 4: http://www.youtube.c.../watch?v=hTYqJO5EpL8
  Part 5: http://www.youtube.c.../watch?v=kel3dK3DL6s
  Part 6: http://www.youtube.c.../watch?v=XdYdQ5E_rWA
  Part 7: http://www.youtube.c.../watch?v=7KT0gTPPaGk
  Part 8: http://www.youtube.c.../watch?v=0ZLZX3tU2SM
  Part 9: http://www.youtube.c.../watch?v=PVe4WjWliRs
  Part 10: http://www.youtube.c.../watch?v=R5bfT1kLzAE
  Steve Wozniak's comments on the movie: http://www.youtube.c.../watch?v=1lx9JsSTklI

 

39

Podcast Radio Show / W T F happened with the music in podcast #4?

« on: August 17, 2006, 02:47 PM »

Whoever deceided it was a good idea to use techno as background music for the fourth DC podcast needs to be run over by several large trucks, then eaten by tigers and have their eyes removed with a spoon.

What happened to the cute piano and organ music?

Other than that,... good job

40

Developer's Corner / The CADT programming model

« on: August 11, 2006, 03:44 PM »

http://www.jwz.org/doc/cadt.html

:S

41

Living Room / Funny 3d animation tutorial.

« on: July 05, 2006, 01:14 AM »

http://www.anim8or.com/movies/movies1/easy_eyes.zip

Funny 3d animation tutorial for anim8tor (http://www.anim8or.com/).
Kinda fun to watch even if you're not into 3d animation.

I think this is a great approach for a tutorial.

(Source: http://irrlicht3d.org/)

42

Living Room / Motivational sticky-notes

« on: July 03, 2006, 02:55 AM »

I found a new way to encourage productivity.
Believe it or not, but it's STICKY NOTES!

A sticky note on your monitor is something you constantly look at while at the computer. A sticky note reminding you to go back to work actually does help, because you are constantly reminded by looking at it!

I started to experiment with this and it seems to help. I spend less time browsing etc,...

(However, I did spend the time writing this post )

43

Developer's Corner / the OS 3d engine Irrlicht just released it's 1.0

« on: April 19, 2006, 02:59 PM »

One point Oh!!

http://www.irrlicht3d.org/pivot/entry.php?id=263
http://irrlicht.sourceforge.net

I've been using this engine for a long time (eg. for spaceduck ( http://gothic.dcmembers.com ) , and finally it's stable!

44

Mini-Reviews by Members / Distributed compiling and clustering

« on: March 29, 2006, 10:30 AM »

Distributed compiling and clustering

Compiling large amounts of code takes time, sometimes even massive amounts of time. Some of us see the test-build as a well deserved coffee break, but after a while it gets old. Having to wait for a long time, in some cases hours, for a compile to finish, is not only counter-productive but makes debuging in many cases alot harder. (eg: you can't just make a quick change in your code, quickly run it, and see what it does.) 

The solution: Gather around those old computers you have lying around everywhere (and I'm sure many of us do), and set up a little compile farm to help your computer crunch some code.

The tools: Finding the right tool for the job can be a bit tricky, and reading up on clustering might make your head explode initially, but there are truely some great tools out there that make this all to easy, and that's what this review is about.

In my short adventure through the distributed compiling and clustering world, I have run in to quite a few options:


The first was building a beowulf cluster with tools such as heartbeat (http://www.linux-ha.org/).
From wikipedia(http://en.wikipedia.org/wiki/Beowulf_cluster):

A Beowulf cluster is a group of usually identical PC computers running a FOSS Unix-like operating system, such as GNU/Linux or BSD. They are networked into a small TCP/IP LAN, and have libraries and programs installed which allow processing to be shared among them.

Unfortionally not all my computers are identical so this was not an option.


Then there is openMosix, which is a kernel patch for linux that lets you share cpu power and memory over any number of machines, or as wikipedia (http://en.wikipedia.org/wiki/OpenMosix) describes:

openMosix is a free cluster management system that provides single-system image (SSI) capabilities, e.g. automatic work distribution among nodes. It allows program processes (not threads) to migrate to machines in the node's network that would be able to run that process faster. It is particularly useful for running parallel and intensive input/output (I/O) applications. It is released as a Linux kernel patch, but is also available on specialized LiveCDs and as a Gentoo Linux kernel choice.

And last but not least there is distcc. Distcc is actually the only one that will work with windows. Distcc is different from all of the above as it focuses on distributed compiling rather than regular clustering. It requires very little setup.  You can use it togeather with ccache, which makes it even faster.  From wikipedia(http://en.wikipedia.org/wiki/Distcc):

distcc works as an agent for the compiler. A distcc daemon has to run on each of the participating machines. The originating machine invokes a preprocessor to handle source files and sends the preprocessed source to other machines over the network via TCP. Remote machines compile those source files without any local dependencies (such as header files or macro definitions) to object files and send them back to the originator for further compilation.

Note that none of the above requires any tampering with makefiles or creating complex build scripts.


The results:

openMossix had a fairly easy setup ( just configure / install the kernel and run the daemon ) and did seem to do a good job with applications that are cpu-intensive (such as a compile job) however I ran into some problems now and then, getting segmentation faults. I assume it's my fault, but after playing with it for a day I was ready to try something new.

Distcc was VERY impressive. It seems like the perfect tool for the job. Setting it up was very easy (just install distcc, and set it as your default compiler, it has a configuration tool that sets the participating hosts, and you just have to start the distccd daemon specifying which ip's to allow) and it worked right away. Required though is that your build-environment has the same versions of things. (like same version of gcc, ld, etc,..) but that's quite easy to deal with. I must say the speedup was significant. distcc comes with a monitoring tool (openMosix does too) that shows the running jobs on the farm. Now I can finally compile things on my slow computer, taking advantage of the speed of my faster computer. I tested distcc with one machine running windows (distcc running in cygwin) and the other running gentoo linux (http://www.gentoo.org ). Because both platforms were different I had to set up a cross compiling envoronment (binutils come in handly) which worked out just fine. Later I tried it with one machine running gentoo, the other gentoo on vmware with windowsXP host. I must say this was the easyest of all, as there was no additional setup needed for cross-compilation.

I was also reading that you can set up distcc to run on openMosix, but i did not get into that. (I'm curious as to what the difference would be in benchmark results with just distcc and distcc+openMosix)


Conclusion:
Distcc seems to be the best tool for the job,  and to save yourself some cross-compiling trouble, the uber easyest is to set it up on the same platform.

Screenshots:

http://images.google.com/images?q=openmosix&svnum=10&hl=en&lr=&client=firefox-a&rls=org.mozilla:en-US:official&sa=N&imgsz=xxlarge

45

Developer's Corner / Source code colors: Contrast / Readability / Ergonomics

« on: February 09, 2006, 06:08 PM »

Why is it that most IDE's have a default white background with black text?

Personally it really hurts my eyes. Especially on CRT monitors. The white of a computer monitor is not like say a white paper, it is light. So especially for those of us who are coding over 8 hours a day, which is better for our eyes?
Personally I use a soft green on a black background because green is one of the colors humans perceive best. Green on black provides high contrast. I use soft green so the contrast isn't too high. Seems to work best for me. I hear yellow on blue is best for bright-lit rooms / daylight.  But black on white is overkill contrast and really bad on the eyes.

Here is an interesting page about this topic:

http://www.writer200...m/colwebcontrast.htm

46

Mini-Reviews by Members / Game Review : Silkroad Online

« on: February 06, 2006, 02:15 AM »

Introduction:
Silkroad Online is a FREE mmorpg (massively multiplayer online roll playing game) developed by a Korean company called JOYMAX. This game is inspired by the oriental silk trade routes and therefore also focuses on trading allot. Players can set up trading posts, and there is a sophisticated trading system.

Features:

• An amazing fantasy world with stunning graphics.
• Quests : if you don't feel like aimlessly bashing monsters, maybe you'll be interested in solving some quests? This game has plenty of them.
• You can work on improving your skills. These are divided in weapon skills and 'alchemy' skills, which are magic-like abilities.
• The combat system is in real time - no turnbased combat.
• Interesting interface concept : There is an action window on which there is several action icons which can be dragged into containers at the bottom of the screen. Each of the containers corresponds with a number 0-9 which you can use as shortcut for the specified action. These containers are then sub-divided in groups with the function keys.
• Animals!! You can ride horses, buy a camel, etc,...

First impressions:
At my first glance of the game I was totally blown away with the quality of the graphics, and how polished the game was for being free. This is truly a high-quality game. I quickly got hooked on the monster-bashing collecting money, buying stuff rpg catch, which is definitely present in this game. After I got bored with that I started exploring the quests a bit, which are also quite interesting. One of the first things I noticed is how MASSIVELY this mmorpg really is. There is really allot of players connected, and it has a well established community with plenty of guilds.

System requirements:
I mentioned before that this game has stunning graphics, so I was a bit worried when I saw the initial screenshots that my poor graphics card would not be able to handle it, but infact it did quite well. This is mainly because the game seems to be highly optimized. This game obviously wasn't written by idiots. Objects are only visible from a certain distance to save memory and keep the fps high. This distance can be manually altered. Also, they aren't magically appearing/disappearing like in some games, nor are you constantly walking around in fog (which is the solution some other games opt for), the objects have a fade/in fade/out effect, which I thought was quite original. As a result the required CPU isn't too high compared to other games with similar graphics.

Required CPU: Intel Pentium 3 - 800 MHz CPU
Required memory: 256 MB RAM
Required graphics card: 3D speed over GeForce2 graphics
Required HD space: 3.0 GB free hard disk space (though the download is 515MB)

Screenshots:



Links:

• Game home page (click JOIN in the upper right corner to create an account and download): 
  
  http://www.silkroado...et/intro/feature.asp
  
  
• Wikipedia page:
  
  http://en.wikipedia....wiki/Silkroad_online