Home | Blog | Software | Reviews and Features | Forum | Help | Donate | About us
topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • December 04, 2016, 02:20:23 AM
  • Proudly celebrating 10 years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Author Topic: secure automated backup on *nix.  (Read 7657 times)

Gothi[c]

  • DC Server Admin
  • Charter Honorary Member
  • Joined in 2006
  • ***
  • Posts: 858
    • View Profile
    • linkerror
    • Donate to Member
secure automated backup on *nix.
« on: May 26, 2007, 01:50:52 AM »
After having lost months of work on various projects(ironically one of them was a backup tool) due to HD crashes and such, I wrote a little Bash shell script that will automatically archive a directory (and all it's sub-folders) and upload it via secure copy (scp) to a remote shell account. I figured I would share this with my dc buddies in case any one can use it. ;)

It reads a simple configuration file in which you can configure a few things, including files to EXCLUDE from the backup. (for example: object files and other garbage files generated while compiling a program).

I will post step-by-step instructions on how to use this, but first, here is the script:


Code: Text [Select]
  1. #!/bin/bash
  2.  
  3. ###########################################################################################
  4. #                                                                                         #
  5. # Simple secure automatic backup script by Gothi[c]                                       #
  6. # http://www.donationcoder.com                                                            #
  7. #                                                                                         #
  8. # Takes 1 argument: settings file                                                         #
  9. #                                                                                         #
  10. # Settings file can have the following directives:                                        #
  11. #                                                                                         #
  12. # - backup (which folder to back up)                                                      #
  13. # - scphostname (scp remote machine host name)                                            #
  14. # - scpusername (scp username on remote machine)                                          #
  15. # - scpdestdir (scp destination folder on remote machine)(optional)                       #
  16. # - except files (which files to exclude from the backup) (can have multiple directives)  #
  17. # - logfile (where to log events)                                                         #
  18. # - scpdestfile (optional filename on remote server, if not specified one will be         #
  19. #                generated based on the current date)                                     #
  20. # - enable scp (whether or not to enable scp upload of backups)                           #
  21. # - enable email (whether or not to enable emailing backups)                              #
  22. # - email (email address to send backups to, if enabled)                                  #
  23. # - sendmail (optional)(path to the sendmail program, used to send email)                 #
  24. #                                                                                         #
  25. # Requires following GNU utilities to be installed (these are present on about any GNU    #
  26. # system): cat,sed,grep,tar,bzip2                                                         #
  27. #                                                                                         #
  28. # Requires following GNU utilities to be installed for scp uploading: scp (openssh)       #
  29. #                                                                                         #
  30. # Requires following GNU utilities to be installed for emailing: sendmail, uuencode       #
  31. #                                                                                         #
  32. ###########################################################################################
  33.  
  34. #### first check the sanity of our user. ####
  35.  
  36. if [ ! -n "$1" ]; then
  37.   echo "ERROR: Missing settings file argument."
  38.   echo "Usage: $0 settingsfile".
  39.   exit $E_BADARGS
  40. fi  
  41.  
  42. #### check if settings file is readable ####
  43.  
  44. if [ ! -r "$1" ]; then
  45.   echo "ERROR: Could not read provided settings file: $1."
  46.   echo "Please ensure the file exists and that you have read permissions to it."
  47.   exit $E_BADARGS
  48. fi
  49.  
  50. #### init variables ####
  51.  
  52. SETTINGS_FILE=$1
  53. LOCAL_SRC_FOLDER=`cat $SETTINGS_FILE | grep "backup:" | sed -e 's/backup:\( \)\?//g' | sed -e 's/\( \)\+//'`
  54. ARCHIVE_FILENAME=`basename $LOCAL_SRC_FOLDER`-`date +%F`.tar.bz2
  55. REMOTE_HOSTNAME=`cat $SETTINGS_FILE | grep "scphostname:" | sed -e 's/scphostname:\( \)\?//g' | sed -e 's/\( \)\+//'`
  56. REMOTE_USERNAME=`cat $SETTINGS_FILE | grep "scpusername:" | sed -e 's/scpusername:\( \)\?//g' | sed -e 's/\( \)\+//'`
  57. REMOTE_DIRECTORY=`cat $SETTINGS_FILE | grep "scpdestdir:" | sed -e 's/scpdestdir:\( \)\?//g' | sed -e 's/\( \)\+//'`
  58. REMOTE_FILENAME=`cat $SETTINGS_FILE | grep "scpdestfile:" | sed -e 's/scpdestfile:\( \)\?//g' | sed -e 's/\( \)\+//'`
  59. LOG_FILE=`cat $SETTINGS_FILE | grep "logfile:" | sed -e 's/logfile:\( \)\?//g' | sed -e 's/\( \)\+//'`
  60. EXCEPTIONS=( `cat $SETTINGS_FILE | grep "except files:" | sed -e 's/except files:\( \)\?//g' | sed -e 's/\( \)\+//' | sed ':a;N;$!ba;$s/\(.*\)\n/\1 /'` )
  61. EMAIL=`cat $SETTINGS_FILE | grep "email:" | sed -e 's/email:\( \)\?//g' | sed -e 's/\( \)\+//'`
  62. SCP_ENABLED=`cat $SETTINGS_FILE | grep "enable scp:" | sed -e 's/enable scp:\( \)\?//g' | sed -e 's/\( \)\+//'`
  63. EMAIL_ENABLED=`cat $SETTINGS_FILE | grep "enable email:" | sed -e 's/enable email:\( \)\?//g' | sed -e 's/\( \)\+//'`
  64. PROG_SENDMAIL=`cat $SETTINGS_FILE | grep "sendmail:" | sed -e 's/sendmail:\( \)\?//g' | sed -e 's/\( \)\+//'`
  65.  
  66. #### create archive ####
  67.  
  68. TAR_COMMAND="-cvjf /tmp/$ARCHIVE_FILENAME $LOCAL_SRC_FOLDER"
  69. for exception in $(seq 0 $((${#EXCEPTIONS[@]} - 1))); do
  70.   TAR_COMMAND="$TAR_COMMAND --exclude ${EXCEPTIONS[$exception]}"
  71. done
  72. tar $TAR_COMMAND &> /dev/null
  73.  
  74. #### scp upload ####
  75.  
  76. if test "$SCP_ENABLED" != ""; then
  77.   if test "$REMOTE_FILENAME" = ""; then
  78.     SCP_REMOTE_FILE="."
  79.   else
  80.     SCP_REMOTE_FILE=$REMOTE_FILENAME
  81.   fi
  82.   SCP_COMMAND="/tmp/$ARCHIVE_FILENAME $REMOTE_USERNAME@$REMOTE_HOSTNAME:.$REMOTE_DIRECTORY/$SCP_REMOTE_FILE"
  83.   scp $SCP_COMMAND &> /dev/null
  84. fi
  85.  
  86. #### email upload ####
  87. echo "email enabled = $EMAIL_ENABLED"
  88. if test "$EMAIL_ENABLED" != ""; then
  89.   echo "email enabled"
  90.   if test "$PROG_SENDMAIL" = ""; then
  91.     PROG_SENDMAIL="sendmail"
  92.   fi
  93.   if test "$EMAIL" != ""; then
  94.     cat /tmp/$ARCHIVE_FILENAME | uuencode /tmp/$ARCHIVE_FILENAME | ${PROG_SENDMAIL} $EMAIL
  95.     echo "email sent"
  96.   fi
  97. fi
  98.  
  99. #### delete temporary archive file ####
  100.  
  101. rm /tmp/$ARCHIVE_FILENAME
  102.  
  103. #### logging ####
  104.  
  105. if test "$LOG_FILE" != ""; then
  106.   touch $LOG_FILE
  107.   if [ -w $LOG_FILE ]; then
  108.     THE_DATE=`date +%F`
  109.     THE_TIME=`date +%T`
  110.     echo "[$THE_DATE] ($THE_TIME): Completed backup for $SETTINGS_FILE." >> $LOG_FILE
  111.   fi
  112. fi
  113.  
  114. exit 0
(you can download it here)



Here is how I recommend you set it up:

  •   Download the script above and copy it to your /usr/local/bin -or- /usr/bin -or- /bin directory
     
  •   Make sure you can execute it, just type autobackup (i am assuming in this guide that you saved the script under that filename) and hit enter, you should get something like: "ERROR: Missing settings file argument." Which is good news, it means the script executes and is in our execute path. (if you get a command not found error, try moving the script to a different bin directory (eg: /bin)) (also note that you need to be root to write into these folders)

    You may or may not have to chmod it (chmod a+rx /usr/local/bin/autobackup) (replace /usr/local/bin with your bindir)
     
  •   Decide where you want to store the configuration files, a good place would be /etc/backups.
      In this guide I will use as example, a project called bpmnotepad of which i want to keep a daily, weekly, and monthly backup.

      So I am going to create the configuration files bpmnotepad-daily, bpmnotepad-weekly and bpmnotepad-monthly.

      First I need to create the /etc/backups directory, so as root, "cd /etc" and then "mkdir backups" (without the quotes of course)

      I will want to run these backups as my regular user (eg: the user gothic) so i am changing the ownership of that folder with "chown gothic /etc/backups" so the user gothic can read and write that folder. (maybe also make sure to chmod it, just in case your default permissions are restrictive (rare though) using the command "chmod +rw /etc/backups"
     
  •   I also want my backed up files to not just be dumped into the home folder on the remote machine where I have shell access, so I am going to ssh connect to the remote machine and create a backups folder in the home directory on the remote machine. (ssh user@remote-host.com,   then mkdir ~/backups) (the ~ tilde is a shortcut to your home directory)
     
  •   We also don't want scp to prompt us for a password, for this to work securely, we'll have to upload our public ssh key to the remote server and tell it to trust us. If you don't have a public ssh key yet, you can create one by running the command "ssh-keygen -t dsa" on your LOCAL machine, then paste the contents of the generated PUBLIC key file on the remote machine into the file ~/.ssh/authorized_keys To test if you no longer need the password, disconnect from the remote machine, and re-establish an ssh connection (ssh user@hostname) and it should not prompt you for a password. If you are having problems with this step, there is an exellent tutorial here
     

Now here are the example configuration files for our bpmnotepad project, which will go in /etc/backups:

/etc/backups/bpmnotepad-daily
backup:        /home/gothic/development/projects/bpmnotepad
logfile:       /var/log/backups/bpmnotepad

except files:  *.d
except files:  *.o

enable scp:   yes
scphostname:   linkerror.com
scpusername:   linkerr
scpdestdir:    /backups
scpdestfile:   bpmnotepad-daily.tar.bz2

/etc/backups/bpmnotepad-weekly
backup:        /home/gothic/development/projects/bpmnotepad
logfile:       /var/log/backups/bpmnotepad

except files:  *.d
except files:  *.o

enable scp:   yes
scphostname:   linkerror.com
scpusername:   linkerr
scpdestdir:    /backups
scpdestfile:   bpmnotepad-weekly.tar.bz2

/etc/backups/bpmnotepad-monthly
backup:        /home/gothic/development/projects/bpmnotepad
logfile:       /var/log/backups/bpmnotepad

except files:  *.d
except files:  *.o

enable scp:   yes
scphostname:   linkerror.com
scpusername:   linkerr
scpdestdir:    /backups
scpdestfile:   bpmnotepad-monthly.tar.bz2

As you can see, the only thing that changes in the different files is the remote filename, so we keep 3 copies at all times, which are overwritten with each scheduled backup. Also don't forget to edit the 'backup' line and have it point to the folder where your project or folder you want to have backed up is located. also notice the 'except files' lines, these tell us that i don't want any files with the .d or .o extention in my backup (c++ object files), these lines are optional, you can add as many exception file masks as you want.
The destdir and destfile lines are also optional. When destfile is not given, a filename will be generated from the name of your project/to_backup folder + the date of the backup. (also see information in the script header)

The logfile line is also optional, i forgot to mention this earlier, if you want to keep a logfile, you can create a directory /var/log/backups (follow the same steps as you did to create the /etc/backup directory, but apply to /var/log instead of /etc) -OR- just make it log to some file in your home directory, put whatever you want. or just remove the logfile line.

  • Now we should have everything set up for our backups. Before we make them run automatically, we should test if it actually works. To do so, in the context of our example, we would issue the following command:

    autobackup /etc/backups/bpmnotepad-daily

    ( the script takes only 1 argument (the configuration file to use) )

    If everything went well, the script should pause for a while (depending on how much you have to backup, it can take a while), and then return back to the prompt. The script is NOT verbose and will not give you any output when successful (as is the GNU standard). There will only be output when errors have occured.

    After the command finishes, doublecheck on the remote server if the backup file was created.
  • If the above test was successful, we can now do the actual scheduling part. For this we will use cron.
    create a new temporary file, and put something like this in it:

    0 1 * * * autobackup /etc/backups/bpmnotepad-dayly
    0 1 * * 1 autobackup /etc/backups/bpmnotepad-weekly
    0 1 1 * * autobackup /etc/backups/bpmnotepad-monthly

    The syntax of this is:
    minute_of_hour hour_of_day day_of_month month_of_year day_of_week command_to_run
    (the example backs up every day at 1 AM for the daily backup, every monday at 1am for the weekly backup and every 1st day of the month at 1am for the monthly backup)
    (more information on this here)

    After creating the temporary file run the command:

    crontab tmpfile

    (where tmpfile = the temporary file you just created)



That's it, now we are set up for automated backups over a secure connection. I know it seems like alot and the above guide may be intimidating to look at, but the only reason why it's so long is because I've tried to be very elaborate.

I hope someone else besides me can make some use of it ;)
I'll put this script + guide up on my website at some point.


(ps: sorry it isn't an ahk script, i'll leave that to skrommel, but us *nix geeks are allowed some fun every now and then too ;) )
« Last Edit: May 26, 2007, 02:51:14 PM by Gothi[c] »

Gothi[c]

  • DC Server Admin
  • Charter Honorary Member
  • Joined in 2006
  • ***
  • Posts: 858
    • View Profile
    • linkerror
    • Donate to Member
Re: secure automated backup on *nix.
« Reply #1 on: May 26, 2007, 01:30:50 PM »
You can also have the script email you the backups, instead of uploading them over scp, by using a config file like this:

backup:        /home/gothic/development/projects/bpmnotepad
logfile:       /var/log/backups/bpmnotepad

except files:  *.d
except files:  *.o

enable email: yes
email:         gothic@dcmembers.com

« Last Edit: May 26, 2007, 01:52:23 PM by Gothi[c] »

Gothi[c]

  • DC Server Admin
  • Charter Honorary Member
  • Joined in 2006
  • ***
  • Posts: 858
    • View Profile
    • linkerror
    • Donate to Member
Re: secure automated backup on *nix.
« Reply #2 on: May 26, 2007, 01:39:20 PM »
Instructions to back up and replace a MySQL database:

Say for example, you have a database called "donationcoder". You should have the folder /var/lib/mysql/donationcoder
That's the folder you will want to backup, so use a configuration file similar to this one:

backup:        /var/lib/mysql/donationcoders
logfile:       /var/log/backups/bpmnotepad

except files:  *.d
except files:  *.o

enable email: yes
email:         gothic@dcmembers.com



To replace a backup:

    • If the database doesn't exist yet, create a new empty database with the name of the database you backed up.
    • If the database already exists, first delete all the old data in it. (or issue rm -fr /var/lib/mysql/mydatabase/*)
  • Now just extract the archive to /var/lib/mysql and it will nicely replace all tables and data from the backup.
« Last Edit: May 26, 2007, 01:52:01 PM by Gothi[c] »

Chris

  • Charter Member
  • Joined in 2005
  • ***
  • default avatar
  • Posts: 40
    • View Profile
    • Donate to Member
Re: secure automated backup on *nix.
« Reply #3 on: June 17, 2007, 07:30:33 PM »
Remote back up is a good idea. You might want to look at rsync to minimise the data transferred when only a few items change. http://rsync.samba.org.

For automated incremental back up there is a good php script RIBS (Rsync Incremental Backup Script) http://www.rustyparts.com/ribs.php. I use this successfully to back up several web servers to another. Even for multiple backups not too much disk space is used.

Similar scripts can be found at http://www.rsnapshot.org and http://www.mikerubel...ters/rsync_snapshots. I have not tried these.

Chris
Adelaide
Australia

Gothi[c]

  • DC Server Admin
  • Charter Honorary Member
  • Joined in 2006
  • ***
  • Posts: 858
    • View Profile
    • linkerror
    • Donate to Member
Re: secure automated backup on *nix.
« Reply #4 on: October 22, 2007, 07:06:53 PM »
It has a home now: http://www.linkerror.com/autobackup.cgi
It can also encrypt backups now, and backup to email (for small stuff?)

mouser

  • First Author
  • Administrator
  • Joined in 2005
  • *****
  • Posts: 36,406
    • View Profile
    • Mouser's Software Zone on DonationCoder.com
    • Read more about this member.
    • Donate to Member
Re: secure automated backup on *nix.
« Reply #5 on: October 22, 2007, 07:18:40 PM »
well done  :up: :up: :up:

DisturbedComputer

  • Participant
  • Joined in 2009
  • *
  • Posts: 2
    • View Profile
    • Donate to Member
Re: secure automated backup on *nix.
« Reply #6 on: January 18, 2009, 04:34:40 AM »
So, is this to back up all your files and installed Programs and settings

Like
I have ObjectDeskTop
can this back up all of ODT and put it back into the right folders after a re-install of XP pro?

and/or
can it back up all my settings for Microsoft Office
Ex:
I have Word set up so I don't have to set it every time I open Word.
So, will it restore my setting after I re-install XP Pro  or MS Office














DisturbedComputer

  • Participant
  • Joined in 2009
  • *
  • Posts: 2
    • View Profile
    • Donate to Member
Re: secure automated backup on *nix.
« Reply #7 on: January 18, 2009, 04:37:36 AM »
PS

Oh, can it or dose it save the Registry


f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,029
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: secure automated backup on *nix.
« Reply #8 on: January 18, 2009, 06:27:54 AM »
DisturbedCompute: this is a backup tool for *u*x (unix, linux...), not Windows :)
- carpe noctem