After having lost months of work on various projects(ironically one of them was a backup tool) due to HD crashes and such, I wrote a little Bash shell script that will automatically archive a directory (and all it's sub-folders) and upload it via secure copy (scp) to a remote shell account. I figured I would share this with my dc buddies in case any one can use it.
It reads a simple configuration file in which you can configure a few things, including files to EXCLUDE from the backup. (for example: object files and other garbage files generated while compiling a program).
I will post step-by-step instructions on how to use this, but first, here is the script:
#!/bin/bash
###########################################################################################
# #
# Simple secure automatic backup script by Gothi[c] #
# https://www.donationcoder.com #
# #
# Takes 1 argument: settings file #
# #
# Settings file can have the following directives: #
# #
# - backup (which folder to back up) #
# - scphostname (scp remote machine host name) #
# - scpusername (scp username on remote machine) #
# - scpdestdir (scp destination folder on remote machine)(optional) #
# - except files (which files to exclude from the backup) (can have multiple directives) #
# - logfile (where to log events) #
# - scpdestfile (optional filename on remote server, if not specified one will be #
# generated based on the current date) #
# - enable scp (whether or not to enable scp upload of backups) #
# - enable email (whether or not to enable emailing backups) #
# - email (email address to send backups to, if enabled) #
# - sendmail (optional)(path to the sendmail program, used to send email) #
# #
# Requires following GNU utilities to be installed (these are present on about any GNU #
# system): cat,sed,grep,tar,bzip2 #
# #
# Requires following GNU utilities to be installed for scp uploading: scp (openssh) #
# #
# Requires following GNU utilities to be installed for emailing: sendmail, uuencode #
# #
###########################################################################################
#### first check the sanity of our user. ####
if [ ! -n "$1" ]; then
echo "ERROR: Missing settings file argument."
echo "Usage: $0 settingsfile".
exit $E_BADARGS
fi
#### check if settings file is readable ####
if [ ! -r "$1" ]; then
echo "ERROR: Could not read provided settings file: $1."
echo "Please ensure the file exists and that you have read permissions to it."
exit $E_BADARGS
fi
#### init variables ####
SETTINGS_FILE=$1
LOCAL_SRC_FOLDER=`cat $SETTINGS_FILE | grep "backup:" | sed -e 's/backup:\( \)\?//g' | sed -e 's/\( \)\+//'`
ARCHIVE_FILENAME=`basename $LOCAL_SRC_FOLDER`-`date +%F`.tar.bz2
REMOTE_HOSTNAME=`cat $SETTINGS_FILE | grep "scphostname:" | sed -e 's/scphostname:\( \)\?//g' | sed -e 's/\( \)\+//'`
REMOTE_USERNAME=`cat $SETTINGS_FILE | grep "scpusername:" | sed -e 's/scpusername:\( \)\?//g' | sed -e 's/\( \)\+//'`
REMOTE_DIRECTORY=`cat $SETTINGS_FILE | grep "scpdestdir:" | sed -e 's/scpdestdir:\( \)\?//g' | sed -e 's/\( \)\+//'`
REMOTE_FILENAME=`cat $SETTINGS_FILE | grep "scpdestfile:" | sed -e 's/scpdestfile:\( \)\?//g' | sed -e 's/\( \)\+//'`
LOG_FILE=`cat $SETTINGS_FILE | grep "logfile:" | sed -e 's/logfile:\( \)\?//g' | sed -e 's/\( \)\+//'`
EXCEPTIONS=( `cat $SETTINGS_FILE | grep "except files:" | sed -e 's/except files:\( \)\?//g' | sed -e 's/\( \)\+//' | sed ':a;N;$!ba;$s/\(.*\)\n/\1 /'` )
EMAIL=`cat $SETTINGS_FILE | grep "email:" | sed -e 's/email:\( \)\?//g' | sed -e 's/\( \)\+//'`
SCP_ENABLED=`cat $SETTINGS_FILE | grep "enable scp:" | sed -e 's/enable scp:\( \)\?//g' | sed -e 's/\( \)\+//'`
EMAIL_ENABLED=`cat $SETTINGS_FILE | grep "enable email:" | sed -e 's/enable email:\( \)\?//g' | sed -e 's/\( \)\+//'`
PROG_SENDMAIL=`cat $SETTINGS_FILE | grep "sendmail:" | sed -e 's/sendmail:\( \)\?//g' | sed -e 's/\( \)\+//'`
#### create archive ####
TAR_COMMAND="-cvjf /tmp/$ARCHIVE_FILENAME $LOCAL_SRC_FOLDER"
for exception in $(seq 0 $((${#EXCEPTIONS[@]} - 1))); do
TAR_COMMAND="$TAR_COMMAND --exclude ${EXCEPTIONS[$exception]}"
done
tar $TAR_COMMAND &> /dev/null
#### scp upload ####
if test "$SCP_ENABLED" != ""; then
if test "$REMOTE_FILENAME" = ""; then
SCP_REMOTE_FILE="."
else
SCP_REMOTE_FILE=$REMOTE_FILENAME
fi
SCP_COMMAND="/tmp/$ARCHIVE_FILENAME $REMOTE_USERNAME@$REMOTE_HOSTNAME:.$REMOTE_DIRECTORY/$SCP_REMOTE_FILE"
scp $SCP_COMMAND &> /dev/null
fi
#### email upload ####
echo "email enabled = $EMAIL_ENABLED"
if test "$EMAIL_ENABLED" != ""; then
echo "email enabled"
if test "$PROG_SENDMAIL" = ""; then
PROG_SENDMAIL="sendmail"
fi
if test "$EMAIL" != ""; then
cat /tmp/$ARCHIVE_FILENAME | uuencode /tmp/$ARCHIVE_FILENAME | ${PROG_SENDMAIL} $EMAIL
echo "email sent"
fi
fi
#### delete temporary archive file ####
rm /tmp/$ARCHIVE_FILENAME
#### logging ####
if test "$LOG_FILE" != ""; then
touch $LOG_FILE
if [ -w $LOG_FILE ]; then
THE_DATE=`date +%F`
THE_TIME=`date +%T`
echo "[$THE_DATE] ($THE_TIME): Completed backup for $SETTINGS_FILE." >> $LOG_FILE
fi
fi
exit 0
Here is how I recommend you set it up:
- Download the script above and copy it to your /usr/local/bin -or- /usr/bin -or- /bin directory
- Make sure you can execute it, just type autobackup (i am assuming in this guide that you saved the script under that filename) and hit enter, you should get something like: "ERROR: Missing settings file argument." Which is good news, it means the script executes and is in our execute path. (if you get a command not found error, try moving the script to a different bin directory (eg: /bin)) (also note that you need to be root to write into these folders)
You may or may not have to chmod it (chmod a+rx /usr/local/bin/autobackup) (replace /usr/local/bin with your bindir)
- Decide where you want to store the configuration files, a good place would be /etc/backups.
In this guide I will use as example, a project called bpmnotepad of which i want to keep a daily, weekly, and monthly backup.
So I am going to create the configuration files bpmnotepad-daily, bpmnotepad-weekly and bpmnotepad-monthly.
First I need to create the /etc/backups directory, so as root, "cd /etc" and then "mkdir backups" (without the quotes of course)
I will want to run these backups as my regular user (eg: the user gothic) so i am changing the ownership of that folder with "chown gothic /etc/backups" so the user gothic can read and write that folder. (maybe also make sure to chmod it, just in case your default permissions are restrictive (rare though) using the command "chmod +rw /etc/backups"
- I also want my backed up files to not just be dumped into the home folder on the remote machine where I have shell access, so I am going to ssh connect to the remote machine and create a backups folder in the home directory on the remote machine. (ssh [email protected], then mkdir ~/backups) (the ~ tilde is a shortcut to your home directory)
- We also don't want scp to prompt us for a password, for this to work securely, we'll have to upload our public ssh key to the remote server and tell it to trust us. If you don't have a public ssh key yet, you can create one by running the command "ssh-keygen -t dsa" on your LOCAL machine, then paste the contents of the generated PUBLIC key file on the remote machine into the file ~/.ssh/authorized_keys To test if you no longer need the password, disconnect from the remote machine, and re-establish an ssh connection (ssh user@hostname) and it should not prompt you for a password. If you are having problems with this step, there is an exellent tutorial here
Now here are the example configuration files for our bpmnotepad project, which will go in /etc/backups:
/etc/backups/bpmnotepad-daily
backup: /home/gothic/development/projects/bpmnotepad
logfile: /var/log/backups/bpmnotepad
except files: *.d
except files: *.o
enable scp: yes
scphostname: linkerror.com
scpusername: linkerr
scpdestdir: /backups
scpdestfile: bpmnotepad-daily.tar.bz2
/etc/backups/bpmnotepad-weekly
backup: /home/gothic/development/projects/bpmnotepad
logfile: /var/log/backups/bpmnotepad
except files: *.d
except files: *.o
enable scp: yes
scphostname: linkerror.com
scpusername: linkerr
scpdestdir: /backups
scpdestfile: bpmnotepad-weekly.tar.bz2
/etc/backups/bpmnotepad-monthly
backup: /home/gothic/development/projects/bpmnotepad
logfile: /var/log/backups/bpmnotepad
except files: *.d
except files: *.o
enable scp: yes
scphostname: linkerror.com
scpusername: linkerr
scpdestdir: /backups
scpdestfile: bpmnotepad-monthly.tar.bz2
As you can see, the only thing that changes in the different files is the remote filename, so we keep 3 copies at all times, which are overwritten with each scheduled backup. Also don't forget to edit the 'backup' line and have it point to the folder where your project or folder you want to have backed up is located. also notice the 'except files' lines, these tell us that i don't want any files with the .d or .o extention in my backup (c++ object files), these lines are optional, you can add as many exception file masks as you want.
The destdir and destfile lines are also optional. When destfile is not given, a filename will be generated from the name of your project/to_backup folder + the date of the backup. (also see information in the script header)
The logfile line is also optional, i forgot to mention this earlier, if you want to keep a logfile, you can create a directory /var/log/backups (follow the same steps as you did to create the /etc/backup directory, but apply to /var/log instead of /etc) -OR- just make it log to some file in your home directory, put whatever you want. or just remove the logfile line.
- Now we should have everything set up for our backups. Before we make them run automatically, we should test if it actually works. To do so, in the context of our example, we would issue the following command:
autobackup /etc/backups/bpmnotepad-daily
( the script takes only 1 argument (the configuration file to use) )
If everything went well, the script should pause for a while (depending on how much you have to backup, it can take a while), and then return back to the prompt. The script is NOT verbose and will not give you any output when successful (as is the GNU standard). There will only be output when errors have occured.
After the command finishes, doublecheck on the remote server if the backup file was created.
- If the above test was successful, we can now do the actual scheduling part. For this we will use cron.
create a new temporary file, and put something like this in it:
0 1 * * * autobackup /etc/backups/bpmnotepad-dayly
0 1 * * 1 autobackup /etc/backups/bpmnotepad-weekly
0 1 1 * * autobackup /etc/backups/bpmnotepad-monthly
The syntax of this is:
minute_of_hour hour_of_day day_of_month month_of_year day_of_week command_to_run
(the example backs up every day at 1 AM for the daily backup, every monday at 1am for the weekly backup and every 1st day of the month at 1am for the monthly backup)
(more information on this here)
After creating the temporary file run the command:
crontab tmpfile
(where tmpfile = the temporary file you just created)
That's it, now we are set up for automated backups over a secure connection. I know it seems like alot and the above guide may be intimidating to look at, but the only reason why it's so long is because I've tried to be very elaborate.
I hope someone else besides me can make some use of it
I'll put this script + guide up on my website at some point.
(ps: sorry it isn't an ahk script, i'll leave that to skrommel, but us *nix geeks are allowed some fun every now and then too )
