Example of setting up scheduled backups

Objective

Create a script that will regularly launch a command-line client to archive and transfer critical data to S3.

Requirements

• a console client (in the example, S3cmd with the crontab automation tool);
• a cloud or dedicated server running Ubuntu 18.04 or higher;
• a user with access to S3.

Result

The script will create a backup of a file or directory using tar and upload the backup to S3 using s3cmd.

Steps

1. Create a script.
2. Transfer files to S3.
3. Configure flow control.
4. Check the script.
5. Optional: automate backups using crontab or Cyberduck.

1. Create a script

1. Open the home directory on your server:

   cd ~

2. Using the nano editor, create an empty file (for example, named bkupscript):

   nano bkupscript.sh

3. Start writing the backup script in a text editor with a shebang:

   #!/bin/bash

   A shebang is an interpreter directive that allows scripts or data files to run as commands and looks like a two-character sequence: # and ! . Thanks to the shebang at the beginning of the script, the shell executes the file's commands in bash.

4. Below the shebang at the top of the text file, add variables to the script:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3

   Where:

   • DATETIME — a timestamp to attach to the resulting file name. Every file backed up in the space will have a unique name. This timestamp is generated by calling the date command and formatting the output to display the last two digits of the year (% y), two digits of the month (% m), two digits of the day (% d), hour (% H), minutes (% M), and seconds (% S);
   • SRC — the source path for the file or folder being backed up. $1 indicates that the value is taken from the first parameter passed to the script;
   • DST — the file destination. In the example, this is the name of the space where the backup is uploaded. This name will be obtained from the second parameter passed to the script, as indicated by $2;
   • GIVENNAME — the user-selected name for the destination file. The resulting file name will start with GIVENNAME, and DATETIME will be appended to it. This name comes from the third parameter passed to the script $3.

5. Add the showhelp function to the backup script to display messages in case of script failure:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   }

6. Collect the necessary files and combine them into a single package that can be uploaded:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   ){
   
   }
   
   tarandzip(){
       echo "\n##### Gathering files #####\n"
       tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC
   }

   When the if statement is called, the script runs the tar command and waits for the result. If the command succeeds, the lines after the then statement will be executed:

   • outputting a message indicating that the tar process completed successfully;
   • returning error code 0 so that the part of the code calling this function knows that everything is working properly.

   The else part of the script will execute only if the tar command encounters an error during execution:

   • outputting a message indicating that the tar command failed;
   • returning error code 1, which indicates that something went wrong.

7. End the if/then/else script with fi. The completed tarandzip function:

   tarandzip(){
       echo "\n##### Gathering files #####\n"
       if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
           echo "\n##### Done gathering files #####\n"
           return 0
       else
           echo "\n##### Failed to gather files #####\n"
           return 1
       fi
   }

2. Transfer files to S3

1. Add the file transfer function movetoSpace to the backup script to transfer files to the selected space. The previously declared variables are used to create the command that will place backup files in the selected space:

   movetoSpace(){
       /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
   }

   Where:

   • /bin/s3cmd — calls s3cmd, a command-line tool used to manage object storage buckets;
   • put — used by s3cmd to upload data to a bucket;
   • $GIVENNAME-$DATETIME.tar.gz — the name of the backup that will be uploaded to the space. It consists of the fourth and first declared variables followed by .tar.gz, and is created by the tarandzip function;
   • s3://$DST — the destination where the file will be uploaded;
   • s3:// — a URI scheme used to describe object storage locations on the network, and $DST is the third variable declared earlier.

2. Add a notification confirming that the file transfer process has started:

   movetoSpace(){
       echo “\n##### MOVING TO SPACE #####\n”
       /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
   }

3. Configure the output of a message regarding the command execution result:

   if /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
       echo "\n##### Done moving files to s3://"$DST" #####\n"
       return 0
   else
       echo "\n##### Failed to move files to the Space #####\n"
       return 1
   fi

3. Configure flow control

If the script is configured correctly, upon execution it should read the input command, assign values from it to each variable, execute the tarandzip function, and then execute movetoSpace.

If the script fails at any step, it should print the output of the showhelp function to help with troubleshooting.

At the end of the file, add the if / then / else conditional statement:

if [ ! -z "$GIVENNAME" ]; then
    if tarandzip; then
        movetoSpace
    else
        showhelp
    fi
else
    showhelp
fi

4. Check the script

1. Check the script:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(){
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   }
   
   tarandzip(){
       echo "\n##### Gathering files #####\n"
       if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
           echo "\n##### Done gathering files #####\n"
           return 0
       else
           echo "\n##### Failed to gather files #####\n"
           return 1
       fi
   }
   movetoSpace(){
       echo "\n##### MOVING TO SPACE #####\n"
       if /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
           echo "\n##### Done moving files to s3://"$DST" #####\n"
           return 0
       else
           echo "\n##### Failed to move files to the Spac #####\n"
           return 1
       fi
   }
   
   if [ ! -z "$GIVENNAME" ]; then
       if tarandzip; then
           movetoSpace
       else
           showhelp
       fi
   else
       showhelp
   fi

2. Before exiting nano, close the file ( **Ctrl+X **) and save your changes ( **Y+Enter **).

5. Optional: automate backups

Crontab

1. Set up a cron job that will use the script for regular backups to the selected space. In this example, the backup will be performed every minute.

2. Make the script executable:

   chmod +x bkupscript.sh

3. Edit the crontab file so that the script runs every minute:

   crontab -e

4. When running the crontab -e command for the first time, you will be prompted to select an editor from the list:

   no crontab for root - using an empty one
   Select an editor. To change later, run "select-editor".
   1. /bin/ed
   2. /bin/nano        <---- easiest
   3. /usr/bin/vim.basic
   4. /usr/bin/vim.tiny
   Choose 1-4 [2]:

5. Choose nano by default or another text editor.

6. In crontab, add the line at the end of the script:

   * * * * * ~/bkupscript.sh ~/backupthis nameofyourspace cronupload

7. Close the file ( **Ctrl+X **) and save your changes ( **Y+Enter **).

8. If you leave the cron job running without changes, a new file will be copied to the selected space every minute. If cron is working successfully, reconfigure crontab to back up files at the required interval.

Cyberduck

There is a directory on your local machine whose contents need to be periodically copied to S3. You can do this using a script and a Cron job that sends a backup to the storage every day at a specified time.

Example job (you can optionally use other Cyberduck options):

#!/bin/bash
SERVERCORE_ID=<servercore_account>
USERNAME=<username>
PASSWORD=<password>
BACKUP_PATH=<bucket_name>/<prefix>
LOCAL_PATH=<path>

duck --upload servercore:/<bucket_name>/<prefix> <path> --existing rename --username <servercore_account>:<username> --password <password> -q -y

Specify:

• <servercore_account> — Servercore control panel account number;
• <username> — service user name;
• <password> — service user password. If you forgot the password, change it;
• <bucket_name> — bucket name;
• <prefix> — object prefix;
• <path> — path to the local directory.

Where:

• the --existing flag of the duck command specifies what to do with files that already exist in the storage;
• the rename option renames the existing backup by appending the time and date to its name.

Cyberduck options

When automating backups using Cyberduck, you can use additional options:

• compare — perform differential backup:

  duck --upload  servercore:/<bucket_name>/<prefix> <path> --existing compare --username <servercore_account>:<username> --password <password>

  The uploaded backup will be compared with the existing one by size, modification date, and checksum. If the parameters differ, the old version will be deleted and the new one uploaded to storage;

• skip — only new files (those that appeared in the folder on the local machine after the previous upload) will be uploaded to the storage. Existing files will not be uploaded even if they have been modified on the local machine;

• overwrite — deletes the existing backup from the storage and uploads a new one.