Example of setting up scheduled backups

Objective

Create a script that will regularly launch a command-line client to archive and transfer critical data to S3.

Requirements

• a command-line client (in the example S3cmd with the crontab automation tool);
• a cloud or dedicated server running Ubuntu 18.04 or higher;
• a user with S3 access.

Result

The script will create a backup of a file or directory using tar and upload the backup to S3 using s3cmd.

Steps

1. Create a script.
2. Transfer files to S3.
3. Configure flow control.
4. Check the script.
5. Optional: automate backups via crontab or Cyberduck.

1. Create a script

1. Open the home directory on your server:

   cd ~

2. Use the nano editor to create an empty file (for example, named bkupscript):

   nano bkupscript.sh

3. Start writing the backup script in a text editor with a shebang:

   #!/bin/bash

   A shebang is an interpreter directive that allows scripts or data files to be executed as commands and looks like a sequence of two characters: # and ! . Thanks to the shebang at the beginning of the script, the shell executes the file's commands in bash.

4. Below the shebang at the top of the text file, add variables to the script:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3

   Where:

   • DATETIME — the timestamp to be attached to the name of the resulting file. Every file backed up to the space will have a unique name. This timestamp is created by calling the date command and formatting the output to display the last two digits of the year (% y), two digits for the month (% m), two digits for the day (% d), hours (% H), minutes (% M), and seconds (% S);
   • SRC — the source path for the file or folder being backed up. $1 indicates that the value is taken from the first parameter passed to the script;
   • DST — the file destination. In this example, it is the name of the space to which the backup is uploaded. This name will be retrieved from the second parameter passed to the script, as specified in $2;
   • GIVENNAME — the user-selected name for the destination file. The resulting filename will begin with GIVENNAME, and DATETIME will be appended to it. This name originates from the third parameter passed to the script $3.

5. Add the showhelp function to the backup script to output messages if the script fails:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   }

6. Collect the necessary files and combine them into a single package that can be uploaded:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   ){
   
   }
   
   tarandzip(){
       echo "\n##### Gathering files #####\n"
       tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC
   }

   When the if instruction is called, the script executes the tar command and awaits the result. If the command executes successfully, the lines after the then: operator will be executed:

   • the output of a message confirming that the tar process finished successfully;
   • returning error code 0, so that the part of the code calling this function knows that everything is working normally.

   The else part of the script will be executed only if the tar command encounters an error during execution:

   • the output of a message stating that the tar command failed;
   • returning error code 1, which indicates that something went wrong.

7. End the if/then/else script with the fi statement. The look of the completed tarandzip function:

   tarandzip(){
       echo "\n##### Gathering files #####\n"
       if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
           echo "\n##### Done gathering files #####\n"
           return 0
       else
           echo "\n##### Failed to gather files #####\n"
           return 1
       fi
   }

2. Transfer files to S3

1. Add the movetoSpace file transfer function to the backup script for the selected space. The previously declared variables are used to create the command that will place backup files into the selected space:

   movetoSpace(){
       /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
   }

   Where:

   • /bin/s3cmd — invokes s3cmd, a command-line tool used to manage object storage buckets;
   • put — used by s3cmd to upload data to a bucket;
   • $GIVENNAME-$DATETIME.tar.gz — the name of the backup that will be uploaded to the space. It consists of the fourth and first declared variables, followed by .tar.gz, and is created by the tarandzip function;
   • s3://$DST — the place where the file will be uploaded;
   • s3:// — the URI scheme type used to describe object storage locations on the network, and $DST — this is the third variable that was declared earlier.

2. Add a notification confirming that the file transfer process has started:

   movetoSpace(){
       echo “\n##### MOVING TO SPACE #####\n”
       /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
   }

3. Configure the output of a message regarding the command execution result:

   if /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
       echo "\n##### Done moving files to s3://"$DST" #####\n"
       return 0
   else
       echo "\n##### Failed to move files to the Space #####\n"
       return 1
   fi

3. Configure flow control

If the script is configured correctly, when launched, it should read the input command, assign values from it to each variable, execute the tarandzip function, and then execute movetoSpace.

If the scenario fails at any step, it should print the output of the showhelp function to assist in troubleshooting.

At the end of the file, add an if / then / else conditional statement:

if [ ! -z "$GIVENNAME" ]; then
    if tarandzip; then
        movetoSpace
    else
        showhelp
    fi
else
    showhelp
fi

4. Check the script

1. Check the script:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(){
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   }
   
   tarandzip(){
       echo "\n##### Gathering files #####\n"
       if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
           echo "\n##### Done gathering files #####\n"
           return 0
       else
           echo "\n##### Failed to gather files #####\n"
           return 1
       fi
   }
   movetoSpace(){
       echo "\n##### MOVING TO SPACE #####\n"
       if /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
           echo "\n##### Done moving files to s3://"$DST" #####\n"
           return 0
       else
           echo "\n##### Failed to move files to the Spac #####\n"
           return 1
       fi
   }
   
   if [ ! -z "$GIVENNAME" ]; then
       if tarandzip; then
           movetoSpace
       else
           showhelp
       fi
   else
       showhelp
   fi

2. Before exiting nano, close the file ( Ctrl+X) and save the changes ( **Y+Enter **).

5. Optional: automate backups

Crontab

1. Set up a cron job that will use the script for regular backups to the selected space. In this example, the backup will be performed every minute.

2. Make the script executable:

   chmod +x bkupscript.sh

3. Edit the crontab file so that the script runs every minute:

   crontab -e

4. The first time you run the crontab -e command, you will be prompted to select an editor from the list:

   no crontab for root - using an empty one
   Select an editor. To change later, run "select-editor".
   1. /bin/ed
   2. /bin/nano        <---- easiest
   3. /usr/bin/vim.basic
   4. /usr/bin/vim.tiny
   Choose 1-4 [2]:

5. Choose nano by default or another text editor.

6. In crontab, add the line at the end of the script:

   * * * * * ~/bkupscript.sh ~/backupthis nameofyourspace cronupload

7. Close the file ( Ctrl+X) and save the changes ( **Y+Enter **).

8. If you leave the cron job running without changes, a new file will be copied to the selected space every minute. If cron is working successfully, reconfigure crontab to back up files at the required interval.

Cyberduck

There is a directory on the local machine whose contents need to be periodically copied to S3. This can be done using a script and a Cron job, which sends the backup to storage every day at a specified time.

Example job (additionally, you can use other Cyberduck options):

#!/bin/bash
SERVERCORE_ID=<servercore_account>
USERNAME=<username>
PASSWORD=<password>
BACKUP_PATH=<bucket_name>/<prefix>
LOCAL_PATH=<path>

duck --upload servercore:/<bucket_name>/<prefix> <path> --existing rename --username <servercore_account>:<username> --password <password> -q -y

Specify:

• <servercore_account> — Servercore Control Panel account number;
• <username> — service user name;
• <password> — service user password. If you have forgotten your password, change it;
• <bucket_name> — bucket name;
• <prefix> — object prefix;
• <path> — path to the local directory.

Where:

• the --existing key of the duck command specifies what to do with files already in storage;
• the rename option renames an existing backup by appending the time and date to the name.

Cyberduck options

When automating backups via Cyberduck you can use additional options:

• compare — perform differential backups:

  duck --upload  servercore:/<bucket_name>/<prefix> <path> --existing compare --username <servercore_account>:<username> --password <password>

  The uploaded backup will be compared with the existing one by size, modification date, and checksum. If the parameters differ, the old version will be deleted and the new one uploaded to storage;

• skip — only new files will be uploaded to storage (those that appeared in the local machine folder after the previous upload). Already existing files will not be uploaded, even if they were modified on the local machine;

• overwrite — deletes the existing backup from storage and uploads a new one.