Example of configuring scheduled backup

Example of configuring scheduled backup

Purpose of customization⁠

Create a script that will regularly run the console client, archive and move important data to object storage.

What you need to customize⁠

• console client (in the example S3cmd with crontab automation tool);
• cloud or dedicated server with Ubuntu version 18.04 or higher installed;
• user with access to object storage.

Customization result⁠

The script will create a backup of the file or directory using tar and upload the backup to the object store using s3cmd.

Customization steps⁠

1. Create a script.
2. Transfer the files to object storage.
3. Configure flow control.
4. Check the script.
5. Optional: automate backups via crontab or Cyberduck.

1. Create a script⁠

1. Open the home directory on your server:

   cd ~

2. Use the nano editor to create an empty file (for example, named bkupscript):

   nano bkupscript.sh

3. Start writing a backup script in a text editor with shebang:

   #!/bin/bash

   Shebang is an interpreter directive that allows scripts or data files to be run as commands and looks like a sequence of two characters: # and !. With a shebang at the beginning of a script, the shell runs the file commands in bash.

4. Under shebang at the top of the text file, add variables to the script:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3

   Here:

   • DATETIME — time stamp to be appended to the received file name. Each file backed up in space will have a unique name. This timestamp is created by calling the command date command and formatting the output to display the last two digits of the year ( % y), the last two digits of the month ( % m), two digits of the day ( % d), hour ( % H), minute ( % M) and seconds ( % S);
   • SRC — The source path for the file or folder to which the backup is being made. $1 specifies that the value is taken from the first parameter passed to the script;
   • DST — The destination location of the file. In the example, this is the name of the space to which the backup is loaded. This name will be obtained from the second parameter passed to the script, as specified in $2;
   • GIVENNAME — the user-selected name for the destination file. The resulting file name will begin with GIVENNAME, and will be appended with DATETIME. This name comes from the third parameter passed to the script $3.

5. Add a showhelp function to the backup script to display messages if the script fails:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   }

6. Gather the files you need and combine them into a single package that can be downloaded:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   ){
   
   }
   
   tarandzip(){
       echo "\n##### Gathering files #####\n"
       tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC
   }

   When the if instruction is called, the script executes the tar command and waits for the result. If the command is successful, the lines after the then statement will be executed:

   • output a message that the tar process has successfully completed;
   • return error code 0, so that the part of the code calling this function knows that everything is working fine.

   The else part of the script will be executed only if the tar command detects an execution error:

   • output a message saying that the tar command was not executed;
   • returning error code 1, indicating that something has gone wrong.

7. Finish the if/then/else script with the phrase fi. View of the completed tarandzip function:

   tarandzip(){
       echo "\n##### Gathering files #####\n"
       if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
           echo "\n##### Done gathering files #####\n"
           return 0
       else
           echo "\n##### Failed to gather files #####\n"
           return 1
       fi
   }

2. Transfer files to object storage⁠

1. Add to the backup script the movetoSpace file transfer function to the selected space. The previously declared variables are used to create a command that will place backup files into the selected space:

   movetoSpace(){
       /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
   }

   Here:

   • /bin/s3cmd — calls s3cmd, a command-line tool used to manage object repository segments;
   • put — is used by s3cmd to load data into the container;
   • $GIVENNAME-$DATETIME.tar.gz — The name of the backup that will be loaded into the space. It consists of the fourth and first declared variables, followed by .tar.gz, and is created by the function tarandzip;
   • s3://$DST; — location where the file will be uploaded;
   • s3:// — URI type scheme used to describe the storage locations of objects in the network, and $DST; — is the third variable declared earlier.

2. Add a notification that the file transfer process has started:

   movetoSpace(){
       echo “\n##### MOVING TO SPACE #####\n”
       /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST
   }

3. Customize the output of the message about the result of command execution:

   if /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
       echo "\n##### Done moving files to s3://"$DST" #####\n"
       return 0
   else
       echo "\n##### Failed to move files to the Space #####\n"
       return 1
   fi

3. Configure flow control⁠

If the script is configured correctly, when it starts, it should read the input command, assign values from it to each variable, execute the tarandzip function, and then execute movetoSpace.

If the script fails at any step, it should print the output of the showhelp function to assist in troubleshooting.

Add an if / then / else conditional instruction to the end of the file:

if [ ! -z "$GIVENNAME" ]; then
    if tarandzip; then
        movetoSpace
    else
        showhelp
    fi
else
    showhelp
fi

4. Check the script⁠

1. Check the script:

   #!/bin/bash
   
   DATETIME=`date +%y%m%d-%H_%M_%S`
   SRC=$1
   DST=$2
   GIVENNAME=$3
   
   showhelp(){
       echo "\n This script will backup files/folders into a single compressed file and will store it in the current folder."
   }
   
   tarandzip(){
       echo "\n##### Gathering files #####\n"
       if tar -czvf $GIVENNAME-$DATETIME.tar.gz $SRC; then
           echo "\n##### Done gathering files #####\n"
           return 0
       else
           echo "\n##### Failed to gather files #####\n"
           return 1
       fi
   }
   movetoSpace(){
       echo "\n##### MOVING TO SPACE #####\n"
       if /bin/s3cmd put $GIVENNAME-$DATETIME.tar.gz s3://$DST; then
           echo "\n##### Done moving files to s3://"$DST" #####\n"
           return 0
       else
           echo "\n##### Failed to move files to the Spac #####\n"
           return 1
       fi
   }
   
   if [ ! -z "$GIVENNAME" ]; then
       if tarandzip; then
           movetoSpace
       else
           showhelp
       fi
   else
       showhelp
   fi

2. Before exiting nano, close the file **(Ctrl+X **) and save your changes ( **Y+Enter **).

5. Optional: automate backups⁠

Crontab

1. Set up a cron job that will use a script to perform regular backups to a selected space. For the purposes of this example, the backup will be performed every minute.

2. Make the script executable:

   chmod +x bkupscript.sh

3. Edit the crontab file so that the script runs every minute:

   crontab -e

4. The first time you run the crontab -e command, you will be prompted to select an editor from the list:

   no crontab for root - using an empty one
   Select an editor.  To change later, run "select-editor".
   1. /bin/ed
   2. /bin/nano        <---- easiest
   3. /usr/bin/vim.basic
   4. /usr/bin/vim.tiny
   Choose 1-4 [2]:

5. Choose the default nano or another text editor.

6. In crontab, add a line at the end of the script:

   * * * * * ~/bkupscript.sh ~/backupthis nameofyourspace cronupload

7. Close the file **(Ctrl+X **) and save your changes ( **Y+Enter **).

8. If you leave the cron job running unchanged, a new file will be copied to the selected space every minute. If cron runs successfully, reconfigure crontab to back up files at the desired interval.

Cyberduck

On the local machine there is a directory whose contents need to be periodically copied to object storage.This can be done using a script and a Cron job that sends a backup copy to the storage every day at a specified time.

Example job (additionally other Cyberduck options can be used):

#!/bin/bash
SERVERCORE_ID=<servercore_account>
USERNAME=<username>
PASSWORD=<password>
BACKUP_PATH=<container_name>/<prefix>
LOCAL_PATH=<path>

duck --upload servercore:/<container_name>/<prefix> <path> --existing rename --username <servercore_account>:<username> --password <password> -q -y

Specify:

• <servercore_account> — Servercore control panel account number;
• <username> — the name of the service user;
• <password> — service user password. If you have forgotten the password, change it;
• <container_name> — container name;
• <prefix> — object prefix;
• <path> — path to the local directory.

Here:

• The --existing key of the duck command specifies what to do with files already in the repository;
• The rename option renames an existing backup by adding the time and date to the name.

Cyberduck options⁠

When automating backups through Cyberduck, you can use additional options:

• compare — perform differential backups:

  duck --upload  servercore:/<container_name>/<prefix> <path> --existing compare --username <servercore_account>:<username> --password <password>

  The uploaded backup will be compared to the existing backup by size, modification date, and checksum. If the parameters are different, the old version will be deleted and the new version will be uploaded to the repository;

• skip — only new files (those that appeared in the folder on the local machine after the previous upload) will be uploaded to the repository. Existing files will not be uploaded, even if they have been changed on the local machine;

• overwrite — removes an existing backup from storage and loads a new one.