Linux Backup Script(s)

using 'rsync'

sample code for
home directory backup

Home > RefInfo menu >

Computer topics menu (Linux section) >

This (old) 'rsync' Backup Scripts page
(scripts used from 2010 to 2019)

! Note !
A newer web page (circa 2019-plus) on
rsync-backup scripts is available.
See the scripts on that page.

Sections Below:

INTRO     RSYNC-SCRIPT(2010apr)     RSYNC-SCRIPT(2010dec)

CLEANUP-SCRIPT(2010oct)     CLEANUP-SCRIPT(2012sep)

BACKUP-DRIVE-PREP(2010oct)     BACKUP-DRIVE-RECOVERY(2010dec)

INFO-SEARCHES     BOTTOM-of-PAGE

INTRODUCTION :

Around 2010, I migrated my files (email, web browser bookmarks, documents, photos, etc.) from a Microsoft Windows personal computer to a home-built computer using Linux (Ubuntu 9.10, 'Karmic Koala').

Around that time, I happened to read about using the Linux 'rsync' command to backup files.

    (Reference:
    Linux Format Magazine #130, Apr 2010, page 105, in the Answers section.
    Although that article may not be available on the internet, you could try a web search for it or for similar articles.)

That 'Answer' showed that the form of the 'rsync' command for backing up files was rather simple:

  rsync   --archive   --delete   /path/to/source/   /path/to/dest/

For my applicaton, the 'source' directory is my home directory --- and the 'destination' directory is a directory on an external USB storage drive.

    NOTE: You need the slash at the end of the source and destination directory names if they already exist. If you leave off the slash, rsync tries to make a new sub-directory with that last sub-directory name.

In my case, I could simply use

    /home/

as the entire 'source' directory name --- which would backup my home directory and any other home directory under the '/home' directory.

And (in the Ubuntu operating system), I could use for the 'destination' directory a name like

    /media/VolumeName/home/

    or

    /media/userid/VolumeName/home/

The '--delete' option removes files not present under the 'source' directory --- that is, it removes files from the 'destination' directory that are no longer in the 'source' directory.

The '--archive' option causes all file permissions and timestamps to be copied.

    See the comments (below), in the backup script(s), for a more detailed description of the 'rsync' parameters being used --- such as '--archive' and '--delete'.

If you start with an empty 'destination' directory (say, on an external USB storage drive), the first run of the 'rsync' command would copy ALL the files from 'source' to 'destination'.

In subsequent runs, this form of the 'rsync' command copies only files that are different.

With large files that have changed, it only copies the parts that have been altered.

So this form of the 'rsync' command is said to do 'incremental backups' --- copying only new files and files that have been changed, to the 'destination' directory --- and deleting files from the 'destination' directory that have been removed from the 'source' directory.

I devised the following 'rsync' scripts, for my main desktop computer and for some netbook computers, based the 'rsync' info summarized above.


When to run the scripts:

When I am doing a lot of web-page or shell-script or Tcl/Tk-script development (almost daily), I may run the 'rsync' script almost daily --- to back up the file changes and new files in my home directory to an external USB disk drive.

Otherwise, if I have changed only a few non-critical files, I may run the 'rsync' script after a week or two.

But BEFORE each 'rsync' run, I run a 'file cleanup' script, in order to get rid of an accumulation of un-needed 'thumbnail' files and web browser files, such as 'Crash reports' and 'minidumps'.


A case when I really needed the backups:

In October 2010, when I fried my video board (and, as I later found out, my CPU) on my main desktop computer when investigating a noisy fan, my 'rsync' backup really saved me.

    A short rant on the fried-board:

    Lousy EVGA Nvidia 7600GS board! Googling revealed that many others have had problems with a noisy fan on this board --- sometimes only a few months after installing the board. Many reported burst capacitors on this board --- and the board kept running, until one too many capacitors blew. My board lasted for about 3 years --- progressively noisier for about the last year of those three.

    It appears that the screws holding a fan onto the board were working their way loose, and the fan was touching the screws. The noise got so bad I could not ignore it anymore. I tried to access the screws to tighten them but could not. Somewhere in the process, it appears that one-too-many capacitors blew.

    I am a fan of fanless video boards now --- pun intended. I do not need a fan-equipped, high-end video board, since I am not a video game fan --- pun intended.

I was able to restore about 45 Gig of backed up data to some of my other computers --- which I used until my desktop CPU and video were replaced/repaired.

I only wish I had backed up more frequently.

After that incident, I made a backup rsync-script an icon on my desktop and I run the script just before I logoff --- especially after login sessions in which I change some of the files in my web site directories and software development directories.


Image of the 'rsync' backup script icon.

Having the backup script as an icon on my desktop --- near a logoff icon --- makes it very easy for me to remember to 'kick off' the backup script.

    (I also run a file-cleanup script before running the 'rsync' script. The 'fileCleanup' script is described below.)


How the scripts are presented, below:

The scripts are presented in two ways below:

  • as a link to a text file
    (click on the link and the file shows in a separate browser window)

  • as a scrollable block of text shown in this window
    (implemented by using the HTML 'object' tag)

You can see the comments, in the backup scripts below, for some details of the parameters being used in the 'rsync' and 'rm' and 'read' commands being used in the scripts.

And you can see the exact way that the directories are being specified in those commands.

    The 'read' command is used here to pause the execution of a script.


If you want to download any of these scripts for use, you can right-click on the '.txt' file link and choose an option such as 'Save Link Target As ...' in the popup menu that appears from your web browser.

After downloading one of these scripts, you can remove the '.txt' suffix from the file name and leave the '.sh' suffix.

Remember to change the permissions on the file to make the file 'executable'.

You can use a command like 'chown 755 filename' or, in a GUI file-manager like MATE-Caja, you can right-click on the filename and choose the 'Properties' option from the menu that pops up. Then open the 'Permissions' panel and click the checkbutton titled 'Allow executing file as program'.

LINK TO SCRIPT 00_rsync_bkup_2maxtor.sh.txt
for backing up home directory, with 'rsync'.


(2010 Apr version, *OLD*;
There is a newer, 2010 Dec version below ---
to show 'rsync' messages in a terminal window)

Alternate display of the old 2010-Apr rsync script:

Link to script

UPDATE:
(2010 Dec version of the rsync script above)

    This new version allows for starting up 'rsync' from a desktop icon. This version starts up 'rsync' in a terminal window --- using the 'gnome-terminal' command --- so that backup messages can be seen in the terminal.

    This helps assure that the proper (recently changed) files are being backed up, and one can see when the backup is finished (when the messages stop and a summary is shown).

    This script uses a little 'start_sh_after_exe.sh' script whose code is shown below this 2010-Dec rsync script.

    For command-syntax details, see comments and code in this rsync script.

LINK TO SCRIPT 00_rsync_bkup_2maxtor.sh.txt
for backing up home directory, with 'rsync'.


(newer, 2010 Dec version --- using
'gnome-terminal' to start the 'rsync')

Link to script

Below is the 'start_sh_after_exe.sh' script code, which is used to hold the terminal window open after 'rsync' runs.


#!/bin/sh
set -x
$@
set -
/bin/bash

I found a forum thread on the internet where someone said they use this technique to cause the 'gnome-terminal' to stay open after executing the command string. (In fact, not only does it stay open, but it can accept additional commands at the command prompt.) (I added the 'set -x' and 'set -' commands --- to show the command string being executed --- the '$@'.)

FILE CLEANUP BEFORE BACKUP :
(A 2010 Oct UPDATE on the 'rsync' backup process before logoff)

After doing several backups just before logoff (via the 'backup_2maxtor' desktop icon shown above), I saw, in the messages from 'rsync', that 'rsync' was repeatedly backing up files from browser 'cache' directories --- and thumbnail image files from a $HOME/.thumbnails directory.

This was a waste. I did not need these files/subdirectories backed up.

    (I have not traced the source of all of the thumbnail files. I have set my Nautilus preferences to NOT show thumbnail icons of image files. However, Nautilus seems to continue making thumbnails, in some cases. The Gnome2 'SaveAs...' dialog, which includes a GUI directories-and-files navigator, seems to be making some of the thumbnail files. It also appears that some image programs, such as the 'Eye of Gnome' ('eog') image viewer program, makes some of the thumbnail files.)

I devised the following 'files-cleanup' script to show me the removal of files from such directories.

LINK TO SCRIPT 00_fileCleanup.sh.txt

(OLD 2010-2011 Version for 'file-cleanup';
newer 2012-Sep version is below)

Alternative display of the file-cleanup script:

Link to script

UPDATE - 2012 Sep Version of 'file-cleanup' script:

    This version uses just one terminal window, instead of one terminal for each of several directories to be cleaned.

    Since the '-e' option of this terminal command is not friendly to a lot of separate commands, the terminal-startup command calls on a separate script, named '00_fileCleanup_ver2_RunInTerminal.sh', below, to do the file cleanup.

    For command-syntax details, see the comments and code in the following two scripts.

LINK TO SCRIPT 00_fileCleanup_ver2.sh.txt

(newer, 2012-Sep version;
a 'wrapper' script)

Alternative display of the file-cleanup 'wrapper' script:

Link to script

LINK TO SCRIPT 00_fileCleanup_ver2_RunInTerminal.sh.txt

(a 'core' script to be called by
the 'wrapper' script above)

Alternative display of the file-cleanup 'core' script:

Link to script

Of course, if I find that I do not need to clean one of these directories, I can comment-out that cleanup code. And I could add other directories to cleanup.

    I hardly ever use Firefox (I use Seamonkey instead), so in a perfect world I would not need to clean out the Firefox cache directories. But there are some applications which startup Firefox (to show help files or whatever), so I keep the Firefox cache directory cleanup code even though it usually finds nothing to delete.

    You might want to tell me that one can clean the Firefox cache directory by setting cache-cleanup-at-exit in Firefox preferences. But I find that the Firefox (and Seamonkey) cleanup is not always complete.

    The reason for the two Seamonkey directories: I never know when a version of Seamonkey is going to use the $HOME/.seamonkey directory or the $HOME/.mozilla/.seamonkey directory. There has been some confusion in use of these directories in a previous Seamonkey 1.x release. (The $HOME/.seamonkey directory does not seem to be used by Seamonkey2.)

To help remind myself to run the file cleanup script before running the rsync backup script (via the desktop icon), I added an icon to the desktop (above the rsync backup icon) to remind myself to do the file cleanup before the backup. Picture follows.

Sometimes I run the 'file-cleanup' script without running the 'backup' script, just before logging off. Hence I keep the two scripts separate, rather than combining them into one 'monolithic' script.

PREPARING AN EXTERNAL BACKUP DRIVE :
(2010 Oct)

(an example of prepping a backup drive)

As the backup directory name (/media/Maxtor160GB) in the rsync script above indicates, the backup drive on that computer (a home-assembled PC) was an external Maxtor 160 Gigabyte drive.

In October 2010, I prepared a 1 Terabyte Seagate Freeagent external drive for use as a backup drive --- to be used with an 'rsync' script on another desktop computer of mine --- a Cyberpower PC that I bought in 2005 from cyberpowerpc.com.

After frying the video board on my main desktop computer, I installed Ubuntu 9.10 on the Cyberpower computer in October 2010 --- overlaying an old Xandros Linux installation, as I described in a web page on installing Ubuntu 9.10 over Xandros Linux, while preserving a dual-boot of MS Windows --- and creating a separate partition for the home directory.

I took this opportunity to take notes on how I prepared this Seagate drive as a backup drive for a Linux system.

A description of the drive preparation process, for this 1 Terabyte Seagate drive, follows.


  1. I plugged in the USB and power cords of the disk drive.

  2. A 'Freeagent Drive' icon showed up on my Ubuntu desktop.

  3. I right-clicked the icon and chose 'Open'. I saw a bunch of Seagate files (for MS Windows Seagate software) --- about 0.5 Gig of directories and files. To backup those files, in case I ever had use for them, I copied those files to my home directory --- into new directory 'DISKseagateFreeagent'.

  4. Then I right-clicked the Seagate drive icon and chose 'Format...'.

      UPDATE NOTE:
      This 'Format...' option may no longer be seen when you right-click on a drive-name or drive-icon. Around 2019, I started using Ubuntu-MATE on my desktop and laptop computers. It uses the MATE desktop environment and the Caja file manager. When you right-click on a drive in MATE, the 'Format...' option does not appear. If you have the 'Applications-Places-System' triplet on your MATE 'top-panel', you can probably use 'System > Preferences > Hardware > Disks' to use the 'gnome-disk-utility' to partition-and-format a drive.

    In the Format GUI, I chose an 'ext3' file system in the 'Type:' drop down menu, and I specified 'SEAGATEfreeagent1000G' at the 'Name:' entry field.

    I clicked on the 'Format' button. It took about 12 to 15 minutes to format the 1 Terabyte drive.

    When the formatting was done, I right-clicked the drive icon and checked 'Properties'. The volume name was shortened to 'SEAGATEfreeagent' --- 16 characters. Apparently, that was the maximum length allowed for the volume name.

    The 'Properties' showed that about 850 Gig was available after formatting the 1 Terabyte drive. (There is about 150 Gig of overhead for the ext3 file system??)

  5. I prepared an rsync script in $HOME/Rsync on the Cyberpower PC --- script name: 00_rsync_bkup_2freeagentSEAGATE.sh

    The one command in that script was (like the original 'rsync' script above):


gnome-terminal -e \
   rsync --archive --delete -vv --modify-window=1 \
   /home/blaze/ \
   /media/SEAGATEfreeagent/CyberpowerPC/home/blaze/

  1. I ran the script. I got 'file does not exist' errors. I had to make the 'to' subdirectories --- CyberpowerPC/home/blaze/ --- on the Seagate drive.

    I right-clicked the Seagate drive icon and chose 'Open'. In the Nautilus file manager that opened, I made a 'CyberpowerPC' directory with 'File > Create Folder'. I opened that directory and made a 'home' directory with 'File > Create Folder'. I opened that directory and made a 'blaze' (username) directory with 'File > Create Folder'.

  2. I ran the script again --- in a terminal window positioned in the directory where I had the script --- $HOME/Rsync.

    The names of files being archived for the first time streamed down the terminal window. There are about 45 Gig of files in the home directory. The copying went on for about 35 minutes. That's a rate of about 45/35 = 1.3 Gig per minute.

  3. I changed the parameter '-vv' to '-v' to speed up the running of the script. '-vv' shows a list of all files being processed, which is many thousands in my 45 GB of files. '-v' just shows the files being deleted and the files being backed up, in the backup directory.

    I was now prepared to run this rsync script almost every day, at the end of a long file-generating and/or file-modifying login session.

    I made a desktop icon for the script and positioned the icon under the Shutdown drop down menu at the right end of the top Gnome panel.

    The 'gnome-terminal -e' part of the command above is intended to cause a terminal window to open, in which I can watch the names of files being archived --- so that I can follow the progress of the backup --- and shutdown the computer when it is done.

    Since I generally have less than 1 Gig of file changes at the end of a login session, I expect the rsync script to run for less than a minute, in most cases. (It turns out it usually runs for a little more than one minute, since it still has to check all 45 Gig of files for differences.)


Actually, the 'gnome-terminal -e' command in step 5 above is a simplification of what I actually use.

I would like the gnome-terminal to stay open after the rsync command executes, so that I can examine the list of files being archived. But 'gnome-terminal' does not support a '-hold' option like 'xterm' does.

I could use 'xterm -hold -bg black -fg white -e'. But, since I prefer the scroll bar of 'gnome-terminal', I use the following command instead.


gnome-terminal -e "$HOME/Rsync/start_sh_after_exe.sh \
   rsync --archive --delete -v --modify-window=1 \
   /home/blaze/ \
   /media/SEAGATEfreeagent/CyberpowerPC/home/blaze/"

where the script 'start_sh_after_exe.sh' contains code shown in an rsync-script section above.


LINK TO SCRIPT 00_rsync_bkup_2freeagentSeagate.sh.txt
the complete 'rsync' script that I use with a Seagate drive.


(complete with the comments that I use
to help remind myself of some things about
doing backup with the 'rsync' command)

    This script uses the 'start_sh_after_exe.sh' script, which is shown in a section above.

Alternative display of the rsync-to-Seagate-drive script:

Link to script

RECOVERY FROM A 'BOTCHED'
BACKUP DISK FOR A NETBOOK :

(A WHOOPS moment --- a few panicky minutes, 2010dec02.
Use of 'fdisk' & 'Gparted' to avoid a 'bricked' USB drive.

After setting up external backup disks for a couple of desktop computers, I decided to set up an external backup disk for an Acer netbook computer on which I installed Ubuntu Linux 9.04 in mid 2009, as described on my Ubuntu Installs web page. I bought a Segate Goflex 320BG portable hard drive at BestBuy.

I connected the drive to a USB port of my netbook and it was automatically recognized by the Ubuntu Linux installation on my netbook. A drive icon appeared on the desktop. I right-click opened the drive to check the contents.

There are about half a Gig of files on most Seagate drives --- mostly Microsoft Windows executables, which I did not need on my Acer netbook on which I had installed Ubuntu Linux 9.04 over the MS Windows OS that came on the netbook.

    (By the way, those articles that keep appearing that say Linux is dead on netbooks --- I strongly disagree. I am not the only one who buys a netbook with Microsoft Windows on it and overlays that MS installation with a Linux install. I have several Acer netbooks on which I overlaid MS Windows installations with Ubuntu Linux installations.)

I moved the Seagate files into a new home subdirectory, just in case there were a file (like a documentation file) that I might need later.

The drive label was a rather generic name like 'Seagate', but I like to use a more descriptive label, like 'SeagateGoflex320GB' that shows the capacity of the drive. So I 'right-clicked' on the drive icon and clicked on the 'Properties' line in the menu that popped up.

I was concerned that the Format option was grayed out. I did not know how I would format the drive from NTFS to a Linux format. But I proceeded to try to change the label. I entered a new name for the drive in a blank entry field that showed up. I unmounted the drive and then tried reconnecting it. WHOOPS! I got the following error in a popup window:

    Cannot mount volume.
    Error org.freedesktop.Hal.Device.UnknownError
    libhal.c 1399: wrong reply from hald.
    Expecting an array.
    org.freedesktop.Hal.Device.Volume.UnknownFailure.

Apparently the Hal (Hardware Abstraction Layer) daemon did not like what Nautilus (the Gnome2 file manager) and I did to the drive. I had a mild panic attack. It seemed that I had made a paper-weight out of this new disk drive.

I Googled keywords like 'usb external hard drive ubuntu'.

On an Ubuntu help page on USB mounts, I found a hint:

Use 'sudo fsdisk -l' to show the disks.

With this command (in a terminal window on my Acer netbook), I saw that /dev/sdb1 was recognized as being a 320GB drive --- in addition to /dev/sda, the 160GB internal drive. The fact that the drive could be detected gave me hope that this mount problem could be overcome.

I saw a web page that suggested using Gparted to fix the label. I used the Ubuntu Software Center to install Gparted on the Acer netbook.

Then I used Gparted (which recognized the existence of the /dev/sdb1 device) to reformat the NTFS-formatted external drive to ext3.

I set the volume label to 'SeagateGoflex320'. (I entered 'SeagateGoflex320GB' but, apparently, only the first 16 characters were used.)

Then I was able to mount the external drive by unplugging the USB connection and reconnecting.

I felt much relieved at this point. I could use the drive for something other than a paper-weight.

I then used the sample scripts above to setup a 'fileCleanup' icon and a 'bkup_2Seagate320GB' icon on my desktop, just below the LogOff/Shutdown button on the right of the Gnome2 'top-panel'.

Now, before I click on the button for the options LockScreen/Suspend/Hibernate/Restart/Shutdown --- to shutdown the netbook --- if I have edited some files in my home directory, I click on the 'fileCleanup' icon to cleanup files before the backup.

Then I connect the backup drive and I use the 'bkup' icon to rsync-backup my files.

Works like a charm --- after that initial mount-problem scare. Thank you Google developers, Ubuntu developers and users, fdisk developers, and Gparted developers.

In addition to being thankful for my family, I felt I had to add these thank-you's in the 2010 Thanksgiving season in which this near-fiasco occurred.

FOR MORE INFORMATION:

To find more information on using 'rsync' for backups (and for info on related file backup topics), you can try WEB SEARCHES on keywords such as the following.

You can also try Wikipedia pages like the following, and following links on those pages for even more information.

Bottom of this
Linux Backup Script(s) - using 'rsync' page.

To return to a previously visited web page location, click on the Back button of your web browser a sufficient number of times.

OR, use the History-list option of your web browser.

OR ...

< Go to Top of Page, above. >

Page history:

Page was created 2010 Oct 14.

Page was changed 2014 Dec 09.

Page was changed 2018 Sep 02.
(Added css and javascript to try to handle text-size for smartphones, esp. in portrait orientation.)

Page was changed 2019 Jan 05.
(Put the object-tag sections in separate table-sections from the text above and below the object-tag section, for better width-management of the various sections. Also added code-tags in the pre-tag sections, for font control.)

Page was changed 2021 Jun 23.
(Added a link to a newer, circa-2019-plus web page on rsync scripts. Also reformatted this page with links to 9 sections of the page.)