Setting up laptop backups using the EECISbak.sh script 1) BACKGROUND This script is designed to back laptops up to the EECIS "smb2.eecis.udel.edu" drive array, though it could be used to back up to any machine with rsync and SSH access. It does not attempt to perform file "versioning", rather, it relies on the underlying backup filesystem to keep older versions of files. For "smb2", it relies on ZFS. This script merely duplicates the files on the local machine to the remote machine. A backup system is, indeally, simple enough to be robust. This script re-uses existing tools to the greatest extent possible. The script is written in "bash", the remote connection is made with "ssh", and the synchronization is performed using "rsync". These are both tools with substantial history in the UNIX arena. OSX Macs have them installed as part of the base system, and they can be installed under Windows using the "Cygwin" environment (described below). The script is intended to be invoked periodically approximately once per hour by the operating-system-dependent periodic-command scheduler ("cron" for Macs and "Task Scheduler" under Windows). The script checks a time-stamp file to see when a backup was last performed, and if it has been long enough (12 hours by default) a backup is made. Laptops are off and on the net, so by trying frequently, there's a good chance that an attempt will eventually be made when the laptop is able to make a net connection to the server. The "rsync" software only transfers the parts of files that have changed so after the first, full copy, updates should be fast even over a relatively slow link. Further, it will do an excellent job of picking up where it left off if it is not able to finish the backup before the laptop is shut down or disconnected from the network. 2) CONFIGURATION All files connected to the script should be kept under the EECISbak directory as directed in installation below. Files with "sys" in the filename should not be modified by end users since updates to this software may overwrite them. The "config" file is "sourced" by the main script and can be used to set the environment for the backup. While complex shell-script logic could be included, it's best to use simple "VARIABLE=value" lines. The "sysconfig" file shows default values (all commented-out with #); if required, this file could be copied to the "config" file and then modified. Configuring which files are included is done by the "include", "sysinclude", "exclude", and "sysexclude" files. Note that "includes" and "excludes" may be listed in any file by using "+ " and "- " respectively (see the sysexclude file for examples). Each file and directory is matched against the list of patterns in these files, and the first match is applied, so if the following two lines are found: + /Users/roosen/ - /Users/* then the directory /Users/roosen and all contents will be included (unless excluded later), and all other contents of /Users/ will be excluded. If the two lines are reordered, however: - /Users/* + /Users/roosen/ Then since /Users/roosen/ will match the first line, it will be excluded. Lines ending in / will only match directories, and lines beginning with / will be matched against the full file pathname. Note for Windows laptops: Files that end in ".txt" use DOS/Windows-style line breaks and those without use UNIX-style line breaks. Under Windows, either may be modified; if you prefer to use Windows-native software for text-file editing, you will probably find the ".txt" versions easier to work with, and the reverse if you prefer to use Emacs or Vi under Cygwin. Changes will be propagated from the .txt to the non-.txt or vice-versa automatically when the EECISbak.sh script runs. 3) INSTALLATION 3.1) Windows 3.1.1) The backup script depends on several "Cygwin" packages. Cygwin is easily obtained and installed, but you must be running as an administrator. If Cygwin is already installed, make sure the required packages have been installed. 3.1.1.1) Begin by going to "cygwin.com" and click on the "Install Cygwin Now" button in the upper-right. 3.1.1.2) Run "setup.exe" to install (or reconfigure) Cygwin. 3.1.1.3) Accept all the defaults (keep clicking "next"). Select "http://mirror.mcs.anl.gov" as the mirror and click "next". 3.1.1.4) In the "select packages" panel, click "view" until it shows the "Full" view. Locate and select "bash", "cygutils", "openssh", and "rsync" are selected for installation. Click "next", and wait. 3.1.2) Download EECISbak.zip from "http://www.eecis.udel.edu/~roosen". Unzip in your folder under "Documents and Settings" so that all the files are stored in a EECISbak folder. Files include: 3.1.2.1) EECISbak.sh: this is the main backup script (bash shell). 3.1.2.2) EECISbak.bat: a DOS script that invokes the above for ease of running directly from Windows. 3.1.2.3) EECISbka.bat: a DOS script that invokes the above for ease of running non-interactively under Windows using "Scheduled Tasks" 3.1.2.4) readme.txt: this help 3.1.2.5) sysconfig, sysconfig.txt: configuration files for use by system staff in "UNIX" and "DOS" formats. Copy to "config" or "config.txt" for your own modifications. 3.1.2.6) sysexclude, sysexclude.txt: rsync exclude files for system staff defaults in "UNIX" and "DOS" formats. 3.1.2.7) crontab: sample crontab entry for use under MacOS. 3.1.2.8) pwget: keychain helper script for use under MacOS. 3.1.2.9) host_key: the backup host's public SSH key. 3.1.3) Run the "EECISbak.bat" script (you can just double-click on it in Windows Explorer). 3.1.3.1) When it finishes the first time, it will write a file called "authorized_keys.txt". Post the contents of that file into a EECIS Help Request ticket asking for authorization to back up. 3.1.4) Once your Help Request has been answered, run the EECISbak.bat once to ensure that backups will go through. 3.1.5) Set up EECISbka.bat to run hourly to insure that regular backups will occur. 3.1.4.1) Select "Start->All Programs->Accessories->System Tools->Scheduled tasks", then double-click "Add Scheduled Task". A "wizard" will start up. Browse to the "EECISbka.bat" file as the program to be run. The basic wizard doesn't let you set it to run hourly, so just say "daily". Otherwise, take all the defaults, and be sure to enter your username and password where asked. In the last page, select "Open Advanced Properties for this Task when I click Finish". 3.1.4.2) In the "Advanced Properties" pane, select the "schedule" tab. Make sure it starts "Daily" at "12PM". Hit the "Advanced" button. Click on "Repeat Task" and set it to "every 1 hour" with a duration of 24-hours. Click "OK". 3.1.4.3) In the "Settings" tab, uncheck both boxes under "Power Mangement". 3.2) MacOS X All required utilities are installed by default. To install: 3.2.1) Download "http://www.eecis.udel.edu/~roosen/EECISbak.zip" and unzip it into your home directory so that the script will be found in "~/EECISbak/EECISbak.sh". Contents are listed in 3.1.2, above. 3.2.2) Modify ~/EECISbak/exclude to fit your needs (this is just a text file). 3.2.3) Open a terminal (Applications->Utilitied->Terminal.app) and run "~/EECISbak/EECISbak.sh". The first time it is run, it will generate an SSH key-pair and construct an authorized_keys.txt file that will need to go onto the server. 3.2.4) Open a help-request ticket at https://www.eecis.udel.edu/service with the subject, "EECIS laptop backup setup" and post the contents of "authorized_keys.txt" into the description. If you have more than one laptop to back up, note that they need to be given different names in the server authorized_keys. 3.2.5) Edit the "EECISbak/crontab" file to reflect the location of the EECISbak directory. If you already have a crontab file (run "crontab -l" to check), add the line in EECISbak/crontab to it. Otherwise, run "crontab ~/EECISbak/crontab". 3.2.6) Once your help request has been answered, run EECISbak.sh again ("by hand") to make sure all is well. You may be asked to accept an SSH key from smb2: do so. 3.2.6) Logs will be created in "EECISbak/log-YYYYMMDDHHMM" (marked by the date and time). Check these logs periodically to ensure backups are actually taking place. 4) KNOWN PROBLEMS 4.1) Under Windows, a window pops up every time the .bat file is run even if there is no output. I'm sure there's a better way to handle running the script that doesn't do that. 4.2) "cron" is outdated for MacOS: "launchd" is the "right way to do it".