Nerd:RSYNC Backup
From ASD Wiki
[edit]
Target Backup Server
The target backup server is located currently to the left of Chris's Desk. The IP address is 10.0.0.20 and it's running SuSE Linux Enterprise Server 10 (SLES10). There are two main functions of this server:
- Target for all of the other servers to send backups
- Run scripts to archive files daily
Basically what happens is a source SERVER will send files/volumes to the TARGET on a repetitive basis (depending on the frequency that each server's SYS:\ETC\CRONTAB designates). On the TARGET, the /etc/rsyncd.conf has a list of sessions that are allowed connection. On the source SERVER, the scripts listed below send all new information to a folder in the TARGET path -- /media/rsync/[SERVER]/current. Then, early each morning, the TARGET /etc/crontab file starts scripts which copy the content into a new date-specific folder. Actually, it creates hard links to all existing, non-changed files and copies new/changed files. This keeps up with the latest files, but also allows you to grab files from a specific date while using less storage space.
[edit]
CRONTAB
The ASDSYNC crontab is as follows:
SHELL=/bin/sh PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin MAILTO=root # # check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly # -*/15 * * * * root test -x /usr/lib/cron/run-crons && /usr/lib/cron/run-crons >/dev/null 2>&1 59 * * * * root rm -f /var/spool/cron/lastrun/cron.hourly 14 4 * * * root rm -f /var/spool/cron/lastrun/cron.daily 29 4 * * 6 root rm -f /var/spool/cron/lastrun/cron.weekly 44 4 1 * * root rm -f /var/spool/cron/lastrun/cron.monthly 0 23 * * 1 root /usr/local/bin/asdcomm_dailybackups.sh 2 23 * * 1-5 root /usr/local/bin/asdapps1_dailybackups.sh 4 23 * * 1-5 root /usr/local/bin/ahsserv1_dailybackups.sh 6 23 * * 1-5 root /usr/local/bin/amsserver_dailybackups.sh 8 23 * * 1-5 root /usr/local/bin/apsserv1_dailybackups.sh 10 23 * * 1-5 root /usr/local/bin/adminserver_dailybackups.sh 12 23 * * 1-5 root /usr/local/bin/aisserv1_dailybackups.sh 14 23 * * 1 root /usr/local/bin/asdapps2_dailybackups.sh 16 23 * * 1-5 root /usr/local/bin/asdtemp_dailybackups.sh 18 23 * * 1-5 root /usr/local/bin/asdlin2_dailybackups.sh 19 23 * * 1-5 root /usr/local/bin/asdalms_dailybackups.sh 20 23 * * 1-5 root /usr/local/bin/apps1hscl_dailybackups.sh 21 23 * * 1-5 root /usr/local/bin/apps1mscl_dailybackups.sh 22 23 * * 1-5 root /usr/local/bin/apps1iscl_dailybackups.sh 23 23 * * 1-5 root /usr/local/bin/apps1pscl_dailybackups.sh 24 23 * * 1-5 root /usr/local/bin/admincl_dailybackups.sh 25 23 * * 1-5 root /usr/local/bin/apps1distcl_dailybackups.sh 26 23 * * 1-5 root /usr/local/bin/apps2jlccl_dailybackups.sh 27 23 * * 1-5 root /usr/local/bin/commcl_dailybackups.sh 28 23 * * 1-5 root /usr/local/bin/webcl_dailybackups.sh 29 23 * * 1-5 root /usr/local/bin/trustees_dailybackups.sh 30 23 * * 4 root /usr/local/bin/saveourbutts.sh
The saveourbutts.sh script backs up the current folders to USB. This happens every Thursday night at 11:30pm.
[edit]
Backup Scripts
The backup scripts all look like the following but have the base directory relative to the path it's backing up:
[edit]
Simple Script - Incremental Daily Backup
#!/bin/bash # COMMCL backup rotation script # BACKUPBASE=/media/rsync/[FOLDER] BACKUPDIR=current DATEDIR=`date +%Y-%m-%d-%a` mkdir $BACKUPBASE/$DATEDIR cp -Rl $BACKUPBASE/$BACKUPDIR/* $BACKUPBASE/$DATEDIR/
[edit]
Advanced Script - Time-sensitive Incremental Daily Backup
#!/bin/bash
# SERVER backup rotation script
LOGFILE=/var/log/shiftday.log
LOCAL_TARGET=/media/rsync/[FOLDER]
FOLDER0=current
FOLDER1=`date +%G-%m-%d-%a`
DISK_MOUNTPOINT=/media/rsync/
THRESHOLD=90
MINBACKUPS=31
FILESPEC=20*
echo ---------------------------- >> $LOGFILE
echo Creating backup for $FOLDER1 >> $LOGFILE
while [[ `df -k | grep $DISK_MOUNTPOINT |awk '{print $5}' |cut -d% -f1` -gt "$THRESHOLD" ]]
echo Checking for necessary disk space...
do
if [[ `ls -atdr $LOCAL_TARGET/$FILESPEC 2> /dev/null | wc -l` -gt $MINBACKUPS ]]
then
DELDIR=`ls -atd /$LOCAL_TARGET/$FILESPEC | tail -n 1`
echo Removing $DELDIR to make room on disk.
echo Removing $DELDIR to make room on disk. >> $LOGFILE
rm -rf $DELDIR
else
break
fi
done
echo Checking for copy of todays Folder...
if [ -a $LOCAL_TARGET/$FOLDER1 ]
then
rm -rf $LOCAL_TARGET/$FOLDER1/*
else
mkdir $LOCAL_TARGET/$FOLDER1
fi
echo Copying $FOLDER0 to $FOLDER1 on `date`>> $LOGFILE
echo Copying $FOLDER0 to $FOLDER1
cp -lurp $LOCAL_TARGET/$FOLDER0/* $LOCAL_TARGET/$FOLDER1
[edit]
/etc/rsyncd.conf
This configuration file is what allows connection from other source servers to the target. It looks like this:
gid = root
read only = false
use chroot = false
transfer logging = true
log format = %h %o %f %l %b
log file = /var/log/rsyncd.log
hosts allow = *.almasd.net
path = /media/rsync
uid = root
secrets file = /etc/rsyncd.secrets
#[Example]
# path = /home/Example
# comment = An Example
# auth users = user
# secrets file = /etc/rsyncd.secrets
[COMMCL]
comment = CLUSTER COMMCL SYS/PUBLIC/SNAPSHOT
hosts allow = *
[AHSSERV1]
comment = LOCAL FILES, BUSINESS_ED, MULTIMEDIA, APPS
hosts allow = 10.0.2.1
[APPS1PSCL]
comment = CLUSTER PS ADM
hosts allow = *
[APPS1HSCL]
comment = CLUSTER HS ADM
hosts allow = *
[APPS1ISCL]
comment = CLUSTER IS ADM
hosts allow = *
[APPS1MSCL]
comment = CLUSTER MS ADM
hosts allow = *
[AMSSERVER]
comment = LOCAL FILES, KPAWS, APPS
hosts allow = 10.0.0.200
[APSSERV1]
comment = LOCAL FILES, APPS
hosts allow = 10.0.4.1
[AISSERV1]
comment = LOCAL APPS, FILES, ADM
hosts allow = 10.0.1.1
[ADMINCL]
comment = CLUSTER ADM, FILES, APPS
hosts allow = *
[WEBCL]
comment = CLUSTER WEB FILES
hosts allow = *
[asdlin2]
comment = mysql backup + /srv/www.tar
hosts allow = 10.0.0.23
[asdalms]
comment = alms sql & data backup
hosts allow = 10.0.0.18
[APPS2JLCCL]
comment = JLC Cluster Backup
hosts allow = *
[APPS1DISTCL]
comment = ZEN app installs for District - Login Includes
hosts allow = *
[TRUSTEES]
comment = Copy trustees over from SERVERS to TARGET
hosts allow = *
[edit]
Regular Servers Currently Backed Up Daily
Scheduled Every Weekday @ XX:XX (Staggered throughout the evening hours from about 10p-12a)
[edit]
AHSSERV1 (10.0.2.1)
- FILES
- home dirs for all students/staff
- MULTIMEDIA
- holds mm students projects... M: drive
- APPS
- Mostly HS application install source for business ed, and all general/public apps
- Card Catalog application
- BUSINESS_ED
- yearbook?
- newspaper?
- FILES
[edit]
AMSSERVER (10.0.0.200)
- FILES
- Teacher & Students Home directories
- APPS
- MS specific application install source and all general/public apps
- Library Card Catalog app
- KPAWS
- KPAWS files for production
- FILES
[edit]
AISSERV1 (10.0.1.1)
- FILES
- Teacher Home Directories
- APPS
- IS Specific install source for applications
- Library card catalog stuff
- ADM
- Should be unused, but keeping it just in case. Shouldn't increase.
- FILES
[edit]
APSSERV1 (10.0.4.1)
- FILES
- Teacher Home Directories
- APPS
- Library Card Catalog stuff and all general/public apps
- FILES
[edit]
ASDLIN2 (in VM) (10.0.0.23)
!!! scripts located in /usr/local/bin/ !!!
- DBBACKUP
- current databases
- dragonfly
- drupal
- test
- counter
- wikidb
- WWWBACKUP
- current /srv/www/* in a gzipped tarball
- DBBACKUP
[edit]
ALMS (10.0.0.19)
- Database
- Other stuff... chris check to see
[edit]
Cluster Server Volumes Currently Backed Up Daily
[edit]
ASDAPPS1HSCL (High School Gradequick)
- /hs/apps - gradequick
- /hs/SharedDocs - HS O: drive
- /hs/USERS - HS ADM home folders
[edit]
ASDAPPS1MSCL (Middle School Gradequick)
- /ms/apps - gradequick
- /ms/Shared - MS O: drive
- /ms/USERS - MS ADM home folders
[edit]
ASDAPPS1ISCL (Intermediate School Gradequick)
- /apps - gradequick
- /SharedDocs - IS O: drive
- /USERS - IS ADM home folders
[edit]
ASDAPPS1PSCL (Primary School Gradequick)
- /apps - gradequick
- /Shared - PS O: drive
- /USERS - PS ADM home folders
[edit]
ADMINCL (Administration Files [O: drive/ADM Home Dirs])
- /Adminbd - O: for Administration
- /USERS - Home Dirs for Administration
[edit]
APPS2JLCCL (JLC Backup)
- /jlc1 - JLC Data
[edit]
COMMCL (Z Drive and SNAPPSHOT Files)
- /public - Z public
[edit]
WEBCL (Local District Website)
- /Apache2 - mainly for conf files
- /webs - all website content
[edit]
APPS1DISTCL (District Application Installs)
- /District - Application Installs
- /login_includes - Dist. login scripts
- /ADMIN - Administration G drive
- /trustees - District volume trustee .csv files.
[edit]
CLUSTER NCF FILES
- CLRSYNC1.NCF
#Copy all APPS1ISCL volume files and force rewrite apps/gqwin everytime sys:\rsync\rsync --volume=APPS1ISCL: -avz --ignore-times --bwlimit=0 --delete /apps 10.0.0.20::APPS1ISCL/APPS1ISCL/current/ --port=873 --progress sys:\rsync\rsync --volume=APPS1ISCL: -avz --whole-file --bwlimit=0 --delete /SharedDocs 10.0.0.20::APPS1ISCL/APPS1ISCL/current/ --port=873 --progress sys:\rsync\rsync --volume=APPS1ISCL: -avz --whole-file --bwlimit=0 --delete /USERS 10.0.0.20::APPS1ISCL/APPS1ISCL/current/ --port=873 --progress
- CLRSYNC2.NCF
#Copy all //asdapps1/adm/MS/* and force rewrite apps/gqwin everytime sys:\rsync\rsync --volume=APPS1MSCL: -avz --ignore-times --bwlimit=0 --delete /ms/apps 10.0.0.20::APPS1MSCL/APPS1MSCL/current/ --port=873 --progress sys:\rsync\rsync --volume=APPS1MSCL: -avz --whole-file --bwlimit=0 --delete /ms/Shared 10.0.0.20::APPS1MSCL/APPS1MSCL/current/ --port=873 --progress sys:\rsync\rsync --volume=APPS1MSCL: -avz --whole-file --bwlimit=0 --delete /ms/USERS 10.0.0.20::APPS1MSCL/APPS1MSCL/current/ --port=873 --progress
- CLRSYNC3.NCF
#Copy all //asdapps1/adm/IS/* and force rewrite apps/gqwin everytime sys:\rsync\rsync --volume=APPS1HSCL: -avz --ignore-times --bwlimit=0 --delete /hs/apps 10.0.0.20::APPS1HSCL/APPS1HSCL/current/ --port=873 --progress sys:\rsync\rsync --volume=APPS1HSCL: -avz --whole-file --bwlimit=0 --delete /hs/SharedDocs 10.0.0.20::APPS1HSCL/APPS1HSCL/current/ --port=873 --progress sys:\rsync\rsync --volume=APPS1HSCL: -avz --whole-file --bwlimit=0 --delete /hs/USERS 10.0.0.20::APPS1HSCL/APPS1HSCL/current/ --port=873 --progress
- CLRSYNC4.NCF
#Copy all //asdapps1/adm/IS/* and force rewrite apps/gqwin everytime sys:\rsync\rsync --volume=APPS1PSCL: -avz --ignore-times --bwlimit=0 --delete /apps 10.0.0.20::APPS1PSCL/APPS1PSCL/current/ --port=873 --progress sys:\rsync\rsync --volume=APPS1PSCL: -avz --whole-file --bwlimit=0 --delete /Shared 10.0.0.20::APPS1PSCL/APPS1PSCL/current/ --port=873 --progress sys:\rsync\rsync --volume=APPS1PSCL: -avz --whole-file --bwlimit=0 --delete /USERS 10.0.0.20::APPS1PSCL/APPS1PSCL/current/ --port=873 --progress
- CLRSYNC5.NCF
#Copy all //asdapps1/adm/IS/* and force rewrite apps/gqwin everytime sys:\rsync\rsync --volume=ADMINCL: -avz --whole-file --delete /Adminbd 10.0.0.20::ADMINCL/ADMINCL/current/ --progress sys:\rsync\rsync --volume=ADMINCL: -avz --whole-file --delete /Users 10.0.0.20::ADMINCL/ADMINCL/current/ --progress
- CLRSYNC6.NCF
#Copy all //asdapps1/adm/IS/* and force rewrite apps/gqwin everytime sys:\rsync\rsync --volume=WEBCL: -avz --whole-file --bwlimit=0 --delete /webs 10.0.0.20::WEBCL/WEBCL/current/ --port=873 --progress sys:\rsync\rsync --volume=WEBCL: -avz --whole-file --bwlimit=0 --delete /Apache2 10.0.0.20::WEBCL/WEBCL/current/ --port=873 --progress
- CLRSYNC7.NCF
#Copy all //asdapps1/adm/IS/* and force rewrite apps/gqwin everytime sys:\rsync\rsync --volume=APPS2JLCCL: -avz --whole-file --bwlimit=0 --delete /jlc1 10.0.0.20::APPS2JLCCL/APPS2JLCCL/current/ --port=873 --progress sys:\rsync\rsync --volume=COMMCL: -avz --whole-file --bwlimit=0 --delete /public 10.0.0.20::COMMCL/COMMCL/current/ --port=873 --progress
- CLRSYNC8.NCF
#Copy all //asdapps1/adm/IS/* and force rewrite apps/gqwin everytime sys:\rsync\rsync --volume=APPS1DISTCL: -avz --whole-file --bwlimit=0 --delete /District 10.0.0.20::APPS1DISTCL/APPS1DISTCL/current/ --port=873 --progress sys:\rsync\rsync --volume=APPS1DISTCL: -avz --whole-file --bwlimit=0 --delete /login_includes 10.0.0.20::APPS1DISTCL/APPS1DISTCL/current/ --port=873 --progress sys:\rsync\rsync --volume=APPS1DISTCL: -avz --whole-file --bwlimit=0 --delete /ADMIN 10.0.0.20::APPS1DISTCL/APPS1DISTCL/current/ --port=873 --progress sys:\rsync\rsync --volume=APPS1DISTCL: -avz --whole-file --bwlimit=0 --delete /trustees 10.0.0.20::APPS1DISTCL/APPS1DISTCL/current/ --port=873 --progress
- CLRSYNC9.NCF
#Copy all //asdapps1/adm/IS/* and force rewrite apps/gqwin everytime sys:\rsync\rsync --volume=SYS: -avz --whole-file --bwlimit=0 --delete /apps1trust 10.0.0.20::TRUSTEES/TRUSTEES/current/ --port=873 --progress sys:\rsync\rsync --volume=SYS: -avz --whole-file --bwlimit=0 --delete /apps2trust 10.0.0.20::TRUSTEES/TRUSTEES/current/ --port=873 --progress sys:\rsync\rsync --volume=SYS: -avz --whole-file --bwlimit=0 --delete /commtrust 10.0.0.20::TRUSTEES/TRUSTEES/current/ --port=873 --progress sys:\rsync\rsync --volume=SYS: -avz --whole-file --bwlimit=0 --delete /temptrust 10.0.0.20::TRUSTEES/TRUSTEES/current/ --port=873 --progress sys:\rsync\rsync --volume=SYS: -avz --whole-file --bwlimit=0 --delete /mail1trust 10.0.0.20::TRUSTEES/TRUSTEES/current/ --port=873 --progress
[edit]
Windows RSYNC Backup
[edit]
PAC
[edit]
Cafeteria?
[edit]
CRON and TRUSTEE.NLM
Copies of the files I used are in O:\Technology Stuff\RSYNC_SOURCE\
[edit]
CRON (Automated kickoffs)
The idea behind this is to have each server within the cluster kickoff the exact CRONTAB file. By having this happen, each server will backup the volumes that it has mounted and fail on the ones it doesn't. This will cover all of the clustered volumes.
This is a list of the cluster CRONTAB commands
# Run perl script to rotate logs each midnight. 0 0 * * * perl sys:/apache2/rotate.pl sys:/apache2/rotate.ini --noscreen # RSYNC stuff see http://wiki.almasd.net for documentation # Run Trustee.nlm for each clustered volume 1 18 * * 1-5 trustadmincl.ncf 3 19 * * 1-5 trustcommcl.ncf 5 19 * * 1-5 trustdistcl.ncf 7 19 * * 1-5 trusthscl.ncf 10 19 * * 1-5 trustiscl.ncf 13 19 * * 1-5 trustjlccl.ncf 15 19 * * 1-5 trustmscl.ncf 17 19 * * 1-5 trustpscl.ncf 20 19 * * 1-5 trustwebcl.ncf # Run rsync for each clustered volume 30 19 * * 1-5 clrsync1.ncf 1 20 * * 1-5 clrsync2.ncf 30 20 * * 1-5 clrsync3.ncf 1 21 * * 1-5 clrsync4.ncf 20 21 * * 1-5 clrsync5.ncf 40 21 * * 1-5 clrsync6.ncf 1 22 * * 1-5 clrsync7.ncf 20 22 * * 1-5 clrsync8.ncf 40 22 * * 1-5 clrsync9.ncf
[edit]
TRUSTEE Backups
This had to be treated a little differently. It functions in much the same way as CRONTAB, but because it collects a file for each volume to be RSYNC'd, we have to copy a server-named folder to the RSYNC server.
Location and content of all trustee NCF files for each server/volume:
### Delay 30 mins to allow for dsrepair to run delay 1800 ### Save trustees load trustee.nlm save [volume]: sys:\[server-specific-folder]\[volume-specific].txt
trustadmincl.ncf is different from the above because it contains the dsrepair launcher:
### Run DSREPAIR with Log file and unattended complete repair DSREPAIR -L sys:\[server-specific-folder]\dsrepairsync.log -U ### Delay 30 mins to allow for dsrepair to run delay 1800 ### Save trustees load trustee.nlm save [volume]: sys:\[server-specific-folder]\[volume-specific].txt
Each server-specific-folder is located in the SYS: volume.
ASDAPPS1 = SYS:\APPS1TRUST\ ASDAPPS2 = SYS:\APPS2TRUST\ MAIL1 = SYS:\MAIL1TRUST\ ASDCOMM = SYS:\COMMTRUST\ ASDTEMP = SYS:\TEMPTRUST\
[edit]
TO DO
- Set up the Trustee folders to RSYNC to the server
- Double-check Regular Servers rsync scripts
- PAC Backups
- CAFE Backups
