SUMMARY: backup script for a stacker?

From: Dan Penrod (penrod@whiplash.er.usgs.gov)
Date: Fri May 05 1995 - 11:49:43 CDT


I got a lot of responses in regard to backup solutions with stackers; 15 in
all. There seemed to be a majority vote that Amanda is the most popular
shareware - unix - nonproprietary - portable - stacker capable backup
solution out there. Many others provided their own home-rolled solutions.

Amanda (Advanced Maryland Automatic Network Disk Archiver) is sophisticated
(for shareware anyway) C source, compiled program that provides backup
automation. It works in conjunction with cron. It has a primitive interface
with stacker capabilities. The ftp site is: ftp.cs.umd.edu Also, there
are three (subltly different) listserv maillists for amanda buffs. Nice.

I found it easy to compile and install on my server but it seems to also
require make and make install on all the clients. Yuck! I found the
documentation on configuration to be confusing and couldn't quite figure
out how I'm supposed to integrate the stacker functionality with my
particular stacker (Andataco EXB-10e RTH system). Also, I have to backup
a mixture of both SunOS 4.1.3 and Solaris 2.4 systems; I couldn't figure
out how to do this elegantly.

I'd like to use amanda eventually, but not until amanda is enhanced a
couple of minor ways.
        1. support for hetrogenous network (SunOS and Solaris).
        2. support for Andataco's RTH system
        3. some organized way to locate appropriate tape for restores
                (file-level index of backups?)

HERE'S WHAT I'M GOING TO DO. For right now, I'm going to slap together
a quick'n'dirty, hard-coded script, to back up each of our of our 15
workstations, partition by partition, using Andataco's RTH robotic commands.
Now that I've had my new stacker in my hands for a week I have a much
better understanding of how it works and how you can write scripts for it.
This way I can run level-0s over this weekend. Also, I'll know exactly what
machines are backed up on what tapes. Over the next couple weeks I plan
to slowly enhance my script with some smarts, borrowing clever chunks of code
from the many scripts sent to me by various sun managers. Finally, I'm going
to keep my eyes open for enhancements to amanada. I see amanda as a good
long term solution... it's just not quite there yet, and I need something now.

ESSENTIALLY THAT'S MY SUMMARY... IF YOU WANT TO KNOW BRIEFLY WHAT INDIVIDUALS
SAID, OR SEE THE CREDITS THEN READ ON...

***************************

From: yamaguch@cqt.com (Bob Yamaguchi)
>We have the exact same stacker from Andataco (10E).
>For the past few months, I've been using the Amanda backup software available
>from ftp.cs.umd.edu. When I first got the stacker, I researched a bunch
>of different software from free to commercial. The commercial stuff cost too
>much for what was offered (IMHO), so I went for the freely available software.
>There were a bunch of them like DeeJay, Backup-2.6, and Amanda. The
>sun-managers archives showed that most people preferred Amanda. But I still
>went through the documentation for each. They all target different solutions;
>one is written to minimize the number tapes. One is written with error
>checking as the focus. Some have file indexing so you know what was backed
>up and when.
>I was initially looking to maximize tape usage, but arguments that it's
>cheaper to waste tape space than to have to make up for a lost tape changed my
>mind. I still would like file indexing, but I can live without it.
>The newest version of Amanda has stacker support in the form of a configurable
>script. The software calls this script to change tapes, etc. I wrote my
>own script that handles Andataco's rth program. I wrote another
>script that's called from cron that handles the tape changing and backups.
>It also mails me a message indicating that I have to update the tapes in
>the changer. It's working pretty well.
>The only problem is with multiple tapes. It's still a little fuzzy to me
>since I haven't had to use this part of it. When I first tested it, I ran
>it with multiple tapes (having the stacker change the tape), and it seemed
>to work. (I didn't thoroughly check the resulting dump, though). Also,
>the docs seem to indicate that multiple tapes isn't really supported yet.
>But I have a slightly older version. You may want to look more in depth into
>this issue of multiple tapes by going through the docs.
>I forgot to mention that Amanda uses the normap dump command to do its
>dumps.
>If you're interested in taking a look at my scripts, let me know.

Thanks Bob. I share some of your confusion about how amanda works. I wonder
if you're backing up both SunOS and Solaris boxes???

From: chitty@polaris.orl.mmc.com (Tom Chitty)
>...and of course, you'll summarize?

Here it is Tom! :-)

From: sjs@sunthing.sjsinc.com (Stefan Jon Silverman)
> attached (sun mailtool uuencoded compressed tar archive) find the
>scripts that i have been using for a few years. they aren't elegant, but
>they do work.
>Basicly, on my system, i have a cron job set up for every night at
>1:00 am to run the auto.backup script in the archive. on sunday nights (monday
>morning really) it gets passed the parameter "zero" which will cause a zero
>dump of the directories listed in the "backup.dirs" file. on other nights,
>a find command compares the modification times of the files in those dirs w/
>a timestamp file created during the last zero-dump and only backs-up the
>newer files. a disadvantage here is that as the week progresses, the incremental
>list begins to get longer, it only takes account of the last zero-dump not the
>last incremental. in all cases, the files listed in the data file "crit.files"
>get backed up every night.
> i'm sure that you will get many other replies to this query (we all
>have our home-rolled scripts hanging around) that require less modifications
>then mine, but i thought i would offer it to you anyway.

Thanks Stefan. I see it's a korn shell script. I don't run korn shell but
I'll definately look it over for suggestions on what I should do in my script.

From: Gene Rackow <rackow@mcs.anl.gov>
>Get the amanda package from ftp.cs.umd.edu. It's not written in
>perl but will do what you want easily. Then if you really want to
>automate things, you could do a simple perl front end for it to
>do several things per day.

Thanks Gene. It does look like a good product. I'll probably migrate
to amanda sometime this year.

From: "Trevor Kirby" <Trevor.Kirby@newcastle.ac.uk>
>The latest version of amanda has bumph in for tape stackers, you may want to
>copy that or even try amanda itself.
>The latest version is 2.2.6 and is ftp'able from cs.umd.edu:/pub/amanda.
>Just in case you've nvever heard of it I've enclosed the bit from the FAQ.
>1. What is AMANDA
> AMANDA is the Advanced Maryland Automatic Network Disk
>Archiver. It allows the administrator of a LAN to set up
>single master backup server to back up multiple hosts to a
>single large capacity tape drive. AMANDA uses native dump
>facilities and can back up a large number of workstations
>running multiple versions of Unix efficiently. For more
>information, see the postscript file in
>cs.umd.edu:/pub/amanda/.... You can also find this in the
>proceedings of the Summer 1991 USENIX conference held in San
>Antonio, TX.

Thanks for the teaser Trevor.

From: Philippe Gesnouin <phil@turing.infobiogen.fr>
>have a look at : http://shaq.pnl.gov:2080/~d3c572/docs/index.html.
>This describes a script wrote by Darren Curtis, we use it every day with
>success.

Thanks Philippe. I looked over this bourne shell script. I like the way
it's implemented.

From: dave@sarah.lerc.nasa.gov (David A. Thompson)
>Here is the script that I use on a sunos 4.1.3 system. The program to
>retreive a disk space estimate may need to be changed for your
>operating system. The perl script assumes that a given filesystem will
>fit on single tape. It is also my first real attempt at writting in
>perl so you may find that there can be a lot of optimizations
>done. The scripts arguments are
        { Dave's Perl script followed - If anyone wants a copy and Dave }
        { doesn't object, I'll gladly forward it on request. }

Thanks Dave. Good job on the script. I'll definately refer to as I put
together my solution.

From: Reto Lichtensteiger <rali@hri.com>
>Why re-invent the wheel? <g>
>OSU-Backup (as described at LISA 6) covers these bases already ...
>It's written largely in perl, supports multi-system backups, has online
>indexing ... Here's part of the readme:
> The OSU CIS backup system consists of a set of Perl scripts that
> strive to make it easy and safe to manage backups and restores for
> a number of hosts. The main design constraints were that it
> should be easy to use, robust in handling errors, complementary to
> human skills and shortcomings, and that it should use native
> backup programs.
> The current version supports full and incremental backups on
> multiple hosts, where backups of many file systems from different
> hosts can be put onto one tape. It maintains a tape library with
> a free tape pool. There are safety checks that verify that you
> are using the right tape in the right drive, that backups haven't
> been skipped, etc. There is support for multiple backup chains.
> There is offsite tape storage support. Restores are easier since
> the system knows what backups are on which files on which tapes,
> and it will ask you to mount tape X and skip to the correct file
> for you. You can backup to disk or tape or both. There's an
> event log so you know who has done what. It comes with support
> for dump and GNU tar, and its easy to add support for other
> archive programs, like cpio or others.
>It was written by Steve Romig of OSU's CIS group and there is a mailing
>list (backup-request@cis.ohio-state.edu to subscribe)
>Ask archie where at OSU to get it -- I can't remember the host for FTP ...

Thanks Reto. I haven't archied for it yet... just been too busy :-)

From: dburwell@telecom.com (David Burwell)
> If so, then there is a propritary interface to the juke box, but the
>tape drive is standard. The program that I have is from Articon called
>stacktool and stackutil. They basicly allow me to write scripts to do
>tape handling and then I use mt / dump / rdump to do backups and stuff.
> If you want them, I can send them.
> If this isn't an OEM version of the Exabyte, then never mind.

Thanks David. Yes, mine is OEM Exabyte (EXB-10e), but apparently Articon
supplies you with their tools (stacktool and stackutil) while Andataco
supplies me with their tools (RTH and SSAPI). It seems that this stacker
(as provided by Andataco) actually has a scsi card in it (they call it the
RTH card) which has two scisi target ids, one used by the tape drive and
one used by the robot. Those ids are on their own scsi bus and seen as a
single scsi target on my workstation. Must be magic. ;-)

From: Todd Pfaff <todd@water.eng.mcmaster.ca>
>I strongly suggest you look at the amanda tape backup system rather than
>roll your own. It's free, portable, the recent versions have support for
>tape stacker systems and it's a great system. The more people that start
>using it, the better it will become. Use archie and search for amanda.

Thanks Todd. I hope to be on the bandwagon soon enough.

From: Duncan White <d.white@surrey.ac.uk>
>I don't know anything about Stacker products, but we use a PERL-script based
>backup and restore regime, using a set of Exabyte drives (some 2.5GB, some
>5GB) on different machines. Each machine dumps a set of filesystems (different
>each night) from machines all around the network... You're welcome to look at
>the scripts and data files, a shar file is enclosed below..

Thanks Duncan. Stackers are a mixed blessing. They allow me to back up my
who site, some 40+GB (Soon to be 80GB!) without having to intervene, but
there doesn't seem to be a lot of good seemless software solutions for them
yet. Definately the wave of the future though. Thanks for the offer but
I think I've got enough scripts now.

From: z055084@uprc.com (Kohler R. P. (Robert))
>I wrote alot of stuff for our site... But it is all in "BOURNE SHELL".
>It uses specific drivers supplied by vendors to "load", "unload", the tapes,
>but my routines tell it when, where to run.
>It is handling roughly 400+ GB (yes Giga-bytes) of data per week.
>I can send you more info if you are interested....

Thanks for the comments and offer Robert. Looks like I've got a grip on
things now.

From: dburwell@telecom.com (David Burwell)
> If its a Exabyte 10e (or 10i) then you can treat it as a 100GB tape. You
>load 10 tapes into the carrier, and start a stacktool script file.
>The drive / juke box will grab the first tape and when the first tape
>fills up, approx. 6 to 8GB per tape, the juke box / tape drive will change
>to the next tape and so on until the tape job is done, or you have exceded
>the capisity of the tapes that are loaded. If you have lots of tape, or you
>only do backups once a week, this can be OK, but if you try to do lots of
>backups per week, you will end up with a lot of tape, and no effective means
>to keep tract of them.
> I use the juke box as an automattic tape loader and treat each tape
>as a seperate 10GB backup session. I use one tape per day to backup
>selected file systems, with all file systems getting backed up twice a week.
>This allows me to change tapes once a week, 7 at a time, and label the tape
>with the week-day (EG: Week of April 24 is week 17 of 1995, Monday is
>day 1, so, todays tape will be 17-1-95). I use the stackutil program,
>which is what move tapes around on the juke box, and dump to create the
>tape. I wrote the scrip files by hand and they run in crontab.
> The stacktool that Articon sends with the drive is a GUI interface
>for creating script files that actually run from crontab. It is an easy
>way to set up the script files and timing. It also creates a simple
>database to help track the tapes.
> Without the Articon programs, stacktool and stackutil, I would not be
>able to have tapes move around inside of the juke box under script command.
>Exabye supplies the manual for the scsi interface to the juke box, but you
>have to write the program that makes those low level scsi calls, not a very
>pleasent job at all. Also, there is a kernel mod required for the Sparc
>machine to "see" the juke box, otherwise, forget it. That kernel mod was
>included with the Articon installation program, it made the mods and helped
>generate a new kernel.

Dave thanks for this description. Yea, it's pretty neat that you can treat
the stacker like a single 100GB tape drive, but as you point out... it makes
it pretty tricky to figure out what tape a specific file-to-be-restored is
on. For now, for simplicity, I just want to say something like "okay, these
three machines are on tape1, these four machines are on tape2, this one
machine with the 9GB disk drive will be all alone on tape3, etc..." and
always be able to quickly go to the right tape.

From: Hans van Staveren <sater@cs.vu.nl>
>See http://www.cs.vu.nl/~sater/

Thanks Hans. It looks like a pretty smart program. I plan to look more
closely at how the smarts were implemented.

From: brent@DOT.IMGEN.BCM.TMC.EDU (Brent Wiese)
>I am not sure this is what you are looking for, but we had three shell scripts
>we used to do our backups on a stacker using GNU tar. You might want
>to take look. One of the reasons for using GNU tar was it allowed one to
>specify a change-tape script if necessary and we did not have access to the
>root password on the systems. We never got around to needing the change
>tape script, but with the new 9gig drives it would be a concern if another
>group here didn't take over doing the backups.

Thanks Brent. I think we need to stick with a dump solution so we can
run various incremental levels.

***
That's it! A million thanks to everyone!
Regards,
-drp

 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| _/ _/ _/_/_/_/ _/_/_/ _/_/_/_/ | Dan Penrod - Unix Administrator |
| _/ _/ _/ _/ _/ | USGS Center for Coastal Geology |
| _/ _/ _/_/_/_/ _/ _/_/ _/_/_/_/ | St. Petersburg, FL 33701 |
| _/ _/ _/ _/ _/ _/ | (813)893-3100 ext.3043 |
|_/_/_/_/ _/_/_/_/ _/_/_/_/ _/_/_/_/ | penrod@whiplash.er.usgs.gov |
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~



This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:10:23 CDT