Managers:
1st. My apologies. I sent the same query twice. After three days I hadn't
received any replies or seen my query so I resent it. The following day I
received about 20 replies to my first query. I guess the listserver is just
getting slow these days. I've been annoyed when others sent the same query
twice... I should have know better. Sorry.
2nd. Thank you, thank you, thank you. I've been banging my head against the
wall on this problem for about a month now. You guys were dead-on with the
solution.
In brief, the question was... is this a valid dump command to tell a remote
machine to dump (backup) and send the results back to my local tape drive:
# rsh -n wacko /usr/etc/dump 5bdsf 126 54000 13000 - /wack1
2>>stderr | dd bs=1024 conv=sync of=/dev/rmt/0hn 2>>stderr
The overwhelming answer was NO! It's not the hardware, it's DEFINATELY my
LOUSY DUMP command.
There were numerous answers and solutions so let me start by telling you
what I settled on as the solution. The dump in the form of...
# rsh -n wacko /usr/etc/dump 5bdsf 126 54000 13000 - /wack1
2>>stderr | dd obs=126b of=/dev/rmt/0hn 2>>stderr
...seems to work quite well. Two things are different here. 1.) I've
now specified my blocksize to 126 blocks which now matches my dump command
instead of the 1024 bytes I was specifying. No mismatch, and it should
be faster with the larger block size. 2.) I've gotten rid of the
conv=sync parameter which was apparently padding my blocks out creating
a corrupted mess.
The most common solution mentioned was to used the rdump syntax...
rsh -n wacko /usr/etc/rdump 5bdsf 125 5400 13000
server:/dev/rmt/0hn wack1 2>>stderr
...which is a great idea, simple, elegant, clean, efficient... but it
suffers one major drawback, which is the fact that it's a major security
hole. When I tested this syntax I received a "Permission Denied" message.
So, I edited my /.rhost file to give the remote machine rpc permissions to
my tape server host's tape device. This fixed the "Permission" problem and
the dump ran great. Quick, reliable, and most importantly restoreable.
BUT this means that anyone who can become root user on a remote machine can
now become root user on my taper server. No way! Policy on our site is that
anyone with a workstation on their desk can become root on their machine
but not on anyone elses machine. We inforce that with the lroot (local
root) convention described in The O'Reilly - Hal Stern classic, "Managing
NFS and NIS". This would totally destroy that security.
There are some solutions to this securiy hole described down below in the
details although they do require a little work to implement. If you're
interested be sure to read through the credits in their entirety.
Using the dd solution doesn't require me to open up root permissions in
any way because of the way that dump dumps to standard output which is
piped back through dd. It's definately more verbose and error prone (it
took me a month...) but once you get it right it works like a charm with
no sacrafice to security.
Essentially that summarizes the solutions. I'll post credits and individual
comments/replies down at the bottom for those who are interested in more
detail. Again, a thousand thanks.
-drp
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| _/ _/ _/_/_/_/ _/_/_/ _/_/_/_/ | Dan Penrod - Unix Administrator |
| _/ _/ _/ _/ _/ | USGS Center for Coastal Geology |
| _/ _/ _/_/_/_/ _/ _/_/ _/_/_/_/ | St. Petersburg, FL 33701 |
| _/ _/ _/ _/ _/ _/ | (813)893-3100 ext.3043 |
|_/_/_/_/ _/_/_/_/ _/_/_/_/ _/_/_/_/ | penrod@whiplash.er.usgs.gov |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
From: seanw@amgen.com (Sean Ward)
>Hi Dan. The only thing I might try differently from a syntax standpoint would
>be to put the part of the command you want to run on wacko within double
>quotes. IOW, the command would look like this:
>
>rsh -n wacko "/usr/etc/dump 5bdsf 126 54000 13000 - /wack1 2>>stderr" |
> dd bs=1024 conv=sync of=/dev/rmt/0hn 2>>stderr
>
>My experience has shown that if you don't use double quotes, the rsh will
>try to run the entire command on the remote machine.
Good point Sean but not the problem in this case. I've had similar problems
though.
--- >From: ers@kestnbaum.com (Eileen Schatz) >This is a line from our dump and it works great. We use a 10GB exabyte >drive from Sun. > > rsh fozz /etc/rdump 0ufsdb hal1:/dev/rmt/1cn 6000 54000 126 / >>&! > $logfile > >Where fozz is a remote host running 4.1.3 and hal1 is a host running >2.4 we append to a logfile that is then mailed to me when the the dump >finishes.Yea, I notice your size param is different but everyone seems to use their own number for that parameter. I'm sure it works fine. Again, my complaint is the security hole.
--- From: szh@zcon.com (Syed Zaeem Hosain) >It is your command, I believe. The 'dump' block size does not match up >with the block size given to 'dd'. In 'dump, you use 126 blocks (the >default for 'dump' is in blocks of 512 bytes each) and with dd, you use >1024 bytes (the default for 'dd' is in bytes). > >This will cause you some *serious* problems with the restore. The >padding of the input records with the "conv=sync" parameter is simply >not used at all - each dump block of 64 kbytes (or so) is TRUNCATED to >1024 bytes before being put onto the tape. No *wonder* you cannot >restore from these dumps! > >I highly and strongly recommend that you use the SAME consistent value >for the block size parameter for both the commands, and not pad the 'dd' >input records. I.e., try the following: > > # rsh -n wacko /usr/etc/dump 5bdsf 126 54000 13000 - /wack1 > 2>>stderr | dd obs=126b of=/dev/rmt/0hn 2>>stderr > >Note that you need to specify the "b" after the "obs" option to get >the 'dd' operation to be set in blocks rather than in bytes.
This is exactly what I did. Seems to work great. Excellent answer. Thanks.
--- From: Ian MacPhedran <Ian_MacPhedran@engr.USask.Ca> >Why not use: > # rsh -n wacko /usr/etc/dump 5bdsf 126 54000 13000 tapehost:/dev/rmt/0hn > /wack1 >and let rmt take care of handling your mag.tape.? > >It's so much easier.
I agree. It is sooo much easier. Again, I don't want to trust root@wacko.
--- From: brunel@delver.iterus.org >I'll have a guess. Instead of using restore, try running something >like: > >rsh tapehost dd bs=1024 if=/dev/rmt/0hn | /usr/etc/restore <bunch of >restore parms> > >Or try > >dd bs=1024 if=/dev/rmt/0hn | /usr/etc/restore <bunch of restore parms> > >on the machine with the tape drive. For reasons I don't understand, >remote dump and restore seem to be picky about block size. Can the >4.1.x restore specify the block size? If so, specify it.
I thought this was a very interesting reply. There was no concern expressed in regard to my bad dump command. Just restore it in reverse and read the tapes as they are. It made me laugh... it might even work. I have to admit I haven't tried it yet, but I do have several weeks worth of level 0s that I may need to retrieve from. I'll probably give a try tomorrow.
--- From: "Bruce Rossiter" <AROSSITE@us.oracle.com> > Try reversing what you used to dump. (dd if=/dev/tape bs=1024 | >restore). > > You probably want to use "dd obs=XX" (XX being your preferred blocking >factor for tape writes) in your dump script.
This is kind of similar to the last reply. Again, I didn't try it and it might just work on my 'bad' level-0s. You're right, I needed to specify my obs=.
--- From: raoul@MIT.EDU >Why aren't you using rdump and rrestore? Trying to wrap dd details around >an Exabyte is an extremely painful task, since Exabytes are notorious >for doing EOF and block grouping differently than other media. > >Do it with rdump: "rdump 5bdsf 126 54000 13000 wack:/dev/rmt/0hn" should >do it. > >And don't redirect stderr in rsh's: Try doing it by hand, wrapped around >your script: "dumpscript 2>&1 | mail root" works for me.
Yup, it is a painful task, but I think, the best solution. My redirection works just fine as-is within the script. Using it within the script allows me more control over what gets redirected where.
--- From: Dave Mitchell <D.Mitchell@dcs.shef.ac.uk> >The problem is with conv=sync. This should in fact be conv=block. > >conv=sync takes each (small) chunk of data it receives, and pads it with >zeroes to the block size specified. conv=block reads in multiple >small chunks, and merges them into a single block before outputting. > >Also, you really want bs=126b to match the block size of the dump. > >Incidentally, why dont you just use rdump?
I dropped my conv=sync param. I didn't try adding conv=block param. It may work, I don't know, but apparently I didn't need it. I set my obs=126b and that works. I didn't try setting bs=126b. It may work. It appears my ibs, by default, is set to something appropriate.
--- From: perryh@pluto.rain.com (Perry Hutchison) >I would try changing that "bs=1024" to "obs=126b". 1024 bytes is very >small for a tape device, and "bs=" will cause dd to copy to the tape >whatever oddball blocksizes it receives from the network pipe set up >by rsh. "obs=" will cause it to reblock to the specified size.
Right on.
--- From: fhb@kestnbaum.com (Fred Berns) >Why aren't you using rdump, something like: > > rsh -n wacko /usr/etc/rdump 5bdsf 125 5400 13000 > server:/dev/rmt/0hn wack1
So many people strongly recommending the rdump command! Maybe I've overlooked something major and it's not the security blunder I've described. I'd be interested to see how everyone using rdump deals with root logins at there sight. Do you allow workstation owners to become root at all? Do you memorize all 200 workstation root passwords? Hal Stern's lroot solution is the way to go for my money but you have to keep a tight reign on your /.rhosts. How can you use rdump without opening up /.rhosts to the whole site (God forbid beyond the site)?
--- From: i.camm@fml.co.uk (Ian Camm) >Try just using > >rsh -n wacko rdump 5bdsf 126 54000 13000 localhost:/dev/rmt/0hn /wack1
I'm so dramatically outnumbered... I MUST be missing something.
--- >From d.white@surrey.ac.uk >We're doing dumps exactly this way (but on SunOS machines mainly). Are the >parameters correct? Well, we have been using Exabytes for a while, and we >use commands like: > >rsh cuillin "sync; sync; /etc/dump 0ucsdf 5190 8900000 - /dev/sd0h" | dd of=/dev/nrst13 bs=56k > >I inherited these parameters from an earlier incarnation of the dump script. >I have no idea why tapesize=5190 and density=8900000 (the latter seems VERY >unlikely!) or dd blocksize=56K. > >They seem to work ok though :-)
Ah! At last someone who's actually doing it this way on purpose! I'm glad to know I'm not totally alone! :-) The 'sync;sync;' is probably an excellent idea. I think I might add that myself. Of course your size and density params are bizarre but everyone seems to have their own favorite random numbers to use there. So long as it's working for you... So you're using a dd blocksize of 56k. Maybe that's a performance enhancer... I wonder...
> >Now, I'm trying to get DAT drives to work on Solaris - whole new set of >parameters to guess :-( In fact, so far it seems to work best with no >parameters (although the percentage of a tape used messages are of course >utter rubbish).
You have my total sympathy. I haven't needed to worry about DAT drives yet.
> >What restore command do you use? We use: > >dd if=/dev/{tapedevice} bs=56k conv=block | restore ivf -
So it appears you have to restore through this dd command in order to get the correct block sizes back off of the tape. That's very interesting but it would make me a little nervous. I think I prefer being able to simply use a 'restore ivf' directly without needing the dd. It must be because you're using an unusual dump block size.
--- From: Michael Baumann <baumann@proton.llumc.edu> >Um... why not just do the following: >Create a user, say, called backups. create a .rhosts for that user. >then do the following: ># rsh -n wacko /usr/etc/dump 5bdsf 112 54000 13000 \ > backups@tapehost:/dev/rmt/ohn >the 112 gives better thruput on an 8mm unit, and 0hn is *with* compression >and no rewind.. 0n is without compression. You might want to consider >bsd semantics.. I dunno. >Anyway.. this does the dump, using rmt, as user backups.
And that's how to avoid the security hole! Good thinking. I had forgotten that you could specify users in the /.rhosts file. I think your mistaken about the "0hn is *with* compression" comment. /dev/rmt/0hn is standard format 8500 5Gb, /dev/rmt/0cn is 8500c, 10Gb *with* hardware compression, /dev/rmt/0mn is old 8200 5Gb, and /dev/rmt/0ln is old 8200, 2.3Gb. I guess if you're using an 8500 you might think of /dev/rmt/0hn as compressed, in comparison to the 2.3Gb format. When you say "0n is without compression", again I have to disagree. On my tape device 0n is the default which may be with or without compression depending on what the default has been set to. In my script I make my device /dev/rmt/0n and then set the default value (compressed or not compressed) in a seperate command in the script depending on whether I'm doing level-0s or incrementals.
>> like..."WARNING: /sbus@1,f8000000/esp@0,800000/st@4,0 (st4): Periodic head >> cleaning required" and "WARNING: /sbus@1,f8000000/esp@0,800000/st@4,0 (st4) >> :Soft error rate (72%) during writing was too high Please, replace tape
>This is bad. Very bad. you could have >0) Buggy dump. >1) bad tapes. >2) bad scsi cables >3) bad scsi termination >4) bad tape unit (yes.. new.. has it *ever* worked since repair? ) > >Note: We has a similar problem with our first 8mm unit, a 8200 >(new technology at the time) Turned out to be a bad power supply.
That's what Andataco said... that it might be the power supply, but we replaced it and the problem didn't go away. Now, with the new dump command I'm not getting the "Soft error rate" messages at all anymore. I am getting the "Periodic head cleaning required" message but I have a feeling that it's the result of dumping to standard output and piping it. It appears to be innocuous.
--- From: dave%endeavour.exar.com@exar.com (Dave Haut) >I don't think that the block size is correct for *dd*. Since dump outputs >data at 126 bytes per block, perhaps you should specify 126 bytes per block >or something that goes into 126 evenly ( 1024 doesn't ) in dd. > >A long time ago, I used to pipe my remote dumps to dd using bs=20b ( 20 Bytes ) >and that seemed to work. Why don't you use this format for remote dumps > >tapehost # rsh remotehost dump 0fbs tapehost:/dev/rmt/0hn 126 165000 /dev/rsdxx > >This is what I use and it seems to work just fine. Of course you can ufsdump >when you are backing up a solaris machine remotely.
Yup, block size was definately a problem. 126 seems to be a favorite so that's what I'm using, both in the dump and in the dd command. Making them match seems to be a good idea.
--- From: leach@OCE.ORST.EDU (Tom Leach) . . . Tom has a terrific full solution which implements the use of rdate and covers all of the security holes. It's really well thought out. I'm not including it here because it's just too verbose. Anyone who want's to see it... let me know and I'll forward it. Good job Tom.
--- From: mike@trdlnk.com (Michael Sullivan) >The problem is your dd command -- conv=sync will hopelessly corrupt the >data. The use of bs=1024 doesn't make much sense either. Your dd command >should be something like: > | dd obs=126b of=/dev/rmt/0hn 2>>stderr >It is important to use obs rather than bs when reading from a network >connection and writing to tape -- otherwise the network connection will >cause dd to input random record sizes up to the input block size, >which it will faithfully replicate on its output, except that your use >of conv=sync means that dd will additionally pad those incorrectly sized >record lengths with zeros.
So here we see why it's important to use obs=126b instead of bs=126b. Some other managers suggested using just the bs=126b. This is a good point that I wasn't aware of. You're right on the money here Michael. Thanks for the input.
--- From: etnibsd!vsh@uunet.uu.net (Steve Harris) >try: > # rsh -n wacko /usr/etc/dump 5bdsf 2 9999999 9999999 - /wack1 > 2>>stderr | dd ibs=2b obs=126b | > dd bs=126b conv=sync of=/dev/rmt/0hn 2>>stderr > >The problem is that reads from a pipe are always small (max 4k as I >recall), and conv=sync pads _input_ buffers. > >In this case, the args to dump are pretty much meaningless, you just >want it _not_ to tell you to change the tape, so I use huge values. > >I think the optimal blocking factor for exabytes is still 126, though >the newer drives may have bigger on-board caches.
Seems a bit drastic... but I believe it works if you tell me it does. :-)
--- From: anderson@neon.mitre.org (Mark S. Anderson) >For starters, your blocking factor for dump is 126. This is 126 * 512 >bytes. You need to tell dd that it's blocking factor should be 64512 >bytes. You are specifying 2 tape blocks (1024 bytes). >Dump knows nothing about your tape drive, because you sending the dumps >to standard output. Therefore, the messages from dump are not exactly >true.
Yup, I wonder if this doesn't explain why I get the "Periodic head cleaning required" message.
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:10:27 CDT