15 answers to my posting :-)
Thanks to all of you !
The (cut down) postings are below.
I've put all of them in - excuse me if you think its too large
Results of my posting:
Most told me that RAID 5 with DiskSuite will be too slow.
The reason for this is that the processor has to do the parity calculations
and that for disks > 3 you will get too many IO-Operations. This will hit
write performance, espacially for writes of large files. RAID 5 might be a
good solution for archive directories or other applications where you don't
have too many writes.
The recommendation voiced most often was to use mirroring (RAID 1) instead,
this will not hit write performance as much, especially if you have the
disks on 2 different controllers (recomended anyway because the controller
is a single point of failure otherwise).
Also using RAID 1 in combination with RAID 0 (stripping) will better
performance - in my case this would mean buying 4*9gB disks an building 2
stripesets which are mirrored. Read the answers to get more tips
(backup,logging,hot spare pool).
The idea of using rsync was mostly answered with 'How strange' and 'Bad idea!' :-)
although some said that if we are prepared to loose a single day of work
we might go with it. Side note (a) read the answer from Kevin Sheehan on
replication software. (b) we use rsync at this time to replicate a 4gB drive
(NFS-shared Software) to another server. This and the replica are mounted by
the clients using NFS-failover which works very well. But this is of course
read-only data.
Our decision will probably be to get 2*18gB for mirroring (not 4*9 because
we might not get 9gB drives in 2 years time easily) and maybe get
a second controller.
Regards,
Sebastian Benoit
------------------------------------------------
My original posting:
I have to decide on a storage solution for about 18gB of diskspace for
users home directories. Currently these homes are on an 8gB disk backuped
on an IBM ADSM Server.
Our concerns are that a) the restore from backup is rather slow and would
be nearly untolerable for 18gB and b) we have had bad experiences here
with a Hardware-RAID Solution where an electrical fault in
one disk hosed the complete array.
My idea was to use Disksuite RAID 5 with 3 or 4 9gB drives.
Might write performance be a problem ? Currently, 'iostat' shows
30.7 kr/s and 25.4 kw/s and 6% busy on the homedirectory volume.
Also 2 * 18gB mirroring (again with DiskSuite) might be possible.
Now a collegue had the idea to use one 18gB drive for the directories
and use 'rsync' every night to sync the files with the second drive
instead of using RAID, and my boss likes the idea because its 'simple'.
What do you think is the best solution ?
Is the 'rsync' idea practical ?
The system is an Ultra 2/300 (one SCSI-Bus) with a Multipack attached.
I have found some bits and pieces in the Archives b.t.w.,
however I can see no conclusive picture.
Answers:
From: "Hill, Alan" <Alan.Hill@microcell.ca>
RAID 0 is striped across multi drives, for speed. Will fail if ANY
drive fails.
RAID 1 Mirror is SAFE, but cost is high, and may have a bit of a speed
impact. Min. 2 drives
RAID 5 is VERY POOR on writes, but is cheaper, and keeps working if one
drive fails. Works best with at least 5 drives.
RAID 0 and 1 is a good combo of speed and Mirror for security, but only
over multi-drives.
I would recommend getting a tape backup unit, rather than the rsync,
because a blown controller could kill both drives.
Or a power problem during rsync can also kill them both.
------------------------------------------
From: Harry Levinson <levinson@ll.mit.edu>
[bad experiences with Hardware-RAID]
I don't see how a Hardware-RAID would be any more suseptible to this
kind of failure than any other solution that has a shared electrical
bus.
[My idea was to use Disksuite RAID 5 with 3 or 4 9gB drives.
Might write performance be a problem ?]
I use Disksuite for my read only or read mostly files systems. We found
that
the heavy writing in the home directories located on RAID 5 filesystems
was intolerably slow in our environment.
This was compounded by NFS delays waiting on write completions. I don't
know if the Presto boards are still available. I have looked for them
without
success. Some similar product might help mask the poor write performance.
[Also 2 * 18gB mirroring (again with DiskSuite) might be possible.]
This should help the write problems and with such large disks available.
I personally like the look of the HP AutoRAID arrays (though I don't own
one yet). Some of the NFS appliance people do something similar. They
dynamically move files between Mirrored sections and RAID 5 sections
depending on access patterns.
[rsync'ing']
If you put the disks on separate systems, perhaps connected with Fibre
Channel if performance dictates, you also get redundant servers.
------------------------------------------
From: Frank Dreilich <dreilich@ira.uka.de>
I wouldn't suggest a RAID 5 software system with only _one_ SCSI-bus.
> Also 2 * 18gB mirroring (again with DiskSuite) might be possible.
We have here a 9 GB Disk ( 2*9Gb mirror with Disksuite) and it works
just fine. Okay, this is not the cheapest solution, but disks are not
such expensive any more.
I think the write performance should be no problem with the mirror.
For a backup you can take one disk from the mirror offline and make
a backup from this one. If you put this disk back online the Disksuite
will re-sync the mirror. Nice :)
But maybe this is not so important for your solution because you use ADSM
and not a local backup-device.
[rsync]
Then You'll have only for a few moments a identical disk. Just let
Disksuite do the job. :)
[best solution ?]
It depends which priority you have (Performance,Availibility,Costs)?
See "DiskSuite User Guide" page 1-3: "How to determine DS use ..."
If you plan to use more than one disks as RAID[0,1,5] get
a second scsi-controller!
------------------------------------------
From: Al Hopper <al@logical-approach.com>
[My idea was to use Disksuite RAID 5 with 3 or 4 9gB drives.]
Don't use host based RAID 5. It's simply too slow. Use RAID 5 only with
hardware RAID controller boxes.
> Might write performance be a problem ? Currently, 'iostat' shows
> 30.7 kr/s and 25.4 kw/s and 6% busy on the homedirectory volume.
Not a problem for RAID 1 (mirroring).
> Also 2 * 18gB mirroring (again with DiskSuite) might be possible.
Yes this would be the preferred solution. Consider a 3-way mirror if you
want some extra redundancy and better read performance.
[rsync]
Bad Idea!
[Is the 'rsync' idea practical ?]
No.
General points:
Use RAID 0,1 or 0+1 for host based RAID
Use RAID 5 only with hardware based RAID solutions
Recommended configuration:
Disksuite with RAID 1 (mirroring)
Consider the cost/benefits of a 3-way mirror
Make sure to set aside disksuite hot-spares (the hot spares will be
automatically brought online to substitute for a failed mirror component).
One point that I wish to emphasize: this is a simple, run-of-the-mill
application for disksuite. Its a typical solution to your problem, used
widely at lots of sites - there is no rocket science involved. It's not
difficult to setup and it's done very widely on Sun platforms.
------------------------------------------
From: Rick von Richter <rickv@mwh.com>
For archive directories and db storage, RAID 5 is a good solution. But,
for home directories I would recommend using a RAID 1 or 0+1 solution.
Disksuite is software RAID so you will have a severe impact on the CPU
if you use RAID 5 for high i/o directories. A 0+1 solution is much
faster and less CPU intensive. If you have four drives, I suggest using
a 0+1 solution (stripe two of the drives and mirror them to the other
two).
Rick
------------------------------------------
From: Scott MacKay <scott_mackay@mail.rte.com>
Simple advantages/disadvantages I can think of are:
DiskSuite:
Pro: All data is constantly synchronized and fail over *should* be transparent
Notes:
A really nice thing is that you can mount the original or mirror slice
if you need to for any bizzare reason (after shutting down DiskSuite).
You can also perform backups with better integrity by breaking the
mirror, backing up the mirror disk, and resynchronizing.
rsync:
Con: You can lose up to a full day's work
Pro: You effectively have a fast restore; You can recover files as they
appeared the day before.
I guess I think it depends on how large a window you are willing to
lose. If the disk dies, you lose up to a full day's work. The only
drawback for DiskSuite is the need to add the software. Proper planning
can make the DiskSuite also appear quite 'simple'.
-Scott
------------------------------------------
From: "Brooke King (6532)" <jbking@sandia.gov>
I you have four 9GB drives and DiskSuite, the simplest and safest
thing to do is to create two stripes, each consisting of the two
9GB drives, and put the stripes in a mirror.
You could go further, plan ahead, and use a tiny portion of each
disk, 32MB, to create another mirror of two 64MB stripes to use
for logging. Then you could put the first and second mirrored
stripes into a "trans" as the data and log portions,
respectively.
The advantages of the first paragraph is run-time performance and
safety. The advantages of the second paragraph are more safety
and quicker boots after the rare system crash because an fsck
will just be a flash instead of what you would expect of such a
huge file system.
Rsync every night cuts your backup window. Don't do it. Shoot,
with mirrors you have a possibility I've never exercised: you can
break the mirror temporarily and backup from the submirror you
took out while the system continues to work on the remaining
submirror. Then you put your submirror back in the mirror; it
resyncs; everything is happy. If you can do this, it doesn't
matter much how long your backup window is, within reason.
------------------------------------------
From: Brion Leary <brion@dia.state.ma.us>
To: Sebastian Benoit <benoit@Mathematik.Uni-Marburg.de>
Subject: Re: should I use RAID (DiskSuite) ?
Sebastian,
Is it acceptable to lose a days work? If yes, use rsync, else use
disksuite. If you can put the two 18GB drives on different controllers
then mirroring would be good - low performance impact, good redunancy.
You could also use disksuite to build an 18GB logical partition out
of two 9GB drives, build it as a stipe, then mirror one striped
drive against another. This would use 4x9GB drives as opposed to
3x9GB drives with raid 5. It would also provide better performance
than a single 18GB drive and with mirroring provide redunancy.
Raid 5 is done on the cpu with disksuite so it will have a performance
impact.
------------------------------------------
From: Kevin Sheehan/Uniq {Consulting Poster Child} <u-kevin@veritas.com>
Disclaimer: I don't work for Veritas, but I am the author of SRFS, which
they license from Uniq.
[Our concerns are that a) the restore from backup is rather slow and would
be nearly untolerable for 18gB and b) we have had bad experiences here
with a Hardware-RAID Solution where an electrical fault in
one disk hosed the complete array.]
That is one of the reasons we did SRFS (formerly known as UPFS). It
replicates the data to another system so that neither site nor system
nor arrays represent a single point of failure.
The other reason we did it is that doing the I/O in real time means
it is spread out over real time, so you don't have the concern of doing
a huge restore. You can switch over immediately to (for a passive config)
or be using (in an active config) the other copy of the data on the
other system.
[My idea was to use Disksuite RAID 5 with 3 or 4 9gB drives.
Might write performance be a problem ? Currently, 'iostat' shows
30.7 kr/s and 25.4 kw/s and 6% busy on the homedirectory volume.]
At that level, you are nowhere near having a problem with I/O bandwidth.
[Also 2 * 18gB mirroring (again with DiskSuite) might be possible.]
True - but you waste 50% of the disk instead of getting the advantages
of RAID in a lower rate of redundancy. That said, mirroring does give
you 2 copies to read from and 2 spindles to seek with, with can be
a big win for reads.
[use 'rsync' every night to sync the files with the second drive
instead of using RAID - What do you think is the best solution ?]
>From an availability point of view, having a second system with the data
is obviously what I think best :-) RAID 5 is a good idea from a price
and performance point of view in your situation - but as you point out,
there are still single points of failure.
[Is the 'rsync' idea practical ?]
Hmmmm. If you are going disk to disk, you might as well mirror. Rsync
between systems isn't bad (though it can take a good long while) but the
major objection would be that the file system should be idle while doing
the rsync. I.e. if you are actively accessing the data, there is no
interlock with the xfer in progress.
That tradeoff, along with the fact you do it in one hit (while idle) and
have a lag until your data is backed up are the movtivations for SRFS.
There you have real time backup to the other system (or local file system
actually, but that is less common as mirroring does that job) and you have
the I/O spread out so it isn't a big hit.
If you are willing to eat an 18GB disk for rsync, my guess is that mirroring
is what you really want. If you have multiple disks, then RAID is probably
a better solution. If you are concerned about single points of failure
or making the copy remote, then SRFS is probably a good idea.
------------------------------------------
From: Seth Rothenberg <SROTHENB@montefiore.org>
Sebastian,
You are right, write performance under RAID5 is
likely to be poor. For read/write applications,
mirrored is fast, striped + mirrored is faster.
We use striped and mirrored under Disk Suite.
With regard to your mirroring ideas...
I have made a similar argument with regard to
our OS upgrade path...because if an OS upgrade
fails, it takes about 3 hours to restore.
If you need the real protection of mirroring,
you can even get 3 disks, and build a 3-way
mirror under Solstice Disk Suite.
Each night, you attach the 3rd mirror,
then detach it again once it is synchronized.
That mirror remains a snapshot. Do you need
to save more than one backup? With disk,
this is hard. With tape, it is easier.
------------------------------------------
From: ian@masuma.com
[Also 2 * 18gB mirroring (again with DiskSuite) might be possible.]
We have been using RAID 5 with Disksuite and the NFS write performance is
poor. So poor we are getting a hardare RAID box. If you have the drives,
use mirroring, there is only a slight write hit and a reading is slightly
faster than a singe drive.
[rsync]
How strange!
[Is the 'rsync' idea practical ?]
Not much use if you get a failure in the afternoon! At least with a mirror,
you don't have any downtime.
------------------------------------------
From: sam.vilain@nz.unisys.com
The rsync sounds hairy - a disk goes, and you're suddenly instantly having to
restore via Rsync. I would consider this a "hack".
There is a good white paper on this sort of thing in SunSolve - look around
for "RAID" in the white papers section.
Remember that with a RAID 5 with (n) disks you get:
- (n) I/O's per update (usually this is not a problem, but unacceptable
for big databases).
- (n-1) disks worth of space
- If you lose one disk, you're instantly down to worst case performance of
(n-1) i/o's per read.
Given your small requirements, I recommend a RAID 1 (mirror) with 2x18GB disks.
The only problem is if your controller goes, I guess. But this doesn't
happen very often in practice.
Sam.
------------------------------------------
From: "Rose, Robert" <Robert.Rose@ag.gov.au>
Sebastian,
I think you should be looking at other Hardware RAID solutions. Disksuite
doing RAID5 is really slow, I've had some bad problems with disksuite
mirroring (the machine ended up being down for a week !) The rsync solution
sounds interesting, but could you handle the loss of a day's work if a disk
failed just before the rsync happened?
Have a look at Sun's A1000, we've just got one here and it's really quick!
cheers,
Robert Rose
Unix Systems Administrator
------------------------------------------
From: Jay Lessert <jayl@lscpdx.latticesemi.com>
We currently run Veritas Volume Manager RAID5 and RAID0+1. To first
order, there is no major performance difference between Disksuite
and Veritas.
[My idea was to use Disksuite RAID 5 with 3 or 4 9gB drives.
Might write performance be a problem ? Currently, 'iostat' shows
30.7 kr/s and 25.4 kw/s and 6% busy on the homedirectory volume.]
RAID5 with 4 drives should be able to do this. Read performance
should be no problem. Make no mistake though, RAID5 write performance
(HW or SW) will suck compared to a single disk write. You can make
your average 25 kw/s, but on those times when a user copies a single
big file (for example) they will definitely notice it is slow.
[Also 2 * 18gB mirroring (again with DiskSuite) might be possible.]
Certainly works, but a 4 * 9GB RAID0+1 (striped and mirrored) would
kick butt. Read performance (especially for multiuser) is sparkling,
and write performance would be 30-50% faster than a single disk
(striping helps more than mirroring hurts).
[What do you think is the best solution ?]
4 * 9GB RAID0+1 :-)
[Is the 'rsync' idea practical ?]
Yesssss, if you're certain none of your users will ever, ever do things
that rsync can't handle at all (fifos, sockets) or doesn't handle well
(dynamic changes during the rsync run).
In general, I would not use RAID5 for a home directory partition. We
do use RAID5 for an application SW partition and an archive partition,
both read-mostly sort of setups. Works fine for that.
Disksuite mirroring (whether you do 4 * 9GB with striping or 2 * 18GB
without striping) is not difficult to set up. Assuming you've never
touched Disksuite before, it'll probably take you a weekend at home
reading the docs and 2-3 days to play with it and get it right. And
it works really, really well.
------------------------------------------
From: Dieter Wurm <dieter.wurm@de.origin-it.com>
hy,
Yes RAID-5 is io-noisy...shoudn't use this Config.
Raid 1 is nice because its mirroring...why not.
For safty you could detach during day and attach (for udate)
at night...
rsync-job does the same as attach/detach...why not using software-
default tools, uh?
one thing about disksuite: for y2k you need a big jumbo-patch as well...
regards
dieter wurm
---------------------------------------------------------------
On a clear disk you can seek forever.
- Sebastian Benoit
- benoit@mathematik.uni-marburg.de
- http://www.mathematik.uni-marburg.de/~benoit
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:13:15 CDT