Greetings Sun Managers,
Please accept my most sincere apologies for being late on this summary.
Been really busy ;-).
My original message is posted below for reference. Many, many thanks to:
Kevin Ying <kevin@ns1.interq.or.jp>
Benjamin Cline <benji@hnt.com>
Jochen Bern <bern@uni-trier.de>
Dale Shaw <Dale.Shaw@select.com.au>
David Lee <T.D.Lee@durham.ac.uk>
Aaron Holtz <aholtz@bright.net>
Special thanks to Adrian Otto, our performance consultant whom we hired to
help us get a grip on the various performance problems were experiencing
simultaneously.
I received many helpful replies from the above users, which I will post
below my original message below.
Our original problem (as you can see from my original post below) had to
with a performance problem we were having on a set of large mail servers
that share /var/mail over NFS. At the time of my post, we were running
Sendmail 8.9.1 with ipop3d spawned from inetd. We were also also having
performance problems with mail.local looping endlessly in contention for
mailbox locks when a given user received several messages that were
attempting delivery at the same time. Another concern was NFS performance
due to cacheing being turned off, as you'll see in my original message.
The status of the system at the time was were were experiencing high load
averages, tons of individual sendmail, mail.local and POP3 processes and
the result was many processes were blocked waiting for I/O at
peak times according to vmstat.
It turned out that running the POP3 daemon from inetd was a bad idea. To
get a picture of which of the two mail services (sendmail or pop3) was
consuming more resources, we split POP3 and SMTP between two machines
using our Cisco LocalDirector. It turned out, that with the exception of
periodic high sendmail loads and very occasional mail.local lock
contention, POP3 by far was using the most system resources. Since we
experienced a high average of short-lived POP3 connections, the ensuing
fork() and exec()'s from inetd caused a high load and increased paging
activity enormously. The solution was to get a multithreaded POP3
application.
The POP3 solution we chose was DPOP, part of the Dmail package available
from NetWin Software (http://www.netwinsite.com). For roughly $400 per
server, you can get both their multi-threaded SMTP (Dmail) and their POP3
(DPOP) servers with unlimited user licenses. Aside from being
multi-threaded, DPOP also supports custom external authentication (with
tunable cacheing), easy administration and installation, command line (cgi
ability, too) POP user addition (without requiring UNIX accounts) and
directory hashing support (Ie, /var/mail/u/s/user). It is totally awesome
and it fixed the great majority of our performance problems ;-).
NetWin's Dmail is also a highly configurable, mult-threaded MTA. We have
not installed it yet as we are in the process of doing more research
before abandoning sendmail.
We still have occasional problems with mail.local looping incessantly in
contention for a given user's mailbox when that user receives multiple
inbound messages at the same time. This is rare, but it happens
nonetheless. The number one recommendation I received to alleviate this
problem is to use procmail as the local delivery agent instead of
mail.local. Also, Aaron Holtz pointed out (see below) that it is possible
to achieve directory hashing with procmail, so this makes it even more
attractive for use in conjunction with DPOP's hashing support (10,000+
files in /var/mail has gotta be another big performance problem) . We will
probably install procmail as an intermediate step before migrating to
Dmail.
We also applied some minor sendmail tweaks to alleviate some problems due
to load spikes. Most notably, we limited the number of children spawned
from sendmail (Thanks David Lee for the reminder), and we set sendmail to
queue only when the load average on the machine approaches 60. Here's the
relevant m4 info for these tweaks:
define(`confQUEUE_LA',`60')
define(`confREFUSE_LA',`100')
define(`confMAX_DAEMON_CHILDREN',`120')
NB: This was set a bit high because I believe it effects all children,
including mail.local processes and what not. I'll have to refresh myself
on this one.
David Lee also recommended investigating the following value
confCONNECTION_RATE_THROTTLE
... I remember investigating this in the past, but I don't
remember why I opted not to use it. Maybe I fell asleep ;-).
With regards to my file locking concerns:
Jochen Bern also pointed out that another good reason for using Procmail
was due to its configurability in the area of file locking (see below).
Note to NetApp users: I may be wrong, but I believe the NetApp may need to
be explicitly configured to pass advisory locks. Jochen also points out
that flock() is local only on SunOS 4.x, but the Solaris 2.6 man page
points out that it has been fixed to work over NFS.
We're currently still using dotlocking which both DPOP and mail.local
support. We are investigating the potential for using flock() or lockf()
and possibly restoring attribute cacheing on our Netapp in an attempt to
curtail the getattr NFS calls that have dominated up to 70% of our NFS
calls in the past. At a minimum, we should be able to turn the cache back
on with dotlocking.
Thanks to all who have replied. Below my original post, please see other
helpful advice provided by the respondents of my original post.
Sincerely,
Jason Hatch
SysAdmin for many places ;-).
---------- Forwarded message ----------
Date: Tue, 16 Feb 1999 09:52:48 -0500 (EST)
From: Admin Staff <admin1@berkshire.net>
To: Sun Managers <sun-managers@sunmanagers.ececs.uc.edu>
Subject: NFS /var/mail; mail.local locking
Greetings Sun Managers,
We are running two SMTP/POP3 mail servers (Ultra10 clones, 300mhz, 1 gig
ram, lotsa swap, Solaris 2.6 patched) that are load balanced with a Cisco
Local Director 410. We are running Sendmail 8.9.1 configured for
mail.local, and for POP3, we're running ipop3d. We wanted CuciPop for it's
NFS awareness and general hackability, but could not get it to compile and
ipop3d was substituted in a pinch. The mail spool is provided by a NetApp
F630. I am having some performance problems, particularly with mail.local
and the contention for file locks.
When we researched running multiple mail servers that share the same
/var/mail (private mqueue) over NFS, we knew that file locking would be a
concern. We researched the Sun Managers archives and FAQs and found that
the general NFS mount options were as follows:
netapp:/export/mail - /var/mail nfs - yes
-O,noac,actimeo=0
Basically, the key was "noac" for no attribute cacheing, and "actime=0",
which is probably a bit redundant. We basically didn't want attribute or
write cacheing enabled. This ensures immediate updates of all information:
file attributes (access and modification times, advisory locking flags,
etc) and writes being immediately available.
The end result is that file locking works well, too well.
One of the immediate things we noticed was mail.local's behavior.
Mail.local, from what we could see with truss, achieves file locking by
writing a lock file (mailbox.lock, I think). If more than one mail.local
process needs to write to the same mailbox, we have a small party going on
that rapidly consumes the machine's resources, like a bunch of lions
fighting over the same piece of meat. It doesn't matter much if these
processes are running on one or both mail server, they seem to behave the
same, in that they are both acting on the NetApp filesystem.
Watching truss, each process is very active, checking for the lock several
times a second. There appears to be no random backoff period, just a bunch
of processes looping as fast as they can until by chance, one of them gets
the prize. Eventually they do, but depending on how many messages are
being delivered to that one user, a mail.local processes could loop for up
to 5 minutes before they score the lock.
Our mail servers, on average, handle about 20 inbound SMTP connections at
any time. I've seen this as high as 300 between 2 servers. What kills us
is not the sendmail processes, or the mail.local processes delivering to
different users, or the POP3 processes. It's when one user gets a ton of
mail at the same time. Woe to us if that mailbox is locked by the POP3
daemon, as the mail.local processes will mercilessly loop until the
mailbox becomes available. To reiterate, we could have 100 mail.local
processes delivering to say 60 users fine, but when we get 100 mail.local
processes delivering to the SAME user, the fireworks fly.
I tested this out by sending myself 100 messages with a simple perl
script. The LocalDirector landed the messages evenly between each mail
server, about 50 each. Each message did arrive (obviously not in order),
but what I witnessed was a feeding frenzy of mail.local processes on the
machines, lasting about 5 minutes and driving the load average above 60.
Once my 100 messages were delivered, the load average dropped to a nice
low level, and responsiveness improved.
I first noticed this when someone ran a KamaKaze mail script on one of our
web servers. This generated a ton of spam being sent from the owner of the
CGI process, www. The lamer had about 1.5 million e-mail addresses in his
possession. The mail was being delivered from the web server, and didn't
really result in a higher than usual load on that machine (in that it's
usually slammed). What alerted is to the whole deal was the BOUNCE
messages coming into user www, some 2000 an hour on the mail servers. The
mail.local contention for the "www" mailbox lock killed our servers and we
eventually had to alias www to /dev/null to preserve resources, as bounces
seemed to be coming in for quite some time after we located and terminated
the user.
Another general side affect we noticed are in NFS statistics:
Version 3: (261887628 calls)
null getattr setattr lookup access
0 0% 173244940 66% 6540510 2% 24642192 9% 10440793 3%
readlink read write create mkdir
484 0% 26472926 10% 3713090 1% 2418652 0% 4251 0%
symlink mknod remove rmdir rename
2238 0% 0 0% 4522609 1% 243 0% 7007 0%
link readdir readdirplus fsstat fsinfo
2193701 0% 752 0% 23004 0% 7660002 2% 171 0%
pathconf commit
63 0% 0 0%
Note how 66% of the traffic is used for "getattr". This statistic is
pretty stable: between 60% and 69%, with these statistics representing
about 18 days worth of traffic. The mount_nfs man page warns of this when
"noac" is used to mount a file system.
One of the things I noticed is that mail.local and ipop3d use the same
lock file mechanism for mailbox locking. This is nice, but the contention
is both time and resource consuming.
So I guess I have a few questions ;-)
1) Is there a local mailer I could use that has better contention
(ie, backoff and retry) mechanisms than mail.local?
2) Is there a better way I could achieve load balanced mail, with
SMTP and POP3 coming into multiple machines sharing the same
mailboxes, even if drastic change is required?
3) Would anyone care to share their methods used to achieving
this?
In exchange for some BIG JUJU, I promise to write a detailed summary to
this list :-)
-Jason
Responses are below:
>From benji@hnt.com Thu Jun 3 08:43:33 1999
Date: Tue, 16 Feb 1999 11:13:31 -0500 (EST)
From: Benjamin Cline <benji@hnt.com>
To: Admin Staff <admin1@berkshire.net>
Subject: Re: NFS /var/mail; mail.local locking
Just a thought, maybe try using procmail as the local mailer?
benji
--
Benjamin R. Cline Harrison & Troxell, Inc. benji@hnt.com
Quis Custodiet Ipsos Custodes?
>From bern@penthesilea.uni-trier.de Thu Jun 3 08:43:33 1999
Date: Tue, 16 Feb 1999 18:18:30 +0100 (MET)
From: Jochen Bern <bern@penthesilea.uni-trier.de>
Reply-To: bern@uni-trier.de
To: admin1@berkshire.net
Subject: Re: NFS /var/mail; mail.local locking
> One of the things I noticed is that mail.local and ipop3d use the same
> lock file mechanism for mailbox locking. This is nice, but the contention
> is both time and resource consuming.
Yet, it is pretty much unavoidable. POP daemons, especially those
that come out of a combined popd/imapd package, need to add an ID
number to the headers of every new mail (usually this header is named
"X-UIDL:" or somesuch), which means that the point where mail.local
could start appending to the mail spool keeps moving.
> 1) Is there a local mailer I could use that has better contention
> (ie, backoff and retry) mechanisms than mail.local?
I wholeheartedly recommend procmail as local mailer, the major reasons
being a) versatility (~/.procmailrc) and b) ability to deal with disk
quotas. I didn't inspect the sources, and it's been a while since I
installed it, but if I'm not very much mistaken, there is a load of
compile-time options for file locking ...
| penthesilea:/home/TI/bern% procmail -v
| procmail v3.11pre3 1995/05/17 written and created by Stephen R. van den Berg
| berg@pool.informatik.rwth-aachen.de
|
| Submit questions/answers to the procmail-related mailinglist by sending to:
| procmail@informatik.rwth-aachen.de
|
| And of course, subscription and information requests for this list to:
| procmail-request@informatik.rwth-aachen.de
|
| Locking strategies: dotlocking, flock()
| Default rcfile: $HOME/.procmailrc
| System mailbox: /usr/spool/mail/$LOGNAME
> 2) Is there a better way I could achieve load balanced mail, with
> SMTP and POP3 coming into multiple machines sharing the same
> mailboxes, even if drastic change is required?
Pure theory on my part: If you could set up sendmail so that incoming
mail *even for local users* would only get queued, to be delivered by
queue runs from cron, you'ld have pretty much a guarantee that there
will be only *one* instance of the local mailer at any point; Having
that one time out sufficiently fast should keep your system basically
alive (at the cost of a growing mail queue, if deadlock strikes).
(Variation: Leaving the sendmail setup unchanged while setting a short
timeout for the local mailer would still solve the problem as long as
SMTP connections to the victimized mailbox don't come in faster than
the timeout.)
Really nasty quick-and-dirty idea: Choose a limit for the number of
local mailers you want to have waiting for a specific mailbox, check
the actual numbers from a cron job regularly, and automatically add/
remove aliases for "overloaded" mailboxes that will communicate a
temporary delivery error back to sendmail (which should cause sendmail
to have the mail enter the queue).
Regards,
J. Bern
--
/\ /""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""\
/ \/ bern@uni-trier.de/bern@ti.uni-trier.de| P.O. box 1203 | Ham: \/\
/ J. \ (Accepting PGP, MIME, SUNAttachments) | D-54202 Trier | DD0KZ/ \
\Bern/ finger bern@informatik.uni-trier.de | Email autoreply \ /
\ /\ http://www.informatik.uni-trier.de/~bern/ | on subject '##' /\/
\/ \____________________________________________________________/
>From Dale.Shaw@select.com.au Thu Jun 3 08:43:33 1999
Date: Wed, 17 Feb 1999 09:25:30 +1000
From: Dale Shaw <Dale.Shaw@select.com.au>
To: Admin Staff <admin1@berkshire.net>
Subject: Re: NFS /var/mail; mail.local locking
Jason,
I have not researched this at all but you might like to take a look at
procmail as an alternative to mail.local.
See the README in the sendmail dist for the required .mc setting (I think it's
FEATURE(local_procmail) or something.
I think you can get it from
ftp.informatik.rwth-aachen.de/pub/packages/procmail/
I believe 3.11pre7 is the latest version.
cheers,
Dale.
>From T.D.Lee@durham.ac.uk Thu Jun 3 08:43:33 1999
Date: Wed, 17 Feb 1999 15:14:17 +0000 (GMT)
From: David Lee <T.D.Lee@durham.ac.uk>
To: Admin Staff <admin1@berkshire.net>
Subject: Re: NFS /var/mail; mail.local locking
On Tue, 16 Feb 1999, Admin Staff wrote:
> [...]
>
> Our mail servers, on average, handle about 20 inbound SMTP connections at
> any time. I've seen this as high as 300 between 2 servers. What kills us
> is not the sendmail processes, or the mail.local processes delivering to
> different users, or the POP3 processes. It's when one user gets a ton of
> mail at the same time. Woe to us if that mailbox is locked by the POP3
> daemon, as the mail.local processes will mercilessly loop until the
> mailbox becomes available. To reiterate, we could have 100 mail.local
> processes delivering to say 60 users fine, but when we get 100 mail.local
> processes delivering to the SAME user, the fireworks fly.
Our system is somewhat similar to yours (but our mailspool machines
are simply Solaris 2.6 direct, not indirect to a NetApp).
We occasionally (once every few months) see something similar (and indeed
we had such an incident today after your message arrived!).
You seem to have looked far deeper into it that I have. I don't have any
proven answers, sadly. So please summarise.
But after this morning's incident, I read through the cf/README file in
the sendmail 8.9.x source and noted:
confMAX_DAEMON_CHILDREN
and:
confCONNECTION_RATE_THROTTLE
These look worthy of further investigation (even though they are somewhat
generalised for handling this per-user type of problem).
> [...]
> One of the things I noticed is that mail.local and ipop3d use the same
> lock file mechanism for mailbox locking. This is nice, but the contention
> is both time and resource consuming.
Nice? Essential I think: both a POP process and an incoming sendmail
(mail.local) process may wish to update the file simultaneously. Indeed,
a few years ago we had occasional reports of mail going missing: not often
enough to be able to debug, but often enough to worry us that the effect
was genuine. A subsequent routine POP daemon upgrade mentioned a bug-fix
for faulty locking: the reports vanished.
> So I guess I have a few questions ;-)
>
> 1) Is there a local mailer I could use that has better contention
> (ie, backoff and retry) mechanisms than mail.local?
The "conf..." options above may help. (Let me know if they do!)
> 2) Is there a better way I could achieve load balanced mail, with
> SMTP and POP3 coming into multiple machines sharing the same
> mailboxes, even if drastic change is required?
The received wisdom appears to be "never do NFS locking with e-mail,
always do it directly within the machine". "Don't believe claims about
perfect NFS locking". Not sure how this relates to the (perhaps slightly
different?) case of a NetApp spool.
> 3) Would anyone care to share their methods used to achieving
> this?
I would if I had anything to share!
> In exchange for some BIG JUJU, I promise to write a detailed summary to
> this list :-)
Yes, please!
Incidentally, are you aware of IMAP (second generation POP)? Many mail
user agents now use IMAP4 instead of POP (Pine, latest Netscape, ...).
--
: David Lee I.T. Service :
: Systems Programmer Computer Centre :
: University of Durham :
: Phone: +44 191 374 2882 (ddi) South Road :
: Fax: +44 191 374 7759 Durham :
: Internet: T.D.Lee@durham.ac.uk U.K. :
>From kevin@ns1.interq.or.jp Thu Jun 3 08:43:33 1999
Date: Wed, 24 Feb 1999 12:11:04 +0900 (JST)
From: Kevin Ying <kevin@ns1.interq.or.jp>
To: Admin Staff <admin1@berkshire.net>
Subject: Re: NFS /var/mail; mail.local locking
On Tue, 23 Feb 1999, Admin Staff wrote:
> Thank you for your response. I'm looking forward to your thoughts.
>
> On Wed, 17 Feb 1999, Kevin Ying wrote:
>
> > Jason, we have a very similar configuration, and I wanted to let you know
> > that I'd be responding with a detailed response and comments... (doing
> > morning e-mail over here).
>
We're currently using the NetApp F630 Fibre Channel filer. We have two
shelves of 14 9GB drives. 12 used for space, 1 parity, and 1 hot spare.
So the total capacity is around 100GB. The NetApp holds user data for
both mail and web, as well as our /usr/local software.
Then, we have clusters of UltraSPARC AXi servers. We chose the SME AXi
servers because of their lower cost than the Sun-branded
Ultra5/10/30/60s...
I have 2 dedicated servers that handle incoming mail, 2 dedicated servers
for outgoing mail and POP3 service, 3 web servers for pages, 3 cgi servers
for user cgis (using CGIWrap so we don't have to audit any CGIs). I plan
to add a 3rd incoming mail and outgoing mail/pop server in the next month
or so.
We export our /etc/passwd and /etc/shadow files to all the servers that
need them for user services. They are mounted read-only to ensure that no
changes occur on our generic front-end servers. It's been working pretty
well with no user complaints about speed. I guess that's a testament to
NetApp's filer speed and Sun's nscd caching.
For the MTA, we are continuing to use sendmail. It's the MTA with which
we are most familiar and the access features make it easy to get rid of
spammers. We use procmail for local delivery that's been patched to use
the Maildir delivery format for safe NFS locks. Truly NFS safe because
each message is a separate file. It's murder on inodes but as long as
you manage it, it's OK.
For POP3, we use qmail-pop3d. It's a really small binary that runs out of
inetd and although I'd like a standalone version eventually, it's working
OK and the speed is sufficient.
Anything else I missed?
>From bern@penthesilea.uni-trier.de Thu Jun 3 08:43:33 1999
Date: Wed, 24 Feb 1999 20:18:45 +0100 (MET)
From: Jochen Bern <bern@penthesilea.uni-trier.de>
Reply-To: bern@uni-trier.de
To: admin1@berkshire.net
Subject: Re: NFS /var/mail; mail.local locking
> I think the locking I was witnessing from the POP3 daemon was a file lock
> in the format of user.lock. This is the same file that mail.local looks
> for.
I don't have an authoritative source for that, but I think that everything
locking a mail spool SHOULD (RFC-ese for "must, or you're outta business" ;-)
do that to claim a lock, and SHOULD respect other locks in ANY form it
recognizes, even if there's only one lock set. That way, you have the
user.lock file pretty much everything should recognize, but yet may use
another locking mechanism (*) for "usual situations".
(*) For example, I think that user.lock locking has a window of overriding
- process A checks, sees no lock, process B checks, sees no lock,
A puts the lock, B puts another OVERRIDING A's LOCK - which is not
the case with kernel-based locking mechanisms.
> Just out of curiosity, what POP3 daemon are you using?
None. :-) I have a working NFS-mount-dat-mail-spool-dir-everywhere (no
prominently Windows based users) installation - which is greatly helped
by the fact that the author of procmail took great pains do make the
file locking both NFS- and mailtool-proof, though the latter needs a
special quirk at compile time - and though I've been playing around
with popd's to fulfill a special user's requests, that guy ended up
running IMAP against a Netscape Messaging Server at another site (no
funds to buy that s/w for *this* site - luckily).
> would flock() work over NFS, or is it system function?
>From the flock() manpage (SunOS 4.1.3_U1B):
BUGS
Locks obtained through the flock() mechanism are known only
within the system on which they were placed. Thus, multiple
clients may successfully acquire exclusive locks on the same
remote file. If this behavior is not explicitly desired,
the fcntl(2V) or lockf(3) system calls should be used
instead; these make use of the services of the network lock
manager (see lockd(8C)).
(Note, however, that we found that Linux boxes do not have, and possibly
won't respect, a lockd for fileservers.)
> Hmmm... I never thought of this... So, when you start a sendmail process
> just to process the queue, it will spawn one mail.local process max?
Not 100% sure about that, however, I *am* sure that it will process one
mail at a time, which at the very least limits the number of local
mailers to the number of local recipients of a piece of mail. (Aren't
there sendmail.cf mailer flags to control concurrency? I can't check
offhand, I'm offsite today. I'm pretty sure that there's a mailer flag
controlling whether a mailer can send to several recipients in one run,
not_quite_there (tm) but close.)
Regards,
J. Bern
--
/\ /""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""\
/ \/ bern@uni-trier.de/bern@ti.uni-trier.de| P.O. box 1203 | Ham: \/\
/ J. \ (Accepting PGP, MIME, SUNAttachments) | D-54202 Trier | DD0KZ/ \
\Bern/ finger bern@informatik.uni-trier.de | Email autoreply \ /
\ /\ http://www.informatik.uni-trier.de/~bern/ | on subject '##' /\/
\/ \____________________________________________________________/
>From aholtz@bright.net Thu Jun 3 08:43:33 1999
Date: Sun, 28 Feb 1999 15:10:45 -0500 (EST)
From: Aaron Holtz <aholtz@bright.net>
To: Admin Staff <admin1@berkshire.net>
Subject: Re: NFS /var/mail; mail.local locking
Jason,
I was searching through the newsgroups and saw this posting. I
was quite intrigued by it as I have a similiar situation to contend with.
We have 65,000 dialup customers that do an inordinate amount of mail. Our
mail machine takes about 400,000 connections per day, if not more. Our
current setup:
Ultra Sparc II dual 300mhz processors
2gb RAM
CMD 5500 Hardware RAID controller
90gb disk space
/var/mail is a separate wide scsi controller into the sparc. Our mail
spool is hashed at /var/mail/letter1/letter2 where the letters are the
first two fields of the username. We are using sendmail 8.9.3 and
procmail for local delivery (much nicer than mail.local from what I've
seen.) We are using cucipop for our popclient (same guy who wrote
procmail wrote this.) However, we are getting seriously CPU bound in the
evenings. Disk I/O is great even during the horrible peak hours. My
solution was this:
Cisco local Director
Several Ultra sparcs that nfs mount /var/mail from a central server
Eventually get a NetApp unit to be that NFS central server.
My concerns regarding locking appear to be the same as yours. From what
I've seen, by using cucipop and procmail compiled to ONLY do dotlocking
(you can choose from lockf, flock and fnctl methods too), then you are
safe in an NFS environment. From what the author claims regarding his
dotlocking method, that it is much less overhead than say what qpopper
does. You seem to already be doing almost all of this (except for the
procmail/cucipop combo.) How is your performance during peak hours? About
how much mail do you do in a day? I'm curious about how cucipop and
procmail would play together over NFS using dotlocking. Again, from what
I've read, it should be very good. I'm trying to create a solution that
will last us for quite a while - the above method looks to be very
scalable. I'm just wondering if NFS mounting /var/mail is the answer - I
really can't see another way of doing it efficiently. I have read about
using maildir instead of mbox format (maildir is where each user gets a
directory for new and old mail - each message is then stored in a separate
file instead of all in one file like mbox.) The big benefit of maildir is
that no locking has to take place since each message is in a separate
file, so it is NFS friendly. But there is a big change to take place to
get existing mail to that format as well as hacking software to support it
(not to mention the extra inodes and disk space used up....) It appears
you have a Netapp - can I ask which one and whether you are happy with it
or not? Maybe we can come to some conclusions and solutions together.
Hope this helps.
--------------------------------------------------------------------------
Aaron Holtz
ComNet Inc.
Manager, Unix Systems Administration
Email: aholtz@bright.net
"It's not broken, it just lacks duct tape."
--------------------------------------------------------------------------
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:13:20 CDT