Here is the (long-awaited?) summary of the answers to my request for NFS
load characterization information. In general, I'm not passing along names
or exact quotes, as some of the people I talked to didn't want their names
given out. If there is interest in seeing most of the conversations that
took place, let me know, and I'll go check with people to make sure that
they don't mind being quoted.
I'm going to talk first about some of the software I heard about, and
what I think some of the problems are with this software. I'll then move on
to discuss some approaches to NFS load monitoring, discuss some other
numbers which are either interesting or which might have some impact,
mention some information on models, and finish up with some information I
got from some of the NFS fileserver vendors. All told, there were about
fifty messages and a few phone calls exchanged on this subject.
Caveat: I haven't had as much time as I'd hoped to look into this
information in detail. You will no doubt wish to dig further...
As it turns out, there's a surprising amount of software floating around
that looks at NFS. Such software includes:
nfswatch -- curses-based promiscuous NFS monitor. This program
prints out a running tally of how many of each type of
request comes in, and of which file systems are the most
heavily used. Nfswatch can be used to look at traffic to
individual files, too. This is anonymously FTPable from
icarus.riacs.edu.
server_stat -- a NFS monitor program that runs on Encore Multimaxes.
This shows information on hosts, users, and NFS request
types performed. This is capable of talking to a
rpc.srvstatd process on another machine, though I don't know
of other machines that support the Encore srvstatd program.
nfsstone -- the Encore NFS benchmark, as presented in:
Shein, B., Callahan, M., Woodbury, P., NFSSTONE: A Network
File Server Performance Benchmark, Usenix Summer 1989
conference proceedings, pp 269-275.
This is a synthetic benchmark load, with an empirically-
determined mix of operations.
nhfsstone -- the Legato NFS benchmark. This is also a synthetic
load generator, based again on a particular observed
load mix. You can get this by sending mail like:
To: request@legato.com
Subject: send nhfsstone
path path_back_to_me
I had some problems getting this, though I was ultimately
successful.
EtherView -- a Sun-based packet spy that is capable of doing some
characterization of NFS load and response times. This is
a commercial product. For more information, contact:
Matrix Computer Systems, Inc.
7 1/2 Harris Rd, Nashua, NH 03062
(603) 888-7790
LANWatch -- another packet spy, from FTP Software, Inc. This can
filter out NFS traffic; I don't know what can be done with
the packets though once they're filtered out. For more
information, call FTP at (617) 246-0900, or send mail to
info@ftp.com.
[ There's lots of other packet spies, too, and I suspect that most
of them can do at least a little bit with NFS packets. ]
The problem with most of the programs above (except for the synthetic
loads, to which this just doesn't apply, since they're not NFS monitors) is
that you don't get raw data points which you can then analyze. You get the
data that the authors thought you might want... and which might not be what
you really want. There's much to be said for the approach of dumping traces
and lots of timestamps into a file, then providing (a) programs that analyze
such files, and (b) the format of the files, so that people can write their
own analysis programs. On a PC-based packet spy, this is a hard thing to
do.
There's a fair number of people (at the major NFS server vendors, Sun,
DEC, and a few universities) who are also poking around at the problem.
Some people are looking at filesystem activity tracers, which (in addition
to being interesting research on its own) could provide still more reams of
interesting statistics when combined with a NFS tracer.
The consensus was that the best way to trace NFS operations is to do so
via a promiscuous packet spy. Such an approach has many advantages. First,
if you don't have kernel sources, you can still get useful information.
Second, because you don't instrument the server kernel, you don't have to
worry about influencing the experiment in adverse ways. However, there's
some chance (depending on your hardware and on how fast you make your
software go) that you'll drop packets. The 'hack the server kernel'
approach won't drop any requests, but violates the above constraints. I
suspect that the best way to gather statistics is by using *both* methods of
measurement, then comparing the results.
I was also referred (twice) to the SunOS 4.1 NFS implementation, and in
particular the adaptive NFS retransmission code therein. These numbers might
be interesting to see, once 4.1 is more easily available.
Of course, the usual Unix file access pattern (i.e., lots of short-lived
files in /tmp, most of which get written, then read once, then deleted)
information applies. This was mentioned by several people; one reference
given was:
Floyd, Rick, Short-Term File Reference Patters in a UNIX Environment,
University of Rochester Department of Computer Science TR 177,
March 1986.
Another good paper (with not much data on NFS, though) is:
Lazowska et al, "File Access Performance of Diskless Workstations",
ACM TOCS, volume 4, number 3, August 1986, pp 238-268.
Not a whole lot was said about general models of NFS access. Most places
that had any models had derived them from some number of studies and from
the output of nfsstat, or so it seemed. It does seem that there's a few
general trends, however. There are some sites (including ours, I suspect)
that fall into the low-utilization, few write model, where the server rarely
satisfies more than one client's NFS requests in some given time slot.
There's also the high-utilization, many write model, which is what I'm sure
a lot of sites see. One responder said that once one's clients have enough
memory, the buffer cache ends up reducing the number of random reads going
on, so one is left with the reads that happen to start up a new process, and
with writes.
Those who talked about models generally said that they think there's
almost as many models as there are networks using NFS. I suspect that this
is true, but that perhaps some useful information (or at least methods) can
be abstracted out, regardless.
A number of people also suggested that I talk to Legato and to Auspex and
see what they've done in this area. I have a couple of papers from Auspex;
at a first glance, I don't think they look too closely at NFS load
characterization (at least, not as I define that), but instead concentrate
on what Auspex did to get better speed out of their NFS file server. The
Auspex paper titled, "Benchmark Methodology and Preliminary Performance
Specifications for the Auspex NS5000 Network Server" (Bruce Nelson, Auspex
TR #2, October 1989) has more load characterization information than do the
other Auspex TRs I have, but it still doesn't have a whole lot. (By the
way, I'm not implying that Auspex hasn't looked at load characterization,
because they obviously have. I just don't have the fine details of their
results.) I also did some talking with people at Legato; their comments and
models show up in the nhfsstone benchmark, or are otherwise repeated above.
-Steve
Spoken: Steve Miller Domain: steve@umiacs.umd.edu UUCP: uunet!mimsy!steve
Phone: +1-301-454-1808 USPS: UMIACS, Univ. of Maryland, College Park, MD 20742
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:05:57 CDT