No more responses for a few days, so here are the results.
My thanks to the list for some very useful information. I've been a watcher
and occasional responder for a while now, but this is the first time I've
asked for data. Sonofagun! It works ;-)!
The original post:
>Reprise of question: user with homebrew software, doing data acquisition from
>random PC card and firing it up to an NFS-mounted filesystem on the Sun.
>Throughput is lower than I expected (low tens of KB/sec, vs. 100-200).
>Is this a PC/NFS limitation, or should I point a finger at his software?
>If PC/NFS, can I tune it?
Special thanks to Stefan Mochnacki for pointing out that there are news
groups for NFS and Unix<->PC networking in general, and for posting my
request to those groups (I have no Internet access).
Such tuning as is possible in software relates to write buffer sizes on
the PC side. Around 8K seems to be near optimum, and delivers 60ish KB/sec.
In particular, byte-at-a-time writes are agreed to be especially dreadful.
This was the key in my case -- my user's homebrew software was writing data
in 256-byte chunks. We are now getting almost 70 KB/sec from a middlin' 386
(25 Mhz) talking to an IPC, which my user calls acceptable. As usual, your
mileage may vary.
The general consensus is that NFS performance is write-limited unless you
add hardware (Prestoserve or other accelerators). Without hardware, 60 to 80
KB/sec is the best you can do. Juergen Keil offered a kernel patch to
SunOS 4.1.1 to enable asynchronous NFS writes, with significant risk of
losing data and not noticing it! Use at your own risk! However, in our
case we may try this; the stuff coming over is masses of data used to
generate statistics, and a couple of bits dropped on the floor here and
there will be lost in the noise. I'll post the results if we get a big
improvement, but as usual priority drops to zero as soon as the user stops
actually yelling in my ear, so it may be a while...
The other proposal was Sneakernet -- store locally on PC (tape, optical or
removeable disk) and carry the data over to the Sun. Probably reasonable,
but inconvenient.
Thanx and a tip o' the hat to:
Stefan Mochnacki princeton!centaur.astro.utoronto.ca!stefan
Alan Arndt aga@Comtech.com
Anatoly M. Lisovsky princeton!kamaz.kazan.su!Anatoly.Lisovsky
Juergen Keil princeton!leo.tools.de!jk
Ove Nygren ove@ies.luth.se
Peter J. Welcher pjw@math30.sma.usna.navy.MIL
Arie Bikker aribi@geo.vu.nl
danny princeton!ews7.dseg.ti.com!danny
Hal Stern princeton!sunne.East.Sun.COM!stern
Ian Phillipps princeton!unipalm.co.uk!ian
--------------------------- full responses follow -----------------------
>Date: Fri, 23 Oct 92 10:58:54 EDT
>From: Stefan Mochnacki <princeton!centaur.astro.utoronto.ca!stefan>
To get useful answers, you need to specify:
* the version of PC-NFS you are running
* the type and speed of the PC,
* the make and model number of the Ethernet
card in the PC
* the version of OS on the Sun, and what patches have
applied.
Stefan Mochnacki
PS: Tens of KB is OK for an old PC with an old card ...
-----------------------------------------------------------------------
>Date: Fri, 23 Oct 92 12:13:48 EDT
>From: Stefan Mochnacki <princeton!centaur.astro.utoronto.ca!stefan>
We have 5 PC-NFS licenses of rather elderly vintage (3.1 I think).
We use PC's for data acquisition, though in practice we have found
it better to store data locally and transfer it later (PC disks
are pretty cheap now). I found that 25KB/sec using old 8088
PC's or "XT's" was about average. AT's will give you 50KB/s. The
newer 386's with 16-bit cards do better, but I haven't measured
their speeds; 100KB/s should be attainable.
Sometimes one gets VERY SLOW transfers. I now know that with NCSA
Telnet the problem is broken hand-shaking; the PC doesn't shut up
when the host tells it to. I think the earlier versions of PC-NFS
may have had a similar problem. A FAST host is probably very
important for good PC-NFS performance. Also, certain versions
of SunOS require patches (e.g. 4.1 needs the "NFS Jumbo Patch";
fixed a lot of our problems).
Stefan Mochnacki
--------------------------------------------------------------------
>Date: Fri, 23 Oct 92 09:17:37 PDT
>From: princeton!Comtech.COM!aga (Alan G. Arndt)
PC-NFS write performance SUCKS. 10-15KB is about it unless the Sun has
a WRITE cache or hardware accelerator for the disk. He is write his software
is fine.
---- Alan Arndt Comtech Labs 415-813-4500 4005 Miranda Avenue, Suite 150 aga@Comtech.com Palo Alto, CA 94304-1218--------------------------------------------------------------------- >Date: Fri, 23 Oct 92 17:15:33+0400 (GMT) >From: princeton!kamaz.kazan.su!Anatoly.Lisovsky ()
You must tune not PC-NFS, you must tune entire NFS to emprove a write performance. On PC-NFS's side check if you use packet driver. Use 16-bit Ethernet card with direct driver from the vendor.
For example, use NFS accelerator on SPARCstation.
- Anatoly M. Lisovsky, Kamaz Inc. KamaFlash SMG Chairperson ------------ Network is computer. Per aspera ad Sun! ------------
---------------------------------------------------------------------
>Date: Fri, 23 Oct 92 20:49:46 +0100 >From: princeton!leo.tools.de!jk (Juergen Keil)
That's normal behaviour for NFS writes, an NFS server may not send a positive acknowledge for an NFS write call until all the data for the write is safely stored on the disk (i.e. the NFS server uses synchronous writes for writing all the data in an NFS write request). If the server would send the client a positive acknowledge before the data is safely stored on the disk (i.e. the data would be in the UNIX buffer cache marked as a modified buffer that needs to written to disk at a later time) and then it crashes, the data is lost; since it already told the client that the write was ok, the NFS client *has* *no* *possibility* to detect this problem (the client got positive answers for all his write requests but some data is missing in the written file!!).
There are two possible solutions for the problem:
- you can buy prestoserve for your NFS server from sun ($$$). Prestoserve writes the data into a non volatile memory on the prestoserve board, then sends the acknowledge to the NFS client. The data is written from the NVRAM to disk at a later time. If the server crashes before writing the data to disk you still have a copy of the data in the NVRAM and can write it to disk during the reboot process.
- If you have a sun4c and SunOS 4.1.1, you can use the following sh script/patch to turn on asynchrounous NFS writes on the server for *all* NFS clients on the running kernel; rfs_write+258 should contain the value 0x96102004 in the original kernel. (I told you above why this is bad idea; NFS write performance is a lot better, though):
#!/bin/sh adb -w -k /vmunix /dev/mem <<E_O_F rfs_write+258/W 96102000 E_O_F -- Juergen Keil jk@tools.de ...!{uunet,mcsun}!unido!tools!jk
--------------------------------------------------------------------
>Date: Fri, 23 Oct 92 17:01:15 EDT >From: Stefan Mochnacki <princeton!centaur.astro.utoronto.ca!stefan>
> > Great! Thanks for the help! (for posting query to Usenet) >
You're welcome.
> Actually, your first reply was a big help in itself. He's got a 386; should > be getting better than we're seeing. I got hold of a few relevant chunks of > source, and I'm looking for funny business (like lots of short writes instead of > buffer of reasonable size).
Care is needed here. Firstly, the PC card may have a limited buffer size, and secondly a patch may need to be applied to the IPX's SunOS. Big NFS packets (or whatever they are called) may NOT be a good idea!
I'll also try to collect some statistics and see > if it's an intermittent hang/retry rather than slow all the time. We know > we're not actually losing data (at least I think we know that... better > check). >
Good idea. However, I think we HAVE lost data; we were trying long data acquisition sequences writing to a BIG file ( ~10MB), and we apparently lost some MB's along the way.... After that, we switched to local storage at acquisition, followed by subsequent transfer.
- Stefan ---------------------------------------------------------------
>From: princeton!ies.luth.se!Ove.Nygren >Date: Sat, 24 Oct 1992 09:00:40 +0100
We had a similar problem when we try to use 3Com cards (10BT) with a HUB from another manufacturer. We did a lot of testing and found nothing to be wrong with software or hardware on the client PC:s. The only thing that helped was swapping HUB to one from 3Com!!!. After doing that, speed gained from 25-50 Kb/sec to 200-250 Kb/sec.
We use a lot of different cards, machines and OS, but this was the first time I saw this problem and I have no real explanation to what really went wrong.
Ove Nygren University of Lulea, Sweden ove@ies.luth.se
---------------------------------------------------------------
>Date: Sat, 24 Oct 92 12:48:36+0400 (GMT) >From: princeton!kamaz.kazan.su!Anatoly.Lisovsky ()
=>Is this a PC/NFS limitation, or should I point a finger at his software? Yes! ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ He must not write byte-by-byte because each writed byte will occupy one datagram!
- Anatoly M. Lisovsky, Kamaz Inc. KamaFlash SMG Chairperson ------------ Network is computer. Per aspera ad Sun! ------------
----------------------------------------------------------------
>Date: Sun, 25 Oct 92 14:27:48 EST >From: pjw@math30.sma.usna.navy.MIL (Peter J. Welcher -- math FACULTY <pjw@math30.sma.usna.navy.MIL>)
Check if he's writing data a byte at a time or using a buffer. This *may* may a big difference. ---------------------------------------------------------------------- Dr. Peter J. Welcher EMAIL: pjw@math30.sma.usna.navy.MIL Mathematics Department, M/S 9E PHONE: (410) 267-3606 572 Holloway Road U.S. Naval Academy Annapolis, MD 21402-5002 FAX: (410) 267-4883 ----------------------------------------------------------------------
>Date: Mon, 26 Oct 92 16:02:50 +0100 >From: Arie Bikker <princeton!geo.vu.nl!aribi>
Hi,
We have been using PC/NFS here for 3 years on 100+ PC, and our experience is that writes will top at 80 kb/sec if conditions are right: - network bandwidth available (ofcourse) - network packet size is large enough; this can be crudely checked with etherfind. (mine is 2048 bytes) - dos write requests are large enough; the application program should preferably do writes in chunks of a whole number of kilobytes; writing byte by byte makes it crawl.
Here is a table of our write iospeed for several sizes of dos writes for a 1 Mb file on a BUSY!! net and slow file server (SUN 3/60!!):
req. size time (sec) speed (kb/sec) 16384 16.0 62.5 8192 16.0 62.5 4096 23.2 43.1 2048 39.2 25.5 1024 63.2 15.8 512 118.4 8.4
test configuration : server SUN 3/60 (8 Mb), dos client 386SX-16Mhz DOS 3.3 (worst case figures ;)
I hope this gives you insight into the problem you have.
P.S. Here's the program i used for testing speed:
#include <stdio.h>
main(argc, argv) int argc; char **argv; { FILE *ofil; int bsiz; long iolen; char *buff; bsiz = atoi(argv[1]); iolen = atoi(argv[3]); iolen *= 1024; ofil = fopen(argv[2], "wb"); buff = calloc(bsiz, 1); while (iolen > 0) { fwrite(buff, bsiz, 1, ofil); iolen -= bsiz; } fclose(ofil); }
Arie Bikker ___=======-----
------------------------------------ -- Drs. Arie Bikker -- -- Instituut voor Aardwetenschappen -- -- Vrije Universiteit -- -- Amsterdam Netherlands -- -- aribi@geo.vu.nl -- ------------------------------------
>From: princeton!ews7.dseg.ti.com!danny >Date: Mon, 26 Oct 92 09:08:12 CST
this may not be relevant, but we have people trying to backup their PCs with the 'RPUT' command in pcnfs; at one time they got good thruput but now it is very slow, for no apparent reason. there may be a generic PCNFS bottleneck triggered by something...
-----------------------------------------------------------
>Date: Tue, 27 Oct 92 09:42:48 EST >From: princeton!sunne.East.Sun.COM!stern (Hal Stern - NE Area Systems Engineer)
it could be PC/NFS or it could be server write performance.
try: (a) sunos 4.1.2 (b) prestoserve board on the sun
you're doing lots of small writes with PC/NFS, which is probably the slowest thing you can do with NFS.
--hal
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:06:51 CDT