Salutations managers!
Once my first summary was posted I have received several useful and
circumstantial replies to my email regarding that summary. I hope my
2nd summary will be helpful for other people that might come accross
with the problem. My original posting is located at the end of this
essay.
I'm very grateful for the following people for their advices:
Brian Mullen <mullen@hurricane.net2.nlu.edu>
Casper Dik <casper@holland.Sun.COM>
Birger Wathne <birger@sdata.no>
Gustavo Chaves <gustavo@cpqd.com.br>
Fred Crowner <crowner@oar.net>
Jim O'Dell <jim@ecs.fullerton.edu>
Special thanks to Gustavo Chaves who sent me two perl scripts spit and
spitd that is very handy in probing your network for different data
transfer speed problems. These scripts are attached to that summary.
As I wrote before, looks like the problem was due to half duplex
connection for one of two serves. If you have read my 1st summary then
you might be aware of the 11th ftp-data packet that was delivered with
very long delay (3.9 seconds) and thus it garbled the overal FTP transfer
speed.
I used spit and spitd scripts to test how do the data flow from one server
to another. Gustavo Chaves noted in his email (you will find it below) that
"by varying the size of the blocks (the fourth argument of spit) the timings
don't vary proportionally as one would expect." Nope, in my case, I didn't
find such discrepancy. Block's size didn't affect the speed of data flow in
both directions. My approximate measures are:
serverA -> serverB about 10Megs/s
serverA <- serverB about 7Megs/s
I installed 2.6 Recommended patches on both servers, and then rebooted and
turned off/on Cisco Catalyst 5000 (software version 2.2). It turned out
that now Cisco shows full duplex to both servers. Then I used spit/spitd to
track data transfer rate for both servers and what would you think? :) It
worked ok! Now my measures slightly go up:
serverA -> serverB about 10Megs/s (practically not changed)
serverA <- serverB about 9Megs/s
Well, it all sounds misteriously, and even now I'm not sure what has
improved the transfer speed. But as for FTP, long lag with the 11th packet
still persist.
I performed some experiments with connecting one server (serverB) to 100
base-TX hub. The hub then has been connected to Cisco Catalyst 5000
switch. At this configuration, serverB automatically switched to half duplex,
and spit/spitd measures were like:
serverA -> serverB about 10Megs/s
serverA <- serverB about 3Megs/s
Well, I suppose the epopee with low transfer speed is moving to its end.
The only thing left to be clarified is the 3.9 seconds delay with 11th
ftp-data packet.
Thanks so much for your generous help, folks!
With kind regards,
Vitaly Beliaev
***********************************************************************
**** Set of replyes I received to my 1st summary *****
***********************************************************************
Ubiquitous Casper Dik <casper@holland.Sun.COM> adviced:
> ... my posting
>
> /usr/sbin/ndd /dev/tcp tcp_xmit_hiwat 65535
> /usr/sbin/ndd /dev/tcp tcp_recv_hiwat 65535
> /usr/sbin/ndd /dev/tcp tcp_cwnd_max 65534
> /usr/sbin/ndd /dev/tcp tcp_deferred_ack_interval 10
> /usr/sbin/ndd /dev/tcp tcp_rexmit_interval_min 1500
BAD! Dont' change that. This means that when a packet does get lost, TCP
will wait 1.5 seconds minimum; this is a workaround setting for earlier
bugs and you really should keep the default 200ms
>Then I snooped FTP transfer in both directions. I found that when you issue
>'get' command, then you get 11th packet being delivered with pretty big
>delay that was around 3.9 sec, while other packets have been delivered in a
>0.00012 sec, just in a flash!
>
Looks like a packet drop and then a retransmission. Smells like serious
switch problems (misconfigured half vs full duplex would still be the
safest bet)
Casper
============================================================================
From: birger@sdata.no (Birger Wathne)
What is the software version on your Cisco's? We just upgraded ours (5 or 6
of them, I think) to 2.4(4). All our Sun's immediately switched to full
duplex. I think we used to be at 2.2. Throughput over the trunk lines
between the Cisco's seems to have gone up after the upgrade. There was a
bug in the bug database related to degraded trunk line performance when
communicating over a trunk between two ports on 10/100 autosense cards.
Almost all our Cisco ports are 10/100 autosense.....
Birger
==========================================================================
From: Gustavo Chaves <gustavo@cpqd.com.br>
Dear Vitaly,
Some months ago I noticed a very similar problem in one of our
SPARCstation-5-Model-70's. I tried to find my notes about it but couldn't.
I remember that the problem was noticed precisely in FTP. When I
started FTP in the buggy machine, gets were fine, but puts were too
slow. When I started FTP in another machine and made a conection to
the buggy machine, the situation was inverted.
I snooped the conversation between the machines and was able to see
big delays between some of the packets, but not all. Exactly what you
are seeing.
I think I must have done the experiments using rcp or something
because I quickly saw it has nothing to do with FTP per se. To better
study the problem, I implemented a couple of Perl scripts. One is a
server (spitd) and the other a client (spit) of a TCP connection.
Starting spitd in another machine I can use spit in the buggy one and
studdy the pattern of their conversation. I'm sending the scripts to
you hoping they can be useful.
When you start spitd in the remote machine, it displays a message like
this:
vespa$ spitd
/home/gustavo/bin/spitd 2690: server started on port 3333 at Mon Mar 16 09:40:10 1998
It sits there listening for a TCP conection. When it's stablished, it
reads information as fast as it can in blocks of at most 10000 bytes.
For each block read, it displays its number and its size.
The client script, spit, must be invoked in the buggy machine. (In
fact, you can reverse this and the results will be reversed.) You
must invoke it with four arguments: the remote machine's name, the TCP
port in which spitd listens, the number of blocks to write, and the
size of each block. Like this:
buggy$ spit vespa 3333 30 1000
Time taken: 1 secs ( 0.00 usr 0.00 sys = 0.00 cpu)
It connects to spitd and tries to perform 30 writes of 1000 bytes
each. The whole process is timed so that you can see how log it
takes. For each read spitd performs, it displays its number and the
number of bytes read. This is what spitd should display:
1: 1000
2: 1000
3: 1000
4: 1000
5: 1000
6: 1000
7: 1000
8: 1000
9: 1000
10: 1000
11: 1
read 10001 bytes in 11 packets from arraia
(Never mind the last 1 byte. I use it for signaling the end of
transmission.)
Now, what's interesting is that varying the size of the blocks (the
fourth argument of spit) the timings don't vary proportionally as one
would expect. In my experiments, packet sizes less than 537 bytes
(I'm not sure about the exact number, but it was about that.) went
pretty fast. Packets of 538 bytes went in bursts, with long delays
between some of them. The delays persisted as I increased the size of
the packets to about 1020 bytes. The upper limit wasn't as clear cut
as the lower, but from 1020 bytes on the strange delays didn't
appeared any more.
Well, I hope you can make similar experiments to see if something
similar happens.
To end the story... I called my Sun representative, we did some
testings, and the problem vanished when we substituted a
SPARCstation-5-Model-85 for the SPARCstation-5-Model-70 main board of
the workstation. I performed some tests in other
SPARCstation-5-Model-70's and was able to reproduce the problem,
although with different frame sizes as lower and upper limit.
I still don't know if this is a hardware problem, or a soft/hard one.
Perhaps the net driver in the kernel has a bug that only manifests
itself in this particular hardware. (BTW, I'm running SunOS-5.5.1
with recomended patches in my machines. The experiments were
performed last November.) The Sun service asked me to perform some
experiments with `sar', to see if the machine isn't overloaded. It's
not. The experiments were performed in the laboratory, without any
other extraneous processes running. I'm afraid to say that I'm
delaying the conversation with Sun because I've not much hope they'll
be able to sove it. I'm expecting to get rid of all
SPARCstation-5-Model-70's in a short while.
Ah! One more thing. The problem was first noticed when we changed
the network medium linking the workstationg. Formerly, they used
Ethernet coaxial 10Mb. We changed it to twisted-pair and hub-switches
giving dedicated 10Mb for each TP. I'm not sure how important this
is.
I'd like very much to hear about your experiments (if you decide to
perform them, that is).
Please, find the scripts attached.
Good luck,
=======================================================================
From: crowner@oar.net (Fred Crowner)
Hi. I missed seeing your original posting, but saw the summary.
Check the version of software on the Cat 5000. I've had several instances
of the problem you are describing with Cat 3000s that were fixed by either
moving the ethernet cable to a different port on the 3000 or installing a
software update on device.
==========================================================================
From: "Jim O'Dell" <jim@ecs.fullerton.edu>
Hello,
I saw your summary and noticed I have a similiar setup. You didn't mention
which version of Solaris you where using. I'm using Solaris 2.5 and don't
seem to have a problem with ftp.Here are my settings. Heck, it shouldn't
hurt to try them :)
ndd /dev/hme adv_100fdx_cap 0
ndd /dev/hme adv_autoneg_cap 1
ndd /dev/tcp tcp_xmit_hiwat 65536
ndd /dev/tcp tcp_recv_hiwat 65536
ndd /dev/tcp tcp_cwnd_max 65535
ndd /dev/tcp tcp_deferred_ack_interval 50
ndd /dev/tcp tcp_rexmit_interval_min 200
****************************************************************
*** ORIGINAL POSTING ***
****************************************************************
Hi Managers!
We have two Ultra Enterprise 5000 & Solaris 2.6 servers. Both are connected
to 100base-TX ports on Cisco Catalyst 5000 switch. Both servers are at 100
Mb/s.
But when I do FTP from one server to another I can see that the file
transfer speed for "get" command is 3Mb/s, and for send is 10Mb/s.
I still can't figure out why it's a big difference? I would highly
appreciate your generous help!
Here is an example of configurations both servers have:
1. While being on whale server and running FTP, netstat shows:
whale.33916 pharaon.eco.mmk.chel.su.ftp-data 24820 0 64240 0 TIME_WAIT
whale.33915 pharaon.eco.mmk.chel.su.ftp 65160 0 66608 0 ESTABLISHED
2. Current values of some TCP parameters:
tcp_xmit_hiwat 65535
tcp_recv_hiwat 65535
tcp_cwnd_max 65534
tcp_deferred_ack_interval 10
No patches were installed on both servers yet. As netstat shows, Swind is
smaller than Rwind, perhaps that is the reason of slow file transfer speed?
If yes, then I wonder how can I increase Swind?
--Vitaly Beliaev Unix Systems Administration. JSC MISW, Magnitogorsk, Russia. voice: +7 3511 335639 mailto://vit@mmk.ru http://www.mmk.ru -===========================================================-
#!/net/gnu/bin/perl -w
require 5.002;
use strict;
use Socket;
use Benchmark;
my ($remote, $port, $iaddr, $paddr, $proto, $n, $size, $buffer);
$remote = shift;
$port = shift;
$n = shift;
$size = shift;
if ($port =~ /\D/) { $port = getservbyname($port, 'tcp') }
die "No port" unless $port;
$iaddr = inet_aton($remote) or die "no host: $remote";
$paddr = sockaddr_in($port, $iaddr);
$proto = getprotobyname('tcp');
socket(SOCK, PF_INET, SOCK_STREAM, $proto) or die "socket: $!";
connect(SOCK, $paddr) or die "connect: $!";
$buffer = ' ' x $size;
my $t0 = new Benchmark;
for (; $n; --$n) { syswrite SOCK, $buffer, $size; }
syswrite SOCK, "1", 1;
my $line = <SOCK>;
close(SOCK) or die "close: $!";
my $t1 = new Benchmark;
my $td = timediff($t1, $t0);
print "Time taken: ", timestr($td), "\n";
exit;
#!/net/gnu/bin/perl -w
require 5.002;
use strict;
BEGIN { $ENV{PATH} = '/usr/ucb:/bin' }
use Socket;
use Carp;
sub logmsg { print "$0 $$: @_ at ", scalar localtime, "\n" }
my $port = shift || 3333;
my $proto = getprotobyname('tcp');
socket(Server, PF_INET, SOCK_STREAM, $proto) or die "socket: $!";
setsockopt(Server, SOL_SOCKET, SO_REUSEADDR, pack("l", 1))
or die "setsockopt: $!";
bind(Server, sockaddr_in($port, INADDR_ANY)) or die "bind: $!";
listen(Server, SOMAXCONN) or die "listen: $!";
logmsg "server started on port $port";
my $paddr;
$SIG{CHLD} = \&REAPER;
my $size = 10000;
my $buffer = ' ' x $size;
for (; $paddr = accept(Client, Server); close Client) {
my ($port, $iaddr) = sockaddr_in($paddr);
my $name = gethostbyaddr($iaddr, AF_INET);
my $n;
my $total = 0;
my $pkts = 0;
while ($n = sysread(Client, $buffer, $size)) {
$total += $n;
++$pkts;
print "$pkts: $n\n";
last if substr($buffer, -1) eq '1';
}
print Client "fim\n";
print "read $total bytes in $pkts packets from $name\n";
}
sub REAPER {
$SIG{CHLD} = \&REAPER; # if you don't have sigaction(2)
my $waitedpid = wait;
logmsg "reaped $waitedpid" . ($? ? " with exit $?" : "");
}
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:12:34 CDT