Got a few answers regarding part of my query, which was admittedly rather broad. The question is repeated at the end of this email.
Late news: We just found out that we are running a version of parityck that can cause problems with our configuration. We just killed multiple copies of it on two servers, and it reduced I/O times considerably. It did not help on one of our E3500s, however, but that doesn't have the same kind of disk configuration as the servers in question. Since this email was almost ready to send when we found out about the parityck problem, it will be sent as was written, FWIW.
Continuing with the originally intended summary:
There was some question whether the A3500 might just be too slow, not the disks. Good question. We've been having some controller problems, including what look like failures that rm6 can't fix...until after an SSE shows up on site, then rm6 can fix it. It's happened more than once.
We also got some tuning parameters to make vxfs/vxvm more efficient. We will be testing these out in the near future. They are (exactly as received):
A few things you can try --
- Stripping the volume over multiple controllers would help a great deal but if you cant do that - (NOTE: We can't, right now. MW)
- Try tuning up VM and VxFS a bit.
/etc/system
set maxphys=8388608
set ufs:ufs_HW=8388608
set ufs:ufs_LW=6291456
* File System support for VXFS 3.2.5+
set lwp_default_stksize=0x8000
set rpcmod:svc_run_stksize=0x4000
* Volume Manager support.
set vxio:vol_maxioctl=131072
set vxio:vol_maxspecialio=10240
set vxio:vol_maxio=10240
set vxio:voliomem_max_memory=12582912
set vxio:voliomem_kvmap_size=15728640
set vxio:voliomem_base_memory=1048576
- Plus tuning the kernel to give more cache to VFS would help - Especially with Mail files.
(Parameters based on 4GB mail server with HW Raid controllers - adjust to your needs) (NOTE: we have about 5% of a 702GB mail data area used - MW)
/etc/system
* Setup priority paging to make better use of limited Ram.
* Must remain in this order for proper loading.
set priority_paging=1
set cachefree=65536
set pages_before_pager=16384
set lotsfree=32768
set desfree=16384
set minfree=8192
* Speed up the scan rate to run through plenty of memory pages.
set fastscan=131072
set handspreadpages=131072
set maxpgio=65536
set slowscan=500
* Slow down pager and fsflush scan rates.
set autoup=600
set tune_t_fsflushr=5
set bufhwm=20480
And the most important thing is a small detail we had forgotten about: the vxstat command. It made our problems with iostat irrelevant, as it showed us statistics by subdisk that showed only two of the seven were getting any traffic.
What we didn't get was any idea of where to find information on how to calculate stripe sizes if we went to a striped plex/volume setup. Our seven subdisks are currently 68GB each, and we wonder if the standard 128KB/stripe value is really quite what we want.
Similarly, no one had any solid input on whether the SSE's suggested plan for switching from concatenated to striped plex/volume was feasible.
Only one person noticed the controller/disk array name switch error (still tired from the previous night's crash?) in the original question:
> Date: Tue, 10 Oct 2000 14:55:03 -0400
> From: Michael Watson <n6wav@netscape.net>
> To: sun-managers@sunmanagers.ececs.uc.edu
> Subject: Disk config poor for I/O? E10K/A3500/rm6/SEVM 2.6
>
> We have a mail server which is showing high I/O wait cpu stats on iostat -c, and users have been known to complain that the system
seems slow. The system is an E10K domain with D1000 controller over five full trays on an A3500, H/W raid by rm6 into 13 LUNs. The
mail user space is RAID5, 10 LUNs showing as 10 physical disks to the system, which are vxfs and concatenated each as a single subdisk
into a single plex and single volume under SEVM/vxvm 2.6.
>
> I/O wait times are generally 60-80% all the time. We have Virtual Adrian running, and it complains about the native cxtxdxsx
volumes, but never about the sd devices that show up under iostat -x (we don't know if it would). In fact, one of our problems is
that we cannot seem to map all the sd devices to virtual devices recognized by SEVM. Our busiest sd device has these cumulative
stats:
> device r/s w/s kr/s kw/s wait actv svc_t %w %b
> sd686 5.1 25.9 60.9 277.3 0.2 1.3 49.2 2 36
>
> and maps to path_to_inst as:
> /sbus@69,0/QLGC,isp@0,10000/sd@5,4 686 sd
>
> but no cxt5d4sx appears in vxva or any command line disk print output (e.g., vxprint, vxdisk list). This is true of about half the
sd devices iostat -x can see.
>
> Our SSE has suggested defrag of the volume, shrinking it, taking the remaining space, creating a new striped volume, then mirroring
the old volume and after it syncs up, breaking the mirror and moving the new volume to the old mount point, and recovering the space
from the old volume into the new.
>
> The part about being able to mirror a volume with another existing volume sounds unfamiliar to us, but does sound feasible? In
fact, does it sound like we would gain much from doing this in the first place?
>
> More details are available for anyone interested.
S
U BEFORE POSTING please READ the FAQ located at
N ftp://ftp.cs.toronto.edu/pub/jdd/sun-managers/faq
. and the list POLICY statement located at
M ftp://ftp.cs.toronto.edu/pub/jdd/sun-managers/policy
A To submit questions/summaries to this list send your email message to:
N sun-managers@sunmanagers.ececs.uc.edu
A To unsubscribe from this list please send an email message to:
G majordomo@sunmanagers.ececs.uc.edu
E and in the BODY type:
R unsubscribe sun-managers
S Or
. unsubscribe sun-managers original@subscription.address
L To view an archive of this list please visit:
I http://www.latech.edu/sunman.html
S
T
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:14:19 CDT