Hi Everyone,
The original message was:
> I have SS2 with 64M memory running 4.1.1 + patch 100232-01, running a
> memory intensive application (Synopsys compiler). My process size is
> about 200M+, and my local swap is around 350M.
>
> Everything seems to work, but once in a while (2-4 times every hour),
> my machine 'freezes' on me for like 2-4 minutes. The symptoms are that
> my merrily noisy swapping disk gets quiet and its access light is
> continuosly on, while the load widget shoots up from 1-2 to 8-10
> (after the system recovers). While 'frozen', no activity is taking
> place on the screen (clock's hands are still, etc.)
>
> Doing ps before and after shows the process taking like 85+ percent of
> memory before, while less than 10% immediately after. Then it starts picking
> up again back to 80+ percents.
>
> My obvious reading is that for some reason the system decides to swap
> out essentially my whole process to the disk. I have no idea why, as
> the only other process around needs a very small (1-2M) memory, although
> it does use CPU a lot. This swap out is the period of 2-4 minutes of
> 'freeze' that I experience.
What I got were 2 main advices:
1. Change the variable 'nosched' in the kernel to 1 using adb, and it
will disable 'preepmtive' swapping out of the process.
The 'cookbook' approach for that is: (as root)
>> # cd /
>> # cp -p /vmunix /vmunix.KEEP
>> # adb -w /vmunix
>> nosched?W 1
>> _nosched: 0x0 = 0x1
>> $q
>> # reboot
This works (more or less) as advertised, but once in a while (like every few
hours) the machine 'froze' on me for extended periods (like 30-60 minutes!),
so I tried the other approach, suggested by Bob Webber.
2. Change the kernel variables of lotsfree ( 100 -> 1024 ),
fastscan ( 1000->16000 ), nopagereclaim ( 0->1 ) and most importantly
maxslp ( 20->few thousands ).
Read "The Design and Implementation of the 4.3BSD UNIX Operating
System" by Leffler, McKusick, Karels, and Quarterman if you want to get
some real insight on what these do (although it probably will totally change
in Solaris 2).
I have object only distribution and I couldn't find all the variables
in the include files (maybe Bob used previous OS version, or he has a
source) so I used adb again and increased ONLY maxslp to like 2000.
The result seemed to be a decent compromise of swapping out very rarely
(maybe once an hour, taking 2-4 minutes), while other activity in the machine
could go on in parallel as before, and I experienced no real long 'freezes'
anymore.
Bottom line - I hope Solaris 2 will be better in handling *large*
applications, and meanwhile find yourself a *real* CPU server with a lot
(128-256M) of physical memory as I did :-)
An excerpt from Bob Weber's reply follows:
...
> The SunOS kernel is derived from the Berkeley 4.3 kernel, and the
> book The Design and Implementation of the 4.3BSD UNIX Operating
> System by Leffler, McKusick, Karels, and Quarterman describes many
> features of the kernel, including the implementation of memory
> management schemes (pages 145 to 161). I made a number of changes
> based on this information; unfortunately I'm not sure which one did
> the trick but Modeler was varying between 75% CPU use and 90% CPU
> use; it is possible that more tuning would result in still higher
> performance.
> I modified parameters as follows: * change "lotsfree" parameter to
> 1024 from 100; this parameter specifies Kbyte at which page daemon
> scans stop * change "fastscan" to 16000 from 1000; this parameter
> specifies how many pages are scanned per invocation of the page
> daemon when little free memory is available * change "nopagereclaim"
> to 1 from 0; suppress reclamation of pages not written to but not
> recently in use * change "maxslp" from 20 to 300 to 4864; this
> parameter specifies how long a process can sleep on a busy system
> before it is swapped and would also appear to be the parameter with
> the greatest single effect on Modeler performance
> The first few parameter changes affect how well the system handles
> large tasks. With a small value of fastscan, too little of the
> systems memory is scanned for activity during any give invocation of
> the page daemon. A small value of lotsfree postpones the use of the
> page daemon until the fraction of memory still free is quite small,
> on a large-memory system.
> The last parameter, however, affects the memory management system's
> operation in a profound way in the case of Modeler. It would appear
> that the Modeler process was trying to reference a page which had
> been swapped, then going into a "sleep" state to wait for the page to
> be brought back. I think that it was sleeping for so long that the
> process was being swapped wholesale to disk, and that most of the
> wall-clock time consumed by the Modeler run was going into swapping
> the Modeler process in and out of physical memory.
> There is a distinction between swapping and paging which is
> important for an understanding of this process. In swapping the
> entire process image (in this case 240 MB) is copied from physical
> memory to the "swap space" on the system's disk. In paging, only a
> "page" (for our systems, about 8KB) is copied. Boosting the maxslp
> parameter allows Modeler to wait for a page for longer without
> getting swapped out, avoiding the performance penalty of waiting for
> the swap out and corresponding swap in when the requested page is
> finally made available and the process resumes execution.
>
...
The honor list is:
webber@alphatech.com - Bob Webber
trinkle@cs.purdue.edu - Daniel Trinkle
rlg@ida.org - Randy Garrett (albeit reverse logic)
dpk@fid.Morgan.COM - Doug Kingston
stern@sunne.East - Hal Stern - NE Area Systems Engineer
plus general type of comments and requests for summaries from:
cyerkes@jpmorgan.com (Chuck Yerkes)
tonytran@leland.stanford.edu
parens@cards.dazixco.ingr.com (Paul Arens)
bwalker@auratek.com (Brad Walker)
Morry Katz <katz@sid.stanford.edu>
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:06:46 CDT