Hi There, My questions was: Currently, our CC VOB server is showing some performance trouble. It's a E3000, two CPU's (248 Mhz) 2GB memory and Solaris 8. Recommendations state that the amount of memory should be: VOB database size/2 Our database size is 14GB ergo, 7GB main mem would be needed. BUT, what we don't understand is, that if we don't have enough memory, than why is the SR (ScanRate) parameter as shown by vmstat -S not greater than zero: procs memory page disk faults cpu r b w swap free si so pi po fr de sr m0 m1 m2 m3 in sy cs us sy id 0 0 0 2917976 1114016 0 0 724 4 4 0 0 0 0 0 0 1575 1123 108 18 21 61 0 0 0 2702088 908768 0 0 501 5 5 0 0 0 0 0 0 1951 18250 3184 27 29 44 0 0 0 2700384 907776 0 0 696 0 0 0 0 0 0 0 0 2248 20124 2622 37 38 26 0 0 0 2705832 910832 0 0 698 2 2 0 0 0 0 0 0 1741 19004 2512 34 29 36 0 2 0 2718848 918688 0 0 2106 2 2 0 0 0 0 0 0 1929 21766 2643 41 31 28 0 1 0 2718848 918992 0 0 1541 2 2 0 0 0 0 0 0 1889 17221 2606 32 29 39 0 1 0 2718832 919264 0 0 1077 8 8 0 0 0 0 0 0 1803 22904 2581 47 33 20 0 0 0 2718848 919368 0 0 576 16 13 0 0 0 0 0 0 2298 20052 3341 42 31 27 0 0 0 2718424 919240 0 0 1077 8 8 0 0 0 0 0 0 1827 21083 2543 41 34 26 0 2 0 2717944 918984 0 0 1453 10 8 0 0 0 0 0 0 2013 18530 2953 41 27 32 0 1 0 2715416 917480 0 0 602 0 0 0 0 0 0 0 0 1540 20359 2190 36 28 36 0 0 0 2715416 917416 0 0 626 13 10 0 0 0 0 0 0 1681 18923 2518 35 29 37 We do see some other issues on te machine, there seems to be an WIO issue, but now I'm trying to find out if we really NEED 7GB in our VOB server. Any help would be greatly appreciated. I got several replies, thanks you all. But I still don't know how to prove whether or not we need more memory. We plugged in another mainboard with 2GB mem and 2 CPU's. Some users reported an improvement, some didn't and vmstat reports 2.5GB free memory now. Reply's: ------------------------------------------------------------ Thomas Autry: Some good information on performance tuning a system as well as capacity planning can be found in the following guide: http://www.sun.com/blueprints/0902/816-7882-10.pdf It is a design guide for the sun fire series but can be used for planning of any system. ------------------------------------------------------------ Matt Harris: It can't hurt. I'd say upgrade those old processors, too. :-) As far stuff to check, you may want to check out your FS cache performance. Depending on what kind of database it is, and what kind of filesystem it's sitting on, there's probably at least one or two simply kernel tunables that could increase your performance, as well. If it's an internal database, chances are they're using DBM or somesuch (just making an educated guess here, I know nothing of this specific product), then FS tunables could really help a lot. If it's Oracle on a raw partition, then that could explain the scan rate thing... Since I don't believe that Oracle's raw data devices go through VFS at all... In reality there's a billion things it could be or that could help, if you can provide more details as to exactly what the system is doing a lot of, what it's doing it to, and how it's doing it, I can probably provide more exacting advice. :-) ------------------------------------------------------------ Ed: We run ClearCase here on an E3K, similar to yours, but with 1GB main memory for a 4.5GB VOB and have no issues with it. I have never seen the VOB/2 recommendation from Rational. What version are you running? Part of the rationale may bet that MVFS structures are memory hungry in newer versions of the software. Beyond that, sr almost always seems to be 0 in Solaris 8 and 9 due to the introduction of the disk page vs memory page caches. SR would often top out when file cache pages took up too much available memory and had to be paged out to make room for process memory pages. This seems to be the real-world result of running high memory, low-to-mid utilization servers under Solaris 8. Ask me again when I get Oracle running on a Solaris 8 box, though. ;) ------------------------------------------------------------ Wanke Matthias: I'm lost about VOB (what's that?) but i doubt that for a total DB size of 14GB you need 50% in memory; normalay 1%-5% should be enough, given all the sophisticated algorithms of todays DB engines; as i see in your output you have nearly 1GB RAM free, so are there measurements or metrics that support the fact that you actually have a problem (on the OS side) - where can one see the problem? I don't seem to be able to see it in the output!? I don't know about your application but sys-time seems a bit high on a 2 processors machine - do you have software raid on your disk subsystem? ------------------------------------------------------------ McIntosh, Alan: Wow, does this sound familiar! We were having the same problems with our builds due to memory limitations on our VOB servers. Don't believe that crap that Rational is feeding you about "Total VOB database size/2", our largest VOB was approaching 37GB with a total of over 80 VOB's at well over 180GB. That would have meant we would need around 90GB of memory according to Rational! I have attached a PowerPoint presentation for the current, planned, and future recommended environments for our builds. We actually only increased our memory on the VOB's to 12GB, rather than the planned 28GB, and added another VOB server to split the load, which had an additional 12GB. Our build times went from almost eight(8) hours to around 2-2 1/2 hours after the switch! By using current equipment obtained from other departments, we were able to make the changes for under $50,000 as opposed to other recommendations that would have cost well over $250,000. Hope this helps. ------------------------------------------------------------ Kevin Buterbaugh: I'm not familiar with ClearCase VOB - does it use shared memory like other databases (Oracle, Sybase) do? If so, what do you have your shminfo_shmmax parameter set to? Generally speaking for DB servers, the more memory, the better. If, on the other hand, CC VOB doesn't use shared memory, then the statistics you include seem to indicate that you do not need more memory. HTH... ------------------------------------------------------------ Karl Vogel: Two things come to mind, swap and scan rate. Try adding the lines below to /etc/system and rebooting. I run with the swapfs_minfree and autoup settings and they've never caused me trouble. * * Swap * System keeps 1/8th of all memory for swap, which is too much for * a 4GB system. Reduce that to 32 Mbytes (4096 8K pages). set swapfs_minfree=4096 * * Memory management * * http://www.carumba.com/talk/random/tuning-solaris-checkpoint.txt * Tuning Solaris for FireWall-1 * Rob Thomas robt@cymru.com * 14 Aug 2000 * * On firewalls, it is not at all uncommon to have quite a bit of * physical memory. However, as the amount of physical memory is * increased, the amount of time the kernel spends managing that * memory also increases. During periods of high load, this may * decrease throughput. * * To decrease the amount of memory fsflush scans during any scan * interval, we must modify the kernel variable autoup. The default * is 30. For firewalls with 128MB of RAM or more, increase this * value. The end result is less time spent managing buffers, * and more time spent servicing packets. set autoup = 120 ------------------------------------------------------------ Jeff Woolsey: > Currently, our CC VOB server is showing some performance trouble. > > It's a E3000, two CPU's (248 Mhz) 2GB memory and Solaris 8. > > Recommendations state that the amount of memory should be: > VOB database size/2 > > Our database size is 14GB ergo, 7GB main mem would be needed. That seems inordinately huge, from my fading recollection of ClearCase. IIRC, they're talking about the VOB _database_, not the files in it. It's sort of like filesystem metadata, except that ClearCase has a lot of metadata, as filesystems go. > BUT, what we don't understand is, that if we don't have enough > memory, than why is the SR (ScanRate) parameter as shown by > vmstat -S not greater than zero: The above would seem to apply. A VOB database is not the whole VOB. ------------------------------------------------------------ Tristan Ball: Clearcase relies on the disk cache for performance. You _must_ have a high cache hit rate for good performance. In particular, think about this as a hypothetical example case: A cache hit takes 1ms, A cache hit takes 100ms. You have 100 data accesses. At a 100% hit rate, that means 100ms total access time. At 99%, thats 199ms access time, at 95% that 595ms, and 80% its 2080ms! 2080ms vs 100ms is obviously huge. :-) In reality,a memory cache hit is generally going be in the order of tens or low hundreds of nanoseconds, and a disk hit is going to be 10-50ms, I used 1 and 100 to make the math easy and clear. Clearcase accesses the metadata a lot, and if you're getting cache misses, performance drops through the floor. You don't have to run out of memory to see this, you just have to have too little to cache effeciently. Rational's recomendation for the memory size is roughly correct, although if most of your metadata was legacy information and not often accessed, you could probably get away with a vobdb/3 ratio. I try and aim for a vobdb/1 ratio, and then wind that back untill the finance guys stop giving me "the look". :-) I can possibly help with your WIO issue, if there really is one. I'd be looking at "iostat -dxcn 1" output during problem periods if I was you. Of course, increasing ram will reduce the number of IO's anyway. :-) For your system, if you've only just started seeing problems, and it was OK before, I think you'll find that adding 2G will return your performance to previous levels. I'd also point out that while the cache example above is pertinant, it's not always cost effective to get the highest possible hit rate. If there's only one access a second, then going from 90%->95% might not even be user noticable! And as for how I know all this, we went through exactly the same process. It really didn't look like the server was loaded. <25% cpu load, 100 ops/sec on a disk array that I knew would sustain 600+, <10% network traffic on a gig eth link. We went from 2g to 4g on our E450 and the engineers actually cheered. As we have ongoing discussions here about our rational infrastructure, I'd love to hear a little about your setup, number of users, types of clients, where your views are stored. We're currently looking at upgrading ourselves... :-) ------------------------------------------------------------ Janis Lykakis. E-mail: janis.lykakis@ehv.pdsl.philips.com Philips PDSL _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagersReceived on Mon Nov 11 06:02:29 2002
This archive was generated by hypermail 2.1.8 : Thu Mar 03 2016 - 06:42:57 EST