Summary: File size issues

Thanks to the following repondents:
Darren Dunham [ddunham@taos.com]
Casper Dik [casper@holland.sun.com]
Rich Kulawiec [rsk@gsp.org]
Thiagarajan, Karthikeyan [kthiagarajan@doitt.nyc.gov]
Dave Gagliardi [dgagliardi@zonecms.net]
Chandramohan Kowdlex , Tidel Park - Chennai [ChandramohanW@msdc.hcltech.com]
Bill Voight [Bill.Voight@fcc.gov]

Darren Dunham provided the most definitive information with the following:

The application itself maintains a filehandle and a current pointer
within it...  After some time, you've got this..

application:  FH=4, FP=1234568
0                             1M
[xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx]  size=1234567 length=1234567

The application has written 1.2MB to the file, and knows that the next
byte goes to position 1234568.   You nuke the file out from under it and
it writes another 10 bytes, but starting at the same position...

application:  FH=4, FP=1234578
0                             1M
[                                  x]  size=10     length=1234577

The size of the file is small (just whatever blocks are necessary to
hold the 10 bytes), but the length is big.  You can't change that
without signalling the application somehow to close and then reopen a
new file.  Usually by restarting the app, or sometimes it's written to
rotate the logs on signal like HUP.

Casper Dik offered this response that proved to be true:

An open file descriptor has an offset associated with it; the process will
continue to write at that offset; the first write after the truncation will
cause a large hole to appear at the start of the file, filled with NULLs.

If you copy the file then you run out of disk space but the large
files themselves do not take space as they have holes int hem which read
as zeros.

The application which writes the files needs to be changed to use O_APPEND
when opening the file; this will cause all writes to do a seek to the
actual end of the file.


The Netegrity support staff (siteminder application) confirmed that Casper
Dik's suggestion has indeed been
applied in their 5.5 release to circumvent this problem in that version.

The issue we are dealing with is that the applications utilizing this
product is a global insurance app for which
there is not a convenient window to cycle the app and rotate the logs.

The solution for our monitoring concerns is that  I wrote a perl script that
utilizes du to determine the actual disk
usage we have used to replace the original sitescope script that used df.


Original inquiry:
I have a file issue I do not understand.

O/S: Solaris 8 Generic_108528-19
Pltfrm: E10K domain

We have an actively running web application whose siteminder webagent is
logging to siteminder.log.  I was under the impression that you could do
something like:

cp siteminder.log otherlocation.log
cat /dev/null > siteminder.log and free up disk space.

It seems that shortly after I do this the filesystem reports that the log
file is at the size it was when I began and still growing.  df reflects this
as well.

I wrote a script that captures a particular case:

+ ls -l siteminder.log
-rwxrwxr-x   1 kintana  iplanet  109625128 Oct 20 14:37 siteminder.log
+ wc -l siteminder.log
   49312 siteminder.log
+ wc -c siteminder.log
 109629860 siteminder.log
+ cp -p siteminder.log /var/tmp/kintana
+ cat /dev/null
+ 1> siteminder.log
+ ls -l siteminder.log
-rwxrwxr-x   1 kintana  iplanet        0 Oct 20 14:38 siteminder.log
+ wc -l siteminder.log
       0 siteminder.log
+ wc -c siteminder.log
       0 siteminder.log
+ sleep 60
+ ls -l siteminder.log
-rwxrwxr-x   1 kintana  iplanet  109635695 Oct 20 14:39 siteminder.log
+ wc -l siteminder.log
      65 siteminder.log
+ wc -c siteminder.log
109647732 siteminder.log

You can see that the line count appears that it may be correct but the
character count shows the file to still be very large.  We have often dealt
with this in the past successfully.  But now it seems to be ineffective.
Can someone help?  We are running out of disk space fast....


g			 ERC
_______________________________
Rick Brashear
Server Systems
Information Technology Department
Employers Reinsurance Corporation
5200 Metcalf
Overland Park, Kansas 66201
*	913 676-6418
*	rick.brashear@ercgroup.com
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers