my original question is:
one my E3500 server, df and du give different result on /var:
# du -ks /var
39853 /var
# df -k /var
/dev/vx/dsk/var 482824 482581 0 100% /var
Many people think that there is a file that was deleted that is still
being held open by some process. they really hit the nail on the head.
when i killed all the cron related processed, the problem is solved,the
space is freed.
The fact is it is not me who deleted a file that was still being opened
by other processes,it is the solaris itself,in my opinion.
solaris uses "mv $LOG $OLOG" in /etc/cron.d/logchecker to update cron
log. Maybe when mv is runed, /var/cron/log is being written.*I'd like to
get more suggestion on it.*
thanks to everyone who give the answer.especially to Thompsom Dave who
sent me some useful materials. I include it in the attachment to share
it with everyone.
Zhou Na
Subject: FW: df verses du
Date: Fri, 28 May 1999 08:54:27 -0400
From: "Thompson, Dave" <ThompsD1@bsci.com>
To: zhouna@asiainfo.com
----------
From: Herman, Yvonne
Sent: Friday, October 23, 1998 11:39 AM
To: Keefe, Matt; Thompson, Dave; Brown, Howard; Nichols, Jason; Herman,
Yvonne
Subject: df verses du
Info Docs article 17509 [ Notify of changes ]
[ Edit/Retrieve Marked Documents ] [ Mark Document ]
Did the search result in the document you were looking for? Send Feedback
------------------------------------------------------------------------
INFODOC ID: 17509
SYNOPSIS: Inodes and the filesystem
DETAIL DESCRIPTION:
Inodes and the filesystem:
1. one explanation for why du and df can disagree
2. how deleting a file may not free up disk space
3. how can I be out of inodes?
4. a diagnostic procedure
5. a diagnostic script
1. one explanation for why du and df can disagree
-------------------------------------------------
df describes disk usage
du adds up the size all the files on a filesystem
They can report wildly different figures.
One reason:
processes write to files
files can be deleted while processes are writing to files
so a file may be gone, but its inode may not be freed up
example:
# cat > bogus
testing
testing
^Z
Stopped (user)
# ls -i bogus
2921 bogus
# fuser bogus
bogus: 14947o
# rm bogus
# ls -i bogus
bogus: No such file or directory
# /usr/proc/bin/pfiles 14947
14947: cat
Current rlimit: 64 file descriptors
0: S_IFCHR mode:0620 dev:32,120 ino:211552 uid:0 gid:7 rdev:24,17
O_RDWR
1: S_IFREG mode:0644 dev:32,5 ino:2921 uid:0 gid:1 size:16
O_WRONLY|O_LARGEFILE
2: S_IFCHR mode:0620 dev:32,120 ino:211552 uid:0 gid:7 rdev:24,17
O_RDWR
#
This example shows how a file can be deleted but still take up room
on a hard drive:
A file (bogus: inode 2921) is created by a process (pid: 14947)
The files is deleted and it is demostrated that it no longer resides
on the file system.
But pfiles shows the inode is still open and is taking up 16 bytes
of disks space
2. how deleting a file may not free up disk space
-------------------------------------------------
This is the same reason a disk may be full and removing a file does not
clear space.
If you do a fuser on a file before removing it you will determine if
removing it will do any good.
Example:
# ls -l /var/adm/messages
-rw-r--r-- 1 root other 218611 Jul 29 02:18 messages
# fuser /var/adm/messages
messages: 164o
# ps -ef | grep 164
root 164 1 0 22:47:49 ? 0:03 /usr/sbin/syslogd
# kill 164
# rm /var/adm/messages
# touch /var/adm/messages
# /usr/sbin/syslogd
# ls -l /var/adm/messages
-rw-r--r-- 1 root other 7592 Jul 29 11:46 messages
#
This example demostrate a way to remove the /var/adm/messages and
actually clear disk space.
Notice how the messages file jumps up to 7k after syslog is restarted.
syslog will dump the contents of the dmesg buffer into the
/var/adm/messages when it is started.
3. how can I be out of inodes?
------------------------------
Not all inodes captured by a processes are normal files.
Many are character devices which take no disk space but will remove a
inode from possible use.
These character devices may take up memory which in turn reduce the
amount of memory available to your tmpfs file systems (ie, /tmp).
Example:
# /usr/proc/bin/pfiles 10234
10234: xterm
Current rlimit: 64 file descriptors
0: S_IFCHR mode:0620 dev:32,120 ino:211538 uid:52475 gid:7 rdev:24,3
O_RDWR|O_LARGEFILE
1: S_IFCHR mode:0620 dev:32,120 ino:210782 uid:52475 gid:7 rdev:0,0
O_WRONLY|O_LARGEFILE
2: S_IFCHR mode:0620 dev:32,120 ino:210782 uid:52475 gid:7 rdev:0,0
O_WRONLY|O_LARGEFILE
3: S_IFCHR mode:0666 dev:32,120 ino:210791 uid:0 gid:3 rdev:13,12
O_RDWR
4: S_IFIFO mode:0666 dev:171,0 ino:1626250456 uid:0 gid:0 size:0
O_RDWR|O_NONBLOCK FD_CLOEXEC
5: S_IFCHR mode:0000 dev:32,120 ino:65232 uid:0 gid:0 rdev:23,13
O_RDWR|O_NDELAY
# ps -ef | grep 10234
dsweet 10234 10220 0 23:16:58 ?? 0:01 xterm
# ls -lL /dev/dsk/c0t0d0s0
brw-r----- 1 root sys 32,120 Jun 24 19:51 /dev/dsk/c0t0d0s0
# df /
/ (/dev/dsk/c0t0d0s0 ): 1097712 blocks 456994 files
# find / -mount -inum 211538
/devices/pseudo/pts@0:3
# find / -mount -inum 210782
/devices/pseudo/cn@0:console
# find / -mount -inum 65232
#
This example shows that dsweet's xterm has captured 3 inodes* on his
root partition but none were normal files.
None of them have size so they are special character files (a raw
device).
The last inode (65232) was not found on the file system.
Therefore inode 65232 can not be freed unless its process lets go of
it or the process is killed.
* inode 210782 is the console and it was captured twice
4. a diagnostic procedure
-------------------------
This is all well and good, but how does a user figure out where all
his inodes have gone?
The procedure:
Determine the major and minor numbers for the troublesome device:
# df /
/ (/dev/dsk/c0t0d0s0 ): 1097710 blocks 456994 files
# ls -lL /dev/dsk/c0t0d0s0
brw-r----- 1 root sys 32,120 Jun 24 19:51 /dev/dsk/c0t0d0s0
#
Determine which inodes on the device are opened by processes:
# ls /proc
0 10220 10235 10251 10282 10386 136 160 227 298
306
1 10222 10236 10253 10300 10429 141 178 235 3
3060
10120 10227 10237 10255 10310 10446 143 184 236 301
308
10130 10228 10238 10266 10314 10484 14778 194 267 302 99
10134 10229 10239 10268 10338 10485 14934 2 270 303
10144 10230 10240 10270 10352 10567 14935 2115 275 304
10167 10231 10241 10272 10353 106 14937 2117 285 305
10169 10232 10242 10276 10364 10605 14947 212 288 3055
10170 10233 10243 10278 10366 108 14969 215 289 3057
10219 10234 10245 10280 10385 119 15008 225 295 3058
# /usr/proc/bin/pfiles 0 | grep "dev:32,120"
# /usr/proc/bin/pfiles 10220 | grep "dev:32,120"
0: S_IFCHR mode:0620 dev:32,120 ino:211538 uid:52475 gid:7 rdev:24,3
1: S_IFCHR mode:0620 dev:32,120 ino:210782 uid:52475 gid:7 rdev:0,0
2: S_IFCHR mode:0620 dev:32,120 ino:210782 uid:52475 gid:7 rdev:0,0
3: S_IFCHR mode:0666 dev:32,120 ino:210791 uid:0 gid:3 rdev:13,12
8: S_IFCHR mode:0000 dev:32,120 ino:62920 uid:0 gid:0 rdev:42,126
9: S_IFCHR mode:0000 dev:32,120 ino:55000 uid:0 gid:0 rdev:41,171
10: S_IFCHR mode:0000 dev:32,120 ino:64688 uid:0 gid:0 rdev:42,130
11: S_IFREG mode:0644 dev:32,120 ino:313393 uid:0 gid:3 size:316
#
Just by checking the first 2 processes I found 7 unique processes
that are open on the file system.
You have to repeat this step for all the processes found in
/proc to get all the open inodes on the file system.
Determine if the inodes exists on the file system:
# find / -mount -inum 211538
/devices/pseudo/pts@0:3
# find / -mount -inum 210782
/devices/pseudo/cn@0:console
# find / -mount -inum 210791
/devices/pseudo/mm@0:zero
# find / -mount -inum 62920
/usr/lib/lpshut
# find / -mount -inum 55000
# find / -mount -inum 64688
# find / -mount -inum 313393
/etc/group
#
Therefore only 2 of the 7 inodes captured by process 10220 are rogue.
Rouge is my way of describing an open inode that doesn't have
cooresponding file in the file system.
You would have to repeat this step for all the inodes found above to
see which open inodes are rogue.
Once done with the last step you can go back to see how many inodes
are rogue, if they are special character devices, and how much space
they are taking up.
5. a diagnostic script
----------------------
There has to be an easier way then the procedure just described.
You could write a script to do the work for you or use mine:
--------------------------------------------------------------
#!/sbin/sh
tmpfile=/tmp/icheck.$$.tmp
if [ "$1" ]
then
device=$1
if [ -b "$device" ]
then
mntpoint=`/usr/sbin/df | /usr/bin/grep $device | /usr/bin/awk ' { print
$1
} '`
if [ $mntpoint ]
then
firsthalf=`/usr/bin/ls -lL $device | /usr/bin/awk -F\, ' { print $1 }
'`
secondhalf=`/usr/bin/ls -lL $device | /usr/bin/awk -F\, ' { print $2 }
'`
major=`/usr/bin/echo $firsthalf | /usr/bin/awk ' { print $5 } '`
minor=`/usr/bin/echo $secondhalf | /usr/bin/awk ' { print $1 } '`
possiblePIDs=""
for pid in `/usr/bin/ls /proc`
do
inodes=`/usr/proc/bin/pfiles $pid 2> /dev/null | /usr/bin/grep
"dev:${major},${minor}" | /usr/bin/awk ' { print $5 } ' | /usr/bin/awk
-F: ' { print $2 } ' | /usr/bin/sort | /usr/bin/uniq`
if [ "$inodes" ]
then
possiblePIDs="$possiblePIDs $pid"
fi
for inode in $inodes
do
/usr/bin/echo $inode >> $tmpfile
done
done
if [ -f $tmpfile ]
then
inodes=`/usr/bin/sort $tmpfile | /usr/bin/uniq`
/usr/bin/rm $tmpfile
fi
inum=0
for ino in $inodes
do
inum=`/usr/bin/echo $inum + 1 | bc`
done
/usr/bin/echo $inum open inodes found on $device
binum=0
inum=0
badinodes=""
for ino in $inodes
do
/bin/printf "\r%d inodes found without files on file " $binum
/bin/printf "system (%d searched)" $inum
filename=`/usr/bin/find $mntpoint -mount -inum $ino 2>/dev/null`
inum=`/usr/bin/echo $inum + 1 | bc`
if [ "$filename" ]
then
/usr/bin/echo do nothing > /dev/null
else
badinodes="$badinodes $ino"
binum=`/usr/bin/echo $binum + 1 | bc`
fi
done
/bin/printf "\r%d inodes found without files on file " $binum
/bin/printf "system (%d searched)\n" $inum
for badino in $badinodes
do
/bin/printf "the following processes have captured rogue "
/bin/printf "inode %s" $badino
notfound="1"
firsttime=1;
for pid in $possiblePIDs
do
response=`/usr/proc/bin/pfiles $pid 2>/dev/null | /usr/bin/grep
"ino:${badino}"`
if [ "$response" ]
then
notfound=""
size=""
if [ $firsttime ]
then
firsttime="";
havesize=`/usr/bin/echo $response | /usr/bin/grep
"size:"`
if [ "$havesize" ]
then
size=`/usr/bin/echo $havesize | awk ' { print $8 } '`
fi
if [ $size ]
then
/usr/bin/echo " (${size}):"
else
/usr/bin/echo ":"
fi
fi
/usr/bin/echo " $pid"
fi
done
if [ $notfound ]
then
echo ":"
fi
done
exit 0
else
/usr/bin/echo error: $device is not mounted
exit 1
fi
else
/usr/bin/echo usage: icheck.sh \<device_file\>
/usr/bin/echo error: $device is not a block special file
/usr/bin/echo example: icheck.sh /dev/dsk/c0t0d0s0
exit 1
fi
fi
/usr/bin/echo usage: icheck.sh \<device_file\>
/usr/bin/echo example: icheck.sh /dev/dsk/c0t0d0s0
exit 1
--------------------------------------------------------------
Sample output from icheck.sh being run on the file system first
shown in section 1's example:
# /usr/local/sbin/icheck.sh /dev/dsk/c1t0d0s5
12 open inodes found on /dev/dsk/c1t0d0s5
1 inodes found without files on file system (12 searched)
the following processes have captured rogue inode 2921 (size:16):
14947
#
PRODUCT AREA: System Administration
PRODUCT: Disk admin
SUNOS RELEASE: Solaris 2.5
HARDWARE: any
------------------------------------------------------------------------
Home | Free Services | Contract Services | Account Services | Table of
Contents
Comments about SunSolve | Help | SunSolve FAQ's | Privacy Policy
Questions or comments regarding this service? webmaster
Copyright 1994-1998 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto,
CA 94303 USA.
All rights reserved.
Yvonne Herman
Unix System Administrator
Andersen Consulting/Boston Scientific Corporation
hermany@bsci.com
508-652-5226
Subject: FW:
Date: Fri, 28 May 1999 08:54:46 -0400
From: "Thompson, Dave" <ThompsD1@bsci.com>
To: zhouna@asiainfo.com
----------
From: Herman, Yvonne
Sent: Friday, October 23, 1998 11:47 AM
To: Keefe, Matt; Brown, Howard; Thompson, Dave; Nichols, Jason; Herman,
Yvonne
Symptoms and Resolutions article 2131 [ Notify of changes ]
[ Edit/Retrieve Marked Documents ] [ Mark Document ]
Did the search result in the document you were looking for? Send Feedback
------------------------------------------------------------------------
SRDB ID: 2131
SYNOPSIS: How to increase inodes when inode table is full
DETAIL DESCRIPTION:
How do you increase inodes when the inode table is full?
SOLUTION SUMMARY:
There are two types of inodes: kernel and disk. You should be able to tell
which inode table is full by the message.
Kkernel inode table that is full:
vmunix: inode table full
Verify this by running pstat -T:
# pstat -T
301/582files
199/199 inodes
67/138 processes
34208/43012 swap
Notice that the line for inodes shows that 199 are
being used and the maximum is also 199.
If the disk inode tabel is full, this message appears:
/:inode table full
Verify this by running df -i -t 4.2:
# df -i -t 4.2
Filesystem iused ifree %iused Mounted on
/dev/sd0a 2794 2794 100% /
/dev/sd0g 7007 29857 19% /usr
/dev/sd2h 301 11219 3% /var
/dev/sd2a 30 9954 0% /home
/dev/sd2g 4656 16848 22% /export
Notice that the root partition has used all the available inodes.
If the disk inode table is full:
- do a dump of the filesystem, as it will have to be
wiped out.
- run newfs -N on the raw disk partition:
# newfs -N /dev/rsd2g
/dev/rsd2g: 92190 sectors in 439 cylinders of 6 tracks, 35
sectors
47.2MB in 28 cyl groups (16 c/g, 1.72MB/g, 768 i/g)
super-block backups (for fsck -b #) at: ....
Notice that this partition has 768 inodes per cylinder group.
- Make sure the partition is not mounted (if root or
/usr, then this will have to be done from miniroot)
- newfs -i xxx /dev/r<partition>(wherexxx is the
number of inodes per cylinder group and
<partition> is sd0h, sd1a, etc.)
- Restore the file system from dumps.
NOTE: Prior to 4.1, a maximum of 2048 inodes per
cylinder group existed. To get around this, run:
newfs -c
and increase the number of cylinder groups (increasing the
number of cylinder groups allows more inodes in the partition).
If kernel inode table is full, do ONE of the following:
NOTE: the instructions on configuring a new kernel can
be found in the System and Network Aministration manual.
1) Edit the kernel configuration file. Increase the
MAXUSERS value (rule of thumb: double the current
value). Make and reboot the new kernel.
2)Run config on the kernel configuration file. Then:
- cd /sys/<arch>/conf
- cd /sys/<arch>/<kernel>
(where <kernel> is the name of the configuration file
and <arch> is the kernel architecture.)
- edit param.c
- search for the line thatcontains "ninode"
This line is the formula that calculates the number
of kernel inodes.
- adjust this formula by increasing some of the
values.
- do a "make" on the kernel, then reboot
the new kernel
BUG REPORT ID: 0
PATCH ID: n/a
PRODUCT AREA: Kernel
PRODUCT: Config
SUNOS RELEASE: 4.1, 4.1.1
UNBUNDLED RELEASE: n/a
HARDWARE: All
------------------------------------------------------------------------
Home | Free Services | Contract Services | Account Services | Table of
Contents
Comments about SunSolve | Help | SunSolve FAQ's | Privacy Policy
Questions or comments regarding this service? webmaster
Copyright 1994-1998 Sun Microsystems, Inc., 901 San Antonio Road, Palo Alto,
CA 94303 USA.
All rights reserved.
Yvonne Herman
Unix System Administrator
Andersen Consulting/Boston Scientific Corporation
hermany@bsci.com
508-652-5226
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:13:20 CDT