Partial Summary, more help needed: /var/adm/messages stops, dissapears.

Thanks to GertJan Hagenaars for helping me troubleshoot so far:

OK: original symptoms:
1 machine, cron stops logging, /var/adm/messages dissapears.
doing a /etc/init.d stop (then start) (doesnt always restart, may start for
10-15 mins, then dies again)


More info:
doing :
script
% strace -f -p proc id
% ^C
% exit

Shows me a great message:   namely: "ERROR: unable to open /dev/log"
and if I look at it, I get:

mbruntel@hummer: /etc $>  ls -la /dev/log
lrwxrwxrwx   1 root     other         27 Mar 17  2000 /dev/log ->
../devices/pseudo/log@0:log
mbruntel@hummer: /etc $>  ls -la /devices/pseudo/log@0:log
crw-r-----   1 root     staff     21,  5 Mar 17  2000
/devices/pseudo/log@0:log
========================^^^^^^^^^=========
staff is my "default" programmer group.

SO I changed it to the way my other systems look;
(chgrp sys)  to get
crw-r-----   1 root     sys       21,  5 Mar 17  2000
/devices/pseudo/log@0:log
and do an ls -la again :  ls -la /var/adm/messages and I get:
/var/adm/messages: No such file or directory


Another suggestion was to see if lsof shows anything
it shows nothing that I can


However, I do get this message:
lastlog_get_entry: Error reading from /var/adm/lastlog: Bad file number
lastlog_get_entry: Error reading from /var/adm/lastlog: Bad file number


Thanks for any additional insights that you can share with me!
Mitch Bruntel

--original message: --


% -----Original Message-----
% From: GertJan Hagenaars [mailto:gj@hagenaars.com]
% Sent: Thursday, September 18, 2003 11:18 PM
% To: Bruntel, Mitchell L, ALABS
% Subject: Re: /var/adm/messages stops, dissapears.
%
%
% Hi Mitch,
%
% um... I wouldn't copy anything on top of messages/syslog/maillog if
% you're having a problem with it.  Use this instead (shorter, because you
% don't have to check if the file is there or not, and quicker, because it
% will create it with the right permissions (if memory serves)).
%
%         umask 133
%         touch /var/adm/message
%         touch /var/adm/syslog
%         touch /var/adm/maillog
%
% Meanwhile (and this will be a hassle), get lsof from www.sunfreeware.com
% and install it.  See if it can tell you if something else is using those
% files.
%
% Also, find syslog's processid, and do this:
%
% script
% strace -f -p 12345
% ^C
% exit
%
% to find out if syslog is doing this by itself (virtually unheard of).
% script will create a file called "typescript" with all the system calls
% that make up syslog.  If syslog is doing it, you should see an "unlink"
% in there somewhere.
%
% Also, see if there's a specific time when this happens (12 minutes past
% the hour?  33 minutes past the hour?, and have a look at your crontabs.
%
% Last thing, if none of the above really help, there's a trick you can
% use.  Syslog opens the log files and keeps them open while it's still
% alive.  If someone removes the file, you will not free up the space,
% because syslog keeps the file descriptor open (lsof will show you this).
% Soooooooooooooooooooooo...  if you do the following (once) :
%
% cd /var/adm
% ln messages safe.messages
% ln syslog safe.syslog
% ln maillog safe.maillog
%
% you have created a new name for the files that (likely) will not be
% removed.  When the real names are gone, the only thing you need to do to
% get them back (before you run logcheck, and before you restart syslog)
% is this:
%
% for i in messages syslog maillog
% do
%         if [ ! -f /var/adm/$i ]; then
%                 ln /var/adm/safe.$i /var/adm/$i
%         fi
% done
%
% actually, you can take the above snippet and run that from cron as well.
%
% It doesn't solve the problem, but at least it shouldn't cause you too
% many headaches.
%
% Apparently, Bruntel, Mitchell L, ALABS wrote:
% % interesting problem with my syslog. (frustrating I mean)
% % On one system I have the /var/adm/messages file keeps disapearing
% % My log shows:
% % Unusual System Events
% % =-=-=-=-=-=-=-=-=-=-=
% % File /var/adm/syslog cannot be read.
% % File /var/adm/messages cannot be read.
% %
% % and I can do a restart (stop/restart) of syslog by doing a
/etc/init.d/syslog
% % stop, then  a start
% %
% % and it is seen (since this problem started I added the following lines to
the
% % stock file:
% % (based on original creation above)
% %              if [ ! -f /var/adm/messages ]; then
% %                         /usr/bin/cp /dev/null /var/adm/messages
% %                         /usr/bin/chmod 0644 /var/adm/messages
% %                 fi
% %                 if [ ! -f /var/adm/syslog ]; then
% %                         /usr/bin/cp /dev/null /var/adm/syslog
% %                         /usr/bin/chmod 0644 /var/adm/syslog
% %                 fi
% %                 if [ ! -f /var/adm/maillog ]; then
% %                         /usr/bin/cp /dev/null /var/adm/maillog
% %                         /usr/bin/chmod 0644 /var/adm/maillog
% %
% % so it seems to restart  ok, if only for another 10 mins or so.
% %
% % I've tried running a log of when it's stopping, lasts about 10-12 mins,
then
% % needs restart.
% % process list when it stops doesnt show any immediate candidates for cause
of
% % problem.
% %
% % I'm kind of stumped?
% % ideas, suggestions, welcome.
% %
% % I've included my /etc/syslog.conf to see if someone ELSE see's something
funky
% % that might be doing this, but I've
% % been over it 5-6 times, and dont see much.
% %
% % Thanks again, and I did try a few searches on the list, but didnt come up
with
% % anything
% %
% % Oh yes, the syslogd daemon doesnt stop running, it's just that
/var/messages
% % disapears!
% %
% % Mitch Bruntel
% % AT&T Labs
% %
% % ++++++++++++++++++++++++++++++++
% % /etc/syslog.conf follows:
% % #ident  "@(#)syslog.conf        1.5     99/02/03 SMI"   /* SunOS 5.0 */
% % #
% % # Copyright (c) 1991-1999 by Sun Microsystems, Inc.
% % # All rights reserved.
% % #
% % # syslog configuration file.
% % #
% % # This file is processed by m4 so be careful to quote (`') names
% % # that match m4 reserved words.  Also, within ifdef's, arguments
% % # containing commas must be quoted.
% % #
% % # next is original
% % #*.info                                         /var/adm/messages
% % *.info;mail.info                                /var/adm/messages
% % #mail.info                                      /var/adm/maillog
% % *.err;kern.notice                               /dev/sysmsg
% % *.err;kern.debug;daemon.notice;mail.crit        /var/adm/messages
% % #kern, mark.debug                               /var/adm/messages
% % # prev line for kernel mark
% %
% % *.alert;kern.err;daemon.err                     operator
% % *.alert
% % *.emerg                                         *
% %
% % # if a non-loghost machine chooses to have authentication messages
% % # sent to the loghost machine, un-comment out the following line:
% % #auth.notice                    ifdef(`LOGHOST', /var/log/authlog,
@loghost)
% %% % #mail.debug                     ifdef(`LOGHOST', /var/log/syslog,
@loghost)
% %
% % #
% % # non-loghost machines will use the following lines to cause "user"
% % # log messages to be logged locally.
% % #
% % ifdef(`LOGHOST', ,
% % user.err                                        /dev/sysmsg
% % user.err                                        /var/adm/messages
% % user.alert                                      `root, operator'
% % user.emerg                                      *
% % )
% % auth.debug                                      /var/log/authlog
% % auth.notice                                     /var/log/authlog
% % auth.info                                       /var/log/authlog
% % _______________________________________________
% % sunmanagers mailing list
% % sunmanagers@sunmanagers.org
% % http://www.sunmanagers.org/mailman/listinfo/sunmanagers
%
% CHeers,
% GertJan.
%
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers