Thank you to all that replied:
blackman@voicenet.com
Craig Whytock
Dennis Martens
Jochen Bern
Kelly Nakashima
Marco Greene
Robert Rose
rye@jtasc.acom.mil
Sabrina Downard
Mike Evans
simon.millard@gemini.barclays.co.uk
Vahid Moghaddasi
We received a lot of really good ideas - most of them we already tried
or thought about - but we were glad to know we were on the same path as
everyone else. We saw, from the accounting files that there were three
init and one shutdown issued from root (but no tty associated with the
commands - and we guessed that the inits were forked from the shutdown).
So, in lieu of being able to firmly pin it down, we decided the only
thing we could do was change the root password and keep looking. The
ideas we received were very good ones:
*SUN could be right, shutdown will not always bring the system down
since it has to wait for all the processes to accept the TERM signal. In
some cases a "-g0" may take 30 minutes or more for the system to go
down.
*1) User with some rootish permissions (i.e. in a group that gives
him/her some admin privileges) ran shutdown.
2) User with full or partial root perms sent a fake "wall" message to
all users then simple killed all the telnetd processes and inetd. To
either simulate a shutdown for malign reasons or possibly just to kick
the users to update something. Hope you find out. A scan of "last" and
all admin user's shell histories might help
*Check you sulog and your lastlog for who was privileged at the time.
Check /.sh_history to see if a shutdown was called.
*Check the permissions on /usr/sbin/wall. It's shipped
world-executable, so any prankster could send out messages saying the
system was going to be shut down. (Although that wouldn't terminate
telnet.) The man page for "shutdown" reports "...that not all user
processes are stopped when transitioning from multi-user state to state
1." (But you would have to have root to issue a shutdown command.)
/usr/ucb/shutdown has a 'fake out' option, -k, which only pretends to
shut the system down. (Again you need root.) Maybe you should change
your root password, and make double sure that nobody who doesn't need
root has the password.
*A shutdown -k will cause this message.
*Do you have an entry in /etc/passwd whose shell is shutdown?
*this is exactly what we see when we do a remote shutdown by logging in
and using shutdown -g0 -i0 but omit the -y flag to confirm shutdown.
The shutdown starts, logs out the user then there is not way to confirm
so the shutdown aborts.
*Check who was logged in at the time, using the 'last' command, then
cross reference the list against the sulog file. You may find that
someone had su'ed at the critical time. Also, if you have process
accounting, try 'lastcomm shutdown' Do you allow remote root access?
Check /.rhosts to see which remote root users have root access to this
system.
**All* TELNET Logins (of *various* Users, I presume?) being terminated
is pretty much *Proof* of root Privileges being used. (Sometimes a
System UId/GId, often named "tty", could do the same, but unless you
FUBAR'd UId/GId Assignments, that wouldn't be much different in Terms of
"should I worry" ...) At least in SunOS, there's an Option "only do as
if" to 'shutdown', maybe someone put too much Trust into such an Option
not to disrupt anything? OTOH, are you running some Kind of automatic
Shutdown, as in a "Power Saver" or per a UPS S/W? (I'm somehow under the
Impression that such a Thing comes with Solaris' Default Config ...)
* I have seen something like this if this is the case: I have sent a
shutdown message to the system but it would not shutdown because I had a
process running on my sybase database. I had to kill the shutdown
process because the shutdown would not take place. It was just sending
messages to the console - waiting for a process to shutdown. And the
system send out the shutdown message to the users.
There also was not any messages sent to the /var/adm/messages log. (I
just checked again to be sure)If this is your case - then it was not a
fake shutdown but for some reason the database will not shutdown with
processes running from a UNIX shutdown - I know that I can shutdown the
database if I log into the database and kill the processes running. Or
do a kill -9 database(PID) which is not what a shutdown
command does.
*I have a couple of ideas: The message bit is an easy thing to
accomplish (wall -a) from any telnet session would do it. As for the
telnet sessions closing, someone would need root permissions to do it.
Check to make sure that kill does not have its UID bit set....ie
-rwsr-xr-x. One thing you could do is turn on system accounting and let
it run for a little while. This will give you an idea of what is being
run on your
system and by which users. Also consider changing the root password in
case it was accidentally leaked. Mind you if you want to catch the
person doing it, I would just turn on system accounting and not change
anything else on the system. That way the next time something like this
happens, you would catch the culprit.
*In my last job, we had a similar thing happen on a SunOS host. Turned
out to be a DBA that had su-ed to userid "oracle" and was trying to shut
down a database. He hadn't started up sqldba, just typed "shutdown" at
the shell prompt, and because "oracle" was in group "operator", the
thing actually shut down to single user mode. I happened to look over
when one of the SunOS machines beside me beep-ed with the wall message
saying the machine was shutting down.
Original Request:
We had a situation which occurred this morning in which a system
shutdown message was sent to all users and then all telnet sessions were
terminated. The system did not go down. The databases and user
applications were still running. The users that were logged in at the
time were familiar users. The root password is restricted to the Tech
Team. I called Sun service and they said that the only way for this to
happen is that someone having root privileges issued a shutdown - (they
also said that maybe the shutdown was terminated due to a process not
allowing the shutdown to finish). We did not see evidence of a shutdown
not being able to complete. Any ideas?
Yvonne Herman
hermany@bsci.com
508-652-5226
This archive was generated by hypermail 2.1.2 : Fri Sep 28 2001 - 23:12:46 CDT