Summary: Documenting infrastructure

Hello,
   I'd like to thank the following for replying to my inquiry about
documenting one's Unix infrastructure - I'm sorry if somehow I've missed
anyone:
 Jason Husk <Jason.Husk@corp.infi.net>,
Richard Grace <rgrace@aapt.com.au>,
"Homan, Charles (NE)" <Charles.Homan@GDC4S.Com>,
Greg Gallagher <ggallag@foc.com>,
gene.g.beaird@mail.sprint.com,
Harvey Wamboldt <harvey@iotek.ns.ca>,
"Thomas M. Payerle" <payerle@physics.umd.edu>,
Jay Lessert <jayl@accelerant.net>,
Jim Southerd <jsoutherd@bakersfield.com>,
Bertrand_Hutin@notes.amdahl.com,
Andrew_Rotramel/CCHLIS@cch-lis.com,
Karl Vogel <vogelke@dnaco.net>

   I have included all of the replies below because I feel this is a bit
more dynamic of a discussion than "how do I get my ABC to XYZ?" - I hope
the folks who replied don't mind their messages getting re-posted to the
list.

The gist of the methods of documentation and things being documented
include:
* a directory structure of text files (this is currently what we have)
accessable from local filesystems and/or the web, consisting of various how-to notes
   * homemade man pages outlining common sys admin tasks and user
frequently asked questions
   * scripts which compile server info (i.e. disk partition layout) into a web tree along with
typescripts of installed software and other How-Tos
   * using xml+DocBook to publish printed and online sys admin docs
   * a combination of docs in databases, MS Excel, and JPG images dealing
with inventory, machine room layout, Etc
   * various html pages documenting "internal" procedures and How-Tos, as
well as instructions for end-user consumption

   Other good points worth summarizing here are: The more trouble it
is to update docs (i.e. work must be done on a specific machine or
specialized software is required) the less likely it is that they will be
updated as frequently as you'd like.  Successor sys admins will most
likely only want to read docs up to a certain point, and will fall back on
them if/when they're at a loss for how something works or why it is
configured in such a way down the road.  The more docs you have, the more
there is to maintain (we may end up writing more docs than
fixing/improving things).

   My game plan at this point is to continue playing with xml+docbook (my
most favorite suggestion), latex, and texinfo to see which yields the best
output in the desired formats (something like postscript for printing and
text and html for online versions).  I think I can modularize the docs
enough (each sub-section living in a separate file) that maintinance is
minimally daunting.  I'll either use Excel spreadsheets or some other
database driven solution to store inventory (model, serial number, RAM,
disk, repair history) which can be imported into the xml/latex document
easily enough (i.e. CSV export).  I like the idea of pictures of the
server room to give people something to look at and associate with while
they're reading some
particular doc.

   I'll post another message as things are progressing into something
meaningful -- Thanks very much for all the feedback!


Ivan Fetch.


 Here is my original post:
> Hello all,
>
>    I'd like to start maintaining more documentation designed for
> myself (reminders), folks who are replacing me (when that day comes), and
> anyone needing to respond to an emergency if I'm unavailable.  I invition
> including such things as where non-packaged software is installed on our
> servers, brief How-Tos on typical tasks (i.e. restoring backups, user
> maintinance, jumpstarting new workstations), caveats on upgrading
> frequently used software which gets patches often (including Solaris), the
> organization of our IP address space, the configuration of our printers
> (jet-direct -> Solaris -> Samba -> Windows clients), et cetera.
>
>    I'm writing to solicit other people's opinions on the creation of
> something like this - if you were starting a position, what kinds of
> things would be helpful to have some background on vs. figuring things out
> on one's own?  How are you folks out there documenting your infrastructure
> (as much for yourselves as for future employment)?  Do most of you use
> something like rcs or cvs to manage your configuration files (/etc,
> /usr/local/etc)?  What do folks use to type set electronic and printed
> documentation -- I've been thinking latex or sgml -- I would like
> automatic referencing (being able to refer to sections by number, title, and/or page number) as well as an automatically
> generated table of contents.
>
>    I appreciate any input you all are willing to give  - I'll definitely
> summarize.  I'd like to start converting the various notes I've made while
> reconfiguring things into a more organized and useful structure.
>
> Thanks,
> Ivan Fetch.

----- responces -----


On Wed, 1 May 2002, Jason Husk wrote:

>
> 	Something that I used to use when I helped maintain such an
> archive is a simple man page.  What we did was we had a rather simple
> utility that would take a plain ascii text file and convert it to a man
> page and then place it into a common directory that we used to store the
> docs in.  This directory had only group permissions on it so we had to be
> in a member of the admin group to be able to read this directory and the
> info docs contained in it.  Then if we needed to look something up all we
> had to do is make sure that directory was in our MANPATH environment
> variable and then we could access the pages via the built in man
> command... ie man boot-net .... or man NT_printers ... you get the idea I
> hope.  It was very helpful for when I was just starting out.
>
> 	We had items such as how we setup user accounts, changed thier
> passwords, located source files that we compiled for public use, how we
> built various types of workstations.  Items such as common user questions
> were stored too like compiler and library locations.  Other items like
> rules pertaining to how we allowed our users to use the computers and
> access the internet.  (It was a university setting).  Also common everyday
> tasks were listed to things that an end user might ask that the sysadmin
> may not necessarily need to know for their every day work.  How to setup a
> public_html directory for a  web page.  How to setup a .htaccess file.
> How to submit grades etc.. These may be some helpful docs to have lying
> around if you have spare time to create them.
>
>
> Anyway I wont ramble on too much but I hope it helps


On Wed, 1 May 2002, Richard Grace wrote:

> Hi there.  What a task, eh?
>
> I use scripts to dump disk, network, memory, and other information
> into a temporary directory tree on a nightly basis.  This is triggered
> by a script which sshs to each server, executes the script, tars up
> the directory, copies and untars into our web tree.  The files are
> linked by indexes in the web tree.  This is the easiest way I've found
> to keep online information on running servers.
>
> We have a similar web tree structure which links to the jumpstart
> config and packages area.
>
> Most of the software we use is packaged, and there are READMEs
> in each package which (should) detail the last configuration options
> used for the installation, and the output from script, as well as the
> source tarball.  I know this wastes a lot of disk space, but it is more
> thorough that way.
>
> We also have a "links" page, which links to as many howtos as the
> operators have found useful in the past.
>
> Richard.


On Wed, 1 May 2002, Homan, Charles (NE) wrote:

> At my current employer, they had a simple text file setup called
> "miscnotes".  Things in it are arranged in sections (by application, mainly)
> and include notes on how things were installed, how to add users, notes on
> special hardware, standard disk partitioning, bugs encountered and their
> workarounds, license info, rarely-used but necessary administrative command,
> etc.  Longer notes are in seperate files in the same directory, with a
> reference in miscnotes.
>
> This has had numerous advantages over "fancier" (i.e.: laid out and/or
> printed) documentation:
>
> o completely searchable (via "grep -i") and human-readable
> o always where I need it (i.e.: on the server(s))
> o can cut and paste lines to command line for seldom-used commands with long
> syntax (read: NIS+)
> o simple to keep updated
>
> This last point is key to me.  I would much rather have an accurate and
> up-to-date text file than a well-laid out, nice looking piece of
> documentation that someone created two years ago and eventually stopped
> maintaining 18 months ago.  KISS. ;-) That said, some things (I'm thinking
> of network layout here) can be much more easily represented (and understood)
> graphically, and others (like inventories) are more useful in spreadsheet
> form.  Use the right tool for the job, but make sure everything is in a
> "findable" place.
>
> Hope this helps!
> Charles


On Wed, 1 May 2002, Greg Gallagher wrote:

> Quite nice of you to take this initiative -- preparing your successor
> and other junior admins, etc.  Not many SA's do!  :-)
>
> It's a little more troublesome, but I'd suggest looking into XML +
> Docbook and organize a CVS repository where you can build an "Admin
> Guide".  I did this at a previous job, and it turned out really cool.
> By the time I left, I had written a little chapter on all my major
> duties such as backups and restores, installing Oracle, etc.  It was a
> perfect handbook to give the poor bloke who took my job :-)
>
> The thing about using XML (or SGML if you're brave enough) is that
> you'll be able to publish it to the web easily, as well as making a
> printable manual.  It's worth the time, and once you have a basic
> structured document setup it's quite easy to keep it up to date.
>
> cheers,
> --
> Greg Gallagher
<followup>

> On Wed, May 01, 2002 at 07:16:56AM -0600, Ivan Fetch wrote:
> > Hi,
> >    Thanks very much for your reply - Might you have any pointers RE:
> > xml+docbook (did this take place under Solaris when you worked on it)?
.....
> Hrm, well.  At the time I was using Linux and actually it was SGML +
> Docbook using openjade.
>
> Now-a-days, you could look at http://www.xmlsoftware.com for some
> software.  Anything java should work on Sun.  Docbook you can get from
> http://www.docbook.org, or a pointer thereoff.  I'd *highly* suggest
> using Emacs + PSGML mode for doing the actual editing and
> composition.  That works good for me.
>
> Also, you could use XSLT to do the transforming.  I think that works
> the best.  Then use SAXON to do the transformation of the XML +
> Docbook XSL to HTML or whatnot.
>
> Take a look at this page, it looked really useful to me:
>
> http://www.dpawson.co.uk/docbook/
>
> Good luck!
>
> Oh, another idea, I just thought of was to use something like Apache
> XML to do transformation on the fly.  There are a few free options to
> do that type of thing, and a few really expensive Commercial products
> as well (such as Documentum and Arbortext).
>
> cheers,
> --
> Greg Gallagher


On Wed, 1 May 2002 gene.g.beaird@mail.sprint.com wrote:

> Ivan,
>
> Please post your results, I am VERY interested.  That said, I have been
> using various things to help with a 'Greenbook' for some time.  I have
> used things as simple as text files and Word documents to databases.
> It seems that for me, at least, no one tool does it all (unless you get
> one of those multi-thousand-dollar monstrosities that are commercially
> available that try to be everything for everyone all the time, but
> really don't do anything well).
>
> I have always tried to keep it as low-tech and generic as possible.
> When I was maintaining a Macintosh network at an oil company, I kept
> everything on a Filemaker Pro database. That way, I could slice and
> dice the information any way I wanted to keep whoever wanted to see it
> happy (or mostly happy).  If I had walked into this present job knowing
> then what I do now, I would have converted their documentation to some
> sort of database, or another.  That way, you can be sure you are
> maintaining the same data on all the systems you have to maintain.  It
> makes things like machine room diagrams, network diagrams and general
> comments difficult, but dome database applications can even handle
> this.
>
> We use a series of Word documents, Excel spreadsheets, jpeg images
> (photos of the server room equipment) and Visio documents.  That way,
> almost anyone in my group can look at the documentation and use it.
> You still, IMHO, need to keep it simple.  If it runs under some
> behemoth app that is difficult to install and use, you will have
> difficulty getting others to document their work.  It will be equally
> difficult to use as you can bet a machine that has the proprietary
> software installed will not be available when you need it most.
>
> I have assembled a 'Greenbook' or 'IT Operations Manual' using a set of
> loose-leaf notebooks.  That way printed material is available for quick
> reference and it is easy to update.  Keep NO passwords in the
> Greenbook, but get pretty specific.  We also control who gets our
> Greenbook, and limit it to the IT staff and their immediate managers.
>
> I have a standard 'Information Sheet' in M$ Word format, if you would
> like to see it.
>
> Regards,
>
> Gene Beaird
> Systems Integrator V


On Wed, 1 May 2002, Harvey Wamboldt wrote:

> Interesting idea.  I'd be very interested in reading your summary.
>
> We have a very small shop, and often I'm called on to "consult" on
> problems when I'm off site.  I have a simple directory full of ASCII
> text files that contain my notes.  If I can get on the Internet I can
> access these whether I'm on the east coast or the west coast.  Our
> other SysAdmin (who's primarily a PC person) knows where to find these
> notes.  I use RCS to maintain versions, but that's more for security
> against accidentally deleting a file (so I don't have to retrieve it
> from the backup).  I use rsync to maintain copies on my home PC.  The
> biggest problem is outdating obsolete information.  Mostly that
> doesn't happen since I don't take the time.  I also have a public_html
> directory where I keep some information, primarily for users.  I write
> the HTML by hand, and try to keep it brief.  Oh yes, I find every
> directory needs an INDEX file and a README file.  It's amazing what a
> difference that makes.
>
> Best Rgds,
>
> -H-
>


On Wed, 1 May 2002, Thomas M. Payerle wrote:

> On Tue, 30 Apr 2002, Ivan Fetch wrote:
>
> > Hello all,
> >
> >    I'd like to start maintaining more documentation designed for
> > myself (reminders), folks who are replacing me (when that day comes), and
> The dark area of sys admin.
>
> As for how much to document, I say can never have too much documentation
> (albeit, you can waste too much of your time documenting).  Basically, the
> new person will decide for himself what he wants to learn from documentation
> and what to discover on his own, and may decide to refer back to documentation
> previously ignored if he discovered something nasty.
>
> The only down side of a lot of documentation is that there is more stuff to
> keep up to date, or as usually is the case, there is more out of date
> documentation.  Not that out-of-date documentation is truly evil (it usually
> has merit for at least understanding historical significance of decisions,
> and usually is applicable with minor changes.  Reader just needs to know that
> cannot blindly follow documentation).
>
> I try to use RCS to track changes to assorted config files, not always very
> successful (not infrequent to need to change something only to find already
> checked out and don't know what last change was, despite being me who checked
> it out).  But at least are able to quickly revert to working versions, and
> beats 500 files of /etc/inetd.conf.sav.20020501 and ilk.
>
> As for format of documents, we generally do html.  Post on a password protected
> page (nothing is really confidential, but don't want users stumbling upon it
> and getting enough rope to hang themselves), and duplicate on a second web
> server, as well as printed copies.  This is mainly because html is most widely
> readable format after ascii, and being on web stresses the ephemeral nature of
> the contents.
>
> I personally like latex, but fewer and fewer people who know how to use, and
> most of the documentation (here at least), is out of date by the time it hits
> the printer, so we prefer on-line (also out of date, but who is stupid enough
> to take as gospel something got off the web:)
>


On Wed, 1 May 2002, Jay Lessert wrote:

> On Tue, Apr 30, 2002 at 11:10:49PM -0600, Ivan Fetch wrote:
> >    I'm writing to solicit other people's opinions on the creation of
> > something like this - if you were starting a position, what kinds of
> > things would be helpful to have some background on vs. figuring things out
> > on one's own?
>
> I've been through this recently/currently; walked into somebody else's
> shop about a year ago, some things were documented, some weren't, I'm
> writing documentation as i go.
>
> 1) How to create a new account.  For us, NIS/NTDC/qmail aliases/phone
>    setup.
> 2) How to add a new host.
>
> Those two are *really* nice to have.
>
> > How are you folks out there documenting your infrastructure
> > (as much for yourselves as for future employment)?
>
> Mostly in HTML.  We used to use FrameMaker (which we already use here
> for product datasheets and app notes), but it's not worth it for
> short simple documents like these.
>
> > Do most of you use
> > something like rcs or cvs to manage your configuration files (/etc,
> > /usr/local/etc)?
>
> RCS (again, keep it simple).  We use RCS for the HTML systems docs as
> well.
>
> --
> Jay Lessert                               jay_lessert@accelerant.net


On Wed, 1 May 2002, Jim Southerd wrote:

> I'm in the same process.
>
> The first thing I have documented is my disk structure on the array for both of my servers.
> I done this by printing the structure right from Veritas from each system, and superimposing some relevant info for our application, along with some pertinent info of each server.
>
> Now I'm 'better' documenting all my scripts and printing them out, along with the printout of the crotab.
>
> Next will be the file structure from the application perspective.
>
> Then we'll see.
>
> Our Director used Visio to document the entire network from information given to him from each sysadmin.
>
> Now he and several of us, are working on documenting the various automated (and manual) ftp's that are going on between servers.
>
> Hope this helps!
>


On Thu, 2 May 2002 Bertrand_Hutin@notes.amdahl.com wrote:

>
> In the days were I was a sys admin, I gathered all the workstation
> description as a set of html pages.
> I was using rcs for some config files and also had copies of some in an
> specific (non-root) account home.
> I had also written a manual (LaTex) to be used by people when i was on
> vacation.


On Thu, 2 May 2002 Andrew_Rotramel/CCHLIS@cch-lis.com wrote:

>
> I am doing this now, but because of the office culture, have to use Word
> for the book. My goal is to document things such that if I am squashed by a
> meteor, someone else can step right into my shoes. I am not worried about
> job security, and think that if anyone tries to secure their job by doing
> secret configurations, they should be fired.
>
> Andrew
>


On 2 May 2002, Karl Vogel wrote:

> I> Do most of you use something like rcs or cvs to manage your
> I> configuration files (/etc, /usr/local/etc)?
>
>    Yes.  I have it done automatically every night.
>
> I> What do folks use to type set electronic and printed documentation --
> I> I've been thinking latex or sgml -- I would like automatic referencing
> I> (being able to refer to sections by number, title, and/or page number)
> I> as well as an automatically generated table of contents.
>
>    You might like texinfo, it can do all of the above and it's a lot more
>    readable than latex.
_______________________________________________
sunmanagers mailing list
sunmanagers@sunmanagers.org
http://www.sunmanagers.org/mailman/listinfo/sunmanagers