Subject: Re: /var/sadm is too damn big
From: Ericka Fowler <[email-masked]>
To: [email-masked]
Date: Thu, 2 Nov 2000 10:02:19 -0700 (MST)

Sun doesn't recommend that you delete things out of /var/sadm - it will make 
things difficult when you want to patch or pkgrm or upgrade your system. I'm 
including the /var/sadm infodoc so that you can have an explanation of what all 
the different files and directories under /var/sadm are...

INFODOC ID: 21481 
SYNOPSIS: /var/sadm PSD 
DETAIL DESCRIPTION: 

Section 1.0- Introduction 
-------------------------

The intention of this document is to outline some of the high level 
concerns that should be considered by System Administrators with regard 
to patching, space issues, and future planning. Each of the directories 
in /var/sadm serves a special purpose that often is not considered when 
changing, updating, or upgrading Solaris systems. This document is 
intended to give you a place to find information relating to these 
directories, their function, structure, and usage. 

The /var/sadm directory is extremely important, especially when changes 
are made and logged to the software on the system in any form.  Some 
systems are more static than others, however, at some point, all 
systems need changes or upgrades. A healthy /var is critical to the 
change management of the system. 

To this end, proper advance planning and space considerations for 
future use are very important. Take a look at Infodoc 18509 for more 
information on space considerations.

One other note on change management of Solaris systems. What makes 
change management more difficult can be the large amount of time spent 
between changes, and often the personnel changes between these 
changes. The times that packages are being added, removed or patched is 
usually a critical one for the System Administrator, the machine, 
the organization as a whole and possible deadlines. One 
example of this is Year 2000 concerns. I point this out because errors 
in your /var/sadm directory can often need wholesale changes to the 
system to correct, such as reinstallation of your OS. Be aware of this 
from the beginning, so your expectations are set correctly.

This document is structured to show you specific examples first, in 
Section 2.0, then general topics starting with Section 3.0.

Section 2.0-  The /var/sadm directory exploded 
----------------------------------------------
The items found in /var/sadm on my 2.7 machine:
total 754
-r--r--r--      1       root    sys     1092    Sep 10  1998    README
dr-xr-xr-x      4       bin     bin     512     Dec  1 17:40    install
drwxr-xr-x      2       root    sys     512     Oct 21 13:14    
install_data
drwxr-xr-x      11      root    root    512     Dec  1 17:41    patch
dr-xr-xr-x      417     root    sys     8192    Dec  1 17:40    pkg
drwxr-xr-x      2       root    sys     512     Apr 20  1999    softinfo
drwxr-xr-x      5       root    sys     512     Apr 20  1999    system

serve a common purpose.  This purpose is to record the changes made 
to the system. Many Solaris change management utilities rely upon the 
information inside /var/sadm to be an accurate picture of what actually 
resides on the machine.

Section 2.01- The individual items in /var/sadm- README
-------------------------------------------------------
-r--r--r--      1       root    sys     1092    Sep 10 1998     README

This file contains important information about the installation of the 
OS on your machine. See Section 2.516 for more info.

Section 2.02  The individual items in /var/sadm- /var/sadm/install
------------------------------------------------------------------
-rw-r--r--      1       root    root    128             Dec  1 17:40
.lockfile
-rw-rw-rw-      1       root    root    0               Apr 20  1999    
.pkg.lock
dr-xr-xr-x      2       bin     bin     512             Apr 20  1999    
admin
-rw-r--r--      1       root    other   2798313         Dec  1 17:40
contents
dr-xr-xr-x      2       bin     bin     512             Dec  1 17:40    
logs

Both .pkg.lock and .lockfile are necessary for differing reasons, 
although they are similar in what they do. These files are necessary 
for the package utilities and should not be deleted.

Section 2.021- Items in /var/sadm- /var/sadm/install/admin
----------------------------------------------------------
-r--r--r--      1       root    sys     193     Sep  1  1998    default

The directory contains only the 'default' file.

Section 2.022- Items in /var/sadm- /var/sadm/install/admin/default
------------------------------------------------------------------
#ident  "@(#)default    1.4     92/12/23 SMI"   /* SVr4.0  1.5.2.1   */ 
mail=
instance=unique
partial=ask
runlevel=ask
idepend=ask
rdepend=ask
space=ask
setuid=ask
conflict=ask
action=ask
basedir=default

The uses of this file are well-described in Infodoc 15378, and the man 
pages for admin (4). An excerpt from the Infodoc:

Each package must have a request script for pkgask utilities to work. 
In the case of multiple packages, a response DIRECTORY must be used. 
The request script and corresponding response files are specific to the 
package installed.  pkgadd itself may require interaction.  The 
behavior of pkgadd is determined by the settings in the 
/var/sadm/install/admin/default file.  This is well-documented in the 
manual pages (man -s 4 admin).  To establish different behavior, create 
an alternative admin file which you can then refer to using the -a 
option of pkgadd.  

>From the man page:
# man -s 4 admin

File Formats                                             admin(4)

/var/sadm/install/admin/default is the default  admin   file
     delivered  with  this release. The default file is not writ-
     able, so to assign values different from this file, create a
     new  admin  file. There are no naming restrictions for admin
     files. Name the file when installing a package with  the  -a
     option  of  pkgadd(1M).  If  the  -a option is not used, the
     default admin  file is used.

Example:  to have pkgadd not check for overwriting a file which may
conflict with another package, you could create a new admin file
and set the value of the "conflict" parameter from "ask" to "nocheck".
This will unconditionally install the new file when a conflict is 
detected. Obviously, changing these settings should be done with 
caution.

Section 2.03-  The individual items in /var/sadm- /var/sadm/install/logs
------------------------------------------------------------------------
-rw-r--r--      1       root    other   252     Dec 13 23:11 
Netscape_Communicator_4.51_install.A12131109
-rw-r--r--      1       root    other   9850    Dec 13 23:11 
Netscape_Communicator_4.51_install.B12131109
-rw-r--r--      1       root    other   817     Dec 15 18:18 
Solaris_7_Documentation_CD_install.A12150553
-rw-r--r--      1       root    other   272     Dec 15 23:50 
Solaris_7_Documentation_CD_install.A12151150
-rw-r--r--      1       root    other   72652   Dec 15 18:01 
Solaris_AnswerBook2_Server_install.B12150553
-rw-r--r--      1       root    other   686     Dec 16 01:00
Solaris_Easy_Access_Server_3.0_Documentation_CD_install.A12161255
-rw-r--r--      1       root    other   62485   Dec 16 01:00
Solaris_Easy_Access_Server_Collections_install.B12161255

This set of logs is taken from a server that has had several 
documentation cdroms installed onto it, as well as Netscape 
Communicator. This area may be used by any number of products to put 
their installation logs.

Section 2.04-  Items in /var/sadm- /var/sadm/install/contents
-------------------------------------------------------------
-rw-r--r--      1       root    other   2798313 Dec  1 17:40    contents

A key file around synchronization issues is the 
/var/sadm/install/contents file. This is a file that can be used several
ways. You can use it to discover what package an individual file belongs
 to, or conversely, use it to find out all the files associated with a 
certain package. For example to find out what things are associated with
the format command:
# cat /var/sadm/install/contents | grep /etc/format

/etc/format=../usr/sbin/format s none SUNWcsr
/etc/format.dat v none 0644 root sys 7603 30297 904647929 SUNWcsr

If you were to instead grep for SUNWcsr, you would normally find quite 
a few more files, too numerous to list here.

Section 2.1- The individual items in /var/sadm- /var/sadm/install_data
----------------------------------------------------------------------
There are a couple of log files in this directory that can be useful:
-rw-r--r--      1       root    other   2612    Oct 21 13:19    
Solaris_7_Recommended_log
lrwxrwxrwx      1       root    root    26      Apr 20  1999    
install_log -> ../system/logs/install_log

The first is one that was used by the Solaris 7 recommended patch 
cluster, but I find this one to of minimal use in debugging patch 
installations. The second can be used after installation to tell you 
much historical information about the way this system was initially 
installed. It may be a system that has been inherited, or a system that 
has not had changes made on it for a while. 

Over the life of a system, especially if it has been ugraded a few 
times, multiple Maintenance Upgrade scripts may be run. The 
Maintenance_Upgrade_logs are always linked to the latest log file 
generated, but all logs are maintained. The same thing happens in 
/var/sadm/system/logs with the begin.log and finish.logs if they exist...

Section 2.2- The individual items in /var/sadm- /var/sadm/patch
----------------------------------------------------------------
>From a 2.7 machine:
total 34
-rw-r--r--      1       root    root    0       Apr 20  1999    
.mu_applied
-rw-r--r--      1       root    other   5550    Dec  1 17:41    
.patchDB
drwxr-x---      2       root    other   512     Dec  1 17:41    
106541-08
drwxr-x---      2       root    other   512     Oct 21 13:16    
106725-01
drwxr-x---      2       root    other   512     Oct 21 13:16    
106793-03
drwxr-x---      2       root    other   512     Oct 21 13:16    
106934-03
drwxr-x---      2       root    other   512     Oct 21 13:15    
107022-04
drwxr-x---      2       root    other   512     Oct 21 13:15    
107171-04
drwxr-x---      2       root    other   512     Oct 21 13:27    
107359-01
drwxr-x---      2       root    other   512     Oct 21 13:16    
107544-02
drwxr-x---      2       root    other   512     Oct 21 13:28    
107587-01

Section 2.21- Items in /var/sadm- /var/sadm/patch/.mu_applied
-------------------------------------------------------------
-rw-r--r--      1       root    root    0       Apr 20  1999 
.mu_applied

The .mu_applied file is a log file that is created as a result of a HW 
Maintenance Update patches that have been installed as a result of a 
hardware release upgrade, install, or maintenance update patch install.
It could be thought of as a sort of "lock" file that denotes a HW 
release has been done on the system.

Section 2.22- Items in /var/sadm- /var/sadm/patch/.patchDB
----------------------------------------------------------
-rw-r--r--      1       root    other   5550    Dec  1 17:41    .patchDB

The .patchDB is the mechanism through which patch installation speed has
improved on 2.6 and later systems. This file does not exist on 2.5.1 
and earlier systems. 
Sometimes, as a troubleshooting step during patch installation, one can 
delete the .patchDB file, and it will be rebuilt during that patch 
installation. This does add extra time for the next patch installation, 
however.

Section 2.23- Items in /var/sadm- /var/sadm/patch/PATCHID#
----------------------------------------------------------
The rest of the directories are numbered in order by patch-ID and can 
contain backoutpatch (2.6 and earlier) scripts needed to backout 
versions of the patches you have installed  on your machine. Patchrm is 
used from 2.6 and later versions of the OS. It also contains log files 
about the installation of that particular patch, and the README file as 
well.  

There is no backout data contained in this directory structure, at least
not with direct instance (DI)  patching. I believe there is still 
backout data contained here if progressive instance (PI)  patches are 
installed that have not been properly built.

NOTE: A complete description of direct instance vs. progressive instance
patches is beyond the scope of this document, however, to sum up:
-The DI patch structure was introduced with Solaris 2.5, and MOST 
patches since then have been constructed this way. Unfortunately, there 
are several patches (compiler and license patches are typical), that 
still use the PI structure, or in some case both PI and DI.
-If the patches are new and are constructed so that both DI and PI are 
available,  you should see a directory in the top level of the patch 
that is labeled "old_style_patch'.

While it's true that patchrm will complain about backing out a patch 
that does not have a patch_id directory in this tree, the directory 
could be empty and patchrm will often work fine, including restoring the
backout data that is actually contained in /var/sadm/pkg/pkg_name/save 
by default. 

It is likely that your system has many more patches than my example, 
depending on the age and version of the OS you have installed.

Section 2.3- The individual items in /var/sadm- /var/sadm/pkg
-------------------------------------------------------------
..................
drwxr-xr-x      4       root    root    512     Dec  1 17:30    SUNWcar
drwxr-xr-x      4       root    root    512     Dec  1 17:31    SUNWcarx
drwxr-xr-x      4       root    root    512     Dec  1 17:34    SUNWcsr
drwxr-xr-x      4       root    root    512     Dec  1 17:34    SUNWcsu
.................

I will not list all the items found in my /var/sadm/pkg directory, 
since there are too many items. The above packages are simply examples.
This is the directory where all the information about your packages is 
kept. It is critical to keep this directory intact and up to date by 
using the standard pkgadd and installpatch/backoutpatch/patchadd 
utilities. Showrev -p gets it's information from here as well, from the 
pkginfo file in each pkg directory.

Backout information for patches is kept in /var/sadm/pkg/pkgname/save 
directory. See Section 2.33 below for a description.

I have seen a problem arise with showrev -p, where you will get an 
error like: 
showrev: get_env_var(filename, SUNW_PATCHID)

This can be for 2 reasons:

1) There is a file in /var/sadm/pkg.
Usually, this file will match the filename listed in the error message.

2) The permissions on /var/sadm/pkg/pkgname/pkginfo file is wrong. 
It should be:
-rw-r--r--      1       root    root    593     Apr 20  1999    pkginfo

In this case, then, the message would say this instead:

showrev: get_env_var (pkgname, SUNW_PATCHID)

Switching the pkgname affected instead of the filename. A fuller 
discussion of this can be found in Indofoc #18217.

Section 2.31- Items in /var/sadm- /var/sadm/pkg/PKGNAME/install
---------------------------------------------------------------
drwxr-xr-x      2       root    root    1024    Dec  1 17:34    install

This directory can contain a number of different sets of information, 
depending on the package. For example, my SUNWcsr has:
-rwxr-xr-x      1       root    other   59      Nov 18 17:33    
copyright
-rwxr-xr-x      1       root    other   17      Nov 18 17:33    
u.devlinktab
-rwxr-xr-x      1       root    other   17      Nov 18 17:33    
u.drvalias
-rwxr-xr-x      1       root    other   1342    Nov 18 17:33
u.initd
-rwxr-xr-x      1       root    other   17      Nov 18 17:33    
u.kclasses
-rwxr-xr-x      1       root    other   17      Nov 18 17:33    
u.minorperm
-rwxr-xr-x      1       root    other   17      Nov 18 17:33    
u.nametomajor
-rwxr-xr-x      1       root    other   17      Nov 18 17:33    
u.preserve
-rwxr-xr-x      1       root    other   17      Nov 18 17:33    
u.rmmconf
-rwxr-xr-x      1       root    other   855     Nov 18 17:33    
u.syslogconf
-rwxr-xr-x      1       root    other   17      Nov 18 17:33    
u.voldconf

These files will be in every package, however. You will notice, however 
from my cat of the /var/sadm/pkg/SUNWcsr/pkginfo file below in Section 
3.5, that the files you see above largely correspond with those found 
in the CLASSES variable.

Section 2.32- Items in /var/sadm- /var/sadm/pkg/PKGNAME/pkginfo
---------------------------------------------------------------
-rw-r--r--      1       root    root    2580    Dec  1 17:32    pkginfo

This file is the most important file in the /var/sadm/pkg/PKGNAME 
directories. The health of this file is critical to package and patch 
installation. It is the main place for many pkg utilities to find their 
info, and is the one key point of failure in many patch and package 
installation and removal problems.

NOTE: Much of the information in this document relates directly to this 
file. Please check the later sections.

Section 2.33- Items in /var/sadm- /var/sadm/pkg/PKGNAME/save
------------------------------------------------------------
SUNWcsr, for example:
drwxr-xr-x      2       root    other   512     Dec  1 17:34    
106541-08
-rw-r--r--      1       root    other   1425    Dec  1 17:33    
driver_aliases.orig
-rw-r--r--      1       root    other   315     Dec  1 17:33    
driver_classes.orig
-rw-r--r--      1       root    other   3231    Dec  1 17:33    
minor_perm.orig
-rw-r--r--      1       root    other   1201    Dec  1 17:33    
name_to_major.orig

This directory contains old information about the package as it existed 
prior to a patch install, in the save directory, and backups of critical
 files for the package.

What you see in this directory will be different depending on the 
package, but the save information for the patch ID's that have bee 
installed should always exist:
-rw-r--r--      1       root    other   2174892 Dec  1 17:34    undo.Z

If it doesn't, then this means the patch was installed with the -d 
option- meaning do not save the backout information and you patch will 
not be able to be backed out.  This file will not always be in .Z 
format. Sometimes it will be just an undo file.

NOTE: Sun recommends you always install patches in backout-able fashion.

Section 2.4 The individual items in /var/sadm- /var/sadm/softinfo
-----------------------------------------------------------------
lrwxrwxrwx      1       root    root    28      Dec  1 16:44    
INST_RELEASE -> ../system/admin/INST_RELEASE

It is critical for patch installation that this link exist, and the 
place it's linked to exists. This can cause a problem with patching that
makes it look as though your patch installation cannot determine what 
OS your machine has, because that is exactly the problem. Here are the 
contents of this file:

root@dartagnean:/var/sadm/softinfo $ cat INST_RELEASE
OS=Solaris
VERSION=2.7
REV=0

Section 2.5- The individual items in /var/sadm- /var/sadm/system
----------------------------------------------------------------
drwxr-xr-x      3       root    sys     512     Apr 20  1999    admin
drwxr-xr-x      2       root    sys     512     Apr 20  1999    data
drwxr-xr-x      2       root     sys    512     Apr 20  1999    logs

There are no special files in this specific directory, but below this,
it gets interesting, as you will see.

Section 2.51- The individual items in /var/sadm- /var/sadm/system/admin
------------------------------------------------------------------------
-rw-r--r--      1       root    root    36153   Apr 20  1999    
.clustertoc
-rw-r--r--      1       root    root    203     Apr 20  1999    
.platform
-rw-r--r--      1       root    root    18      Apr 20  1999    
CLUSTER
-rw-r--r--      1       root    root    29      Apr 20  1999    
INST_RELEASE
drwxr-xr-x      2       root    sys     512     Apr 20  1999    
services

Section 2.511- Items in /var/sadm- /var/sadm/system/admin/.clustertoc
---------------------------------------------------------------------
The .clustertoc is a file that contains the information for the package 
clusters used to install your system. You will notice a difference 
between the packagenames as they appear in the Meta-Cluster (aka. 
configuration cluster) Contents and the actual package names you find on
your system. You may also see this info when installing software from 
AdminSuite or -Tool. Here is an example entry from the .clustertoc:

CLUSTER=SUNWCacc
NAME=System Accounting
DESC=System accounting utilities
VENDOR=Sun Microsystems, Inc.
VERSION=7.2
SUNW_CSRMEMBER=SUNWaccr
SUNW_CSRMEMBER=SUNWaccu
END

What we see here is a cluster package, and then the packages that 
comprise that cluster package. If you were attempting to compare a 
machine with a Core installation and a Entire Distribution installation,
you could see the differences between their configuration clusters up 
close.

There are also man pages on packagetoc and clustertoc that you can look
at for more detailed information.

Section 2.512- Items in /var/sadm- /var/sadm/system/admin/.platform
-------------------------------------------------------------------
PLATFORM_GROUP=sun4u
INST_ARCH=sparc
PLATFORM_NAME=SUNW,SPARCstation-fusion
PLATFORM_ID=SUNW,SPARCstation-fusion
IN_PLATFORM_GROUP=sun4u
PLATFORM_NAME=FJSV,GP
PLATFORM_ID=FJSV,GP
IN_PLATFORM_GROUP=sun4u

This file is included in cd images of Solaris CD's that need multiple 
platform support, and the file left over in /var/sadm/system/admin/ is a
file that denotes what hardware platform was determined to be 
appropriate for your system The man page fully describes it:
# man -s 4 platform

File Formats                                          platform(4)

DESCRIPTION
     The Solaris 2.5 release includes the .platform directory,  a
     new  directory on the Solaris CD image.  This directory con-
     tains files (created  by  SunSoft  and  Solaris  OEMs)  that
     define   platform  support.   These  files  are  generically
     referred to as platform definition files.   They  provide  a
     means to map different platform types into a platform group.

     Platform definition files in  the  .platform  directory  are
     used  by  the  installation software to ensure that software
     appropriate for the  architecture  of  the  system  will  be
     installed.

     SunSoft  provides   a   platform   definition   file   named
     .platform/Solaris  .  This  file  is  the  only one that can
     define platform groups to which  other  platform  definition
     files  can  refer.  For  example, an OEM platform definition
     file can refer  to  any  platform  group  specified  in  the
     Solaris platform definition file.
<snipped>

Section 2.513-Items in /var/sadm- /var/sadm/system/admin/CLUSTER
----------------------------------------------------------------
-rw-r--r--      1       root    root    18 Apr 20  1999 
/var/sadm/system/admin/CLUSTER

A Configuration Cluster is the level you select during installation 
where you can select from the following options:

1) Entire Distribution Plus OEM (for Sun clones, for example)
2) Entire Distribution (puts in everything and the kitchen sink)
3) Developer (medium level targetted to the needs of developers)
4) End User (for networked workstations that will never be servers)
5) Core (the bare-bones OS without network connectivity)

The way to tell what "Configuration Cluster" level the machine has been 
installed with is to look at the /var/sadm/system/admin/CLUSTER file. 

root@dartagnean:/ $ more /var/sadm/system/admin/CLUSTER
CLUSTER=SUNWCXall

Shows that I have installed the Entire Distribution Configuration 
Cluster on my system.

Section 2.514- Items in /var/sadm- /var/sadm/system/admin/INST_RELEASE
----------------------------------------------------------------------
-rw-r--r--      1       root    root    29      Apr 20  1999    
INST_RELEASE

This file must exist and not be empty. 

It is symbolically linked from /var/sadm/softinfo/INST_RELEASE as 
explained earlier in the document.

Section 2.515- Items in /var/sadm- /var/sadm/system/admin/services
------------------------------------------------------------------
-rw-r--r--      1       root    root    29      Apr 20  1999    
Solaris_2.7

If you cat the file, it looks exactly like 
/var/sadm/softinfo/INST_RELEASE described above.
I don't have information on what uses this file, yet I would not delete 
it on the basis of that.

Section 2.516- The individual items in /var/sadm- /var/sadm/system/data
-----------------------------------------------------------------------
drwxr-xr-x      2       root    sys     512     Apr 20  1999    data

My /var/sadm/system/data directory contains no files. However, it is 
possible that it can contain data:
Checking /var/sadm/README you will find that /var/sadm/system/data 
contains the upgrade_cleanup and upgrade_space_report when and if they 
are generated, and following definition:

system/data     Data files generated by system installation
                and configuration utilities containing system-relevant
                configuration data.

Section 2.517-The individual items in /var/sadm- /var/sadm/system/logs
----------------------------------------------------------------------
drwxr-xr-x      2       root    sys     512     Apr 20  1999    logs

On my system, this directory contains:
lrwxrwxrwx      1       root    root    20      Dec  1 16:44    
begin.log -> begin.log_1999_04_20
-rw-r--r--      1       root    root    90      Apr 20  1999    
begin.log_1999_04_20
lrwxrwxrwx      1       root    root    21      Dec  1 16:44    
finish.log -> finish.log_1999_04_20
-rw-r--r--      1       root    root    41      Dec 21 12:41    
finish.log_1999_04_20
-rw-r--r--      1       root    root    31183   Apr 20  1999    
install_log
-rw-r--r--      1       root    root    11658   Apr 20  1999    
sysidtool.log

Over the life of a system, especially if it has been ugraded a few 
times, multiple begin and finish scripts may be run. The begin.log and 
finish.log are always linked to the latest log file generated, but all 
logs are maintained. The same thing happens in /var/sadm/install_data 
with the Maintenance_Upgrade_log if one exists...

Section 2.518-The individual items in /var/sadm- /var/sadm/system/logs
----------------------------------------------------------------------
-rw-r--r--      1       root    root    31183   Apr 20  1999    
install_log

The install_log is from your OS installation, the options used for 
newfs'ing your filesystems initally, packages installed, etc. This is 
useful if you want to know what the specific original configuration of 
the system was when it was first installed.

Section 2.519-The individual items in /var/sadm- /var/sadm/system/logs
----------------------------------------------------------------------
-rw-r--r--      1       root    root    11658   Apr 20  1999    
sysidtool.log

The sysidtool.log was populated during the System Identification step 
during installation. The files may have different names than these, 
depending on what was installed and when.

Section 2.6- Summary of Section 2.x
-----------------------------------
This ends our detailed explosion of the /var/sadm directory, hopefully 
this has been useful and informative. See Section 3.0 for further 
discussion of higher-level System Administrator concepts with /var/sadm.

Section 3.0- The Role of the /var/sadm Directory 
------------------------------------------------

Quite often, those unfamiliar with the role of /var/sadm will look in 
there and remove the files, since their /var has possibly become full.
With the complexity of the repartitioning process, removing these files
looks like a viable option.  In fact, the opposite is true. The removal 
of these files may not impact the system for quite some time, but as 
soon as a patch or package needs to be applied or removed from the 
system you will run into a variety of  problems.

Since /var/sadm contains many log files, the /var/sadm directory must be
 considered as a single entity, a snapshot in time of the system as it 
exists. Backups of this directory will also be snapshots. This becomes 
important in a number of different ways, as you will see.

Section 3.1- To Patch or not to Patch 
-------------------------------------

Because of the long uptime capabilities of Solaris machines, I note two 
separate philosophies had by sys admins.  One is the "If it isn't broke,
don't fix it" model and the other is "stay up to the minute on all
patches" model.  One could effectively argue for a "catch-as-catch-can" 
model, but that would be a hybrid, as are most philosophies about 
technical things.

Really, it's a difference of reactive vs.  proactive, and it is 
generally Sun's position to take the proactive approach.  This is why we
always recommend you put on the recommended and Y2K patch clusters, etc.
on a regular basis.

The speed of change of hardware and software that runs within and on 
Solaris is updated at a very fast pace.  Many sys admins will want to 
"freeze" their systems in a certain way, but inevitably, some moment 
will come when major upgrades or changes will need to take place on the 
system, and the /var/sadm directory will play a prominent role.  It is 
at these times that you will want to have used the extra disk space that
may not have seemed important a long time ago, or have not deleted or 
manually added.  Often, when taking calls in the Solutions Center, we 
see that the actions of a sys admin years ago (who may no longer be with
the company, or might be in another department now) affect the sys admin
 who is now taking care of the system. If you are the current System 
Administrator, you will want to consider the needs of the System 
Administrator that may be caring for this machine in the future.

Section 3.2- Log File Synchronization 
-------------------------------------
In understanding package and patch installation, you must consider two 
separate variables.  First is the copying of new files or binaries to 
the system which contain bug fixes.  The second is the accurate 
recording of the information regarding what changes have been made to 
the system. This is the role of the /var/sadm directory, among other 
things.

This conceptually makes the maintenance of accurate information in 
/var/sadm of utmost importance to the system administrator. In System 
Administration classes taught at Sun, they emphasize the point that you 
should always use the standard utilities when modifying the system, they
 may not explain fully why.

Section 3.2.1- De-synchronization Examples
------------------------------------------
There are many examples of this de-synchronization phenomenon many of 
which might not seem like a problem, and some that are fairly obvious:

1) Modifying your /etc/passwd file. 

This common act could not be considered by anyone as unusual, but since 
it involves manual modifications of the file, then your contents file 
will not reflect these changes, nor does it need to. Keep in mind that 
any and all manual changes to the system are not reflected in the 
information in /var/sadm/install/contents.

2) Restoring /usr (or another "root" filesystem) from backup.

 In this case, few of the sizes or  permissions of the files will be the
same as the ones found in the contents file, unless those files have not
changed since the last backup happened. Since many times your 
permissions may change when restoring files from backup, you must rely 
on your contents file for a direction to head in. The condition of #1 
will also be true here with respect to modified files.

3) Restoring /var from backup. 

In this case, binaries and files that have been modified by the package 
and patch utilities since your last backup would not be reflected in the
log files in /var as a whole, as well as the contents file.

4) Someone removed SUNWcsu from the system, and now my linker patch 
won't install. 

AND the files show up in /var/sadm/install/contents as being there. You 
can do a manual pkgadd of this pkg, but if that doesn't work, a 
reinstall of the OS is needed to get this back, because it's a core 
package.

NOTE: I have highlighted these examples, but there are many other cases 
of ways people commonly de-synchronize their systems without knowing it.
Every System Administrator should be aware of the implications of their
actions on a server with regard to that action's impact on change 
management.

Section 3.3- General Filesystem Health 
--------------------------------------

Since the synchronization of information between the system and it's log
files is of critical importance, checking the general /var filesystem 
health is very important also.  You should never fsck a mounted
filesystem, even if booted in single-user mode.  It is necessary to boot
from cdrom to check /var, or any of the other so-called 'root' 
filesystems such as / and /usr.

This can be further complicated by the addition of disk management 
software such as Veritas Volume Manager, or Solstice Disksuite.  Follow 
the procedures for those products to make the filesystems available for 
fsck'ing.  Additionally, vxfs ships it's own version of fsck, so be 
aware of that as well.

Section 3.4- /var: To Link or Not To Link
-----------------------------------------

Many times, we see people who run out of space attempt to link /var or 
some directory within /var to get them more space. We do not recommend 
this,  because there are problems you can run into as a result. There 
are many  different workarounds I have heard regarding this, all because
a repartition is undesirable for one reason or another. It is always 
your  best solution to have a lot of space in /var for some generations
of patches and packages. Many times, if /var is not a separate 
filesystem (and we generally recommend that it should be) you may have 
another unused partition or disk that you can off-load this onto. 

This process is relatively simple, and can be done with minimal 
downtime. Make the new partition, mount it, then use the dash (-) and 
p (for retain permissions) options with tar to copy the files to the 
new filesystem, boot from cdrom, modify the directory names, create a 
new /var mount point, change the vfstab to reflect the changes and 
reboot. There are many infodocs in SunSolve that can assist you with 
these various processes.

Section 3.5- Managing pkginfo files 
-----------------------------------

The pkginfo files for packages in /var/sadm/package really contain the
most information about the packages.  I would be certain to keep a
backup of these files handy, in case of a problem during changes you
make to your system. It is for this reason that we always recommend you
do a good backup before installing packages or patches. 

NOTE: Sun Microsystems does not support the manual editing of the
pkginfo  file, even in situations where it may be necessary to reinstall
the OS. It can be possible, in certain circumstances, to restore the
pkginfo file of a package from backup, but there are possible drawbacks
to this depending on what changes have occurred with this package since
your last backup.

There are high level concerns about the structure of the pkginfo file 
that should be understood.  I like to use the SUNWcsr package as an 
example. Most of the variables contained in this file are beyond the 
scope of this document.  We will primarily discuss the PKG, ARCH, 
VERSION, and PATCH_INFO variables.

This example is taken from my own patched 2.7 machine:
I like to use SUNWcsr as the example, because it is often patched.

root@dartagnean:/ $ more /var/sadm/pkg/SUNWcsr/pkginfo
CLASSES=none initd devlinktab kclasses minorperm nametomajor drvalias 
renamenew preserve cronroot passwd tiservices inetdconf iuap etcsystem 
tzonecvt etcremote nsswitch netconfig deflogin defsu syslogconf ttysrch 
group inittab automaster etcrpc etcprofile mailxrc shadow vol dconf 
rmmconf sock2path renameold protocols etcskel
BASEDIR=/
TZ=US/Pacific
PATH=/sbin:/usr/sbin:/usr/bin:/usr/sadm/install/bin
OAMBASE=/usr/sadm/sysadm
PKG=SUNWcsr
NAME=Core Solaris, (Root)
ARCH=sparc
VERSION=11.7.0,REV=1998.09.01.04.16
SUNW_PRODNAME=SunOS
SUNW_PRODVERS=5.7/Generic
SUNW_PKGTYPE=root
MAXINST=1000
CATEGORY=system
DESC=core software for a specific instruction-set architecture
VENDOR=Sun Microsystems, Inc.
HOTLINE=Please contact your local service provider
EMAIL=
SUNW_PKGVERS=1.0
PSTAMP=on99819980901043908
PATCHLIST=106832-02 106924-01 106944-01 106948-01 106936-01 106985-01 
107121-01 107148-01 106963-01 106541-02
PATCH_INFO_106832-02=Installed: Tue Jan 26 13:49:24 PST 1999 From: 
fern  Obsoletes:  Requires: Incompatibles:
PATCH_INFO_106924-01=Installed: Tue Jan 26 13:59:25 PST 1999 From: 
fern  Obsoletes:  Requires: Incompatibles:
PATCH_INFO_106944-01=Installed: Tue Jan 26 14:02:05 PST 1999 From: 
fern Obsoletes:  Requires: Incompatibles:
PATCH_INFO_106948-01=Installed: Tue Jan 26 14:03:07 PST 1999 From: 
fern  Obsoletes:  Requires: Incompatibles:
PATCH_INFO_106936-01=Installed: Tue Jan 26 14:12:42 PST 1999 From: 
fern Obsoletes:  Requires: Incompatibles:
PATCH_INFO_106985-01=Installed: Tue Jan 26 14:18:07 PST 1999 From: 
fern Obsoletes:  Requires: Incompatibles:
PATCH_INFO_107121-01=Installed: Tue Jan 26 14:50:19 PST 1999 From: 
fern Obsoletes:  Requires: Incompatibles:
PATCH_INFO_107148-01=Installed: Tue Jan 26 14:53:30 PST 1999 From: 
fern Obsoletes:  Requires: Incompatibles:
PATCH_INFO_106963-01=Installed: Tue Jan 26 15:13:43 PST 1999 From: 
fern Obsoletes:  Requires: Incompatibles:
PATCH_INFO_106541-02=Installed: Tue Jan 26 15:36:27 PST 1999 From: 
fern Obsoletes: 106976-01 Requires: Incompatibles:
PKGINST=SUNWcsr
PKGSAV=/var/sadm/pkg/SUNWcsr/save
INSTDATE=Apr 20 1999 19:11

Section 3.6- Lab Exercise
-------------------------

To illustrate some things about the way various utilities work, I 
suggest running several package related commands such as pkginfo -l, 
showrev -p, patchadd -p (functionally duplicates showrev -p), or 
pkgparam.

1. If I compare showrev -p, patchadd -p and PATCH_INFO lines, I will see
a pattern (I have <snipped> the output for space considerations):

# cat /var/sadm/pkg/SUNWcsr/pkginfo | grep PATCH_INFO | sort
PATCH_INFO_106541-02=Installed: <snipped>                    
PATCH_INFO_106832-02=Installed: <snipped>
PATCH_INFO_106924-01=Installed:<snipped>
PATCH_INFO_106936-01=Installed:<snipped>
PATCH_INFO_106944-01=Installed: <snipped>
PATCH_INFO_106948-01=Installed: <snipped>
PATCH_INFO_106963-01=Installed: <snipped>
PATCH_INFO_106985-01=Installed: <snipped>
PATCH_INFO_107121-01=Installed:<snipped>
PATCH_INFO_107148-01=Installed: <snipped>
root@dartagnean:/ $ showrev -p | grep SUNWcsr | sort
Patch: 106541-02 <snipped>
Patch: 106832-02 <snipped>
Patch: 106924-01<snipped>
Patch: 106936-01<snipped>
Patch: 106944-01 <snipped>
Patch: 106948-01<snipped>
Patch: 106963-01<snipped>
Patch: 106985-01<snipped>
Patch: 107121-01 <snipped>
Patch: 107148-01<snipped>

root@dartagnean:/ $ patchadd -p | grep SUNWcsr | sort
Patch: 106541-02 <snipped>
Patch: 106832-02<snipped>
Patch: 106924-01<snipped>
Patch: 106936-01<snipped>
Patch: 106944-01 <snipped>
Patch: 106948-01<snipped>
Patch: 106963-01<snipped>
Patch: 106985-01<snipped>
Patch: 107121-01<snipped>
Patch: 107148-01<snipped>

You will notice that the patch ID's are identical. I piped them all 
through "sort" simply so they would appear in the same order. They don't
always appear in the same order, depending on the utility you use to get
the information.

This example shows what these utilities use to get their information. 
Without these lines in the pkginfo, showrev -p will not show any 
information. This illustrates the importance of using standard patching
methodologies to install you patches, and maintain you /var/sadm 
directory.

Section 3.7- Corrupt pkginfo files
----------------------------------

When there are problems installing patches or packages, it can be 
related to a corrupt log file, such as the pkginfo file. This is a 
relatively infrequent occurrence, but merits mention due to the fact 
that we will often recommend drastic actions to redress the problem.

The Solaris patch and packaging facilities, as of 2.7, have no 
redundancy or error checking beyond the messages you may see when 
installing patches.  Often, my /var/sadm/SUNWcsr/pkginfo file can become
corrupted, I will need to restore that file from backup, or to reinstall
the system from scratch, depending on the package. If this is a core 
package,  reinstallation will be your only supported option, besides 
restoring from backup.  For some other examples of troubleshooting this 
issue, take a look at Infodoc 15579.

Patches and packages are closely related.  For a fuller discussion of 
this, please read Infodoc 17973, or the larger Patch PSD 20597, which 
is really a compilation of patch related Infodocs.

In the case of restoring the pkginfo file from backup, it will need to 
be considered as a small part of a larger /var/sadm entity.  For 
example, if I restore my SUNWcsr/pkginfo file from a backup that I made 
previous to several patches being applied, I will be de-synchronizing 
the /var/sadm directory from the way the system actually exists.  This 
is why hand-editing many of the files in /var/sadm is not supported.

Section 3.71- Corrupt pkginfo file and compilers
------------------------------------------------
One common problem is found after upgrading the system, then leaving 
/var alone, but then restoring all the binaries or files in /usr , /opt 
or another filesystem from backup.  In this case your /var/sadm has no 
knowledge whatsoever of the actual contents of the system itself, much 
less actual packages and patch levels, etc.  In this case, it is better 
to restore only special or specific files from backup and reinstall your
compilers.

This is a common call in the group that supports our compiler products,
because often a customer will upgrade the system before upgrading the 
compilers on that system. 

Section 3.8- The "Contents" File
--------------------------------

The /var/sadm/install/contents file is a large file that is responsible 
for  noting the changes made via the standard patching and packaging 
utilities.  There are many uses for it, such as telling what files are 
associated with a particular package, like so:

# cat /var/sadm/install/contents | grep SUNWcsu

And you will get a long list of files, their permissions, file sizes, 
etc. Conversely, you can find what package a file belongs to like this:

# cat /var/sadm/install/contents | grep /bin/csh
/usr/bin/csh f none 0555 bin bin 151976 51498 907659673 SUNWcsu

Where /bin/csh is the example file searched for. Of course, you can do 
general searches as well as this specific one by widening your search 
criteria.

NOTE: You should never simply 'cat' your whole contents file. This file 
is enormous, and would simply scroll off your screen many times over.

The /var/sadm/install/contents file becomes important when large errors 
are inadvertently made on the system.  For example:

#chmod -R /

causes many problems that can be hard to recover from.  Yet pkgchk -f 
can help tremendously in these types of worst-cases. Other problems that
result from some recursive changes can be solved using this. However, 
having a healthy /var/sadm directory is critical when running this 
command, as that command relies upon the accuracy of the information 
there. Also know that many false positives come from running pkgchk -f.

Section 3.9- The Definition of a Package
-----------------------------------------

I am including this seemingly simple explanation for one reason- 
Packages appear to be different depending on how you look at them. 
Here's what I mean:

1) A package is a collection of files and directories.
 (From the Drake test for Solaris Certification)

2) From a patch's point of view, a package is a set of variables: 
PKG, ARCH, and VERSION, taken from the pkginfo files for the packages to
be patched. If you see the error message: none (or some) of the packages
 to be patched are not on the system. This means something different to 
you than it does to the patch installations script. If the PKG, ARCH, 
and VERSION variables do not match exactly, it will not install simply 
because a patch defines a package differently.

3) Showrev -p:
Showrev -p thinks a package is a collection of PATCH_INFO lines,  found 
in the pkginfo file for packages.

4) pkginfo and pkginfo -l: pkginfo without arguments simply takes the 
information it needs  from the pkginfo file and displays it. Pkginfo -l 
must do a little more work, and this is why it takes longer to execute. 
It checks the /var/sadm/install/contents file for the additional 
information such as STATUS and FILES, does the calculations, and 
displays the information. If you check the man page for pkginfo, you 
will notice many of the options deal directly with the variables you 
will see in the pkginfo file.

5) Human perspective:
We think a package is simply a packagename, but it isn't always, as you 
can see from above.

NOTE: One helpful item about using the pkginfo command:
 If you have many packages to install from CDROM, usually you must do a 
'pkgadd -d /path/to/pkg pkgname' for each one. If you do a pkginfo -d 
once, this will set your default package installation directory from 
/var/spool/pkg to whatever long directory tree on the CDROM you need.

In summary, we have fully discussed in general and in detail the many 
aspects of the /var/sadm directory as it relates to overall system 
maintenance, patching,  packages, and change management in general. We 
have shown specific examples of various problems and uses of file one 
can encounter in attempting to keep one's system healthy over time. We 
have discussed high level aspects of the structure of the /var/sadm 
directory and what many of the files and directories there do. 

There are many aspects of System Administration that require many years
of experience to understand. I have attempted to summarize the majority
of the issues around the /var/sadm directory as I see it. There may be
other things to know about the importance of the this directory, however
I hope this has given you a place to begin to understand the structure 
and function of a large part of why /var exists.

> Delivered-To: [email-masked]
> Delivered-To: [email-masked]
> From: "Erin Jones" <[email-masked]>
> To: "sun managers" <[email-masked]>
> Subject: /var/sadm is too damn big
> Date: Thu, 2 Nov 2000 11:39:02 -0500
> Content-Transfer-Encoding: 7bit
> X-Priority: 3
> X-MSMail-Priority: Normal
> X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400
> 
> hey all,
> 
> I went through the archives but i can't find the article pertaining to
> this.....
> my /var/sadm is too damn big.  the tech that originally set up this machine
> gave / the following geometry:
> /dev/dsk/c0t0d0s0     144172  103938   25817    81%    /
> ok, I am getting annoyed that when any of my logs or mail spool gets any
> size to them it fills up the filesystem.  upon investigation I am seeing the
> following culprits on my / partition
> 5556    etc
> 7364    kernel
> 17251   opt
> 5255    platform
> 5019    sbin
> 62926   var
> 
> there are a few others but those compromise the bulk of it.  the big one,
> /var, is almost entirely /var/sadm:
> 59607   /var/sadm
> is there anyway I can prune this directory safely?
> 
> thanx for your help
> ej
> 
> Erin Jones                               Director of Information Services
> [email-masked]           (212) 387 0310 x19
>    Unix: Where /sbin/init is Job #1.
> 
> 
> 
> 
> 
> S
> U   BEFORE POSTING please READ the FAQ located at
> N       ftp://ftp.cs.toronto.edu/pub/jdd/sun-managers/faq
> .   and the list POLICY statement located at
> M       ftp://ftp.cs.toronto.edu/pub/jdd/sun-managers/policy
> A   To submit questions/summaries to this list send your email message to:
> N       [email-masked]
> A   To unsubscribe from this list please send an email message to:
> G       [email-masked]
> E   and in the BODY type:
> R       unsubscribe sun-managers
> S   Or
> .       unsubscribe sun-managers [email-masked]
> L   To view an archive of this list please visit:
> I       http://www.latech.edu/sunman.html
> S
> T

================================================================================
                 ______
                /_____/\             Ericka Fowler
               /_____\\ \            Technical Support Engineer
              /_____\ \\ /           
             /_____/ \/ / /          Sun Microsystems
            /_____/ /   \//\         500 Eldorado Blvd
            \_____\//\   / /         MS: UBRM04-131
             \_____/ / /\ /          Broomfield, CO  80021
              \_____/ \\ \           Phone 303-464-4666
               \_____\ \\            Sun Internal Ext. 50666
                \_____\/	     email: [email-masked]

================================================================================