Return-Path: Received: from mercury.Sun.COM (mercury.Sun.COM [192.9.25.1]) by msg42.younetwork.com (8.9.3/8.9.3) with ESMTP id MAA21921 for ; Thu, 2 Nov 2000 12:15:57 -0500 Received: from bast.Central.Sun.COM ([129.147.4.36]) by mercury.Sun.COM (8.9.3+Sun/8.9.3) with ESMTP id JAA17868 for ; Thu, 2 Nov 2000 09:02:20 -0800 (PST) Received: from mothra (mothra [129.147.40.102]) by bast.Central.Sun.COM (8.9.3+Sun/8.9.3/ENSMAIL,v1.7) with SMTP id KAA04451 for ; Thu, 2 Nov 2000 10:02:19 -0700 (MST) Message-Id: <200011021702.KAA04451@bast.Central.Sun.COM> Date: Thu, 2 Nov 2000 10:02:19 -0700 (MST) From: Ericka Fowler Reply-To: Ericka Fowler Subject: Re: /var/sadm is too damn big To: erin@internationalcomputing.com MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii Content-MD5: OOOsQ2HjudWnTBl871kVMQ== X-Mailer: dtmail 1.3.0 @(#)CDE Version 1.3.2 SunOS 5.7 sun4u sparc X-UIDL: LIl!!9>9!!7Rg"!N3*"! Sun doesn't recommend that you delete things out of /var/sadm - it will make things difficult when you want to patch or pkgrm or upgrade your system. I'm including the /var/sadm infodoc so that you can have an explanation of what all the different files and directories under /var/sadm are... INFODOC ID: 21481 SYNOPSIS: /var/sadm PSD DETAIL DESCRIPTION: Section 1.0- Introduction ------------------------- The intention of this document is to outline some of the high level concerns that should be considered by System Administrators with regard to patching, space issues, and future planning. Each of the directories in /var/sadm serves a special purpose that often is not considered when changing, updating, or upgrading Solaris systems. This document is intended to give you a place to find information relating to these directories, their function, structure, and usage. The /var/sadm directory is extremely important, especially when changes are made and logged to the software on the system in any form. Some systems are more static than others, however, at some point, all systems need changes or upgrades. A healthy /var is critical to the change management of the system. To this end, proper advance planning and space considerations for future use are very important. Take a look at Infodoc 18509 for more information on space considerations. One other note on change management of Solaris systems. What makes change management more difficult can be the large amount of time spent between changes, and often the personnel changes between these changes. The times that packages are being added, removed or patched is usually a critical one for the System Administrator, the machine, the organization as a whole and possible deadlines. One example of this is Year 2000 concerns. I point this out because errors in your /var/sadm directory can often need wholesale changes to the system to correct, such as reinstallation of your OS. Be aware of this from the beginning, so your expectations are set correctly. This document is structured to show you specific examples first, in Section 2.0, then general topics starting with Section 3.0. Section 2.0- The /var/sadm directory exploded ---------------------------------------------- The items found in /var/sadm on my 2.7 machine: total 754 -r--r--r-- 1 root sys 1092 Sep 10 1998 README dr-xr-xr-x 4 bin bin 512 Dec 1 17:40 install drwxr-xr-x 2 root sys 512 Oct 21 13:14 install_data drwxr-xr-x 11 root root 512 Dec 1 17:41 patch dr-xr-xr-x 417 root sys 8192 Dec 1 17:40 pkg drwxr-xr-x 2 root sys 512 Apr 20 1999 softinfo drwxr-xr-x 5 root sys 512 Apr 20 1999 system serve a common purpose. This purpose is to record the changes made to the system. Many Solaris change management utilities rely upon the information inside /var/sadm to be an accurate picture of what actually resides on the machine. Section 2.01- The individual items in /var/sadm- README ------------------------------------------------------- -r--r--r-- 1 root sys 1092 Sep 10 1998 README This file contains important information about the installation of the OS on your machine. See Section 2.516 for more info. Section 2.02 The individual items in /var/sadm- /var/sadm/install ------------------------------------------------------------------ -rw-r--r-- 1 root root 128 Dec 1 17:40 .lockfile -rw-rw-rw- 1 root root 0 Apr 20 1999 .pkg.lock dr-xr-xr-x 2 bin bin 512 Apr 20 1999 admin -rw-r--r-- 1 root other 2798313 Dec 1 17:40 contents dr-xr-xr-x 2 bin bin 512 Dec 1 17:40 logs Both .pkg.lock and .lockfile are necessary for differing reasons, although they are similar in what they do. These files are necessary for the package utilities and should not be deleted. Section 2.021- Items in /var/sadm- /var/sadm/install/admin ---------------------------------------------------------- -r--r--r-- 1 root sys 193 Sep 1 1998 default The directory contains only the 'default' file. Section 2.022- Items in /var/sadm- /var/sadm/install/admin/default ------------------------------------------------------------------ #ident "@(#)default 1.4 92/12/23 SMI" /* SVr4.0 1.5.2.1 */ mail= instance=unique partial=ask runlevel=ask idepend=ask rdepend=ask space=ask setuid=ask conflict=ask action=ask basedir=default The uses of this file are well-described in Infodoc 15378, and the man pages for admin (4). An excerpt from the Infodoc: Each package must have a request script for pkgask utilities to work. In the case of multiple packages, a response DIRECTORY must be used. The request script and corresponding response files are specific to the package installed. pkgadd itself may require interaction. The behavior of pkgadd is determined by the settings in the /var/sadm/install/admin/default file. This is well-documented in the manual pages (man -s 4 admin). To establish different behavior, create an alternative admin file which you can then refer to using the -a option of pkgadd. >From the man page: # man -s 4 admin File Formats admin(4) /var/sadm/install/admin/default is the default admin file delivered with this release. The default file is not writ- able, so to assign values different from this file, create a new admin file. There are no naming restrictions for admin files. Name the file when installing a package with the -a option of pkgadd(1M). If the -a option is not used, the default admin file is used. Example: to have pkgadd not check for overwriting a file which may conflict with another package, you could create a new admin file and set the value of the "conflict" parameter from "ask" to "nocheck". This will unconditionally install the new file when a conflict is detected. Obviously, changing these settings should be done with caution. Section 2.03- The individual items in /var/sadm- /var/sadm/install/logs ------------------------------------------------------------------------ -rw-r--r-- 1 root other 252 Dec 13 23:11 Netscape_Communicator_4.51_install.A12131109 -rw-r--r-- 1 root other 9850 Dec 13 23:11 Netscape_Communicator_4.51_install.B12131109 -rw-r--r-- 1 root other 817 Dec 15 18:18 Solaris_7_Documentation_CD_install.A12150553 -rw-r--r-- 1 root other 272 Dec 15 23:50 Solaris_7_Documentation_CD_install.A12151150 -rw-r--r-- 1 root other 72652 Dec 15 18:01 Solaris_AnswerBook2_Server_install.B12150553 -rw-r--r-- 1 root other 686 Dec 16 01:00 Solaris_Easy_Access_Server_3.0_Documentation_CD_install.A12161255 -rw-r--r-- 1 root other 62485 Dec 16 01:00 Solaris_Easy_Access_Server_Collections_install.B12161255 This set of logs is taken from a server that has had several documentation cdroms installed onto it, as well as Netscape Communicator. This area may be used by any number of products to put their installation logs. Section 2.04- Items in /var/sadm- /var/sadm/install/contents ------------------------------------------------------------- -rw-r--r-- 1 root other 2798313 Dec 1 17:40 contents A key file around synchronization issues is the /var/sadm/install/contents file. This is a file that can be used several ways. You can use it to discover what package an individual file belongs to, or conversely, use it to find out all the files associated with a certain package. For example to find out what things are associated with the format command: # cat /var/sadm/install/contents | grep /etc/format /etc/format=../usr/sbin/format s none SUNWcsr /etc/format.dat v none 0644 root sys 7603 30297 904647929 SUNWcsr If you were to instead grep for SUNWcsr, you would normally find quite a few more files, too numerous to list here. Section 2.1- The individual items in /var/sadm- /var/sadm/install_data ---------------------------------------------------------------------- There are a couple of log files in this directory that can be useful: -rw-r--r-- 1 root other 2612 Oct 21 13:19 Solaris_7_Recommended_log lrwxrwxrwx 1 root root 26 Apr 20 1999 install_log -> ../system/logs/install_log The first is one that was used by the Solaris 7 recommended patch cluster, but I find this one to of minimal use in debugging patch installations. The second can be used after installation to tell you much historical information about the way this system was initially installed. It may be a system that has been inherited, or a system that has not had changes made on it for a while. Over the life of a system, especially if it has been ugraded a few times, multiple Maintenance Upgrade scripts may be run. The Maintenance_Upgrade_logs are always linked to the latest log file generated, but all logs are maintained. The same thing happens in /var/sadm/system/logs with the begin.log and finish.logs if they exist... Section 2.2- The individual items in /var/sadm- /var/sadm/patch ---------------------------------------------------------------- >From a 2.7 machine: total 34 -rw-r--r-- 1 root root 0 Apr 20 1999 .mu_applied -rw-r--r-- 1 root other 5550 Dec 1 17:41 .patchDB drwxr-x--- 2 root other 512 Dec 1 17:41 106541-08 drwxr-x--- 2 root other 512 Oct 21 13:16 106725-01 drwxr-x--- 2 root other 512 Oct 21 13:16 106793-03 drwxr-x--- 2 root other 512 Oct 21 13:16 106934-03 drwxr-x--- 2 root other 512 Oct 21 13:15 107022-04 drwxr-x--- 2 root other 512 Oct 21 13:15 107171-04 drwxr-x--- 2 root other 512 Oct 21 13:27 107359-01 drwxr-x--- 2 root other 512 Oct 21 13:16 107544-02 drwxr-x--- 2 root other 512 Oct 21 13:28 107587-01 Section 2.21- Items in /var/sadm- /var/sadm/patch/.mu_applied ------------------------------------------------------------- -rw-r--r-- 1 root root 0 Apr 20 1999 .mu_applied The .mu_applied file is a log file that is created as a result of a HW Maintenance Update patches that have been installed as a result of a hardware release upgrade, install, or maintenance update patch install. It could be thought of as a sort of "lock" file that denotes a HW release has been done on the system. Section 2.22- Items in /var/sadm- /var/sadm/patch/.patchDB ---------------------------------------------------------- -rw-r--r-- 1 root other 5550 Dec 1 17:41 .patchDB The .patchDB is the mechanism through which patch installation speed has improved on 2.6 and later systems. This file does not exist on 2.5.1 and earlier systems. Sometimes, as a troubleshooting step during patch installation, one can delete the .patchDB file, and it will be rebuilt during that patch installation. This does add extra time for the next patch installation, however. Section 2.23- Items in /var/sadm- /var/sadm/patch/PATCHID# ---------------------------------------------------------- The rest of the directories are numbered in order by patch-ID and can contain backoutpatch (2.6 and earlier) scripts needed to backout versions of the patches you have installed on your machine. Patchrm is used from 2.6 and later versions of the OS. It also contains log files about the installation of that particular patch, and the README file as well. There is no backout data contained in this directory structure, at least not with direct instance (DI) patching. I believe there is still backout data contained here if progressive instance (PI) patches are installed that have not been properly built. NOTE: A complete description of direct instance vs. progressive instance patches is beyond the scope of this document, however, to sum up: -The DI patch structure was introduced with Solaris 2.5, and MOST patches since then have been constructed this way. Unfortunately, there are several patches (compiler and license patches are typical), that still use the PI structure, or in some case both PI and DI. -If the patches are new and are constructed so that both DI and PI are available, you should see a directory in the top level of the patch that is labeled "old_style_patch'. While it's true that patchrm will complain about backing out a patch that does not have a patch_id directory in this tree, the directory could be empty and patchrm will often work fine, including restoring the backout data that is actually contained in /var/sadm/pkg/pkg_name/save by default. It is likely that your system has many more patches than my example, depending on the age and version of the OS you have installed. Section 2.3- The individual items in /var/sadm- /var/sadm/pkg ------------------------------------------------------------- .................. drwxr-xr-x 4 root root 512 Dec 1 17:30 SUNWcar drwxr-xr-x 4 root root 512 Dec 1 17:31 SUNWcarx drwxr-xr-x 4 root root 512 Dec 1 17:34 SUNWcsr drwxr-xr-x 4 root root 512 Dec 1 17:34 SUNWcsu ................. I will not list all the items found in my /var/sadm/pkg directory, since there are too many items. The above packages are simply examples. This is the directory where all the information about your packages is kept. It is critical to keep this directory intact and up to date by using the standard pkgadd and installpatch/backoutpatch/patchadd utilities. Showrev -p gets it's information from here as well, from the pkginfo file in each pkg directory. Backout information for patches is kept in /var/sadm/pkg/pkgname/save directory. See Section 2.33 below for a description. I have seen a problem arise with showrev -p, where you will get an error like: showrev: get_env_var(filename, SUNW_PATCHID) This can be for 2 reasons: 1) There is a file in /var/sadm/pkg. Usually, this file will match the filename listed in the error message. 2) The permissions on /var/sadm/pkg/pkgname/pkginfo file is wrong. It should be: -rw-r--r-- 1 root root 593 Apr 20 1999 pkginfo In this case, then, the message would say this instead: showrev: get_env_var (pkgname, SUNW_PATCHID) Switching the pkgname affected instead of the filename. A fuller discussion of this can be found in Indofoc #18217. Section 2.31- Items in /var/sadm- /var/sadm/pkg/PKGNAME/install --------------------------------------------------------------- drwxr-xr-x 2 root root 1024 Dec 1 17:34 install This directory can contain a number of different sets of information, depending on the package. For example, my SUNWcsr has: -rwxr-xr-x 1 root other 59 Nov 18 17:33 copyright -rwxr-xr-x 1 root other 17 Nov 18 17:33 u.devlinktab -rwxr-xr-x 1 root other 17 Nov 18 17:33 u.drvalias -rwxr-xr-x 1 root other 1342 Nov 18 17:33 u.initd -rwxr-xr-x 1 root other 17 Nov 18 17:33 u.kclasses -rwxr-xr-x 1 root other 17 Nov 18 17:33 u.minorperm -rwxr-xr-x 1 root other 17 Nov 18 17:33 u.nametomajor -rwxr-xr-x 1 root other 17 Nov 18 17:33 u.preserve -rwxr-xr-x 1 root other 17 Nov 18 17:33 u.rmmconf -rwxr-xr-x 1 root other 855 Nov 18 17:33 u.syslogconf -rwxr-xr-x 1 root other 17 Nov 18 17:33 u.voldconf These files will be in every package, however. You will notice, however from my cat of the /var/sadm/pkg/SUNWcsr/pkginfo file below in Section 3.5, that the files you see above largely correspond with those found in the CLASSES variable. Section 2.32- Items in /var/sadm- /var/sadm/pkg/PKGNAME/pkginfo --------------------------------------------------------------- -rw-r--r-- 1 root root 2580 Dec 1 17:32 pkginfo This file is the most important file in the /var/sadm/pkg/PKGNAME directories. The health of this file is critical to package and patch installation. It is the main place for many pkg utilities to find their info, and is the one key point of failure in many patch and package installation and removal problems. NOTE: Much of the information in this document relates directly to this file. Please check the later sections. Section 2.33- Items in /var/sadm- /var/sadm/pkg/PKGNAME/save ------------------------------------------------------------ SUNWcsr, for example: drwxr-xr-x 2 root other 512 Dec 1 17:34 106541-08 -rw-r--r-- 1 root other 1425 Dec 1 17:33 driver_aliases.orig -rw-r--r-- 1 root other 315 Dec 1 17:33 driver_classes.orig -rw-r--r-- 1 root other 3231 Dec 1 17:33 minor_perm.orig -rw-r--r-- 1 root other 1201 Dec 1 17:33 name_to_major.orig This directory contains old information about the package as it existed prior to a patch install, in the save directory, and backups of critical files for the package. What you see in this directory will be different depending on the package, but the save information for the patch ID's that have bee installed should always exist: -rw-r--r-- 1 root other 2174892 Dec 1 17:34 undo.Z If it doesn't, then this means the patch was installed with the -d option- meaning do not save the backout information and you patch will not be able to be backed out. This file will not always be in .Z format. Sometimes it will be just an undo file. NOTE: Sun recommends you always install patches in backout-able fashion. Section 2.4 The individual items in /var/sadm- /var/sadm/softinfo ----------------------------------------------------------------- lrwxrwxrwx 1 root root 28 Dec 1 16:44 INST_RELEASE -> ../system/admin/INST_RELEASE It is critical for patch installation that this link exist, and the place it's linked to exists. This can cause a problem with patching that makes it look as though your patch installation cannot determine what OS your machine has, because that is exactly the problem. Here are the contents of this file: root@dartagnean:/var/sadm/softinfo $ cat INST_RELEASE OS=Solaris VERSION=2.7 REV=0 Section 2.5- The individual items in /var/sadm- /var/sadm/system ---------------------------------------------------------------- drwxr-xr-x 3 root sys 512 Apr 20 1999 admin drwxr-xr-x 2 root sys 512 Apr 20 1999 data drwxr-xr-x 2 root sys 512 Apr 20 1999 logs There are no special files in this specific directory, but below this, it gets interesting, as you will see. Section 2.51- The individual items in /var/sadm- /var/sadm/system/admin ------------------------------------------------------------------------ -rw-r--r-- 1 root root 36153 Apr 20 1999 .clustertoc -rw-r--r-- 1 root root 203 Apr 20 1999 .platform -rw-r--r-- 1 root root 18 Apr 20 1999 CLUSTER -rw-r--r-- 1 root root 29 Apr 20 1999 INST_RELEASE drwxr-xr-x 2 root sys 512 Apr 20 1999 services Section 2.511- Items in /var/sadm- /var/sadm/system/admin/.clustertoc --------------------------------------------------------------------- The .clustertoc is a file that contains the information for the package clusters used to install your system. You will notice a difference between the packagenames as they appear in the Meta-Cluster (aka. configuration cluster) Contents and the actual package names you find on your system. You may also see this info when installing software from AdminSuite or -Tool. Here is an example entry from the .clustertoc: CLUSTER=SUNWCacc NAME=System Accounting DESC=System accounting utilities VENDOR=Sun Microsystems, Inc. VERSION=7.2 SUNW_CSRMEMBER=SUNWaccr SUNW_CSRMEMBER=SUNWaccu END What we see here is a cluster package, and then the packages that comprise that cluster package. If you were attempting to compare a machine with a Core installation and a Entire Distribution installation, you could see the differences between their configuration clusters up close. There are also man pages on packagetoc and clustertoc that you can look at for more detailed information. Section 2.512- Items in /var/sadm- /var/sadm/system/admin/.platform ------------------------------------------------------------------- PLATFORM_GROUP=sun4u INST_ARCH=sparc PLATFORM_NAME=SUNW,SPARCstation-fusion PLATFORM_ID=SUNW,SPARCstation-fusion IN_PLATFORM_GROUP=sun4u PLATFORM_NAME=FJSV,GP PLATFORM_ID=FJSV,GP IN_PLATFORM_GROUP=sun4u This file is included in cd images of Solaris CD's that need multiple platform support, and the file left over in /var/sadm/system/admin/ is a file that denotes what hardware platform was determined to be appropriate for your system The man page fully describes it: # man -s 4 platform File Formats platform(4) DESCRIPTION The Solaris 2.5 release includes the .platform directory, a new directory on the Solaris CD image. This directory con- tains files (created by SunSoft and Solaris OEMs) that define platform support. These files are generically referred to as platform definition files. They provide a means to map different platform types into a platform group. Platform definition files in the .platform directory are used by the installation software to ensure that software appropriate for the architecture of the system will be installed. SunSoft provides a platform definition file named .platform/Solaris . This file is the only one that can define platform groups to which other platform definition files can refer. For example, an OEM platform definition file can refer to any platform group specified in the Solaris platform definition file. Section 2.513-Items in /var/sadm- /var/sadm/system/admin/CLUSTER ---------------------------------------------------------------- -rw-r--r-- 1 root root 18 Apr 20 1999 /var/sadm/system/admin/CLUSTER A Configuration Cluster is the level you select during installation where you can select from the following options: 1) Entire Distribution Plus OEM (for Sun clones, for example) 2) Entire Distribution (puts in everything and the kitchen sink) 3) Developer (medium level targetted to the needs of developers) 4) End User (for networked workstations that will never be servers) 5) Core (the bare-bones OS without network connectivity) The way to tell what "Configuration Cluster" level the machine has been installed with is to look at the /var/sadm/system/admin/CLUSTER file. root@dartagnean:/ $ more /var/sadm/system/admin/CLUSTER CLUSTER=SUNWCXall Shows that I have installed the Entire Distribution Configuration Cluster on my system. Section 2.514- Items in /var/sadm- /var/sadm/system/admin/INST_RELEASE ---------------------------------------------------------------------- -rw-r--r-- 1 root root 29 Apr 20 1999 INST_RELEASE This file must exist and not be empty. It is symbolically linked from /var/sadm/softinfo/INST_RELEASE as explained earlier in the document. Section 2.515- Items in /var/sadm- /var/sadm/system/admin/services ------------------------------------------------------------------ -rw-r--r-- 1 root root 29 Apr 20 1999 Solaris_2.7 If you cat the file, it looks exactly like /var/sadm/softinfo/INST_RELEASE described above. I don't have information on what uses this file, yet I would not delete it on the basis of that. Section 2.516- The individual items in /var/sadm- /var/sadm/system/data ----------------------------------------------------------------------- drwxr-xr-x 2 root sys 512 Apr 20 1999 data My /var/sadm/system/data directory contains no files. However, it is possible that it can contain data: Checking /var/sadm/README you will find that /var/sadm/system/data contains the upgrade_cleanup and upgrade_space_report when and if they are generated, and following definition: system/data Data files generated by system installation and configuration utilities containing system-relevant configuration data. Section 2.517-The individual items in /var/sadm- /var/sadm/system/logs ---------------------------------------------------------------------- drwxr-xr-x 2 root sys 512 Apr 20 1999 logs On my system, this directory contains: lrwxrwxrwx 1 root root 20 Dec 1 16:44 begin.log -> begin.log_1999_04_20 -rw-r--r-- 1 root root 90 Apr 20 1999 begin.log_1999_04_20 lrwxrwxrwx 1 root root 21 Dec 1 16:44 finish.log -> finish.log_1999_04_20 -rw-r--r-- 1 root root 41 Dec 21 12:41 finish.log_1999_04_20 -rw-r--r-- 1 root root 31183 Apr 20 1999 install_log -rw-r--r-- 1 root root 11658 Apr 20 1999 sysidtool.log Over the life of a system, especially if it has been ugraded a few times, multiple begin and finish scripts may be run. The begin.log and finish.log are always linked to the latest log file generated, but all logs are maintained. The same thing happens in /var/sadm/install_data with the Maintenance_Upgrade_log if one exists... Section 2.518-The individual items in /var/sadm- /var/sadm/system/logs ---------------------------------------------------------------------- -rw-r--r-- 1 root root 31183 Apr 20 1999 install_log The install_log is from your OS installation, the options used for newfs'ing your filesystems initally, packages installed, etc. This is useful if you want to know what the specific original configuration of the system was when it was first installed. Section 2.519-The individual items in /var/sadm- /var/sadm/system/logs ---------------------------------------------------------------------- -rw-r--r-- 1 root root 11658 Apr 20 1999 sysidtool.log The sysidtool.log was populated during the System Identification step during installation. The files may have different names than these, depending on what was installed and when. Section 2.6- Summary of Section 2.x ----------------------------------- This ends our detailed explosion of the /var/sadm directory, hopefully this has been useful and informative. See Section 3.0 for further discussion of higher-level System Administrator concepts with /var/sadm. Section 3.0- The Role of the /var/sadm Directory ------------------------------------------------ Quite often, those unfamiliar with the role of /var/sadm will look in there and remove the files, since their /var has possibly become full. With the complexity of the repartitioning process, removing these files looks like a viable option. In fact, the opposite is true. The removal of these files may not impact the system for quite some time, but as soon as a patch or package needs to be applied or removed from the system you will run into a variety of problems. Since /var/sadm contains many log files, the /var/sadm directory must be considered as a single entity, a snapshot in time of the system as it exists. Backups of this directory will also be snapshots. This becomes important in a number of different ways, as you will see. Section 3.1- To Patch or not to Patch ------------------------------------- Because of the long uptime capabilities of Solaris machines, I note two separate philosophies had by sys admins. One is the "If it isn't broke, don't fix it" model and the other is "stay up to the minute on all patches" model. One could effectively argue for a "catch-as-catch-can" model, but that would be a hybrid, as are most philosophies about technical things. Really, it's a difference of reactive vs. proactive, and it is generally Sun's position to take the proactive approach. This is why we always recommend you put on the recommended and Y2K patch clusters, etc. on a regular basis. The speed of change of hardware and software that runs within and on Solaris is updated at a very fast pace. Many sys admins will want to "freeze" their systems in a certain way, but inevitably, some moment will come when major upgrades or changes will need to take place on the system, and the /var/sadm directory will play a prominent role. It is at these times that you will want to have used the extra disk space that may not have seemed important a long time ago, or have not deleted or manually added. Often, when taking calls in the Solutions Center, we see that the actions of a sys admin years ago (who may no longer be with the company, or might be in another department now) affect the sys admin who is now taking care of the system. If you are the current System Administrator, you will want to consider the needs of the System Administrator that may be caring for this machine in the future. Section 3.2- Log File Synchronization ------------------------------------- In understanding package and patch installation, you must consider two separate variables. First is the copying of new files or binaries to the system which contain bug fixes. The second is the accurate recording of the information regarding what changes have been made to the system. This is the role of the /var/sadm directory, among other things. This conceptually makes the maintenance of accurate information in /var/sadm of utmost importance to the system administrator. In System Administration classes taught at Sun, they emphasize the point that you should always use the standard utilities when modifying the system, they may not explain fully why. Section 3.2.1- De-synchronization Examples ------------------------------------------ There are many examples of this de-synchronization phenomenon many of which might not seem like a problem, and some that are fairly obvious: 1) Modifying your /etc/passwd file. This common act could not be considered by anyone as unusual, but since it involves manual modifications of the file, then your contents file will not reflect these changes, nor does it need to. Keep in mind that any and all manual changes to the system are not reflected in the information in /var/sadm/install/contents. 2) Restoring /usr (or another "root" filesystem) from backup. In this case, few of the sizes or permissions of the files will be the same as the ones found in the contents file, unless those files have not changed since the last backup happened. Since many times your permissions may change when restoring files from backup, you must rely on your contents file for a direction to head in. The condition of #1 will also be true here with respect to modified files. 3) Restoring /var from backup. In this case, binaries and files that have been modified by the package and patch utilities since your last backup would not be reflected in the log files in /var as a whole, as well as the contents file. 4) Someone removed SUNWcsu from the system, and now my linker patch won't install. AND the files show up in /var/sadm/install/contents as being there. You can do a manual pkgadd of this pkg, but if that doesn't work, a reinstall of the OS is needed to get this back, because it's a core package. NOTE: I have highlighted these examples, but there are many other cases of ways people commonly de-synchronize their systems without knowing it. Every System Administrator should be aware of the implications of their actions on a server with regard to that action's impact on change management. Section 3.3- General Filesystem Health -------------------------------------- Since the synchronization of information between the system and it's log files is of critical importance, checking the general /var filesystem health is very important also. You should never fsck a mounted filesystem, even if booted in single-user mode. It is necessary to boot from cdrom to check /var, or any of the other so-called 'root' filesystems such as / and /usr. This can be further complicated by the addition of disk management software such as Veritas Volume Manager, or Solstice Disksuite. Follow the procedures for those products to make the filesystems available for fsck'ing. Additionally, vxfs ships it's own version of fsck, so be aware of that as well. Section 3.4- /var: To Link or Not To Link ----------------------------------------- Many times, we see people who run out of space attempt to link /var or some directory within /var to get them more space. We do not recommend this, because there are problems you can run into as a result. There are many different workarounds I have heard regarding this, all because a repartition is undesirable for one reason or another. It is always your best solution to have a lot of space in /var for some generations of patches and packages. Many times, if /var is not a separate filesystem (and we generally recommend that it should be) you may have another unused partition or disk that you can off-load this onto. This process is relatively simple, and can be done with minimal downtime. Make the new partition, mount it, then use the dash (-) and p (for retain permissions) options with tar to copy the files to the new filesystem, boot from cdrom, modify the directory names, create a new /var mount point, change the vfstab to reflect the changes and reboot. There are many infodocs in SunSolve that can assist you with these various processes. Section 3.5- Managing pkginfo files ----------------------------------- The pkginfo files for packages in /var/sadm/package really contain the most information about the packages. I would be certain to keep a backup of these files handy, in case of a problem during changes you make to your system. It is for this reason that we always recommend you do a good backup before installing packages or patches. NOTE: Sun Microsystems does not support the manual editing of the pkginfo file, even in situations where it may be necessary to reinstall the OS. It can be possible, in certain circumstances, to restore the pkginfo file of a package from backup, but there are possible drawbacks to this depending on what changes have occurred with this package since your last backup. There are high level concerns about the structure of the pkginfo file that should be understood. I like to use the SUNWcsr package as an example. Most of the variables contained in this file are beyond the scope of this document. We will primarily discuss the PKG, ARCH, VERSION, and PATCH_INFO variables. This example is taken from my own patched 2.7 machine: I like to use SUNWcsr as the example, because it is often patched. root@dartagnean:/ $ more /var/sadm/pkg/SUNWcsr/pkginfo CLASSES=none initd devlinktab kclasses minorperm nametomajor drvalias renamenew preserve cronroot passwd tiservices inetdconf iuap etcsystem tzonecvt etcremote nsswitch netconfig deflogin defsu syslogconf ttysrch group inittab automaster etcrpc etcprofile mailxrc shadow vol dconf rmmconf sock2path renameold protocols etcskel BASEDIR=/ TZ=US/Pacific PATH=/sbin:/usr/sbin:/usr/bin:/usr/sadm/install/bin OAMBASE=/usr/sadm/sysadm PKG=SUNWcsr NAME=Core Solaris, (Root) ARCH=sparc VERSION=11.7.0,REV=1998.09.01.04.16 SUNW_PRODNAME=SunOS SUNW_PRODVERS=5.7/Generic SUNW_PKGTYPE=root MAXINST=1000 CATEGORY=system DESC=core software for a specific instruction-set architecture VENDOR=Sun Microsystems, Inc. HOTLINE=Please contact your local service provider EMAIL= SUNW_PKGVERS=1.0 PSTAMP=on99819980901043908 PATCHLIST=106832-02 106924-01 106944-01 106948-01 106936-01 106985-01 107121-01 107148-01 106963-01 106541-02 PATCH_INFO_106832-02=Installed: Tue Jan 26 13:49:24 PST 1999 From: fern Obsoletes: Requires: Incompatibles: PATCH_INFO_106924-01=Installed: Tue Jan 26 13:59:25 PST 1999 From: fern Obsoletes: Requires: Incompatibles: PATCH_INFO_106944-01=Installed: Tue Jan 26 14:02:05 PST 1999 From: fern Obsoletes: Requires: Incompatibles: PATCH_INFO_106948-01=Installed: Tue Jan 26 14:03:07 PST 1999 From: fern Obsoletes: Requires: Incompatibles: PATCH_INFO_106936-01=Installed: Tue Jan 26 14:12:42 PST 1999 From: fern Obsoletes: Requires: Incompatibles: PATCH_INFO_106985-01=Installed: Tue Jan 26 14:18:07 PST 1999 From: fern Obsoletes: Requires: Incompatibles: PATCH_INFO_107121-01=Installed: Tue Jan 26 14:50:19 PST 1999 From: fern Obsoletes: Requires: Incompatibles: PATCH_INFO_107148-01=Installed: Tue Jan 26 14:53:30 PST 1999 From: fern Obsoletes: Requires: Incompatibles: PATCH_INFO_106963-01=Installed: Tue Jan 26 15:13:43 PST 1999 From: fern Obsoletes: Requires: Incompatibles: PATCH_INFO_106541-02=Installed: Tue Jan 26 15:36:27 PST 1999 From: fern Obsoletes: 106976-01 Requires: Incompatibles: PKGINST=SUNWcsr PKGSAV=/var/sadm/pkg/SUNWcsr/save INSTDATE=Apr 20 1999 19:11 Section 3.6- Lab Exercise ------------------------- To illustrate some things about the way various utilities work, I suggest running several package related commands such as pkginfo -l, showrev -p, patchadd -p (functionally duplicates showrev -p), or pkgparam. 1. If I compare showrev -p, patchadd -p and PATCH_INFO lines, I will see a pattern (I have the output for space considerations): # cat /var/sadm/pkg/SUNWcsr/pkginfo | grep PATCH_INFO | sort PATCH_INFO_106541-02=Installed: PATCH_INFO_106832-02=Installed: PATCH_INFO_106924-01=Installed: PATCH_INFO_106936-01=Installed: PATCH_INFO_106944-01=Installed: PATCH_INFO_106948-01=Installed: PATCH_INFO_106963-01=Installed: PATCH_INFO_106985-01=Installed: PATCH_INFO_107121-01=Installed: PATCH_INFO_107148-01=Installed: root@dartagnean:/ $ showrev -p | grep SUNWcsr | sort Patch: 106541-02 Patch: 106832-02 Patch: 106924-01 Patch: 106936-01 Patch: 106944-01 Patch: 106948-01 Patch: 106963-01 Patch: 106985-01 Patch: 107121-01 Patch: 107148-01 root@dartagnean:/ $ patchadd -p | grep SUNWcsr | sort Patch: 106541-02 Patch: 106832-02 Patch: 106924-01 Patch: 106936-01 Patch: 106944-01 Patch: 106948-01 Patch: 106963-01 Patch: 106985-01 Patch: 107121-01 Patch: 107148-01 You will notice that the patch ID's are identical. I piped them all through "sort" simply so they would appear in the same order. They don't always appear in the same order, depending on the utility you use to get the information. This example shows what these utilities use to get their information. Without these lines in the pkginfo, showrev -p will not show any information. This illustrates the importance of using standard patching methodologies to install you patches, and maintain you /var/sadm directory. Section 3.7- Corrupt pkginfo files ---------------------------------- When there are problems installing patches or packages, it can be related to a corrupt log file, such as the pkginfo file. This is a relatively infrequent occurrence, but merits mention due to the fact that we will often recommend drastic actions to redress the problem. The Solaris patch and packaging facilities, as of 2.7, have no redundancy or error checking beyond the messages you may see when installing patches. Often, my /var/sadm/SUNWcsr/pkginfo file can become corrupted, I will need to restore that file from backup, or to reinstall the system from scratch, depending on the package. If this is a core package, reinstallation will be your only supported option, besides restoring from backup. For some other examples of troubleshooting this issue, take a look at Infodoc 15579. Patches and packages are closely related. For a fuller discussion of this, please read Infodoc 17973, or the larger Patch PSD 20597, which is really a compilation of patch related Infodocs. In the case of restoring the pkginfo file from backup, it will need to be considered as a small part of a larger /var/sadm entity. For example, if I restore my SUNWcsr/pkginfo file from a backup that I made previous to several patches being applied, I will be de-synchronizing the /var/sadm directory from the way the system actually exists. This is why hand-editing many of the files in /var/sadm is not supported. Section 3.71- Corrupt pkginfo file and compilers ------------------------------------------------ One common problem is found after upgrading the system, then leaving /var alone, but then restoring all the binaries or files in /usr , /opt or another filesystem from backup. In this case your /var/sadm has no knowledge whatsoever of the actual contents of the system itself, much less actual packages and patch levels, etc. In this case, it is better to restore only special or specific files from backup and reinstall your compilers. This is a common call in the group that supports our compiler products, because often a customer will upgrade the system before upgrading the compilers on that system. Section 3.8- The "Contents" File -------------------------------- The /var/sadm/install/contents file is a large file that is responsible for noting the changes made via the standard patching and packaging utilities. There are many uses for it, such as telling what files are associated with a particular package, like so: # cat /var/sadm/install/contents | grep SUNWcsu And you will get a long list of files, their permissions, file sizes, etc. Conversely, you can find what package a file belongs to like this: # cat /var/sadm/install/contents | grep /bin/csh /usr/bin/csh f none 0555 bin bin 151976 51498 907659673 SUNWcsu Where /bin/csh is the example file searched for. Of course, you can do general searches as well as this specific one by widening your search criteria. NOTE: You should never simply 'cat' your whole contents file. This file is enormous, and would simply scroll off your screen many times over. The /var/sadm/install/contents file becomes important when large errors are inadvertently made on the system. For example: #chmod -R / causes many problems that can be hard to recover from. Yet pkgchk -f can help tremendously in these types of worst-cases. Other problems that result from some recursive changes can be solved using this. However, having a healthy /var/sadm directory is critical when running this command, as that command relies upon the accuracy of the information there. Also know that many false positives come from running pkgchk -f. Section 3.9- The Definition of a Package ----------------------------------------- I am including this seemingly simple explanation for one reason- Packages appear to be different depending on how you look at them. Here's what I mean: 1) A package is a collection of files and directories. (From the Drake test for Solaris Certification) 2) From a patch's point of view, a package is a set of variables: PKG, ARCH, and VERSION, taken from the pkginfo files for the packages to be patched. If you see the error message: none (or some) of the packages to be patched are not on the system. This means something different to you than it does to the patch installations script. If the PKG, ARCH, and VERSION variables do not match exactly, it will not install simply because a patch defines a package differently. 3) Showrev -p: Showrev -p thinks a package is a collection of PATCH_INFO lines, found in the pkginfo file for packages. 4) pkginfo and pkginfo -l: pkginfo without arguments simply takes the information it needs from the pkginfo file and displays it. Pkginfo -l must do a little more work, and this is why it takes longer to execute. It checks the /var/sadm/install/contents file for the additional information such as STATUS and FILES, does the calculations, and displays the information. If you check the man page for pkginfo, you will notice many of the options deal directly with the variables you will see in the pkginfo file. 5) Human perspective: We think a package is simply a packagename, but it isn't always, as you can see from above. NOTE: One helpful item about using the pkginfo command: If you have many packages to install from CDROM, usually you must do a 'pkgadd -d /path/to/pkg pkgname' for each one. If you do a pkginfo -d once, this will set your default package installation directory from /var/spool/pkg to whatever long directory tree on the CDROM you need. In summary, we have fully discussed in general and in detail the many aspects of the /var/sadm directory as it relates to overall system maintenance, patching, packages, and change management in general. We have shown specific examples of various problems and uses of file one can encounter in attempting to keep one's system healthy over time. We have discussed high level aspects of the structure of the /var/sadm directory and what many of the files and directories there do. There are many aspects of System Administration that require many years of experience to understand. I have attempted to summarize the majority of the issues around the /var/sadm directory as I see it. There may be other things to know about the importance of the this directory, however I hope this has given you a place to begin to understand the structure and function of a large part of why /var exists. > Delivered-To: sun-managers-tiaowodt@sunmanagers.ececs.uc.edu > Delivered-To: sun-managers@sunmanagers.ececs.uc.edu > From: "Erin Jones" > To: "sun managers" > Subject: /var/sadm is too damn big > Date: Thu, 2 Nov 2000 11:39:02 -0500 > Content-Transfer-Encoding: 7bit > X-Priority: 3 > X-MSMail-Priority: Normal > X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 > > hey all, > > I went through the archives but i can't find the article pertaining to > this..... > my /var/sadm is too damn big. the tech that originally set up this machine > gave / the following geometry: > /dev/dsk/c0t0d0s0 144172 103938 25817 81% / > ok, I am getting annoyed that when any of my logs or mail spool gets any > size to them it fills up the filesystem. upon investigation I am seeing the > following culprits on my / partition > 5556 etc > 7364 kernel > 17251 opt > 5255 platform > 5019 sbin > 62926 var > > there are a few others but those compromise the bulk of it. the big one, > /var, is almost entirely /var/sadm: > 59607 /var/sadm > is there anyway I can prune this directory safely? > > thanx for your help > ej > > Erin Jones Director of Information Services > erin@internationalcomputing.com (212) 387 0310 x19 > Unix: Where /sbin/init is Job #1. > > > > > > S > U BEFORE POSTING please READ the FAQ located at > N ftp://ftp.cs.toronto.edu/pub/jdd/sun-managers/faq > . and the list POLICY statement located at > M ftp://ftp.cs.toronto.edu/pub/jdd/sun-managers/policy > A To submit questions/summaries to this list send your email message to: > N sun-managers@sunmanagers.ececs.uc.edu > A To unsubscribe from this list please send an email message to: > G majordomo@sunmanagers.ececs.uc.edu > E and in the BODY type: > R unsubscribe sun-managers > S Or > . unsubscribe sun-managers original@subscription.address > L To view an archive of this list please visit: > I http://www.latech.edu/sunman.html > S > T ================================================================================ ______ /_____/\ Ericka Fowler /_____\\ \ Technical Support Engineer /_____\ \\ / /_____/ \/ / / Sun Microsystems /_____/ / \//\ 500 Eldorado Blvd \_____\//\ / / MS: UBRM04-131 \_____/ / /\ / Broomfield, CO 80021 \_____/ \\ \ Phone 303-464-4666 \_____\ \\ Sun Internal Ext. 50666 \_____\/ email: ericka.fowler@central.sun.com ================================================================================