Mastering Unix Shell Scripting phần 6 pps

unknown to the system because DNS is not configured on this system. The mrranger node is powered down so it is known but not reachable. Notice the difference in the outputs for these two similar, but very different, situations. Please study the code related to both of these tests in the ping_nodes function. Other Options to Consider As always, we can improve on any shell script, and this one is no exception. I have listed some options that you may want to consider. $PINGLIST Variable Length Limit Problem In this scripting solution we gave the user the capability to comment out specific nodes in the $PINGFILE. We assigned the list of nodes, which is a list without the comments, to a variable. This is fine for a relatively short list of nodes, but a problem arises when the maximum variable length, which is usually 2048 characters, is exceeded. If you have a long list of nodes that you want to ping and you notice that the script never gets to the end of the ping list, you have a problem. Or if you see a funny-looking node name, which is probably a hostname that has been cut off by the variable limit and associated with a system error message, then you have a problem. To resolve this issue, define a new file to point to the PINGLIST variable, and then we will use the file to store the ping list data instead of a variable. To use PINGLIST as a file, add/ change the following lines: ADD THIS LINE: PINGLIST=/tmp/pinglist.out CHANGE THIS LINE: PINGLIST=$(cat $PINGFILE | grep -v ‘^#’) TO THIS LINE: cat $PINGFILE | grep -v ‘^#’ > $PINGLIST CHANGE THIS LINE: for HOSTPINGING in $(echo $PINGLIST) TO THIS LINE: for HOSTPINGING in $(cat $PINGLIST) 332 Chapter 12 Using the file to store the ping list data changes the limit to the maximum file size that the system supports or when the filesystem fills up, which should be plenty of space for anyone. This modified shell script is located on this book’s companion Web site. The script name is pingnodes_using_a_file.ksh. Ping the /etc/hosts File Instead of a List File This may be overkill for any large shop, but it is easy to modify the shell script to accomplish this task. You want to make the following change to the shell script after completing the tasks in the previous section “$PINGLIST Variable Length Limit Prob- lem” to the shell script shown in Listing 12.1. CHANGE THESE LINES: if [ -s $PINGFILE ] then PINGLIST=$(cat $PINGFILE | grep -v ‘^#’) TO THESE LINES: if [ -s /etc/hosts ] then # Ping all nodes in the /etc/hosts file cat /etc/hosts | sed /^#/d | sed /^$/d | grep -v 127.0.0.1 \ | awk ‘{print $2}’ > $PINGLIST In this changed code we cat the /etc/hosts file and pipe the output to a sed statement, sed /^#/d. This sed statement removes every line in the /etc/hosts file that begins with a pound sign (#). The output of this sed statement is then piped to another sed statement, sed /^$/d, which removes all of the blank lines in the /etc/hosts file (the blank lines are specified by the ^$). This sed output is sent to a grep command that removes the loopback address from the list. Finally, the remaining output is piped to an awk statement that extracts the hostname out of the second field. The resulting output is redirected to the $PINGLIST file. This modified shell script to ping the /etc/hosts file is included on the Web site that accompanies the book. The filename is pinghostsfile.ksh. Logging I have not added any logging capability to this shell script. Adding a log file, in addi- tion to user notification, can help you find trends of when nodes are unreachable. Adding a log file is not too difficult to do. The first step is to define a unique log file- name in the definitions section and assign the filename to a variable, maybe LOGFILE. In the script test for the existence of the file, using a test similar to the following state- ment will work. Automated Hosts Pinging with Notification 333 ADD THESE LINES: LOGPATH=/usr/local/log LOGFILE=${LOGPATH}/pingnodes.log if [ ! -s $LOGFILE ] then if [ ! -d $LOGPATH ] then echo “\nCreating directory ==> $LOGPATH\c” mkdir /usr/local/log if (( $? != 0 )) then echo “\nUnable to create the $LOGPATH directory EXITING \n” exit 1 fi chown $USER /usr/local/log chmod 755 $LOGPATH echo fi echo “\nCreating Logfile ==> $LOGFILE\c” cp /dev/null > $LOGFILE chown $USER $LOGFILE echo fi After adding these lines of code, use the tee -a $LOGFILE command in a pipe to both display the text on the screen and log the data in the $LOGFILE. Notification of “Unknown Host” You may want to add notification, and maybe logging too, for nodes that are not known to the system. This usually occurs when the machine cannot resolve the node name into an IP address. This can be caused by the node not being listed in the /etc/hosts file or failure of the DNS lookup. Check both conditions when you get the Unknown host message. Currently, this shell script only echoes this information to the screen. You may want to add this message to the notification. Notification Method In this shell script we use email notification. I like email notification, but if you have a network failure this is not going to help you. To get around the network down problem with email, you may want to set up a modem, for dial-out only, to dial your alpha- numeric pager number and leave you a message. At least you will always get the message. I have had times, though, when I received the message two hours later due to a message overflow to the modem. 334 Chapter 12 You may just want to change the notification to another method, such as SNMP traps. If you execute this shell script from an enterprise management tool, then the response required back to the program is usually an SNMP trap. Refer to the docu- mentation of the program you are using for details. Automated Execution Using a Cron Table Entry I know you do not want to execute this shell script from the command line every 15 minutes yourself! I use a root cron table entry to execute this shell script every 15 min- utes, 24 hours a day, Monday through Saturday, and 8:00 A.M. to midnight on Sunday; of course, this requires two cron table entries. Because weekly backups and reboots happen early Sunday morning, I do not want to be awakened every Sunday morning when a machine reboots, so I have a special cron entry for Sunday. Both root cron table entries shown execute this script every 15 minutes. 5,20,35,50 * * * 1-6 /usr/local/bin/pingnodes.ksh >/dev/null 2>&1 5,20,35,50 8-23 * * 0 /usr/local/bin/pingnodes.ksh </dev/null 2>&1 The first entry executes the pingnodes.ksh shell script at 5, 20, 35, and 50 minutes of every hour from Monday through Saturday. The second entry executes the ping-nodes.ksh shell script at 5, 20, 35, and 50 minutes from 8:00 A.M. until 11:59 P.M., with the last ping test running at 11:50 P.M. Sunday night. Summary In this chapter we took a different approach than that of some other shell scripts in this book. Instead of creating a different function for each operating system, we created a single shell script and then used a separate function to execute the correct command syntax for the specific operating system. The uname command is a very useful tool for shell scripting solutions for various Unix flavors in a single shell script. I hope you enjoyed this chapter. I think we covered some unique ways to solve the scripting problems that arise when programming for multiple Unix flavors in the same script. In the next chapter we will dive into the task of taking a system snapshot. The idea is to get a point-in-time system configuration for later comparison if a system problem has you puzzled. See you in the next chapter! Automated Hosts Pinging with Notification 335 337 Have you ever rebooted a system and it came up in an unusual state? Any time you reboot a system you run a risk that the system will not come back up properly. When problems arise it is nice to have before and after pictures of the state of the machine. In this chapter we are going to look at some options for shell scripts that execute a series of commands to take a snapshot of the state of the machine. Some of the things to con- sider for this system snapshot include filesystems that are mounted, NFS mounts, processes that are running, network statistics and configuration, and a list of defined system resources, just to name a few. This is different from gathering a snapshot of performance statistics, which is gathered over a period of time. All we are looking for is system configuration data and the system’s state at a point in time, specifically before the system is rebooted or when it is running in a normal state with all of the applications running properly. With this information captured before a system reboot, you have a better chance of fixing a reboot problem quickly and reducing down time. I like to store snapshot infor- mation in a directory called /usr/local/reboot with the command names used for filenames. For this shell script all of the system information is stored in a single file with a section header added for each command output. Overall, this is not a difficult shell script to write, but gathering the list of commands that you want to run can some- times be a challenge. For example, if you want to gather an application’s configuration you need to find the commands that will produce the desired output. I always prefer having too much information, rather than not enough information, to troubleshoot a problem. Taking a System Snapshot CHAPTER 13 In this chapter I have put together a list of commands and created a bunch of func- tions to execute in the shell script. The commands selected are the most critical for trou- bleshooting an AIX machine; however, you will need to tailor this set of commands to suit your particular needs, operating system, and environment. Every shop is different, but they are all the same in some sense, especially when it comes to troubleshooting a problem. Let’s look at some commands and the syntax that is required. Syntax As always, we need the commands and the proper syntax for these commands before we can write a shell script. The commands presented in this section are just a sample of the information that you can gather from the system. This set of commands is for an AIX system, but most apply to other Unix flavors with modified syntax. The list of AIX commands is shown in Listing 13.1. # Hostname of the machine hostname OR uname -n # Unix flavor uname -s # AIX OS version oslevel # AIX maintenance level patch set instfix -i | grep AIX_ML OR oslevel -r # Time zone for this system cat /etc/environment | grep TZ | awk -F’=’ ‘{print $2}’ # Real memory in the system echo “$(bootinfo -r)KB” OR lsattr -El -a realmem | awk ‘{print $2}’ # Machine type/architecture uname -M OR - Depending on the architecture uname -p # List of defined system devices lsdev -C # Long directory listing of /dev ls -l /dev # List of all defined disks lsdev -Cc disk # List of all defined pdisks for SSA disks lsdev -Cc pdisk # List of defined tape drives Listing 13.1 System snapshot commands for AIX. 338 Chapter 13 lsdev -Cc tape # List of defined CD-ROMs lsdev -Cc cdrom # List of all defined adapters lsdev -Cc adapter # List of network routes netstat -rn # Network adapter statistics netstat -i # Filesystem Statistics df -k AND mount # List of defined Volume Groups lsvg | sort -r # List of varied-on Volume Groups lsvg -o | sort -r # List of Logical Volumes in each Volume Group for VG in $(lsvg -o | sort -r) do lsvg -l $VG done # Paging space definitions and usage lsps -a AND lsps -s # List of all hdisks in the system lspv # Disk drives listed by Volume Group assignment for VG in $(lsvg -o | sort -r) do lsvg -p $VG done # List the HACMP configuration, if installed if [ -x /usr/sbin/cluster/utilities/cllsif ] then /usr/sbin/cluster/utilities/cllsif echo “\n” fi if [ -x /usr/sbin/cluster/utilities/clshowres ] then /usr/sbin/cluster/utilities/clshowres fi # List of all defined printers lpstat -W | tail +3 AND cat /etc/qconfig Listing 13.1 System snapshot commands for AIX. (continues) Taking a System Snapshot 339 # List of active processes ps -ef # Show SNA configuration, if installed sna -d s if (($? != 0)) then lssrc -s sna -l fi # List of udp and x25 processes, if any ps -ef | egrep ‘udp|x25’ | grep -v grep # Short listing of the system configuration lscfg # Long listing of the system configuration lscfg -vp # List of all system installed filesets lslpp -L # List of broken or inconsistant filesets lppchk -v 2>&1 # List of the last 100 users to log in to the system last | tail -100 Listing 13.1 System snapshot commands for AIX. (continued) As you can see in Listing 13.1, we can add anything that you want to the snapshot shell script to get as much detail as needed to troubleshoot a problem. Every environ- ment is different, so this list of commands should be modified, or added to, to suit the needs of your shop. Additional tests include a list of databases that are running, appli- cation configurations, specific application processes that are critical, and a ping list of machines that are critical to the operation of any applications. You can add anything that you want or need here. Always try to gather more information than you think you may need to troubleshoot a problem. Using this snapshot technique allows us to go back and look at what the system looked like under normal conditions and load. By looking at the snapshot script out- put file, the problem usually stands out when comparing it to the currently running system that has a problem. Creating the Shell Script For this shell script we are going to take the commands shown in Listing 13.1 and cre- ate a function for each one. Using functions greatly simplifies both creating and modi- fying the entire shell script. When we want to add a new test, or configuration output, we just create a new function and add the function-name in the main body of the shell script exactly where we want it to run. In this shell script all of the function definitions use the C-like function statement, as shown here. 340 Chapter 13 get_last_logins () { Commands to execute } A lot of script programmers like this function definition technique. I prefer defining a function using the function statement method, as shown here. function get_last_logins { Commands to execute } This last method of defining a function is more intuitive to understand for the peo- ple who will follow in your footsteps and modify this shell script. I hope you noticed the use of the word will in the last sentence. No matter what the shell script does, there is always someone who will come along, after you have moved on to bigger and better things, who will modify the shell script. It is usually not because there is a problem with the script coding, but more likely a need for added functionality. For the people who follow me, I like to make sure that the shell script is easy to follow and under- stand. Use your own judgment and preference when defining functions in a shell script; just be consistent. Because we have all of the commands listed in Listing 13.1 let’s look at the entire shell script in Listing 13.2 and see how we created all of these functions. #!/bin/ksh # # SCRIPT: AIXsysconfig.ksh # AUTHOR: Randy Michael # REV: 2.1.P # DATE: 06/14/2002 # # PLATFORM: AIX only # # PURPOSE: Take a snapshot of the system for later comparision in the # event of system problems. All data is stored in # /usr/local/reboot in the file defined to the $SYSINFO_FILE # variable below. # # # REV LIST: # 7/11/2002: Changed this script to use a single output file # that receives data from a series of commands # within a bunch of functions. # # Listing 13.2 AIXsysconfig.ksh shell script listing. (continues) Taking a System Snapshot 341 [...]... 19, 19, 19, 19, 10, 10, 13, 10, 11, 13, 13, 7, 7, 7, 7, 5, 5, 1, 18, 18, 26, 26, 2, 0 1 2 3 4 5 6 7 0 10 14 10 0 15 30 0 1 3 2 0 1 0 0 1 0 1 3 Jun Mar Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jul Jun Jul Jul Jul 23 29 26 26 26 26 26 26 26 29 26 29 26 26 26 26 26 26 26 26 26 24 24 23 26 26 26 15:23 13:49 2001 2001 2001 2001 2001 2001 2001 2001 2001 2001 2001 2001... UGc 0 10.10/ 16 10.10.10.1 U 37 127/8 127.0.0.1 U 5 Use 0 135807 264 Route Tree for Protocol Family 24 (Internet v6): Listing 13.3 AIXsysconfig.ksh shell script in action (continues) If PMTU Exp en0 en0 lo0 - - 355 3 56 Chapter 13 ::1 ::1 UH 0 0 lo0 168 96 - ################################################# Network Interface Statistics Name en0 en0 lo0 lo0 lo0 Mtu 1500 1500 168 96 168 96 168 96 Network link#2... 10924 67 % 61 680 96% 10 568 81% 70184 35% 3892 5% 164 60 98% 252 96 76% 3504 56 15% node mounted mounted over options -/dev/hd4 / jfs /dev/hd2 /usr jfs /dev/hd9var /var jfs /dev/hd3 /tmp jfs /dev/hd1 /home jfs /proc /proc procfs /dev/hd10opt /opt jfs /dev/scripts_lv /scripts jfs /dev/lv_temp /tmpfs jfs Iused %Iused Mounted on 1854 12% / 40941 12% /usr 67 3 6% /var 223 1% /tmp 55 6% /home... Ipkts Ierrs 2 .60 .8c.2d.75.b1 112330 yogi 112330 28302 loopback 28302 28302 0 0 0 0 0 Opkts Oerrs 10 869 7 10 869 7 28304 28304 28304 Coll 0 0 0 0 0 0 0 0 0 0 ################################################# Filesystem Statistics Filesystem 1024-blocks /dev/hd4 32 768 /dev/hd2 1449984 /dev/hd9var 53248 /dev/hd3 1 064 96 /dev/hd1 40 96 /proc /dev/hd10opt 65 5 360 /dev/scripts_lv 102400 /dev/lv_temp 40 960 0 Free %Used... root 267 2 1 0 Jul 23 root 3140 1 0 Jul 23 root 364 2 464 4 0 17:11:20 root 3882 1950 0 Jul 23 connections root 4 168 1950 0 Jul 23 root 4388 1950 0 Jul 23 root 464 4 1950 0 Jul 23 nobody 49 06 5418 0 Jul 23 daemon 8798 1950 0 Jul 23 root 9034 1950 0 Jul 23 root 92 96 1950 0 Jul 23 root 9554 1950 0 Jul 23 root 9814 1950 0 Jul 23 root 103 36 1 0 Jul 23 root 10588 1950 0 Jul 23 root 10842 1 0 Jul 23 root 11 360 ... Space Information Page Space hd6 Physical Volume hdisk0 Total Paging Space 336MB Volume Group Size rootvg 336MB %Used Active Auto 10 yes yes Percent Used 10% ################################################# Hard Disks Defined hdisk0 hdisk1 0000 367 7cf 068 b62 00012 560 8a48c132 ################################################# Volume Group Hard Drives Listing 13.3 AIXsysconfig.ksh shell script in action (continues)... 40941 12% /usr 67 3 6% /var 223 1% /tmp 55 6% /home - /proc 162 60 10% /opt 887 4% /scripts 26 1% /tmpfs vfs date -Jul 23 18: 56 Jul 23 18: 56 Jul 23 18: 56 Jul 23 18: 56 Jul 23 18:57 Jul 23 18:57 Jul 23 18:57 Jul 23 18:57 Jul 23 18:57 ################################################# Defined Volume Groups rootvg Listing 13.3 AIXsysconfig.ksh shell script in action (continued) rw,log=/dev/hd8 rw,log=/dev/hd8... 10 09 09 27 26 26 26 13:28 12:17 17:35 17:35 17:11 17:11 17:09 21:53 00:25 23:41 19:38 16: 07 20:55 20:55 20:55 - 13:29 (00:00) - 12:18 (00:00) still logged in - 17:35 (00:00) still logged in still logged in - 17:11 (00:01) - 21:53 (00:00) - 23:41 (00:00) - 20: 56 - 20: 56 - 20: 56 (00:00) (00:00) (00:00) Jul 31 18:20 This report is saved in: /usr/local/reboot/sys_snapshot.yogi.072502_09 465 8 Listing 13.3... Drive Manufacturer IBMRISC Machine Type and Model 066 4M1H Part Number .86F0101 ROS Level and ID 5 5A Serial Number .00221833 EC Level 8951 86 FRU Number 86F0118 Device Specific.(Z0) 000002029F00001E Device Specific.(Z1) 75G 364 4 Device Specific.(Z2) 0983 Device Specific.(Z3) 95123 Device Specific.(Z4) 0002 Device Specific.(Z5) 22 Device Specific.(Z6) 895172 rmt0 00-00-0S-5,0 5.0 GB 8mm Tape Drive... ################################################# Logical Volume Information by Volume Group rootvg: LV NAME POINT hd5 hd6 hd8 hd4 hd2 hd9var hd3 hd1 hd10opt scripts_lv lv_temp TYPE LPs PPs PVs LV STATE MOUNT boot paging jfslog jfs jfs jfs jfs jfs jfs jfs jfs 2 84 1 8 354 13 26 1 160 25 100 2 84 1 8 354 13 26 1 160 25 100 1 1 1 1 2 2 1 1 2 1 1 closed/syncd open/syncd open/syncd open/syncd open/syncd open/syncd open/syncd . useful tool for shell scripting solutions for various Unix flavors in a single shell script. I hope you enjoyed this chapter. I think we covered some unique ways to solve the scripting problems. overkill for any large shop, but it is easy to modify the shell script to accomplish this task. You want to make the following change to the shell script after completing the tasks in the previous. modified shell script to ping the /etc/hosts file is included on the Web site that accompanies the book. The filename is pinghostsfile.ksh. Logging I have not added any logging capability to this shell