Managing File Counts

211 ■ ■ ■ CHAPTER 32 Managing File Counts S ystem administrators have to manage the computing resources available to their users. One resource consists of the directory queues for e-mail and print servers. These directo- ries contain the files waiting to be processed by the server. The number of files in these directories varies depending on the volume of work on the server. There are times when no one is using the service and the queue empties out. There are also times when the server is heavily loaded and the queue will be full. Usually, though, the number of files will stay within stipulated limits. Of course, from time to time you will have problems with a queue filling up faster than the server can process the requests. This might be due to a bubble of extremely heavy load or because of some type of problem that prevents the server from processing the requests. You will want to monitor the queue directories so you can take the proper measures when a problem arises. The script presented in this chapter is a simple monitor that counts the files in the specified directories. The script gives you the necessary core functionality; you will, how- ever, want to modify the directory locations and notification methods for your own site and needs. File-Count Monitor First we define the directories to be monitored. It would be a bit cleaner to place this definition in a separate file so you don’t have to modify the script to make configuration changes. However, to keep the script as simple as possible I have not done that. Each entry in the FILENUM variable is a colon-delimited set of fields. The fields specify the direc- tory name to monitor, the warning threshold, the maximum limit, and finally the e-mail address to which to send notifications. #!/bin/sh FILENUM="/var/spool/mqueue:50:100:root \ /var/spool/lp:50:100:root@yourdomain.net /:50:100:rbpeters" 212 CHAPTER 32 ■ MANAGING FILE COUNTS Now we start looping through each of the configured directories to check the file counts. First sed swaps colons with spaces in the configuration entry; then the set com- mand assigns, to each of the positional parameters $1 through $4, the corresponding element of the output, which consists of the items in the original colon-separated fields. Once the fields are assigned to the variables, the script determines the number of files in the directory. for monitor in $FILENUM do set -- `echo $monitor | sed -e 's/:/ /g'` file_count=`ls $1 | wc -l` The script completes by comparing the current file count to the three possible situations: Everything is normal. The file count is lower than the warning threshold. A warning state. The count is between the warning threshold and maximum limit. An error state. The count is above the maximum limit. In either the warning or error state, the script will send a notification via e-mail specify- ing the condition of the directory. There is one inherent problem with the e-mail notification type. If you’re monitoring a mail queue directory that is filling up, it is likely that the e-mail notification will not be sent. In this case, you may want to change your notification method to use something other than e-mail such as an alpha-numeric pager or phone. if [ $file_count -lt $2 ] then continue elif [ $file_count -ge $2 -a $file_count -lt $3 ] then echo "Warning: File count in $1 is $file_count, should be less \ than $2" | mail -s "Directory file count warning for $1" $4 else echo "Error: File count in $1 is $file_count, should be less \ than $2" | mail -s "Directory file count error for $1" $4 fi done To make this script more useful, you would need to call it with cron or possibly sur- round the code with an infinite loop that a perpetual process would run to continually monitor your machine. The script’s main function is carried out in the assignment of the file_count variable by performing an ls on the target directory and then piping the out- put to the wc utility. This is one of a number of ways to get queue-status information. Two CHAPTER 32 ■ MANAGING FILE COUNTS 213 other ways of determining the file count in a directory are to use the echo and find com- mands. Here are some possible solutions: file_count=`ls $1 | wc -l` file_count=`find $1 -type f | wc -l` file_count=`echo $1/* | wc -w` Note that the wc -l switch has been replaced with -w for the last solution. This is because the result of the shell’s expansion of the asterisk will appear in the form of a single line of filenames separated by spaces. Testing File-Count Methods The different methods of counting the files in a directory are not all created equal. Some older operating systems cannot handle large numbers of files when the ls or find com- mand is used. Sometimes the find-oriented solution will return incorrect results on directories containing between 12,000 and 15,000 files. Also, I have worked on systems with file counts of more than one million in a single directory; in these cases ls returns an “argument list too long” error. I have not yet found a situation in which the echo solu- tion fails. On my current Gentoo Linux system, both find and ls seem not to suffer from these earlier limitations, and I have used both to successfully count files in a single directory numbering well above 800,000 files. While testing, I noticed that there is a speed differ- ence between the three solutions. The find solution is by far the fastest. It took about .4 seconds with just over 300,000 files in a directory. The second fastest is echo *. It took about 1.5 seconds. This measure is most accurate if you work in the directory whose files are to be counted. If a full path is supplied with the asterisk (e.g., /some/full/path/*), the command takes longer because each element in the shell expansion contains the full path and all that extra text needs to be processed. The slowest in my tests was ls. It took about 4 seconds to complete. Both find and ls seemed to take similar amounts of time regardless of whether the full path is supplied. Realistically speaking, with a current operating system any of these solutions should work fine. It is not very likely that you will deal with this extreme number of files in a single directory, so the performance differences between the solutions will almost surely be insignificant. . phone. if [ $file_ count -lt $2 ] then continue elif [ $file_ count -ge $2 -a $file_ count -lt $3 ] then echo "Warning: File count in $1 is $file_ count,. #!/bin/sh FILENUM="/var/spool/mqueue:50:100:root /var/spool/lp:50:100:root@yourdomain.net /:50:100:rbpeters" 212 CHAPTER 32 ■ MANAGING FILE COUNTS