Removing Large Files and Log Rolling

261 ■ ■ ■ CHAPTER 40 Removing Large Files and Log Rolling T his chapter gives a few tips relating to moving or removing files that are consuming large amounts of disk space. In some cases you may have a file system that’s filling up because a process adds copious amounts of entries to a log file. When you run into a large log file that is found to be the primary cause of a full file system, your first inclination may be to remove the file to reclaim space. However, this may not work as you might intend. Before I go into why, let me first talk a little about files and their structure. The file name is the visible representation of an inode that allows you to access the data in the file. The inode contains all the important information about a file, such as its ownership, permis- sions, modification times, and other items of interest. An inode also holds information about the location of the data on the hard disk. A file name is a user-friendly way of access- ing an inode, which is represented by a number. You can determine the inode number of any file by running the command ls -i filename. Here is an example of a full directory: $ ls -i 474322 37l5152.pdf 214111 rsync.tar.gz 474633 770tref.pdf 215939 yum-2.0.5-1.fd.fr.noarch.rpm 215944 openbox-3.1-1.i386.rpm Note that the inode number precedes each of the five files listed here. As just men- tioned, part of the data contained in an inode is a pointer to the data on the physical disk. If the file has been opened by a process for writing, the process writes to this location on the hard disk. The potential problem with removing a large file to clean up disk space is that if a user or an administrator removes the file, the inode may still be kept open by a process that is writing the data, and the disk space will not be returned to the system. The operating sys- tem won’t realize it should release the disk space for reuse until the process closes the file. At that time the disk space will definitely be reclaimed. One way of finding out if a process is keeping a file open is to use the fuser command, which will display a list of process IDs that are accessing a given file. You also can use the lsof command to find this information. lsof is designed to list open files and the 262 CHAPTER 40 ■ REMOVING LARGE FILES AND LOG ROLLING processes that are accessing them. Once you know the processes that are using the file, you can stop them before deleting the file. However, you may not be able (or want) to do this. The other problem with needing to stop a process that is holding a file open is that doing so may be against existing site policies. The process or application that is holding the file open might be production-critical and impact business needs if it is halted. Another way to clean up the space used by a file is to zero out the file by redirecting /dev/null into the file. This trims the file down to zero bytes while leaving the file itself in place. The file remains open and accessible to any process that might be using it. How- ever, the operating system will release the disk space in a timely fashion. Keep in mind that some processes may keep a file open for writing for a very long time. Here is a sample of a directory listing, including an offending log file that is consuming large amounts of disk space. We have a choice of several possible commands that will zero out the file. $ ls -l total 113172 -rw-rw-r-- 1 rbpeters rbpeters 1057862 Jun 7 18:21 37l5152.log -rw-rw-r-- 1 rbpeters rbpeters 449184 Jun 7 18:21 770tref.log -rw-rw-r-- 1 rbpeters rbpeters 1104249096 Jun 7 21:22 really_big.log The following command is one, cp /dev/null really_big.log is another, and echo > really_big.log is yet another. All of these commands overwrite a large file with nothing. $ >really_big.log This is the resulting directory listing after the file has been zeroed out: $ ls -l total 1484 -rw-rw-r-- 1 rbpeters rbpeters 1057862 Jun 7 18:21 37l5152.log -rw-rw-r-- 1 rbpeters rbpeters 449184 Jun 7 18:21 770tref.log -rw-rw-r-- 1 rbpeters rbpeters 0 Jun 7 21:25 really_big.log This procedure can be used to rotate log files. For example, you may have an applica- tion for which you would like to keep one week’s worth of log information. You would then rotate it to an older version to keep up to one month. After a month, you would delete the file. One solution includes the logrotate command, which is part of multiple Linux distributions. The following is a simple script that copies the existing log file to a backup with a ver- sion number and then zeroes out the file. During the time the script is being run, any existing processes that are writing to the file can still access it. The code first defines the file in question and then checks whether it exists. If it does exist, execution continues. CHAPTER 40 ■ REMOVING LARGE FILES AND LOG ROLLING 263 #!/bin/sh LOGFILE=./my.log if [ ! -f $LOGFILE ] then echo "Nothing to do . exiting." else You’ve already seen the use of the redirect (>) to zero a file. The main addition to this idea here is the cp -p command. cp -p $LOGFILE ${LOGFILE}.0 > $LOGFILE fi This command preserves the original file’s ownership and time stamp for the new copy. Thus the new copy maintains the attributes of the original file and it appears as if the file had been moved. This is also a useful technique when modifying scripts, configu- ration files, or any other file for which preserving the original file’s attributes would be valuable. 265 ■ ■ ■ CHAPTER 41 Core Finder I n a production environment, it is a good idea to know whether your applications are dumping core often. It is also considered good housekeeping to know about your core files so your hard disk won’t fill up with unnecessary files. The small script in this chapter tracks down and cleans up core files. The script was intended to be run as an hourly cron job, although you could change the schedule to fit your needs. The job also has its priority lowered using the nice command so that it won’t interfere with the performance of regular processes on the machine. The notifications I’ve received from this script have characterized chronic issues with applications on more than one occasion. Without the script, I would have never seen the patterns. This script steps through each of the locally mounted file systems and finds all core files. It determines the applications that created the core files and moves the core files to a central location for later examination. The script also logs its actions and cleans up old saved files. First we have to set up some straightforward variables. #!/bin/sh HOWOLD=30 FSLIST="/ /boot" UNAME=`uname -n` DATE=`date +%m%d%I%M` DATADIR="/usr/local/data/cores" LOGFILE="$DATADIR/cor_report" The HOWOLD variable is used as the maximum age, in days, for saved core files. Any files older than this will be removed. The FSLIST variable contains the list of file systems that will be checked for core files. The list will vary from system to system. You could set FSLIST dynamically by using the df command to determine the locally mounted file systems. The command might look something like this: FSLIST=`df -l | grep '^/dev' | awk '{print $6}'`, which gathers the lines starting with /dev and then prints the field containing the file-system name. The other four variables contain the name of the system that the script is running on (UNAME), the current date (DATE), the directory to which the core files will be saved (DATADIR), and the name of the log file (LOGFILE). 266 CHAPTER 41 ■ CORE FINDER Now we determine whether the data directory exists. This is the directory where the core files will be saved. If it doesn’t exist, you have to create it. The -p option to mkdir adds any omitted parent directories to the path of the directory being created. if [ ! -d $DATADIR ] then mkdir -p $DATADIR fi Then you need to find all previously saved core files that don’t need to be kept around anymore, and remove them. find $DATADIR -name \*\.core\.\* -mtime +$HOWOLD -exec rm {} \; The following is the script’s main loop. It finds all the core files created since the last time the script was run. find $FSLIST -mount -type f -name core -print | while read file do if [ -s $file ] then coretype=`file $file | cut -d\' -f2` mv $file $DATADIR/$UNAME.core.$coretype.$DATE echo "$coretype $UNAME $DATADIR/$UNAME.core.$coretype.$DATE" | \ tee -a $LOGFILE | mailx -s "core_finder: $coretype core file \ found on $UNAME" sysadmins else rm $file fi done If a core file’s size is zero bytes, we remove it. If it is larger than zero bytes, we use the file command to determine the application that created the core file. Then the core file is saved in the archive directory and its name is changed to include the type of core and the date when the file was found. Then we add an entry to the log file noting the action that was taken, and send an e-mail notice to the administrators. . ■ CHAPTER 40 Removing Large Files and Log Rolling T his chapter gives a few tips relating to moving or removing files that are consuming large amounts. this information. lsof is designed to list open files and the 262 CHAPTER 40 ■ REMOVING LARGE FILES AND LOG ROLLING processes that are accessing them. Once