4 – Managing data growth 105 Figure 4.13: Free space in data file after truncate table statement. Figure 4.14: Minimal log file growing with data load. 4 – Managing data growth 106 This time there were no Auto Grow events for the data file, and only 20 for the log file. The net effect is that the average time to load 50,000 records is reduced from 2.5 seconds to 1.3 seconds. A time saving of just over 1 second per load may not seem significant at first, but consider the case where the same process normally takes an hour. Just by ensuring log and data growth was controlled, you have cut the process down to under 30 minutes, and saved a lot of I/O processing at the same time. Handling space problems I've shown that having incorrectly sized data and log files and inappropriate Auto Grow properties, both inherited from the model database, can significantly increase the I/O load during bulk insert processes. I've also demonstrated the dangers of unabated log file growth, unless you change the default recovery model or perform regular log backups. Even for a database that is subject to as few as 50K transactions per day, I have seen the database log file grow to over 220G over the course of a few months, because no log backups have been taken. The reason for this is that, generally, there are databases with low level SLAs, meaning that a full nightly backup is all that is required. As I've stressed previously, handling these space issues is mainly about planning. The DBA needs to: • Correctly size the files – if you know that the database you are managing can expect a 2 Gig growth per month, size the data file(s) at 4G initially, not the 3 MB size that will be the default from the Model database. • Set correct auto grow properties – while 10% growth for data and log files may be sufficient for low utilization databases, typically I set at least 500 MB for the auto growth settings for the data and log files. Unless I expect there to be unusually high data growth, 500 MB represents a good average growth rate, and keeps space utilization at a manageable level but allows for growth over time without heavy I/O impact. • Make sure only those databases that need FULL recovery are using it – you will determine this from the business and will be part of the SLA for the application and database. If point-in-time recovery is required, make sure you have regular log backups taken of the databases in Full recovery mode. • Switch to bulk-logged mode for bulk insert operations (see Chapter 3) – bulk loading is a common practice and, if done correctly, will incur minimal log growth, while reaping the performance benefits bulk loading brings. However, make sure you understand the consequences of 4 – Managing data growth 107 changing the recovery models while bulk loading data. For instance, you will be unable to perform a point-in-time recovery for the bulk transactions. If you fail to plan properly, or are simply subject to unexpected and unpredictable file growth, what does this mean for the DBA? Suppose a database has been inadvertently set to Full recovery with no log backups. The log file has gown massively in size and, ultimately, the drive will run out of space. If you are lucky enough, as I am to have an alerting system (see Chapter 6), the problem will be caught before that happens and I will get an alert, predictably at 2:30 AM when I have just gone to bed after resolving a different issue. What I do in such situations, after cursing myself or other innocent people on my team for not catching this sooner, is to issue the following simple statement: BACKUP LOG <databasename> WITH Truncate_Only This statement has the net effect of removing all of the inactive transactions from the log file that would have otherwise been removed with a standard log backup. Next, I shrink the log file via the GUI (or, if I am not too tired, with code) and then change the recovery model to Simple and go back to bed. Doing this will generally reclaim the necessary disk space to clear all alerts, and ensure that no further log growth will ensue. You can use DBCC to physically shrink a data or log file, as follows: DBCC SHRINKFILE (filename, target_size) Many of the situations that require you to shrink a log file can be avoided simply by planning accordingly and being diligent and fastidious in your installation process (see Chapter 1), in particular by making sure the model database is always set to Simple and not Full recovery mode. It only needs to happen to you once or twice. I quote George W. Bush, "Fool me once … shame on … shame on you … Fool me can't get fooled again." Take that, SQL Server Model Database. Indexes and large row counts All DBAs know that indexes are necessary for Olympic style query performance. We also know that they come at a price; and that price is paid in the currency of space and maintenance time. As much as I desperately yearn for the developer's queries to work efficiently, the DBA is still the gatekeeper of the data and feels 4 – Managing data growth 108 obliged to point out the specifics of why queries will and will not benefit from the indexes that the developers suggest. Often, these index recommendations come from sources like the Database Tuning Advisor (DTA), so we DBAs often eschew them in favor of our own. I do not mean to seem high-minded on this point, my DBA nose pointed straight up in the air. However, rightly or wrongly, DBAs want to control the types of objects (triggers, temp tables, linked servers, and so on) that are added to their servers, and indexes are just another type of object that DBAs must understand, manage and maintain. I am all in favor of a clustered index on almost every table, backed by a healthy volume of covering non-clustered indexes, but I also know from experience that indexes, for all their good, will only be utilized when proper code is executed that will take advantage of them. It is always worthwhile to explain to SQL developers why their queries do not perform as they expect, with their proposed indexes. In this section, I am going to add indexes to the Book_List table in order to find out: • How much extra space is required in order to add a clustered index to a table containing 2.9 million rows. • Whether this space consumption is justified, by examining the proposed queries that intend to take advantage of the indexes. Let's first get a "before" glimpse of space utilization in our Book_List table, using the sp_spaceused stored procedure, as shown in Figure 4.15. Notice the 8K of index size. Figure 4.15: index_size of Book_List table. Before I can add a clustered index, I need to add an identity column, called Read_ID, on which to place the clustered index. Adding the identity column is, in itself, an expensive task for 2.9 million records. The code is as follows: ALTER TABLE Book_list ADD Read_ID INT IDENTITY We can now create the clustered index on this Read_ID column, as shown in Listing 4.1. 4 – Managing data growth 109 USE [All_Books_Ever_Read] GO CREATE UNIQUE CLUSTERED INDEX [Read_ID] ON [dbo].[Book_List] ( [Read_Date] ASC ) WITH ( STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] GO Listing 4.1: Creating a clustered index on the Read_ID column of the Book_List table. As you can see from Figure 4.16, building a clustered index on almost 3 million records takes some time and processing power. Figure 4.16: It takes over 12 minutes to build the clustered index. . "Fool me once … shame on … shame on you … Fool me can't get fooled again." Take that, SQL Server Model Database. Indexes and large row counts All DBAs know that indexes are necessary. DBAs want to control the types of objects (triggers, temp tables, linked servers, and so on) that are added to their servers, and indexes are just another type of object that DBAs must understand,. proper code is executed that will take advantage of them. It is always worthwhile to explain to SQL developers why their queries do not perform as they expect, with their proposed indexes. In