ptg 1474 CHAPTER 39 Monitoring SQL Server Performance Even though many of the performance counters and relevant information are now avail- able in the SQL Server Data Collector, as mentioned previously, the Data Collector does incur some overhead on SQL Server. Performance Monitor, on the other hand, incurs significantly less impact on SQL Server performance. In addition, the SQL Server Data Collector currently doesn’t have a built-in alerting capa- bility. As mentioned previously, you can set up performance counter alerts in Performance Monitor. In addition, Performance Monitor enables you to monitor more than what is provided with SQL Server Data Collector, including all aspects of the operating system as well as other applications. One other feature that’s very useful with Performance Monitor logs is the capability to import performance counter logs into SQL Server Profiler. NOTE For more information on importing and viewing performance counter logs in SQL Server Profiler, see Chapter 6. SQL Server Performance Counters For each SQL Server instance installed, Performance Monitor has a number of SQL Server–specific performance objects added to it, each with a number of associated coun- ters. Each SQL Server instance has its own set of monitoring objects because you certainly wouldn’t want to mix monitoring values across multiple instances. Performance counters for named instances use the naming convention MSSQL$ followed by the instance name (for example, MSSQL$SQL2008DEV:General Statistics). Performance counters for the default instance of SQL Server use the naming convention of SQLSERVER followed by the counter name (for example, SQLServer:General Statistics). Table 39.1 provides a list of the SQL Server performance counters available for SQL Server 2008 TABLE 39.1 SQL Server Performance Objects Performance Object Description SQLServer:Access Methods Information on searches and allocations of SQL Server database objects (for example, the number of index searches or number of pages allocated to indexes and data). SQLServer:Backup Device Information about backup devices, such as the throughput of the backup device. SQLServer:Buffer Manager Information about the memory buffers used by SQL Server. SQLServer:Buffer Partition Information about buffer free page accesses. ptg 1475 Performance Monitoring Tools 39 TABLE 39.1 SQL Server Performance Objects Performance Object Description SQLServer:CLR Information about common language runtime (CLR) SQLServer:Cursor Manager by Type SQLServer:Cursor Manager Total Information about cursors. SQLServer:Database Mirroring Information about database mirroring. SQLServer:Databases Database-specific information such as the amount of free log space available or the number of active transactions in the database. SQL Server:Deprecated Features Information on the number of times deprecated features are used. SQLServer:Exec Statistics Execution statistics information. SQLServer:General Statistics General server-wide activity, such as the number of logins per second. SQLServer:Latches Information about the latches on internal resources, such as database pages. SQLServer:Locks Information about the individual lock requests made by SQL Server, such as lock timeouts and dead- locks. SQLServer:Memory Manager Information about SQL Server memory usage, such as the total number of lock structures currently allo- cated. SQLServer:Plan Cache Information about the SQL Server cache used to store objects such as stored procedures, triggers, and query plans. SQLServer: Resource Pool Stats Information about Resource Governor resource pool statistics. SQLServer:SQL Errors Information about SQL Server errors. SQLServer:SQL Statistics Query statistics, such as the number of batches of T-SQL statements received by SQL Server. SQLServer:Transactions Transaction statistics, such as the overall number of transactions and the number of snapshot trans- actions. SQLServer:User Settable Custom counters that can be a custom stored procedure or any T-SQL statement that returns a value to be monitored. ptg 1476 CHAPTER 39 Monitoring SQL Server Performance User-Defined Counters You can extend the range of information that Performance Monitor displays by creating up to 10 of your own counters. These user-defined counters appear under the SQLServer:User Settable:Query object, which contains the 10 counters as instances, starting with User Counter 1. You define your own counters by calling stored procedures with the names sp_user_counter1 through sp_user_counter10, which are located in the master database. These counters work differently than they did under previous versions of SQL Server and require you to call the stored procedures to update the information they return to Performance Monitor. To make any real use of these stored procedures, you now need to call them within a loop or as part of a job that is scheduled on some recurring basis. Using these counters allows you to monitor any information you want, whether it is system, database, or even object specific. The only restriction is that the stored procedure can take only a single integer value argument. The following sample user-defined counter procedure sets the counter value to the average connection time for all user connections. Processes that have a session_id less than 50 are internal system processes (checkpoint, Lazy Writer, and so on): DECLARE @value INT SELECT @value = AVG( DATEDIFF( mi, login_time, GETDATE())) FROM sys.dm_exec_sessions WHERE session_id > 50 EXEC sp_user_counter1 @value TABLE 39.1 SQL Server Performance Objects Performance Object Description SQLServer: Wait Statistics Information about waits. SQLAgent:Alerts Information about SQL Server Agent alerts. SQLAgent:Jobs Information about SQL Server Agent jobs. SQLAgent:JobSteps Information about SQL Server Agent job steps. SQLAgent:Statistics General information about SQL Server Agent. SQLServer:Replication Agents SQLServer:Replication Snapshot SQLServer:Replication Logreader SQLServer:Replication Dist. SQLServer:Replication Merge Information about replication agent activity. ptg 1477 A Performance Monitoring Approach 39 You could further extend this information by creating additional user procedures for return- ing the minimum and maximum times connected, as well as database usage. Your only limi- tation is that you can monitor only a maximum of 10 pieces of information at one time. Accessing Performance Counters via T-SQL Most of the SQL Server–oriented performance counter values can also be seen at any point in time via the system catalog view named sys.sysperfinfo: SELECT * from sys.sysperfinfo This view shows the performance object name, counter name, and current counter value as of the time the system view is executed. You should keep in mind that many of the performance counters are accumulation coun- ters, and you have to run them at intervals and determine the difference (change) from one interval to the next. Others are current values of aspects such as transaction rates, memory usage, and hit ratios. A Performance Monitoring Approach If you take a closer look at the performance monitoring areas depicted in Figure 39.1, you can see that SQL Server spans them all. SQL Server must process requests submitted to it via the network, service those requests with one or more processors, and rely on accessing a request’s data from both memory/cache and the disk system. If you maximize utilization on these resources from the point of view of SQL Server and the operating system, you end up with a well-tuned database server. However, an optimally tuned system doesn’t guarantee good performance. It’s still important to have a good database design and to implement optimal index strategies. The whole picture is important to tuning your SQL Server implementation, but the database, table designs, indexing strategy, and SQL state- ment tuning are described in much more detail in other chapters. This section focuses on the SQL Server instance as it sits on the OS and the hardware, along with the major moni- toring capabilities available to you. One area of interest is the amount of network traffic handled by SQL Server and the size of these network requests. Another area of interest is the capability of the available proces- sors to service the load presented to them by SQL Server without exceeding the CPU capacity of the system. This section also looks at SQL Server memory utilization of the available system memory and how effectively SQL Server is utilizing the disk system. In general, you want to start from the bottom, with the network, and work your way up into the SQL Server–specific elements. This allows you to quickly isolate certain issues that are paramount in performance tuning. In each of these areas, this section provides a list of minimum detail performance handles or counters that can be examined. This approach can be summarized into the following steps: 1. Understand and monitor network request characteristics as they relate to SQL Server and the machine on which SQL Server has been installed. This means a ptg 1478 CHAPTER 39 Monitoring SQL Server Performance complete profile of what is coming into and being sent back out over the network from SQL Server. 2. Understand processor utilization. Processing power might be the biggest issue affect- ing performance. You need to get a handle on this aspect of performance early. 3. Understand and monitor memory and cache utilization. This is the next step, monitoring the overall memory usage from the operating system level point of view and monitoring the memory that SQL Server is using for data buffers, plan cache, and so on. 4. Understand and monitor disk system utilization. You are often rewarded for imple- menting a simple (and less expensive) disk configuration or data storage approach. However, a simple configuration may not provide the best performance for your sys- tem. And you often don’t know you have a problem unless you look for it. Techniques that are often used to alleviate disk performance issues include disk strip- ing, isolation of logs from data, and so on. You need to repeat steps 1 through 4 on a regular basis. Your continued success (and, it is hoped, salary increases) will reflect your diligence here. For each step, certain tools and facilities are available for you to use that gather the information needed to identify and monitor performance issues. Let’s look now at how to use these tools to monitor and analyze the network, processor, memory, and disk utilization and performance. Monitoring the Network Interface One area of possible congestion is the network card or network interface; it does not matter how fast the server’s work is if it has to queue up to go out through a small pipe. Remember that any activity on the server machine might be consuming some of the bandwidth of the network interface card. With the increasing implementations of gigabit networks and the increases in network bandwidth over wide area networks (WANs), network throughput is not as much of a bottleneck as it used to be. That’s not to say network performance issues do not arise. For example, a bad or misconfigured router can cause all sorts of network performance issues, especially if your users and your SQL Server are on different sides of the router. If you are not detecting query performance issues in SQL Server but users are still complaining about slow query performance, the network is a likely culprit. You can monitor network activity via Performance Monitor. Table 39.2 shows the typical network performance object and counters you can use to monitor network interface activity. TABLE 39.2 Network Interface Performance Objects and Counters Performance Monitor Object Description Network Interface: Bytes Received The rate at which bytes are received on the interface. Network Interface: Bytes Sent The rate at which bytes are sent on the interface. ptg 1479 A Performance Monitoring Approach 39 In general, if the SQL Server packet sends or receives are grossly lower than the overall server’s packet sends and receives, other activity on the server is occurring that is poten- tially bogging down this server or not allowing SQL Server to be used optimally. The rule of thumb here is to isolate all other functionality to other servers if you can and let SQL Server be the main application on a machine. The sp_monitor system stored procedure, as well as several SQL Server system variables, can be used to see how many requests are queuing up, waiting to make use of the network interface. The following SELECT statement retrieves a current picture of what is being handled by SQL Server from a network packet’s point of view: SELECT @@connections as Connections, @@pack_received as Packets_Received, @@pack_sent as Packets_Sent, getdate() as ‘As of datetime’ go Connections Packets_Received Packets_Sent As of datetime 39407 395569 487258 2010-05-16 22:34:19.650 TABLE 39.2 Network Interface Performance Objects and Counters Performance Monitor Object Description Network Interface: Bytes Total The rate at which all bytes are sent and received on the interface. Network Interface: Current Bandwidth The bits per second (bps) of the interface card. Network Interface: Output Queue Length The length of the output packet queue (in packets). If this is longer than two, delays are occurring, and a bottleneck exists. Network Interface: Packets Received The rate at which packets are received on the network interface. Network Interface: Packets Sent The rate at which packets are sent on the network interface. Network Interface: Packets The rate at which packets are sent and received on the network interface. Server: Bytes Received The number of bytes the server has received from the network. This is the big-picture indicator of how busy the server is. Server: Bytes Transmitted The number of bytes the server has sent/transmitted to the network. Again, this is a good overall picture of how busy the server is. ptg 1480 CHAPTER 39 Monitoring SQL Server Performance The sp_monitor system stored procedure provides packets sent and received as a running total and since the last time it was run (difference in seconds). Here’s an example of what would result (the network- and packets-related results): EXEC sp_monitor GO last_run current_run seconds 2010-04-02 17:34:58.817 2010-05-16 22:35:08.940 3819610 cpu_busy io_busy idle 318(318)-0% 62(62)-0% 182952(182928)-4% packets_received packets_sent packet_errors 395781(395753) 487483(487455) 1(1) total_read total_write total_errors connections 8084(8084) 69921(69921) 0(0) 39432(39418) The values within the parentheses are the amounts since the last time sp_monitor was run, and the seconds column shows how long that period was. You can actually see the rate at which traffic is coming into and out of SQL Server. Monitoring the Processors The main processors of a server do the majority of the hard work, executing the operating system code and all applications. This is the next logical place to start looking at the performance of a system. With SQL Server 2008, you can identify the number of CPUs you want to utilize on your physical machine. If your physical machine has 32 CPUs, you might not necessarily want to make all 32 CPUs available to SQL Server. In fact, in some cases, this would be a detriment; some CPU processing should be reserved for the OS and the network management on heavily loaded servers (5% of the CPUs). SQL Server allows you to identify how many CPUs it can use from what is available on the physical machine. In Figure 39.27, you can see the number of CPUs available to SQL Server on a typical server from the SSMS Server Properties page. In this example, all CPUs are being made available to SQL Server. In a 32-CPU server example and using the 5% number just mentioned, you should let SQL Server use 30 of the CPUs and reserve 2 CPUs for dedicated network- and OS-related activ- ity (0.05 × 32 = 1.6, rounded up to 2 CPUs). This also allows SQL Server to utilize SQL parallelism effectively. ptg 1481 A Performance Monitoring Approach 39 FIGURE 39.27 Processor (CPU) properties of a SQL Server instance. Keep in mind that from a multitasking point of view, Windows servers often move process threads among different processors. This process thread movement activity can reduce Microsoft SQL Server performance under heavy system loads because each proces- sor cache is repeatedly reloaded with data. It is possible to assign processors to specific threads, which can improve performance under these types of conditions by eliminating processor reloads. This association between a thread and processor is called processor affin- ity. SQL Server 2008 supports processor affinity by means of two affinity mask options: affinity mask (also known as CPU affinity mask) and affinity I/O mask. If you do nothing, SQL Server is allowed to use each CPU for all its processing, with no affinity whatsoever. The operating system distributes threads from instances of SQL Server evenly among these CPUs. The affinity I/O mask option binds SQL Server disk I/O to a specified subset of CPUs. In high-end SQL Server online transaction processing (OLTP) environments, this extension can enhance the performance of SQL Server threads issuing a high number of I/Os. This enhancement does not support hardware affinity for individual disks or disk controllers, though. Perhaps this will be a future enhancement to SQL Server. ptg 1482 CHAPTER 39 Monitoring SQL Server Performance NOTE A side effect of specifying the affinity mask option is that the operating system does not move threads from one CPU to another. Most systems obtain optimal performance by letting the operating system schedule the threads among the available CPUs, but there are exceptions to this approach. The only time we have used this affinity setting was to isolate CPUs to specific SQL Server instances on the same box that had numer- ous CPUs to utilize. Typically, you should leave the settings to have SQL Server auto- matically set processor and I/O affinity as needed unless your processor monitoring indicates manually configuring the affinity may be necessary. Monitoring Processors in Performance Monitor From a Performance Monitor point of view, the emphasis is on seeing if the processors that are allocated to the server are busy enough to maximize performance but not so satu- rated as to create a bottleneck. The rule of thumb here is to see whether your processors are working at between 20% and 50%. If this usage is consistently above 80% to 95%, you should consider splitting off some of the workload or adding processors. Table 39.3 indi- cates some of the key Performance Monitor objects and counters for measuring processor utilization. The counters System: % Total Processor Time, System: Processor Queue Length, and Processor: % Processor Time are the most critical to watch. If the percentages are consistently high (above that 80% to 95% level), you need to identify which specific TABLE 39.3 Processor-Related Performance Objects and Counters Performance Monitor Object Description Processor: % Processor Time The rate at which bytes are received on the interface. System: Processor Queue Length The number of threads in the processor queue. A sustained processor queue of greater than two threads indicates a processor bottleneck. System: Threads The number of threads executing on the machine. A thread is the basic executable entity that can execute instructions in a processor. System: Context Switches The rate at which the processor and SQL Server has to change from executing on one thread to executing on another. This costs CPU resources. Processor: % Interrupt Time The percentage of time that the processor spends receiving and servicing hardware interrupts. Processor: Interrupts/sec The average number of hardware interrupts the processor is receiv- ing and servicing. ptg 1483 A Performance Monitoring Approach 39 processes and threads are consuming so many CPU cycles. The ideal Windows setup is to run SQL Server on a standalone member server to the Windows domain. You should not install SQL Server on a primary domain controller (PDC) or backup domain controller (BDC) because the PDC and BDC run additional services that consume memory, CPU, and network resources. Before you upgrade to the latest processor just because the % Processor Time counter is constantly high, you might want to check the load placed on the CPU by your other devices. By checking Processor: % Interrupt Time and Processor: Interrupts/Sec, you can tell whether the CPU is interrupted more than normal by devices such as disk controllers. The % Interrupt Time value should be as close to 0 as possible; controller cards should handle any processing requirements. The System: Context Switches counter can reveal when excessive context switching occurs, which usually directly affects overall performance. In addition, the System: Threads counter can give a good picture of the excessive demand on the CPU of having to service huge numbers of threads. In general, you should look at these counters only if processor queuing is happening. By upgrading inefficient controllers to bus-mastering controllers, you can take some of the load from the CPU and put it back on the adapter. You should also keep the controller patched with the latest drivers from the hardware vendor. Monitoring Processor Performance in SSMS The Activity Monitor in SSMS provides some basic information on processor performance, such as the % Processor Time that SQL Server is consuming and a list of the current user connections into SQL Server. If you’ve enabled the SQL Server Utility for your SQL Server instance, you can get a high- level view of CPU utilization on the CPU Utilization page of the Utility Explorer (see Figure 39.28). This view displays CPU utilization for the SQL Server instance and the overall CPU utilization for the server. If you are collecting performance data into the MDW using the Data Collector for a SQL Server instance, you can view more detailed CPU utilization and CPU wait information in the Server Activity History report (refer to Figure 39.12). If you see high CPU utilization, you can click on the CPU Usage graph to drill down and examine the details of the top 10 processes consuming CPU resources. If there is an indication of excessive CPU waits, you can click on the SQL Server Waits graph to drill down into the details regarding the queries experiencing high CPU waits. Dynamic Management Views or System Views for Monitoring Processor Items Within SQL Server, you can execute a simple SELECT statement that yields the SQL Server processes and their corresponding threads: . about SQL Server Agent jobs. SQLAgent:JobSteps Information about SQL Server Agent job steps. SQLAgent:Statistics General information about SQL Server Agent. SQLServer:Replication Agents SQLServer:Replication. pool statistics. SQLServer :SQL Errors Information about SQL Server errors. SQLServer :SQL Statistics Query statistics, such as the number of batches of T -SQL statements received by SQL Server. SQLServer:Transactions Transaction. the counter name (for example, SQLServer:General Statistics). Table 39.1 provides a list of the SQL Server performance counters available for SQL Server 2008 TABLE 39.1 SQL Server Performance Objects Performance