6 – Monitoring and notifications 155 In order to demonstrate notifications for backup failures, I will need to set up a backup job and intentionally make it fail, which is something that makes me squeamish even when it's just for demonstration purposes. To keep things, simple, for this demonstration, I'm going to use a script that backs up just a single database, using the Red Gate extended stored procedure sql_backup. The script is shown in Listing 6.2. DECLARE @exitcode INT DECLARE @sqlErrorCode INT EXECUTE master sqlbackup N'-SQL "BACKUP DATABASE [DBA_Rep] TO DISK = ''D:\Backups\<AUTO>.sqb'' WITH COMPRESSION = 1, MAILTO_ONERROR = ''rlandrum13@cox.net''"', @exitcode OUTPUT IF( @exitCode > 0 OR @sqlErrorCode > 0 ) BEGIN Raise Error RAISERROR ('SQL Backup failed', 16, 1) END Listing 6.2: Red Gate SQL Backup statement. An important point to note about the backup code is use of the Mailto_Onerror parameter. This tells the Red Gate extended stored procedure to use its own native SMTP client to send an email notification, if there are any errors with the backup. This is first line of defense for the DBA. If I were to run this code and produce an error, I should immediately receive a detailed notification telling me not only what database failed to backup (in this case, DBA_Rep) but also what the cause of the failure was. That information is critical to resolving the issue going forward so that is does not happen again. If instead of using a third party tool I were to use native T-SQL code, for example BACKUP DATABASE, I would capture any errors in a variable string and then email the failures when the backup job completed using sp_send_dbmail, which I have mentioned previously. This type of backup code requires some additional scripting, of course. If you are familiar with SSIS, it is also possible, via a Database Maintenance Plan, to add a step in to send mail upon failure. There are really many options available to the DBA to get notifications from failed jobs. OK, now to produce the backup failure. This is not actually quite as easy as it might sound as it's generally a pretty robust process, and I would venture that 98% of backups succeed. To produce the required failure, I simply started a backup of the DBA_Rep database to my 350G USB external hard drive and then pulled out the connecting USB cable (subtle, I know). In my dream world, the resulting error would say "Rodney Pulled the Plug on the USB Hard Drive". It doesn't, of course, but the error message still provides some 6 – Monitoring and notifications 156 useful information, as you can see from Figure 6.6, which states that a "non- recoverable I/O error occurred". Figure 6.6: Error mail message from Red Gate SQL Backup. At this point, I know that it was an I/O error that caused the failure, and I can respond by attempting to backup the database again, and then looking deeper into the issue. If this error were caused by lack of disk space , as it often is, I would need to free up enough disk space on the destination to accommodate the backup file, and then re-run the backup. The failure message also contains key words like "error" that I can use to trigger a higher level alert on my mobile device, associated with a really obnoxious ring tone of the sort you will want to avoid in theatres or quiet dinners with your loved one. However, what if this message did not get delivered for whatever reason … who knows, we might have changed mail servers and forgotten to update the Red Gate mail client properties. I still need to get a notification if the backup fails. You will have seen in Listing 6.2 that I intentionally wrapped error checking around the backup statement. If the backup script fails, it should report this fact to the calling process, which is generally a SQL Agent job, which can then send a notification, via Database Mail. In order to enable this notification mechanism, we first need to create a SQL Agent Job that will run our custom backup script. To do this, simply right-click on "SQL Server Agent" in SSMS and select New | Job. In the "General" section, remember to name the job, in my case "Backup Database Test Failure" and then give the job a valid category, in this case "Database Maintenance". Use of meaningful job categories is valuable for reporting. In the DBA Repository, for example, I run job reports based on each category of job. Next, in the "Steps" section, select "New" and paste in the backup code from Listing 6.2, so that your step looks as shown in Figure 6.7. 6 – Monitoring and notifications 157 Figure 6.7: Creating Job Step for backup code. Click OK and then select the "Notifications" section. Click the box to enable the E-mail action, and then from the drop down select the "DBA Team" operator that we created earlier. The default action, which we'll accept, is to notify "When the Job Fails," as shown in Figure 6.8. Other options are available, such as mailing when the job succeeds or completes. Perhaps you want to receive a notification when the backup job completes, regardless of whether it fails. Click OK to create the job, and we are done. For the sake of this demo, rather than schedule the job, I'm simply going to right- click on it and select "Start Job at Step". Once the job starts, I yank out the USB cable one more time. As expected, I have forced another backup failure, as you can see in Figure 6.9. 6 – Monitoring and notifications 158 Figure 6.8: Selecting to be notified when the backup job fails. Figure 6.9: Forcing the backup job to fail. 6 – Monitoring and notifications 159 Now, let me check my mail. Yep, everything worked as expected, and I receive, almost instantaneously, two separate email notifications; one from the SQL Agent job telling me the job itself failed and the other more detailed mail comes from the code inside the job, as shown side-by-side in Figure 6.10. Figure 6.10: Two separate mail notifications from two separate sources. Performance issues If, over a sandwich, you were to ask a DBA to describe the performance issues that he or she has faced during the preceding year, you will, as the lunch drifts into the late afternoon, probably start to regret not being more focused in your line of questioning. All DBAs are faced with all manner of performance issues. If you were to ask, "What is the worst performance issue you had in the past year," you will get a contemplative stare to the ceiling, hand on chin, eyes scanning and finally, "Oh yeah, there was that one time when … I found code from a website application that was flagrantly misusing MARS (Multiple Active Result Sets) connections and instead of having an expected 300 logical connections, there were over 8500, each simultaneously eating into available RAM until the server slowed to a crawl." In other words, the question is not if DBAs are going to face performance issues in their environment, but what types of problems they are going to encounter, and how they can deal with them. . Gate extended stored procedure sql_ backup. The script is shown in Listing 6.2. DECLARE @exitcode INT DECLARE @sqlErrorCode INT EXECUTE master sqlbackup N' -SQL "BACKUP DATABASE [DBA_Rep]. OUTPUT IF( @exitCode > 0 OR @sqlErrorCode > 0 ) BEGIN Raise Error RAISERROR (&apos ;SQL Backup failed', 16, 1) END Listing 6.2: Red Gate SQL Backup statement. An important. process, which is generally a SQL Agent job, which can then send a notification, via Database Mail. In order to enable this notification mechanism, we first need to create a SQL Agent Job that will