Hands-On Microsoft SQL Server 2008 Integration Services part 59 pps

558 Hands-On Microsoft SQL Server 2008 Integration Services backups and reduce the size of the backup file. As SQL Server backup compression compresses the backup, the device I/O decreases due to reduced size, but the CPU usage increases due to compression overhead. However, the effect of reduced size and I/O makes backups much faster. If you find that the CPU usage has increased to the level that it starts affecting other applications, you can create a low-priority compressed backup in a session whose CPU usage is limited by the Resource Governor. The Resource Governor is another new component that is released in SQL Server Enterprise Edition to enable you to manage SQL Server workload and resources by specifying limits on resource consumption by the incoming requests. Refer to Books Online for more details on the Resource Governor. Though the backup compression is a feature in Enterprise Edition, you can restore compressed backups on any edition of SQL Server. There are some restrictions that you should be aware of; for instance, you cannot mix compressed and uncompressed backups on the same media, and you don’t get much compression of the database when you use transparent data encryption at the same time; hence the use of both together is not recommended. You can use backup compression in large data warehouse implementations to achieve the following goals: Reduce the size of SQL backups in the vicinity of 50 percent or more. c Save hard disk space to keep backups online. c Be able to keep more copies of backups online for the same storage space. c Reduce the time required to back up or restore a database. c Backup compression configuration is specified at the server level but can be overridden. The default setting is off for the server. You can change this setting in SSMS by checking the Compress Backup option in the Database Settings page of Server Properties or by using the sp_configure stored procedure to set the default value of backup compression and then execute the reconfigure statement. You can override the server level setting for one-off or single backups by using one of the following methods: Specify the Compress Backup or Do Not Compress Backup option in the Set c Backup Compression field while using the Back Up Database task in an SSIS package. Refer to the bottom of Figure 5-30 in Chapter 5 to see this option. Specify the Compress Backup or Do Not Compress Backup option in the Options c page of the Back Up Database dialog box while backing up a database in SSMS. Specify the WITH NO_COMPRESSION or WITH COMPRESSION switch c in the Backup statement. Chapter 12: Data Warehousing and SQL Server 2008 Enhancements 559 MERGE Statement SQL Server 2008 includes this new T-SQL statement that can be used to perform multiple DML operations—i.e., INSERT, UPDATE, and DELETE operations—on a table or view in a single statement. As the operations are applied in one statement, by their very nature they are executed under a single atomic operation. Using a MERGE statement, you join a source table with a target table or view and then perform multiple DML operations against the target table based on the results of that join. Typically, you apply these operations using a batch or a stored procedure individually, which means the source and the target tables are evaluated and processed multiple times, once for each operation. For a MERGE statement this evaluation happens only once for all the operations, thus making it more efficient when compared to applying the DML operations individually. The MERGE statement specifies a target table in the MERGE INTO clause and the source table in the USING clause. Then it uses the ON clause to specify the join condition for the source and target tables, which primarily determines the rows in the source table that have a match in the target table, the rows that do not, and the rows in the target table that do not have a match in the source table. For the ON clause, it is recommended that at least you have unique indexes on join columns in both the source and target tables for better performance. After evaluation of match cases, you can use any or all of the three WHEN clauses to perform a specific DML action on a given row. The clauses are: WHEN MATCHED THEN c You can UPDATE or DELETE the given row in the target table for every row that exists in both the target table and the source table. WHEN NOT MATCHED [BY TARGET] THEN c You can INSERT the given row in the target table for every row that exists in the source table, but not in the target table. WHEN NOT MATCHED BY SOURCE THEN c You can UPDATE or DELETE the given row in the target table for every row that exists in the target table, but not in the source table. You can also specify a search condition with each of the WHEN clauses to choose the rows to apply to the DML operation. The MERGE statement also supports the OUTPUT clause, enabling you to return attributes from the modified rows. The OUTPUT clause includes a virtual column called $action, which returns the action that modified the row—i.e., INSERT, UPDATE, or DELETE. 560 Hands-On Microsoft SQL Server 2008 Integration Services You can embed a MERGE statement inside the Execute SQL task to use it in your SSIS package. Typically, you would stage data to a staging table to change capture before loading it in the data warehouse with an Integration Services package. Such an SSIS package would include a Lookup task to identify whether a row is a new row or an update, an SQL Server Destination to INSERT new rows, and OLE DB Command transformations to perform UPDATE and DELETE operations. The Lookup transformation and the OLE DB Command transformation work on a row- by-row basis; thus they perform at a speed that can’t match the set-based operation of a MERGE statement. If you stage data for change capture and make it as a source table for a MERGE statement, you will find that the data is loaded at a far better rate than using SSIS with or without staging data. This performance becomes better especially on servers where lookup is working against large data sets and is running short of memory. You can also replace a Slowly Changing Dimension (SCD) transformation with a MERGE statement in some cases. Again, as the MERGE evaluates the source and the target data only once, it performs much better than the SCD transformation, which otherwise has been recognized as a performance pain point in SSIS. In the following example code, DimProduct is updated with the changes being received in the ProductChanges table. The changes might include changes in price for some existing products that need to be updated and some new products that need to be added. Considering the changes as Type 2, the IsCurrent flag has to be reset to 0 for the updates and new rows for them need to be inserted along with the new products. Look at the inner query where the MERGE statement updates the IsCurrent flag to 0 under the WHEN MATCHED clause and adds a new row under the WHEN NOT MATCHED clause for the new products. Finally, it outputs the rows affected with the action taken for them and then the outer query inserts the rows that have been updated back into DimProduct table. INSERT INTO DimProduct (ProductID, ProductName, Price, IsCurrent) SELECT ProductID, ProductName, Price, 1 FROM ( MERGE DimProduct as TGT USING ProductChanges AS SRC ON (TGT.ProductID = SRC. ProductID and TGT.IsCurrent = 1) WHEN MATCHED THEN UPDATE SET TGT.IsCurrent = 0 WHEN NOT MATCHED THEN INSERT VALUES (SRC.ProductID, SRC.ProductName, SRC.Price, 1) OUTPUT $action, SRC.ProductID, SRC.ProductName, SRC.Price ) AS Changes (action, ProductID, ProductName, Price) WHERE action = 'UPDATE'; Chapter 12: Data Warehousing and SQL Server 2008 Enhancements 561 Just as MERGE can replace of some of the SSIS components for performance reasons, it can be used with change data capture (CDC) functionality to perform inserts and updates, thus replacing parameterized OLE DB Command transformation and resulting in a considerable gain in performance. One such application could be moving staged data into production servers. GROUP BY Extensions The GROUP BY clause has been enhanced in SQL Server 2008 and now adds a new operator: GROUPING SETS. Actually, the GROUP BY clause in SQL Server 2008 now supports ROLLUP, CUBE, and GROUPING SETS operators in line with ANSI SQL-2006 standards. So, an ISO-compliant syntax has been adopted, though a non-ISO-compliant syntax is still supported for backward compatibility—e.g., the earlier operators supported in SQL Server 2005 (WITH CUBE and WITH ROLLUP) are still supported but will be removed from future versions. As you know, a GROUP BY clause enables you to select one summary row for each group of rows created from a selected set of rows on the basis of values of one or more columns or expressions. Adding the operators to a GROUP BY clause provide an enhanced result set. The ROLLUP operator generates simple aggregate groupings with expressions rolled up from right to left and the number of groupings in the result set equal to number of expressions plus one. The CUBE operator generates groupings for every combination of the expressions without any regard to expression order and generates 2 n groupings, where n is the number of expressions used with the CUBE operator. The GROUPING SETS allow you to produce multiple groupings of data only for the specified groups in a single result set. This is simple and better than the ROLLUP and CUBE operator that generate the full set of aggregations. GROUPING SETS can specify groupings equivalent to those returned by ROLLUP or CUBE and can also work in conjunction with ROLLUP and CUBE. The result set thus generated is equivalent to a UNION ALL of the specified groups. So using GROUPING SET, you can generate only the required levels of groupings and the information can be readily made available to reports or requesting applications in a ready-to-digest format. You might use a pivot table to display the results. However, the important thing to take away is that using the GROUPING SETS operator can allow you to retrieve aggregated information in the required levels of grouping all in one single statement and above all efficiently. This could enhance your data analysis experience and improve reporting performance. If you are using an Aggregate transformation to produce aggregations for a report, you may find using GROUPING SETS performs better in some cases, especially where you are running a data flow task only for performing aggregations. 562 Hands-On Microsoft SQL Server 2008 Integration Services Star Join Query Processing Enhancement This is one of those improvements in SQL Server 2008 Enterprise Edition that is applied automatically and does not require users to make any changes to their queries. If you are using a dimensional data model for your warehouse, you may realize this performance gain just by upgrading to SQL Server 2008 and without making any change to your database structure or the queries. Microsoft claims to have achieved 15 to 30 percent improvement in the whole of the star join payload on the dimensional database server, whereas some individual queries can benefit from it by a factor of seven. The star join optimization is provided in Microsoft SQL Server 2008 Enterprise Edition. In a dimensional data warehouse, most of the queries perform aggregations on columns of a large fact table and join it to one or many smaller dimensional tables, apply filter conditions on the non-key dimensional table columns and form groups by one or many dimensional table columns. Such queries are called star join queries. These queries follow a similar processing pattern that you can find out by looking at the execution query plan of some of these queries. Typically, the fact table will be scanned for a range of data by running a seek on the clustered index that will be hash- joined with the results of the seek on one of the dimension table. These hash joins are repeated for as many times as there are dimensions involved in the query, and finally the results are sent to an aggregation hash. If you check out this plan on an SQL Server 2008 server, you will notice some bitmap filters have also been applied. The bitmap filters, also called the bloom filters, are generated as a by-product of the hash joins. The bloom filters are data structures that can probabilistically test whether an element belongs to a set. Though this technology can allow some false positives, it does not allow any false negatives. So, once a fact table row fails the test, it will be excluded from further processing. SQL Server 2008 can generate multiple bloom filters, as against SQL Server 2005, which could do only one. These filters are pushed down in the query execution path to the scan of the fact table to eliminate the nonqualifying rows. Also, the SQL Server 2008 query execution engine can change the sequence in which these filters are applied on the basis of their selectivity—i.e., by placing the most selective filter first, and then the next most selective filter, and so on—thus extracting maximum performance out of this enhancement. So, the bitmap filters enable SQL Server to eliminate nonqualifying fact table rows from further processing quite early on during the query evaluation. This avoids unnecessary processing of the data rows that would have been dropped later in the query processing anyway and results in saving a considerable amount of CPU time. Change Data Capture When you are loading data into a data warehouse system from a source system, you need to know about the rows in the source system that have been either changed or Chapter 12: Data Warehousing and SQL Server 2008 Enhancements 563 inserted or deleted. Until now, the options were to either extract complete source system data to a staging table to compare it against the data warehouse, so as to determine the changes or to use alternate techniques such as timestamp columns, triggers, or complex queries for this purpose. Though these methods work, they have some issues. The first option is very inefficient due to its doing lot of work to find out the changes. Implementing the second option is also not a good choice, as timestamp columns require application changes and triggers or complex queries are not efficient. SQL Server 2008 Enterprise Server (and Developer Edition) has provided an efficient feature called Change Data Capture (CDC) to track data changes such as inserts, updates, and deletions. So, instead of comparing staged data with data warehouse data or using intrusive methods to find the changes in data, you can use CDC to identify changes and eventually load the data warehouse in incremental steps. The Change Data Capture feature is not enabled by default, so you have to enable CDC first in order to use it. The CDC feature is enabled at two levels—the database and table levels. When you enable CDC on a database, it creates a new schema, CDC, and creates a user account, CDC, in the database. The CDC schema is used to store all the change tables and their metadata. When you decide to track a table for changes, you enable CDC on that table. Enabling CDC on first table creates the following: A change table to capture changes to data and metadata c Up to two query functions to allow you to retrieve data from the change tables. By c default, all changes will be captured and only one query function is created in this case; however, if you want to track only the net changes over a period of time, you can do so. You can specify a parameter @support_net_changes = 1 in the CDC enabling command to return only the net changes. is setting creates the second query function that allows you to return only the net changes. is feature can also potentially reduce the number of updates you perform on your data warehouse. A group of change data capture metadata tables to retain metadata configuration c detail Two SQL Server jobs: capture job and cleanup job. e SQL Server Agent service c must be running when you enable CDC tracking on a table. When the changes are applied to the tracked source table, the database engine as usual writes the changes in to the transaction log. The CDC capture job reads the log automatically and adds information about the changes into the associated change table. This table holds the capture columns from the source table to capture the changed data and five additional metadata columns to provide information relevant to the captured change. Two columns are of particular importance and worth mentioning here. The first column, __$start_lsn, records the commit log sequence number (LSN) that 564 Hands-On Microsoft SQL Server 2008 Integration Services was assigned to the change. The commit LSN not only identifies changes that were committed within the same transaction but orders them as well. The second column we want to discuss is __$operation, which records the operation that is associated with the change, for instance, 1 = delete, 2 = insert, 3 = source record prior to update, and 4 = source record after update. This column makes the ETL very efficient by eliminating the need to identify whether a row is an update, a delete, or an insert. Refer to Figure 12-5, which shows methods to load a data warehouse both with and without CDC. With CDC, instead of performing an expensive lookup operation, you just split data conditionally using a Conditional Split transformation. The split condition uses the __$operation column to divert the rows to appropriate output. Partitioned Table Parallelism In SQL Server 2005, when you run a query against large tables that have been partitioned across several disks probably based on dates, it has been observed that the executor allocates threads in an inconsistent manner. If a query touches only one partition, the executor allocates all the threads to the query; however, if a query needs to touch more Read data from source systems Stage data for change capture Read data from stage database Perform lookup operation against data warehouse Update data warehouse Insert into data warehouse Map the starting and ending LSNs for the increment capture interval Prepare query Read data from CDC schema Perform conditional split opertion Update data warehouse Insert into data warehouse Update data warehouse Loading a Data Warehouse without CDC Loading a Data Warehouse with CDC Figure 12-5 Loading a data warehouse using Change Data Capture Chapter 12: Data Warehousing and SQL Server 2008 Enhancements 565 than one partition, it gets one thread allocated per partition, though additional threads might still be available on the server. Sometimes this has produced inconsistent results for the same or similar queries, depending on when they have been executed. In SQL Server 2008, the parallel query plans against partitioned tables have been improved to utilize more threads regardless of how many partitions a query touches. The SQL Server query executor allocates all the available threads to partition table queries in a round robin fashion, keeping CPUs fully utilized at all times so that the performance of the same or similar queries over time is more comparable. The performance boost achieved with this round robin–style thread allocation is noticeably high, especially when more processor cores are available compared to the number of partitions a query touches. You don’t need to configure anything in the SQL Server, as this feature works by default. Partition-Aligned Indexed Views In SQL Server 2005, both indexed views and the partition tables can be used together. For instance, if you have a very large fact table that has been partitioned, say, on a yearly basis and you want to create summarized aggregates on this fact table, you use indexed views to achieve this. So far, so good; however, there is an issue with the way people use fact table partitioning. The fact table is generally partitioned on the basis of a time period, which could be a month, quarter, or year, depending upon your business needs, and as a time period completes, the oldest partition need to be switched out and a new partition has to be switched in to keep data for a fixed period of time. For example, if you are required to keep data for five years and you are keeping partitions on year-by-year basis, then at the beginning of a year you need to remove the oldest partition (six years old) and add a new partition for the new running year. The point to note here is that switching a partition in and out from the fact table is very efficient and is best possible way you can manage data in terms of retention. This way, your data warehouse will always keep the data for the defined period of time and the retained data is partitioned for performance. This is known as a sliding window scenario. The issue with SQL Server 2005 is that you cannot switch a partition in or out from a partitioned table that has indexed views created on top of them. You will end up dropping the indexed view, switching the partition in or out, and then recreating the indexed view. The creation of an indexed view could be a very expensive process that can delay data delivery to your end users. SQL Server 2008 aligns both these features and makes them work together. You don’t need to drop and recreate indexed views when switching a partition in or out of it. With the enhancement of partition-aligned indexed views, you save lot of extra processing that would otherwise be required to rebuild aggregates on the entire partitioned table. 566 Hands-On Microsoft SQL Server 2008 Integration Services Summary This chapter has covered basics about data warehousing and some of the enhancements provided in the SQL Server database engine. The two new editions that have been introduced in the R2 release are interesting to watch, as they realize an approach to bring together software, hardware, and best practices to achieve the highest performance. This chapter is targeted to wet your feet, so if you feel interested, go and take a deep dive in the area of data warehousing. For now, stay on as we still have some interesting topics to cover: migration, package deployment, troubleshooting, and performance tuning. Deploying Integration Services Packages Chapter 13 In This Chapter c Package Configurations c Deployment Utility c Deploying Integration Services Projects c Custom Deployment c Summary . performing aggregations. 562 Hands-On Microsoft SQL Server 2008 Integration Services Star Join Query Processing Enhancement This is one of those improvements in SQL Server 2008 Enterprise Edition. 558 Hands-On Microsoft SQL Server 2008 Integration Services backups and reduce the size of the backup file. As SQL Server backup compression compresses the. table. 566 Hands-On Microsoft SQL Server 2008 Integration Services Summary This chapter has covered basics about data warehousing and some of the enhancements provided in the SQL Server database