478 Hands-On Microsoft SQL Server 2008 Integration Services 30. Drop an Excel Destination just below the Splitting Fuzzy Grouped Duplicates component and join both of these components using the green arrow. Select Fuzzy Grouped Matches in the Output field of the Input Output Selection dialog box and click OK. Rename this Excel destination Fuzzy Grouped Duplicates. 31. Double-click the Excel destination to configure it. Select DuplicateOwners Connection in the OLE DB Connection Manager field if it’s not already selected. 32. Click the New button next to the “Name of the Excel sheet” field and verify in the CREATE TABLE statement that it is creating the Fuzzy Grouped Duplicates table—i.e., worksheet in Excel—then click OK to accept. Select Fuzzy_Grouped_ Duplicates in this field. 33. Go to the Mappings page to create the necessary column mappings automatically. Note that in addition to _key_in, _key_out, and _score columns, the Fuzzy Lookup transformation has added one _Similarity_ColumnName column for each column that participated in the fuzzy grouping. Click OK to close this editor. 34. Drop an OLE DB destination on the Data Flow surface below the Splitting Fuzzy Grouped Duplicates component and join the component to the OLE DB destination using the available green arrow. Select Canonical Row in the Output field of Input Output Selection dialog box and click OK. 35. Double-click the OLE DB destination and select [dbo].[Owner] table in the Name of the table or the view field. Go to the Mappings page and verify that the necessary mappings have been automatically created. Click OK to close this editor. Rename the OLE DB Destination Owner. Press -- to save all the files in the project. Exercise (Execute Removing Duplicates Package) Finally, execute the package to see how the various transformations remove the duplicate data. 36. You can add the data viewers after each transformation to see the records that have been removed from the pipeline. Later, I’ve explained how you can re-run the package to see the workings of various transformations over and over. As a first go, just run it without any data viewer. 37. Press 5 to execute the package. When the package completes execution, note the number of records after each transformation (see Figure 10-32). Following is an explanation of the execution results: e Excel source brings 13 rows into the data flow. c e Sort transformation removes two exact duplicate records—one for Johnathon c Skinner and one for Kathrine Morris. Chapter 10: Data Flow Transformations 479 e Lookup transformation then matches Johnathon Skinner in the reference c table and diverts it to the Excel destination; the remaining ten rows are then passed on to the Fuzzy Lookup transformation. e Fuzzy Lookup transformation fuzzy matches John, Jonothon, and Jonathon c with the Johnathon in the reference table and marks all three records with a high similarity and confidence score. is high similarity and confidence score is then used by the Conditional Split transformation to filter out these three records to the Excel destination. Figure 10-32 Executing Removing Duplicates package 480 Hands-On Microsoft SQL Server 2008 Integration Services e remaining seven records are then passed on to Fuzzy Grouping transformation, c which looks for records that are likely duplicates in the data flow and groups them together using _key_in, _key_out, and _score column values. ese values are then used by the conditional Split transformation to filter out records for Kathy, Kath, and Kathey that are fuzzy-grouped together. e remaining four unique records are then sent to the OLE DB destination for loading into the Owner table. Review You’ve seen various types of duplicates and the methods for removing them in this package. This package will give you a kick start into real-life de-duplication problems. However, bear in mind that whenever you use Fuzzy Lookup and Fuzzy Group transformations, you need to find the similarity threshold values by running the package on sample data that correctly represents the main data. The more effort you put into finding the value of the similarity threshold that works with your data, the more accurate your results will be. If you want to re-run the package, you need to execute the following SQL statement against the Campaign database in the SQL Server Management Studio. This will refresh the Owner table for you to run the package with the same results. USE Campaign; DROP TABLE [dbo].[Owner]; SELECT * INTO [dbo].[Owner] FROM Owner_original; Summary Having used lots of Data Flow transformations, you must by now feel a lot more confident and ready for real-life challenges. You have used various transformations to perform functions such as pivoting; sorting; performing an exact lookup for de- duplication of data; and standardizing data, fuzzy lookup, and fuzzy groups to eliminate duplicates in a pipeline, aggregate data, and load a slowly changing dimension table. You’ve also studied several other preconfigured transformations that are straightforward to use in your packages. Having come so far, you can now create control flow and data flow in your packages to perform workflow and transformation functions, store and manage your SSIS packages, and secure them as well. In the next chapter, you will study how to deploy your packages in an enterprise environment. Programming Integration Services Chapter 11 In This Chapter c The Two Engines of Integration Services c Programming Options c Extending Packages with Scripting c The Legacy Scripting Task: ActiveX Script Task c Script Task c Script Component c Script Task vs. Script Component c Summary 482 Hands-On Microsoft SQL Server 2008 Integration Services B y now you have worked with almost all the preconfigured tasks and components that Integration Services provides and can appreciate the power and ease it provides to developers for building enterprise-wide solutions. However, businesses are doing so many different things that it is sometimes not easy or even possible to build a solution for every scenario that can exist in an enterprise using the preconfigured tasks and components. SSIS does provide a way to cover even those complex scenarios. You can extend SSIS by writing your own custom code. Not only can you extend SSIS using custom code, Microsoft has tried to make it easier to extend your custom solution and, hence, provides you various programming levels. SSIS provides a much enhanced object model that can be easily programmed with different options to choose from, based on the problem you’re trying to solve. You can choose to extend your packages using scripting, you can develop custom components that can be deployed into SSIS and used as preconfigured components, or you can program your packages all over from scratch. In this chapter you will learn more about these options and when to choose the one that is more appropriate for your particular scenario. However, at this point I want to clarify that as programming Integration Services is a vast subject, it will be very difficult to cover it completely or even do some justice to it in just one chapter. This subject probably requires a complete book in itself. So, in this chapter scripting SSIS has been covered in detail, as I think it is the one area that will be of interest to most readers, and other options have been covered only up to an introduction level so that you can choose the best method to extend SSIS. Refer to the Books Online for other programming options. The Two Engines of Integration Services As you know, all of your packages contain Control Flow tasks and most of them also contain a Data Flow task, which is a special task and has its own components such as sources, destinations, and transformations. You also understand that the work flow and the management of the tasks are designed in Control Flow pane while the data movement and the transformations are designed in Data Flow pane. If you refer back to Figure 1-1, you can notice that top half of the figure—the object model of the Integration Services run time—includes connection managers, event handlers, and log providers, along with tasks and containers. This represents the Integration Services run-time engine and, as you can see, provides the necessary infrastructure for package execution and management support such as execution order, logging, event handling, connections, breakpoints, and transactions. The second engine of Integration Services, shown in the lower half of the Architecture diagram, manages the data movement and the transformations. The Data Flow task that performs the actual work of data movement and transformation runs under the management of the Data Flow Chapter 11: Programming Integration Services 483 engine, also popularly called the pipeline. When you drop the first Data Flow task on the Control Flow Designer surface, you invoke the data flow engine. Though you’ll have only one Control Flow within a package, you can include multiple Data Flow tasks in the package, with each Data Flow task able to support multiple data sources, transformations, and destinations. These two engines of Integration Services provide complete control over the execution of a package and the flexibility to deal with buffer-oriented data movement and transformations in a very efficient way. You may also notice in the architecture diagram that both engines provide scope for building custom objects such as custom tasks, custom log providers, and custom connection managers for the run-time engine, and custom data flow components such as custom sources, custom transformations, and custom destinations for the pipeline. In fact, whenever you extend Integration Services programmatically, you will be working with these engines using different classes, methods, and properties exposed by the engines. Some of the tasks or components provided in Integration Services are written in managed code, where the run-time engine and the data flow engine have been written in the native code for providing enhanced performance, but they are exposed for development through the managed object model of Integration Services and providing ease of extension. The run-time engine represents the Microsoft.SqlServer.Dts.Runtime namespace that contains the classes and interfaces to create packages, custom tasks, and other package control flow elements. And the data flow engine represents the Microsoft.SqlServer.Dts.Pipeline namespace that contains the classes used to develop managed data flow components. Programming Options The object model of Integration Services allows you to program any aspect of SSIS; for instance, you can extend the prebuilt functionality and manage interaction of SSIS with other applications by building interfaces or SSIS packages created programmatically by your custom-built application. This is possible because Integration Services fully supports the Microsoft .NET Framework and allows you to choose any of the .NET- compliant languages. The SSIS development team has done an excellent job by making it easier to program SSIS by extending the packages, yet more powerful by enabling the development of custom objects that can be developed outside the package, but can be included in the package just like prebuilt objects once deployed into the Integration Services object model. You can use Microsoft Visual Studio or any other development environment with your preferred .NET-compliant language to write custom code. So, you can choose from the options of simple one-off scripting or custom develop SSIS objects or in fact build complete packages depending upon your requirements and ability to program. Let’s explore these options in detail. 484 Hands-On Microsoft SQL Server 2008 Integration Services Scripting As mentioned earlier, when you have a need that can’t be met using prebuilt tasks or components, you’ll need to create the required functionality. Now, if your need is one off—i.e., the particular functionality is not going to be required in other packages and you are looking for the least development effort—scripting SSIS should be your choice. The code developed using scripting is generally reused within the development team working on the same project. The developers who have worked with SQL Server Data Transformation Services (versions 7.0 and 2000) might have used the scripting option already. The ActiveX Script task is the only method to extend Data Transformation Services (DTS), so many developers have used it extensively and some have actually built quite complex scripting solutions that were deployed throughout the enterprise. Later it has been realized that the cost of maintaining such solutions is quite high, as the ActiveX Script task was not designed to create enterprise solutions. Integration Services has overcome this limitation and includes many flexible scripting solutions along with the possibility of developing reusable custom objects that are easy to maintain. Though the ActiveX Script task has been provided in Integration Services, you should refrain from using it for new development work. This is provided only for backward compatibility, as an interim support while migrating your packages to Integration Services. The two scripting objects, the Script task and the Script component, replace the Dts scripting functionality with a much better and more powerful programming environment, Microsoft Visual Studio Tools for Applications (VSTA). This embedded scripting environment allows you to choose Microsoft Visual Basic 2008 or Microsoft Visual C# 2008 as your preferred language for scripting. You can create a custom task for use in the Control Flow with the Script task or a custom component such as a source or a transformation, or else a destination for use in the Data Flow task with the Script component. When you use the VSTA environment to write scripts for either of these script objects, the scripting environment creates lots of infrastructure code for you and leaves you to focus on writing the code for the required functionality. This makes writing scripts much easier using VSTA. There are several other benefits to use of this powerful IDE, such as extensive debugging and testing of the written code. Due to .NET Framework support, you can use the .NET namespaces, take advantage of the class libraries, and also reference external .NET assemblies quite easily in your scripts. This is a very powerful feature that can save your many man-days of redevelopment effort for the assemblies that already exist. You can simply reference existing assemblies and use already-developed business rules or functionality within your package. All this power and ease of use doesn’t come free, though the cost of these benefits is very minimal in this case. The code that you write in the Script task or Script component resides in the package and is not available to other packages. If you want to Chapter 11: Programming Integration Services 485 reuse the code, you will have to copy the script to other packages. To explain it further, when you deploy a package that has been developed using only the prebuilt objects to a different server, the code for prebuilt objects is not sent along with the code for your package. The prebuilt objects are available as a compiled binary library to the package within Integration Services, while the custom scripts obviously have not been published and hence are not available for use with other packages. You can copy the code quite easily to script objects in other packages if you need to; however, that will increase the maintenance cost. Think about the code that needs to be modified and has been used in hundreds of packages all over in the enterprise. It won’t be a welcome thing to do. The facility to script yourself out of a requirement should be used carefully and should be used where you know that the requirement is unique and will not be used in many packages. If that’s not the case, you’ll be better off with the custom-built extensions that have been developed from scratch by deriving from the base classes provided by the Integration Services object model. Developing Custom Objects from Scratch If you do not want to use scripting to extend your packages because the custom code might be used in multiple packages and you don’t want to undertake the hassle of fixing several packages later on, you can build custom extensions in the managed code from scratch. Using the managed object model of Integration Services, you can develop extensions such as control flow tasks, connection managers, log providers, enumerators, data flow sources, data flow transformations, and data flow destinations. To develop a custom object, you will inherit from the appropriate base class as provided for the functionality and build on that. For example, to develop a control flow task, you will inherit from the Task base class, and for a data flow component you will inherit from the PipelineComponent base class. The provision of a base class as a starting point makes it much easier to develop custom extensions. Once the object development has been completed, you will then build and deploy the object assembly into the appropriate Integration Services and global assembly cache (GAC). The object then can be added into the Toolbox within Visual Studio and can be used as any other prebuilt object. You will need to deploy the custom extensions on all the servers wherever you want to use them. For example, suppose a developer builds a package using a custom component that has been installed on his computer and wants to share this package with another team member. This new team member cannot use the package until he or she installs the custom component on his or her computer, as the component will be referenced locally on the computer. The availability of the custom extension in the SSIS designer makes it very easy to reuse it. As mentioned earlier, the code for the prebuilt components does not get copied into the package; rather, the package references the objects only. This also applies to the custom-built extensions, 486 Hands-On Microsoft SQL Server 2008 Integration Services and you do not need to worry about the deployment of the custom objects within your packages. The custom-built extensions are not deployed with your packages; rather, they are handled separately and keep your package deployments simple. This means that if you need to make an enhancement or a change into a custom extension, you do not need to modify all your packages, but will need to make change only in the custom extension and the packages will automatically pick up this changed object at their next run. Building Packages Programmatically When you want to work with your packages programmatically, the object model allows you to create, configure, load, and execute packages. You can create packages dynamically and define the sources, transformations, and metadata of the selected columns and destinations. Just to explain, think of a CRM application that you may want to extend with ETL capabilities so that you can create a reporting data mart. This CRM application is configured with different metadata for different clients, so you can create SSIS packages programmatically reading metadata of the deployed application from your application interfaces and avoid configuring SSIS packages manually for each client. Such extended applications can save you and the customer a lot of time and effort. Considering your requirements, you can create a grand application by creating packages from scratch, including package objects, you can simplify your solution by loading a template package and configuring it for the relevant changes, or else you can simply load and run an existing package. Extending Packages with Scripting Now that you’ve an overview of programming options, let’s explore them in detail and try some Hands-On exercises along the way as we proceed. I would like to clarify some of the concepts about my approach in this chapter before we get deep into the exercises. The focus will be to demonstrate on how you can implement your code in Integration Services rather than on how to write the code, and to keep things simple and within reason, only Visual Basic 2008 code will be listed. The Legacy Scripting Task: ActiveX Script Task If you have used DTS 2000, you might have used the ActiveX Script task to extend your Dts packages. This powerful task was provided in DTS 2000 and helped database developers to develop packages that otherwise wouldn’t be possible. Many database developers and information analysts have exploited this task to customize data transformation; apply business logic in the Dts package; manage files and folders; dynamically set properties on tasks, connections, or global variables; and perform Chapter 11: Programming Integration Services 487 complex computations on the data. To help smooth migration from DTS 2000 to SSIS, Microsoft provided the ActiveX Script task in SSIS to run those custom-build scripts until such time when the scripts can be upgraded to a more advanced scripting task, simply called the Script task in SSIS. The ActiveX Script task provided in SSIS is quite different than the one provided in DTS 2000 in look and feel. The basic purpose of the ActiveX Script task in SSIS is to allow you to run existing scripts, not develop new scripts. In fact, this task will be removed from future releases of Integration Services. Better not to use this task to develop new scripts, and opt instead for use of the more advanced and efficient Script task for new development work. Here are some of the benefits of using the Script task over the ActiveX Script task: e Script task uses a much powerful development environment—Visual Studio c Tools for Applications—which provides an integrated development environment (IDE) rich in features such as IntelliSense, color-coded syntax highlighting, line- by-line debugging support, and online help. It is easier to develop scripts in the Script task using either Visual Basic 2008 c or Visual C# 2008, both of which are fully capable of referring external .NET assemblies in addition to .NET Framework classes and libraries. All the scripts developed in the Script task (and in Script component) are precompiled c and hence, yield enhanced performance due to fast execution at run time. If you have to use this task to use an existing ActiveX script, follow these steps: 1. Drop the ActiveX Script task on the Designer surface and double-click it to configure it. 2. Specify a Name and a Description for the task in the General page. 3. In the Script page Language field drop-down list, choose a scripting language that was used to write the ActiveX script. The default choices available are the VB Script Language and the JScript Language, though the ActiveX Script task can support other scripting languages, depending on the scripting engines installed on the local computer. 4. The Script field provides a simple interface where you can paste or type in your ActiveX script. If you have an ActiveX script saved into a file, you can click Browse and select the file, and your script will be read in by the task and shown in the Script field. Click Save to save the contents of the Script field to a file, and click Parse to parse the script. 5. The EntryMethod specifies the name of the method that is called from the ActiveX Script task at run time. . Component c Summary 482 Hands-On Microsoft SQL Server 2008 Integration Services B y now you have worked with almost all the preconfigured tasks and components that Integration Services provides and. Excel destination. Figure 10-32 Executing Removing Duplicates package 480 Hands-On Microsoft SQL Server 2008 Integration Services e remaining seven records are then passed on to Fuzzy Grouping. 478 Hands-On Microsoft SQL Server 2008 Integration Services 30. Drop an Excel Destination just below the Splitting Fuzzy Grouped