Hands-On Microsoft SQL Server 2008 Integration Services part 20 ppt

168 Hands-On Microsoft SQL Server 2008 Integration Services If you have chosen ADO or ADO.NET Connection Manager in the ConnectionType field earlier, the IsQueryStoredProcedure field becomes available for use and you will be able to specify that the SQL statement to be run is a stored procedure. When you select OLE DB Connection Manager in the ConnectionType field, you can specify in the BypassPrepare field whether the task should skip preparing the query step before running it. When SQL queries are run, the SQL statement is parsed and compiled in the first step and then executed based on the execution plan prepared earlier in the compilation step. By default the BypassPrepare field is set to True, which means when the query is run the first time, it parses and compiles the SQL statement in the first step and keeps the execution plan in the cache of the database engine and in the next step uses this execution plan to execute the query. So when the query is run the next time, it doesn’t parse and compile the statement but rather uses the existing execution plan, saving time required to parse and compile. If you’re running a query multiple times, it is faster and efficient if you use a prepared execution—however, if you’re running the query only once, it is not a recommended option. Parameter Mapping Page If you are running a parameterized SQL statement, you can map variables to the parameters in this page. The interface on this page is relatively simple and provides self-explanatory fields and options. You can click Add to add a parameter mapping and then click in each field to select the available values from the drop-down lists; however, in the Parameter Name field, you have to type in a value. The configuration of this page has been covered in the “Using System Variables to Create Custom Logs” Hands-On exercise in Chapter 3. Result Set Page If you are using a query that will return a result set and you have selected a value other than None in the ResultSet field on the General page, you can use this page to map the row set to a variable, which can be used later in the package. Expressions Page Using this page, you can build a property expression using SSIS expression language to update the property dynamically at run time. In the Expression Builder, you can use System or User variables, functions, type casts, and operators exposed by expression language to build an expression. Chapter 5: Integration Services Control Flow Tasks 169 Bulk Insert Task When you get large amounts of data in the flat files, say from the mainframe systems or from third parties, and you want to import this data from the flat files into an SQL Server, you can use the Bulk Insert task. This task is the quickest way to copy large amounts of data into an SQL Server table or view. Note the following when deciding whether to use this task: e Bulk Insert task does not perform any validation or transformation on the c data while importing data from the source text file to the SQL Server database table or view. So, it is more suitable in cases where the data is coming from a reliable source or you intend to apply business processes in later stages. A similar component—SQL Server Destination—is provided in the data flow task that can provide the performance of a bulk insert. You will be using SQL Server Destination if you want to perform transformations on data before loading and your target server is the local server. SQL Server Destination is covered in Chapter 9 in detail. When you embed this task in an SSIS package Control Flow, only the members c of the sysadmin fixed server role can run the package. You can use an XML- or non-XML-formatted file in the Bulk Insert task. c A formatted file is used to store format information for each field in a data file in relationship to a specific table and provides all the format information that is required to bulk-import data. is format file must be located on the server executing the SSIS package containing the Bulk Insert task. During the import process, some of the rows may fail to import and in turn can fail c the task. You can still run this task successfully by increasing the allowed number of errors by specifying a value in the MaxErrors option. However, the rows that fail to import cannot be logged or extracted out. If you want to capture the failing rows, you need to consider alternative ways of importing data using the Data Flow task and capture the failing rows in the exception files by way of error outputs. Message Queue Task Microsoft Message Queuing (MSMQ) is a Windows service that allows applications or SSIS packages to send and receive messages using a queue. Message queuing is available in Windows 2000 and later released as a standard operating system component that has the advantages of Active Directory Integration. This service can be extended using the Message Queuing Connector to heterogeneous data sources such as CICS (IBM’s Customer Information Control System) or UNIX. 170 Hands-On Microsoft SQL Server 2008 Integration Services In the SSIS world, message queuing means one package will be sending a message to a queue while another package will be receiving that message from the queue. These packages can be on different servers and may not be running simultaneously. Using MSMQ, you can reliably exchange data between packages that may not be running on the same server and might be separated in time throughout your enterprise. Following are some of the scenarios in which you may be using the Message Queue task: Send a message from an executing package to the package that is waiting for it to c complete so that it can start running. If you have a small window of time to finish a large workload and you decide to c distribute your workload across many servers to utilize the processing power, you can coordinate the operations on different servers using message queuing between packages. SSIS packages can communicate with the applications that utilize message c queuing. Send output from a processing package to a waiting package on the other computer, c where the data enclosed in the message will be processed further. You can configure this task into either send mode or receive mode. Select Send Message in the Message field to configure it into send mode or select Receive Message in the Message field to configure it into receive mode. The package that is running the message queue task in the receive mode can be configured to either run and wait for the message in the queue using a looping construct or run on a schedule that is later than when the message is dropped in the queue just to pick up the message. SSIS is more suitable for the applications that can be satisfied with the latter implementation. If you need to wait for messages, for example, your application is sending data in message format as the transactions happen, Biztalk is more suitable for such implementations. Biztalk server is designed to support such requirements and puts negligible load on the server if it has to wait for messages, whereas SSIS will consume resources, as it has to keep looping and stay in memory. The message queue task is provided in SSIS to transfer the information that is relevant for the remaining ETL to complete rather than read or load data from a transactional system. You can choose from one of the different types of messages to send or receive using the Message Queue task: Data file message, String message, or Variable message. When the task is configured in the receiving mode, you also get an additional String message to variable MessageType choice. As in other Integration Services tasks, the Chapter 5: Integration Services Control Flow Tasks 171 available fields in this task are dynamic and change depending upon your choice to send or receive a message or on message type. Data file message c Used to send or receive a file that contains the message. To send a data file message, you specify Data File Message in the MessageType field and specify the path of the file in the DataFileMessage field on the Send page of the Message Queue Task Editor. When receiving this type of messages, you select Data File Message in the MessageType field and specify the name of the file in the SaveFileAs field into which you want to save the message. Also, you can choose to overwrite the existing file, and optionally apply a filter to receive the message only from the specified package. You specify the full path and package name in the Identifier field on the Receive page. Variable message c Used to send or receive one or more variables. To send variables, you specify Variable Message in the MessageType field and one or more variables in the VariableMessage field on the Send page of the Message Queue Task Editor. While receiving the variables, you specify Variable Message in the MessageType field, specify the name of the variable in the Variable field to receive the message into, and optionally choose to apply a filter to receive the message only from the package specified in the Identifier field on the Receive page. String c message Used to send a text string. To send a text string, you specify String message in the MessageType field and type a text string in the StringMessage field on the Send page of the Message Queue Task Editor. While receiving the text string, you specify String message in the MessageType field and optionally specify to compare the incoming string in the Compare field with a user-defined string specified in the CompareString field on the Receive page. e string comparison options in the Compare field can be Exact Match for an exact comparison, Ignore Case for case-insensitive comparison, or Containing for a substring match. String message to variable c Used to pass the source message that has been sent as a string to a destination variable and is available only when receiving messages. To configure a text string message to be passed to a variable, you specify String Message To Variable in the MessageType field, specify the name of the variable in the Variable field to receive the text string into, and optionally specify to compare the incoming string in the Compare field with a user-defined string specified in the CompareString field on the Receive page. e string comparison options in the Compare field can be Exact Match for exact comparison, Ignore Case for case-insensitive comparison, or Containing for a substring match. 172 Hands-On Microsoft SQL Server 2008 Integration Services To use the Message Queue task to send and receive messages, you first need to install the Message Queuing service and then create messaging queues. You can create either a public or a private queue, depending upon whether you have installed the Active Directory Integration component of the Message Queuing service. A public queue is created in an Active Directory environment to publish its properties and description to the Active Directory. A private queue does not publish itself to the Active Directory but works on the local computer that holds the queue. The Message Queuing service has the following components available in Windows 2008 Server for installation: Message Queuing Server c is component enables you to perform Message Queuing functions such as guaranteed message delivery, efficient routing, improved security, support for sending messages within transactions, and priority-based messaging. It can be used to implement both synchronous and asynchronous messaging solutions. Directory Service Integration c Provides integration with Active Directory whenever the computer belongs to a Windows domain (Windows 2000 and later). With this component, you can publish queue properties to the directory, authenticate and encrypt messages using digital certificates registered in the directory, and route messages across directory sites. Public queues are configured and used when using this component. Message Queuing Triggers c Associates the arrival of messages at a queue with triggering functionality of a COM component or a standalone application, depending upon the filters that you define for the incoming messages in a given queue. HTTP Support c Enables sending or receiving of messages over HTTP transport with proxy settings configured using the proxycfg.exe tool. Multicasting support c Enables queuing and sending of multicast messages to a multicast IP address. Routing Service c Available in Windows 2008 Server to provide routing of messages using the store and forward mechanism in Active Directory transport between different sites or within the same site. Windows 2000 Client Support c Available in Windows 2008 Server to provide support for Message Queuing clients on Windows 2000 computers in the domain. Message Queuing DCOM Proxy c Enables a computer to act as a DCOM client of a remote Message Queuing Server. Chapter 5: Integration Services Control Flow Tasks 173 For proper functioning of the Message Queue task, make sure you have installed the Message Queuing Server and the SQL Server Integration Services service. When you install SQL Server 2008 without specifically selecting the Integration Services service, you may still be able to use BIDS to design and run Integration Services packages. However, as this is a partial installation of SSIS, not all tasks will run properly, and the Message Queue task is one of those that won’t. For the Message Queue task to be functioning properly, you must install Integration Services fully during installation of SQL Server 2008. Next, you will import the expanded files into the Campaign database in SQL Server using the Message Queue task and the Bulk Insert task in the following Hands-On exercises. Hands-On: Importing Expanded Files Dealers of Adventure Works Bikes submit the sales reports to an FTP server, and they need to be downloaded from this server and imported into the SQL Server database. You have already downloaded, expanded, and archived these report files in the previous exercises, and now you have DealerSales01.txt and DealerSales02.txt files in the Downloads folder ready to be imported to the Campaign database. However, you want to run this new package independent of earlier packages, and this may occur at a different time during the day. Method We can meet this objective in several ways: We can use the Bulk Insert task to import the data to our existing table—but we have multiple files, so we can use the Foreach Loop Container to read the filenames one by one and pass them to the Bulk Insert task to import multiple files one at a time. However, as we want to keep the packages independent from one another and want to run the second package only when the first has completed successfully, the use of messages from the first package to pass the information to the other will be a better solution. Though in this case you will not be following the shortest and easiest solution option to achieve your goal, the solution you are going to follow is quite interesting, and I’m sure it will be relevant in real-life scenarios. You will add the Message Queue task at the end of the Archiving Downloaded Files package to send filenames in the messages, which will be read by the Importing Expanded Files package using the Message Queue task in the receiving mode. As the package won’t have to read filenames from the file system, you will not be using 174 Hands-On Microsoft SQL Server 2008 Integration Services the Foreach Loop Container; instead, you will use the For Loop Container to read multiple messages one by one. Here’s the step-by-step method: 1. Install the Message Queuing service and create message queues. 2. To keep it simple and applicable to most of the users, for the sake of this exercise I’ve used a Windows Server 2008 machine with a private queue only. If you’ve a different environment, refer to Microsoft SQL Server 2008 Books Online for more details to create a message queuing environment suitable to your requirements. 3. Configure the Archiving Downloaded Files package to send filenames. In this step, we will add Message Queue task in the already created Archiving Downloaded Files package. 4. Build the Importing Expanded Files package. And the final step is to execute this package and check that the two messages have been picked up from the queue and the text files have been imported into the SQL Server Campaign database. Exercise (Install Message Queuing Service) Let us start this exercise by installing the Message Queuing service. You will also create message queues while learning about private and public queues. 1. From the Server Manager, click the Add Features link to start the Add Features Wizard. 2. In the Select Features dialog box, expand the Message Queuing group and select the Message Queuing Server from the list. Click Next to go to Confirm Installation Selections page. 3. Verify the selection and click Install to install the Message Queuing Server component. 4. When the installation is completed successfully, click the Close button in the Results page to close the wizard. 5. Again in the Server Manager, expand the Features node in the left pane of the window to see the installed Message Queuing feature. 6. Right-click the Private Queues folder and choose New | Private Queue from the context menu. This will open the New Private Queue window. In the Queue name field, type SSISprivQ and then click OK. Note that another difference between a private queue and a public one is that the private$ is attached to the full name of a private queue. The New Private Queue window shows that the queue will be created in your computer name with private$ added in front of the name you type in. So, the full path of the queue will be ComputerName\private$\SSISprivQ. If this were the public queue in which we specify the queue name as SSISpubQ, the full path would have been ComputerName\SSISpubQ. Chapter 5: Integration Services Control Flow Tasks 175 7. Expand the Private Queues folder under the Message Queuing from the left pane of the Computer Management window. Further expand the SSISPrivQ folder and you will see a Queue messages subfolder. This is where the Message Queue task will be delivering the messages and the messages will wait for a receiving Message Queue task to pick them up. Exercise (Configure Archiving Downloaded Files Package to Send Filenames) Remember that in the Archiving Downloaded Files package, the Foreach Loop Container reads the filenames of zipped files lying in the C:\SSIS\downloads folder and then passes those names to the File System task, which moves the zipped files to the Archive subfolder. To accomplish this, the package uses a variable named fname to pass the zipped files names. Because you want to import text files, not the zipped files, you will derive a new variable called txtfname to contain the text files names from fname variable. This new variable will then be sent through the Message Queue task as a variable message. 8. Run BIDS and open the Control Flow Tasks project. In the Solution Explorer window, double-click the Archiving downloaded files.dtsx package to open it. 9. Open the Variables window. Click the Auto Hide pushpin to dock the Variables window on the left side of the screen. On the Control Flow surface, click the Enumerating Files Foreach Loop Container and switch to the Variables window. Add a new variable named txtfname in the Enumerating Files Scope and set Data Type to string. Press  to open the Properties window. The Properties window will open up by default on the right side of the screen, showing properties of the txtfname variable. (Sometimes you may not see properties for the object in which you are interested—this is because of your selection of the items on the designer. If this happens, make sure you’ve selected the Enumerating Files Container and then clicked the txtfname variable to see the properties for this variable.) Scroll through the properties and locate the EvaluateAsExpression property. This property allows you to enable the variable to use the results of an expression as its value. That is, you can write an expression, which will be evaluated at run time and the result of this evaluation will be used as the value of the variable. Select True for the EvaluateAsExpression property. Type the following text in the Expression property: SUBSTRING(@[User::fname] , 1, LEN(@[User::fname]) -3) + "txt" This expression will evaluate txtfname as C:\SSIS\downloads\DealerSales01.txt from the fname variable when it equals C:\SSIS\downloads\DealerSales01.zip. If you look further down in the properties window in the Value field, you will notice 176 Hands-On Microsoft SQL Server 2008 Integration Services (see Figure 5-9) that the full filename in fact has been resolved as txt. You can see that nothing appears in the name portion of the file path name because the fname variable has not yet been populated (by the Foreach Loop Container) in the design mode. However, at run time, the Foreach Loop Container provides a value for the fname variable and the name portion of the filename gets populated in the file path name. 10. Drag the Message Queue task from the Toolbox and drop it in the Enumerating Files (Foreach Loop) Container below the Archive Downloaded Files (File System) task. Drag and drop the green arrow, the precedence constraint, from below the Archive Downloaded Files task on to the Message Queue task. The package will look as shown in Figure 5-10 after you’ve configured the Message Queue task. 11. Right-click the Message Queue task and choose Edit from the context menu. Type the following on the General Page of the task editor: Name Send variable message Description This task sends the txtfname variable to the Importing expanded files. Figure 5-9 Deriving the txtfname variable using property expressions Chapter 5: Integration Services Control Flow Tasks 177 If you want to send this message in the format that is acceptable to SQL Server 2000 Message Queue task, you can use the Use2000Format field. For now, leave the default False value set in this field. 12. In the MSMQConnection field, select to create a new connection. In the MSMQ Connection Manager Editor, type the following: Name SSISprivQ Description Connection Manager for private Queue SSISprivQ Path YourComputerName\private$\SSISprivQ You can use a dot (.) instead of YourComputerName to indicate your local computer. Figure 5-10 Archiving the downloaded files package after adding Message Queue task . Integration Services service. When you install SQL Server 200 8 without specifically selecting the Integration Services service, you may still be able to use BIDS to design and run Integration Services. for case-insensitive comparison, or Containing for a substring match. 172 Hands-On Microsoft SQL Server 200 8 Integration Services To use the Message Queue task to send and receive messages, you. 168 Hands-On Microsoft SQL Server 200 8 Integration Services If you have chosen ADO or ADO.NET Connection Manager in the ConnectionType