ptg 2144 CHAPTER 52 SQL Server Integration Services (location) of the package to be signed. Hash specifies a certificate identifier expressed as a hexadecimal string of varying length. . /~SourceP[assword] Password—This option provides the password used with the SQL and SOURCEUSER options to connect to a SQL Server instance that uses SQL Server authentication. . /~SourceS[erver] Server—This option provides the name of the server where the package is to be stored. . /~SourceU[ser] User Name—This option provides the SQL Server username to use to access the SSIS package. . /~SQ[L] PathName—This option specifies the path (location) of the SSIS package stored in the msdb database. Next, let’s look at various examples of running dtutil. dtutil Examples The following example copies an existing package in SQL to the SSIS package store: C:> Dtutil /SQL ExistingPackage /COPY DTS;destPackage The following example copies an existing package from one location on the filesystem to another location on the filesystem: C:> dtutil /FILE c:\Unleashed\HotCustomersPlus.dtsx /COPY FILE;c:\UnleashedProduction\HotCustomersPlus.dtsx The following example creates a new GUID (usually after you copy a package): C:> dtutil /I /FILE HotCustomersPlus.dtsx The following example deletes a package stored in the local server (msdb database): C:> dtutil /SQL HotCustomersPlus /SOURCEUSER PBertucci /SOURCEPASSWORD xyz /DELETE The following example deletes a package stored in the filesystem: c:> dtutil /FILE c:\UnleashedProduction\HotCustomersPlus.dtsx /DELETE The following example verifies whether a package exists in a local server (msdb database): C:> dtutil SQL HotCustomersPlus /SOURCEUSER Pbertucci /SOURCEPASSWORD xyz /EXISTS The following example verifies whether a package exists on the local filesystem: C:> dtutil /FILE c:\UnleashedProduction\HotCustomersPlus.dtsx /EXISTS ptg 2145 Connection Projects in Visual Studio 52 The following example moves a package from one server (msdb database) to another server ( msdb database): C:> dtutil /SQL HotCustomersPlus /SOURCEUSER Pbertucci /SOURCEPASSWORD xyz /MOVE SQL;HotCustomersPlus /DESTUSER sa /DESTPASSWORD zwx The following example moves a package from one filesystem location to another: c:> dtutil /FILE c:\Unleashed\HotCustomersPlus.dtsx /MOVE FILE;c:\UnleashedProduction\HotCustomersPlus.dtsx The following example signs a package on the filesystem: dtutil /FILE c:\Unleashed\HotCustomersPlus.dtsx /SIGN FILE; c:\Unleashed\HotCustomersPlus.dtsx;987377773999af33df399999333 Connection Projects in Visual Studio Microsoft is trying to make it easier to get data sources or data targets defined and ready to utilize with SSIS packages. From Visual Studio, you can now create an Integration Services Connection project. A wizard takes you through defining the data sources and connections for an SSIS package. As you can see in Figure 52.44, this wizard allows you to jumpstart the data source/connection managers you might need for a new SSIS package. FIGURE 52.44 The Visual Studio Integration Services Connection Project Wizard. ptg 2146 CHAPTER 52 SQL Server Integration Services You can then step through the buildup and testing of the connection one connection manager at a time. Figure 52.45 shows the various data providers available through this wizard and the creation and testing of one to SQL Server. After you finish using this wizard, you are set to start defining your SSIS package with the data sources and connection managers ready for use in your data flows, as you can see in Figure 52.46. FIGURE 52.45 Select Data Providers Integration Services Connection project wizard. FIGURE 52.46 The Visual Studio Designer with connections ready for use. ptg 2147 Using bcp 52 Change Data Capture Addition with R2 With SQL Server 2008 R2, Microsoft has introduced some additional capability around grabbing changes from tables as a part of SSIS. A series of new adapters are now available from Attunity; they have implemented a streaming agent that detects changes in several different database platforms and makes these changes available to SSIS package process- ing. Attunity CDC products for SQL Server Integration Services provide end-to-end opera- tional data replication (ODR) solutions for many heterogeneous sources. Using these products, you can enable efficient and real-time data warehousing, set operational busi- ness intelligence requirements, replicate data from one source to another, and synchro- nize two or more data sources to ensure data consistency. Attunity CDC Suite for SSIS includes products for the following: . SQL Server (Attunity CDC Accelerator) . Oracle (Attunity Oracle-CDC for SSIS) . DB2 on IBM iSeries (Attunity iSeries DB2-CDC for SSIS) . DB2 on IBM Mainframe (Attunity z/OS DB2-CDC for SSIS) . SQL/MP on HP NonStop (Attunity SQL/MP-CDC for SSIS) . Nonrelational legacy data sources (Attunity CDC Accelerator) These adapters in SSIS allow you to feed the deltas (changes) to any target data platform as opposed to full refreshes—very much like data replication approaches but via SSIS package control. Using bcp As you have seen in this chapter, it is fairly easy to create and implement SSIS packages to do data transformations from one or more data sources to one or more data destinations. However, many organizations still really just need a vanilla and very fast mechanism to export data out of SQL Server or import data into SQL Server. bcp fills this need well (and has done so from the beginning of SQL Server). The following sections outline the primary execution methods of bcp, the many switches of bcp, the format file, and ways to improve performance when using bcp. By the end of these sections, you will be able to optimally execute bcp successfully for several common production scenarios. Microsoft has added a new execution switch that generates an XML format file. Most other features of bcp have remained the same. First, you need to see whether you have the right version of bcp. A quick check of your version of bcp guarantees that you won’t run into any limitations from older versions of bcp that might be left on your servers. You can do this by executing bcp at the command prompt with the –v option and no other parameters. (Note that all bcp switch options are case sensitive; for example, –v and –V are two very different switches.) Here’s an example: C:> bcp –v ptg 2148 CHAPTER 52 SQL Server Integration Services BCP - Bulk Copy Program for Microsoft SQL Server. Copyright (C) Microsoft Corporation. All Rights Reserved. Version: 10.0.1600.22 This is version 10.0, which is distributed with MS SQL Server 2008 (SQL Server 10.0). If a version other than 10.x is present here, you must re-install bcp immediately. At any time, you can see the proper usage and bcp switch options available by executing bcp at the command prompt with a question mark (?): C:> bcp ? usage: bcp {dbtable | query} {in | out | queryout | format} datafile [-m maxerrors] [-f formatfile] [-e errfile] . . . You use the following syntax for bcp, along with one or more switches: bcp {dbtable | query} {in | out | queryout | format} datafile In this syntax, dbtable is the database_name, schema, and table_name | view_name (for example, AdventureWorks2008.Production.Product or ”AdventureWorks2008.Production.Product”): . database_name—This is the name of the database in which the specified table or view resides. If not specified, this is the default database for the user. . owner—This is the name of the schema of the table or view. . table_name | view_name—This is the name of the destination table or view when copying data into SQL Server ( in), and it is the name of the source table when copy- ing data from SQL Server ( out). query is a T-SQL query that returns a result set. queryout must also be specified when bulk-copying data from a query. in | out | queryout | format specifies the direction of the bulk copy (in copies from a file in to the database table or view, out copies from the database table or view to a file). queryout must be specified when bulk-copying data from a query. format creates a format file based on the switch specified ( –n, –c, –w, –V, or –N) and the table or view delimiters. If format is used, the –f option must be specified as well. data_file is the data file used when bulk-copying a table or view into or out of SQL Server. All the available bcp switches are listed in Table 52.1. ptg 2149 Using bcp 52 TABLE 52.1 bcp Switches Switch Description Example -m Specifies the maximum number of errors to allow before stopping the transfer. The default is 10. [–m max_errors] -f Specifies the format file used to customize the load or unload data in a specific style. [-f format_file] -e Specifies the file to write error messages to. [-e err_file] -F Specifies the first row in the data file to start copying from when importing. The default is 1. [-F first_row] -L Specifies the last row in the data file to end copying with when importing. The default is 0, which indicates the last row in the file. [-L last_row] -b Specifies the number of rows to include in each commit- ted batch. By default, all data rows in a file are copied in one batch. [-b batch_size] -n Specifies that native (database) data type formats are to be used for the data. [-n] -c Specifies that character data type format is to be used for the data. In addition, \t (tab character) is used as the field separator, and \n (newline character) is used as the row terminator. [-c] -w Specifies that the Unicode data type format is to be used for the data. In addition, \t (tab character) is used as the field separator, and \n (newline character) is used as the row terminator. [-w] -N Specifies to use Unicode for character data and native format for all others. This can be used as an alternative to the –w switch. [-N] -V Specifies to use data type formats from earlier versions of SQL Server. [-V (70 | 80|90)] -q Tells bcp to use quoted identifiers when dealing with table and column names. [-q] -C If you are loading extended characters, allows you to specify the code page of the data in the data file. [-C code_page] -t Specifies the terminating character(s) for fields. The default is \t (tab character). [-t field_term] ptg 2150 CHAPTER 52 SQL Server Integration Services TABLE 52.1 bcp Switches Switch Description Example -r Specifies the terminating character(s) for rows. The default is \n (newline character). [-r row_term] -i Specifies a file for redirecting input into bcp (the response file containing the responses to the command prompts). [-i input_file] -o Specifies the file for receiving redirected output from bcp. [-o output_file] -a Specifies the network packet size (in bytes) used to send to or receive from SQL Server. Can be between 4,096 and 65,535 bytes. The default size is 4,096. [-a packet_size] -S Specifies the SQL Server name to connect to. Local is the default. [-S server_name | server_name\instance_ name] -U Specifies the user account to log in as; this account must have sufficient privileges to carry out either a read or a write of the table. [-U login_id] -P Specifies the password associated with the user account. [-P password] -T Makes a trusted connection to the server, using the network user/security credentials instead of the login_id/password. [-T] -v Displays the bcp version information. [-v] -R Uses the regional format for currency, date, and time data, as defined by the locale settings of the client computer. [-R] -k Overrides a column’s default and enforces NULL values being loaded into the columns as part of the bcp opera- tion. [-k] -E Uses the identity values in the import file rather than generating new ones. [-E] -h Specifies special hints to be used during the bcp opera- tion. They include the following: the sort order of the data file, number of rows of data per batch, number of kilobytes of data per batch, whether to acquire a table- level lock, whether to check constraints, and whether to fire insert triggers. [-h hint_type, ] ptg 2151 Using bcp 52 Fundamentals of Exporting and Importing Data One of the great things about bcp is its ease of use. This section runs through a couple simple examples and provides full explanations. All tables used here can be found in the AdventureWorks2008 sample database supplied by Microsoft in SQL Server 2008. Let’s start by exporting product data from AdventureWorks2008 that may be needed by a sales team for reference in Excel format (a .csv file). To do this, you simply export the Product table data into a comma-delimited file. You need to specify the following with bcp in this case: . The full table name (in this case, AdventureWorks2008.Production.Product) . The direction of bcp (OUT in this case because it is exporting data out) . Data filename to hold the exported data (in this case, products.dat) . The server name DBARCH-LT2\SQL08DE01 for this example (in this case, –S DBARCH- LT2\SQL08DE01) . The username SA (in this case, –U sa) . The password (in this case, –P xyz) . A comma as the column delimiter (in this case, –t “,”) . That this should be exported in character data format (in this case, –c) At the command prompt, you execute the following: C:> BCP AdventureWorks2008.Production.Product OUT products.dat –S DBARCH-LT2\SQL08DE01 –U sa –P xyz –t “,” –c Starting copy 504 rows copied. Network packet size (bytes): 4096 Clock Time (ms.): total: 10 Average: (50400.00 rows per sec.) Here’s a sample of the data in the Products.dat file that was just exported: 1,Adjustable Race,AR-5381,0,0,, 2,Bearing Ball,BA-8327,0,0,, 3,BB Ball Bearing,BE-2349,1,0,, 4,Headset Ball Bearings,BE-2908,0,0,, . . . TABLE 52.1 bcp Switches Switch Description Example -x Generates an XML format file. [-x] ptg 2152 CHAPTER 52 SQL Server Integration Services Now let’s look at importing data into SQL Server 2008. Let’s say that each salesperson is providing a flat file that contains his or her new sales orders summaries. These files are emailed to a person in the ordering department and need to be imported into SQL Server every week. The file that you will import will be a comma-delimited file ( .csv) that the salesperson created using Excel. The new entries will be the salesperson’s hottest customer’s totals. A sample input data file (named BCPHotCustomersPlus.csv) is included on the book’s CD for this chapter. You need to specify the following with bcp in this case: . The full table name (in this case, AdventureWorks2008.Sales.HotCustomersPlus). . The direction of bcp (IN in this case because it is importing data). . The names of the data files that contain the import data (in this case, BCPHotCustomersPlus.csv). The following is a sample of the input data file ( BCPHotCustomersPlus.csv): 268,Cycle Parts and Accessories,S,9,Australia,954,Touring-1000 Yellow,7152.210000 268,Cycle Parts and Accessories,S,9,Australia,955,Touring-1000 Yellow,7152.210000 268,Cycle Parts and Accessories,S,9,Australia,956,Touring-1000 Yellow,5721.768000 268,Cycle Parts and Accessories,S,9,Australia,968,Touring-1000 Blue,5721.768000 . . . . The server name (in this case, –S DBARCH-LT2\SQL08DE01) . The username SA (in this case, –U sa). . The password (in this case, –P xyz). . A comma as the column delimiter (in this case, –t “,”). . That this should be exported in character data format (in this case, –c). . The –q option (in this case, –q), to be sure quoted identifiers are handled properly. At the command prompt, you execute the following: C:> BCP AdventureWorks2008.Sales.HotCustomersPlus IN BCPHotCustomersPlus.csv –S DBARCH-LT\SQL08DE01 –U sa –P xyz –t “,” –c –q Starting copy 4 rows copied. Network packet size (bytes): 4096 Clock Time (ms.): total : 241 Average : (99.59 rows per sec.) A quick SELECT * from the HotCustomersPlus table shows the success of this operation: CustomerID CustomerName CustomerType TerritoryID TerritoryName 268 Cycle Parts and Accessories S 9 Australia 954 268 Cycle Parts and Accessories S 9 Australia 955 ptg 2153 Using bcp 52 268 Cycle Parts and Accessories S 9 Australia 956 268 Cycle Parts and Accessories S 9 Australia 968 . . . The sales team can now send in their sales orders as they make sales. This brief example illustrates the beauty and power of using bcp. The following sections look at how bcp can work with basic data representations (charac- ter, native, or Unicode), the use of a format file, and a few other extended bcp capabilities. File Data Types bcp can handle data in one of three forms: character (ASCII), native, or Unicode. You have the choice of which character format is used, depending on the source or destination of the data file: . The character format ( –c) is the most commonly used of the three data types because it reads or writes using ASCII characters and carries out the appropriate data type conversion for the SQL Server representations. The CHAR data type is the default storage type; it uses tabs as field separators and the newline character as the row terminator. . The native format ( –n) is used for copying data between servers. This format allows bcp to read and write using the same data types used by server, which results in a performance gain. This format does, however, render the data file unreadable by any other means. . The Unicode option ( –w) uses Unicode characters rather than ASCII characters. The NCHAR data type is the default storage type; it uses tabs as field separators and the newline character as the row terminator. Format Files By using a format file, you can customize the data file created by bcp or specify complex field layouts for data loads. There are two ways to create a format file: by using interactive bcp and by using the format switch. Customizing a Format File by Using Interactive bcp If you do not specify one of the –n, –c, or –w data type format switches, bcp (in or out) prompts you for the following information for each column in the data set: . File storage type . Prefix length . Field length . Field terminator bcp offers a default for each of these prompts that you can either accept or reject. If you accept all the defaults, you wind up with the same format file you would have by specify- ing the native format (with the –n switch). The prompts look like this: . used with the SQL and SOURCEUSER options to connect to a SQL Server instance that uses SQL Server authentication. . /~SourceS[erver] Server This option provides the name of the server where the package. connections ready for use. ptg 2147 Using bcp 52 Change Data Capture Addition with R2 With SQL Server 2008 R2, Microsoft has introduced some additional capability around grabbing changes from tables. very fast mechanism to export data out of SQL Server or import data into SQL Server. bcp fills this need well (and has done so from the beginning of SQL Server) . The following sections outline