Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Timothy Peterson 201 West 103rd St., Indianapolis, Indiana, 46290 USA Microsoft SQL Server 2000 Data Transformation Services 00 0672320118 fm 11/13/00 5:02 PM Page i Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Microsoft SQL Server 2000 Data Transformation Services Copyright © 2001 by Sams Publishing All rights reserved. No part of this book shall be reproduced, stored in a retrieval system, or transmitted by any means, electronic, mechanical, photo- copying, recording, or otherwise, without written permission from the pub- lisher. No patent liability is assumed with respect to the use of the information contained herein. Although every precaution has been taken in the preparation of this book, the publisher and author assume no responsibility for errors or omissions. Nor is any liability assumed for damages resulting from the use of the information contained herein. International Standard Book Number: 0-672-32011-8 Library of Congress Catalog Card Number: 00-103543 Printed in the United States of America First Printing: December 2000 03 02 01 00 4 3 2 1 Trademarks All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized. Sams Publishing cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark. Microsoft is a registered trademark of Microsoft Corporation. SQL Server is a trademark of Microsoft Corporation. Warning and Disclaimer Every effort has been made to make this book as complete and as accurate as possible, but no warranty or fitness is implied. The information provided is on an “as is” basis. The author and the publisher shall have neither liability nor responsibility to any person or entity with respect to any loss or damages aris- ing from the information contained in this book or from the use of programs accompanying it. ACQUISITIONS EDITOR Sharon Cox DEVELOPMENT EDITOR Kevin Howard EXECUTIVE EDITOR Rosemarie Graham MANAGING EDITOR Charlotte Clapp PROJECT EDITOR Elizabeth Finney COPY EDITOR Sean Medlock INDEXER Tina Trettin PROOFREADER Tony Reitz TECHNICAL EDITORS Todd Meister Ivan Oss TEAM COORDINATOR Pamalee Nelson MEDIA DEVELOPER JG Moore INTERIOR DESIGNER Anne Jones COVER DESIGNER Anne Jones 00 0672320118 fm 11/13/00 5:02 PM Page ii Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Overview PART I Getting Started with DTS 1A Quick Look at DTS 5 2 DTS Enhancements for SQL Server 2000 41 3 DTS and the Flow of Data Through the Enterprise 59 4 Using DTS to Move Data into a Data Mart 77 P ART II DTS Connections and the Data Transformation Tasks 5 DTS Connections 101 6The Transform Data Task 125 7Writing ActiveX Scripts for a Transform Data Task 179 8The Data Driven Query Task 213 9The Multiphase Data Pump 227 10 The Parallel Data Pump Task 247 P ART III Other Data Movement and Manipulation Tasks 11 The Bulk Insert Task 267 12 The Execute SQL Task 291 13 The Copy SQL Server Objects Tasks 309 14 The File Transfer Protocol (FTP) Task 325 15 The Transfer Database and Other Transfer Tasks 333 P ART IV Control Tasks 16 Writing Scripts for an ActiveX Script Task 345 17 The Dynamic Properties Task 365 18 The Execute Package Task 377 19 The Message Queue Task 391 20 The Send Mail Task 405 21 The Analysis Services Tasks 411 22 The Execute Process Task 425 00 0672320118 fm 11/13/00 5:02 PM Page iii Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. PART V DTS Packages and Steps 23 The DTS Package and Its Properties 435 24 Steps and Precedence Constraints 469 25 Rapid Development with the Copy Database Wizard and the DTS Import/Export Wizard 501 26 Managing Packages with Visual Basic and Stored Procedures 525 27 Handling Errors in a Package and Its Transformations 553 28 High-Performance DTS Packages 565 29 Integrating DTS with Meta Data Services 587 P ART VI Extending the Power of DTS 30 Programming with the DTS Object Model 611 31 Creating a Custom Task in VB 633 32 Creating a Custom Transformation with VC++ 659 Index 679 00 0672320118 fm 11/13/00 5:02 PM Page iv Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. Contents Introduction 1 Using DTS in SQL Server 7.0 and SQL Server 2000 1 The Code Samples 1 Keeping Current with Information on DTS 2 PART I Getting Started with DTS 1A Quick Look at DTS 5 A High-Performance Data Pump—and a Whole Lot More 6 Using the DTS Designer to Create Complex Data Transformations 7 The Structure of a DTS Package 8 Creating Connections 10 Creating Tasks 11 Setting the Workflow 12 Saving and Retrieving DTS Packages 14 SQL Server Storage 14 SQL Server Meta Data Services Storage 15 Structured Storage File 15 Visual Basic File 15 Package Versions 16 How to Execute a DTS Package 16 From the DTS Designer 16 From the Wizards 17 From the Enterprise Manager 17 From Visual Basic Code 18 With the OLE Automation System Stored Procedures 18 As a Data Provider 19 Using the DTSRun Command Prompt Utility 19 Using the DTSRun Utility for Windows 22 From Another DTS Package 23 Scheduling a DTS Package 24 The Execution Context for a DTS Package 25 Using the DTS Object Model for Programmatic Control 27 The DTS Object Model 27 Using the DTS Object Model with the Dynamic Properties Task 28 Using the DTS Object Model with Disconnected Edit 29 Programming DTS with VBScript 31 Programming DTS with Visual Basic 31 Programming DTS with VC++ 32 00 0672320118 fm 11/13/00 5:02 PM Page v Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. MICROSOFT SQL SERVER 2000 DATA TRANSFORMATION SERVICES vi DTS Templates 32 Using Templates 32 Creating Templates 32 Using Wizards for Rapid Application Development 33 Copy Database Wizard 33 DTS Import/Export Wizard 34 Practical Uses for DTS 36 Upgrading a Database from Microsoft Access to SQL Server 36 Consolidating Data from Multiple Sources 36 Initial and Periodic Loading of a Data Mart or a Data Warehouse 37 Reporting on Data from Transaction Processing Systems 38 Building an Interface to the Web for a Legacy System 38 Archiving a Database 38 Analyzing Internet Clickstream Data 39 Importing and Exporting with XML 40 Conclusion 40 2 DTS Enhancements for SQL Server 2000 41 Top Ten New DTS Features 42 The New DTS Tasks 43 Dynamic Properties Task 43 The File Transfer Protocol (FTP) Task 44 The Execute Package Task 44 The Message Queue Task 44 The Parallel Data Pump Task 44 The Analysis Services Processing Task 44 The Data Mining Prediction Query Task 45 The Transfer Databases Task 45 The Transfer Master Stored Procedures Task 45 The Transfer Jobs Task 46 The Transfer Logins Task 46 The Transfer Error Messages Task 46 Transformation Task Enhancements 47 Multiple Phases in the Data Transformation Tasks 47 Package Object Browser When Writing Scripts 47 The New Data Transformations 49 Using Parameters in the Source Query 49 New Error File Options 49 Lookups Can Now Modify Data 50 New Features in Other Tasks 50 Using Input Parameters in the Execute SQL Task 50 Assigning the Result of the Execute SQL Task to Parameters 51 Assigning the Result of the Execute SQL Task to Recordset 51 00 0672320118 fm 11/13/00 5:02 PM Page vi Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. CONTENTS Package Object Browser for Writing ActiveX Scripts 52 Auto-Generating a Format File in the Bulk Insert Task 52 Other New Features in DTS 52 Copy Database Wizard 52 Save To Visual Basic File 53 Setting Values of the Package Global Variables with DTSRun 53 DTSRunUI—A Windows Interface for DTSRun 53 Package Templates 54 Disconnected Edit 54 Option Explicit for Global Variables 55 Just-In-Time Debugging 55 Turn On Package Cache 56 Executing Individual Steps in the Package Designer 56 DTS Package Logs 56 Other New Features in SQL Server 2000 That Enhance Data Transformation 56 Integration with XML 56 Indexed Views 57 User-Defined Functions 57 Conclusion 57 3 DTS and the Flow of Data Through the Enterprise 59 Enterprise Data Structure and Data Flow 61 Business Drivers for Enterprise Data Transformation 61 Ways to Use Data 63 Transaction Processing 63 Business Analysis 64 Maintaining History 68 Sources of Data 68 On Line Transaction Processing (OLTP) Systems 68 Web Clickstream Data 69 Data from Outside the Organization 69 The Data Warehouse 69 Results from Business Analysis Systems 69 Meta Data 70 Types of Data Storage 71 Text Files 72 XML 72 Spreadsheets 74 Relational Database Management Systems 74 Multidimensional Database Management Systems (OLAP) 76 Conclusion 76 vii 00 0672320118 fm 11/13/00 5:02 PM Page vii Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. MICROSOFT SQL SERVER 2000 DATA TRANSFORMATION SERVICES 4 Using DTS to Move Data into a Data Mart 77 Multidimensional Data Modeling 78 Differences Between Relational Modeling and Multidimensional Modeling 80 The Fact Table 82 Choosing the Measures 83 Choosing the Level of Summarization for the Measures 83 The Dimension Tables 84 The Primary Key in a Dimension Table 85 Levels of the Dimension Hierarchy 86 Attributes of the Dimension 86 The Time Dimension 87 Subordinate Dimension Keys 88 Loading the Star Schema 88 Loading Data into a Staging Area 89 Loading the Dimension Tables 90 Updating the Subordinate Dimension Keys 92 Loading the Fact Table 92 Using the Data 93 Avoiding Updates to Dimension Tables 94 Conclusion 98 PART II DTS Connections and the Data Transformation Tasks 5 DTS Connections 101 Creating DTS Connections 103 DTS Connection Properties 104 Properties That Identify a DTS Connection 104 Properties That Determine Access to the Data Source 108 Other Properties 112 Creating Connections for Different OLE DB Providers 114 The SQL Server OLE DB Provider 114 Text File Connections 115 The OLE DB Provider for OLAP Services 119 Other OLE DB Providers 119 Modifying Data Connections Dynamically 121 Performance Issues with DTS Connections 122 Conclusion 123 6 The Transform Data Task 125 When to Use the Transform Data Task 126 Creating a New Transform Data Task 128 Using the Package Designer 128 Using the DTS Import/Export Wizard 129 Using Code 129 viii 00 0672320118 fm 11/13/00 5:02 PM Page viii Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. CONTENTS The Description and Name of the Task 129 The Source of a Transform Data Task 130 Text File Source 131 SQL Table, View, or Query for a Relational Database Source 131 MDX Query for a Multidimensional Cube Source 133 Using XML as the Source 134 Using Parameters in a Source Query 135 DataPumpTask Source Properties 136 The Destination of a Transform Data Task 137 Creating a New Destination Table 137 Text File Destination 138 DataPumpTask Destination Properties 139 Mapping Source Columns to Destination Columns 139 Transformation Flags 144 Testing a Transformation 146 The Collections That Implement a Transformation 146 Other Properties of a Transformation 147 The Transformation Types 148 The DateTime String 148 Uppercase Strings, Lowercase Strings, and Copy Column 149 Middle of String and Trim String 149 Read File 151 Write File 151 ActiveX Script 152 Custom Transformation 152 Other Properties of the Transform Data Task 153 Error Handling Properties 154 Data Movement Properties 156 Options for Improving Performance with SQL Server Destinations 157 Column Properties 159 Creating a Transform Data Task in Visual Basic 160 Creating Transformations in Code 165 Choosing a Transformation Type in Code 165 Creating the Transformation and Its Columns 167 Copy Column, Uppercase, and Lowercase Transformations 169 Trim String and Middle of String Transformations 169 Read File and Write File Transformations 170 DateTime Transformations 171 ActiveX Transformations 172 A Sample Application with All the Transformations 172 Using the Transform Data Task as a FreeStanding Icon 173 Conclusion 177 ix 00 0672320118 fm 11/13/00 5:02 PM Page ix Please purchase PDF Split-Merge on www.verypdf.com to remove this watermark. [...]... Using DTS in SQL Server 7.0 and SQL Server 2000 I believe DTS in SQL Server 7.0 is a great tool But DTS in SQL Server 2000 has a number of improvements: • More tasks • More transformations • More convenient design environment • Better access to object properties This book is about DTS in SQL Server 2000 Whenever possible, I have pointed out differences between SQL Server 7.0 and SQL Server 2000 to help... current information about DTS: • The Microsoft DTS newsgroup microsoft. public.sqlserver.dts on msnews .microsoft. com • The SQL Server section of Microsoft s Web site—www .microsoft. com • The SQL Server Magazine Web site—www.sqlmag.com • SQL Server Professional’s Web site—www.pinpub.com • The Professional Association of SQL Server Users Web site—www.sqlpass.org • The SQL Server section of www.swynk.com PART... DTS Packages 565 DTS Transformation Goals 567 The Goal of Rapid Execution 567 Rapid Development 568 Self-Documenting Data Transformations 568 Maintainable Code 568 DTS Task Performance Comparison 569 Loading from a Text File to a SQL Server Database 569 Loading from SQL Server 2000 to SQL Server 2000 571 Performance Statistics for the Transformation Tasks... Parallel Data Pump Task 252 The Transformation Modes 252 Hierarchical Mode 253 Flattened Mode 253 Data Driven Query Mode 253 The TransformationSetOptions Property 253 Creating a Parallel Data Pump Task in Visual Basic 254 A User Interface for the Parallel Data Pump Task 257 Conclusion 263 xi xii MICROSOFT SQL SERVER 2000 DATA TRANSFORMATION. .. Car Rental, and Microsoft He worked on his first data warehousing project in 1995, using SQL Server 6.0 Tim is the lead author of Microsoft OLAP Unleashed, published by Sams in 1999 He wrote the sections about DTS, data modeling, enterprise data structure, creating cubes, and programming the OLAP server Tim is a Microsoft Certified Trainer who teaches courses in the field of SQL Server data warehousing... use DTS with a wide variety of data stores, including relational databases such as Microsoft SQL Server and Oracle, multidimensional databases such as Microsoft OLAP Services, text files, and spreadsheets DTS Connections use OLE DB providers but can also connect through ODBC drivers and Data Link files Some of the data sources that are normally installed with SQL Server 2000 are shown in Figure 1.4 A...x MICROSOFT SQL SERVER 2000 DATA TRANSFORMATION SERVICES 7 Writing ActiveX Scripts for a Transform Data Task 179 When You Should Use the ActiveX Script Transformation 180 Deciding Between One Task and Many 181 Using the Variety of Transformation Types 181 Transformation ActiveX Scripts Basics 182 The Transformation ActiveX Script Development Environment... Pump—and a Whole Lot More Data Transformation Services in Microsoft SQL Server 2000 is a high-performance data pump It is a tool for copying, moving, consolidating, cleansing, and validating data The data pump loads each row from the data source, manipulates the values in that row, and inserts the row into the data destination DTS is also a Rapid Application Development tool for data- oriented programming... environment We set all the connections to the local server using integrated security Hopefully those settings will work for you without the need for adjustments 2 MICROSOFT SQL SERVER 2000 DTS Keeping Current with Information on DTS This book is based on DTS in the initial release of SQL Server 2000 DTS developers will be learning much more about DTS in SQL Server 2000 in the months and years ahead We would... 103rd Street Indianapolis, IN 46290 USA Introduction Data Transformation Services in Microsoft SQL Server 2000 is a powerful tool for moving data It’s easy to use, especially when you’re creating transformation packages with the DTS Wizard But it’s also a flexible tool that you can customize to gain a high degree of control over the transformation of your data The size of this book reflects the flexibility . from a Text File to a SQL Server Database 569 Loading from SQL Server 2000 to SQL Server 2000 571 Performance Statistics for the Transformation Tasks 573 Comparing. watermark. MICROSOFT SQL SERVER 2000 DATA TRANSFORMATION SERVICES xvi Package Logs and Error Files 451 Logging to SQL Server 454 DTS Packages as Data Sources