ETL Techniques • Chapter 8 367 3. In the Project Types list select the left Business Intelligence Projects and then in the Templates list on the right, select Integration Services Project. 4. Finally, you can complete the New Project window by giving your project a name, selecting the location of the parent folder that the project will be created under, and creating a solution for the project to belong to. Head of the Class… BIDS Projects and Solutions Remember that Business Intelligence Development Studio (BIDS) is really Visual Studio. Visual Studio is the development environment that devel- opers use when working with a variety of technologies. With the right products installed, Visual Studio could be used by Windows developers writing programs in Visual Basic.Net, by Web developers creating Web sites in C#, or by business intelligence developers working with Integration Services, Analysis Services, or Reporting Services. Each different type of work, or development, that can be done in Visual Studio is represented by a project template. When working with SQL Server Integration Services, for example, you would choose from the Integration Services Connection Project, or the Integration Services Project templates. Again, it depends on what products you have installed on your workstation as to what project templates will appear when creating a new project in BIDS. Each project will have files that need to be stored on the developer’s workstation. For SSIS projects, the package files will be stored locally on the developer’s workstation. Projects use a folder structure to organize the files related to each project. So when you are naming a project in BIDS, you are naming not only the project but also the folder in which the projects resources will be stored. When creating a project you also need to choose under which parent folder, your project’s folder will be created. By default this will be under your My Documents folder in a Visual Studio 2008\Projects subfolder. However, you can store them wherever you like. Continued 368 Chapter 8 • ETL Techniques Solutions allow developers to work with multiple projects at once. Often developers need multiple project types to complete a business requirement. For example a developer may be using C# to create a custom task to be used in SSIS. The developer would benefit from having both the C# project and SSIS project open at the same time in BIDS. That’s where solutions come in. A solution is a container of one or more projects. Whenever you work with a project in BIDS, it is always a part of a solution. Most of the solutions you work with will have only a single project; however it is possible to add multiple projects to a solution. The benefit is that when the developer opens the solution, all the projects that are part of the solution are available within BIDS. The developers can then seamlessly move between projects as they complete their development work. If you use BIDS to create a new project, you will find two projects related to Integration Services in the Business Intelligence Projects category. The Integration Services Connections Project template starts a new project, but it runs a wizard to prompt you for the connection information for the various data sources to which you will need to connect. This configures a package with the connection managers for the data sources you specify already set up. This may be helpful if you are new to creating packages in BIDS, but it configures only one package with the connection managers. Any additional packages will have to be created from scratch. The other project template is just the Integration Services Project template. This template starts you out with a single empty package. You build up from there. Integration Services projects can have multiple packages. Each package is stored in the project folder as an XML file with the .dtsx extension. Developers can create new packages or edit existing ones. When you open a package in BIDS, the package designer is displayed. Figure 8.4 shows BIDS with a package open. ETLTechniques•Chapter8 369 Figure 8.4 BIDS Package Designer—Control Flow The screen shot shown in Figure 8.4 shows you a number of important elements, BIDS, and an SSIS Package. As you look at the screenshot in Figure 8.4, you see that it is broken into a number of windows. The left-most window is the Toolbox. Depending on what tab developers are on in the package designer, the toolbox changes to show items that are appropriate for that tab. Currently, the toolbox is showing the tasks that are available to place in a package. The largest section in the middle of the screen is the package designer, and you can see that the Export Data.dtsx package file has been opened for editing. The various boxes and lines in the package designer represent tasks and how they are sequenced. The window at the top right is the Solution Explorer. It displays all the projects in the current solution, and all the resources in those projects. Finally, the window in the lower right is the Properties window. The properties window shows all the properties of whatever is currently selected in the package designer. If you focus on the package designer in the middle of the screenshot, you can see it is broken down into some subtabs, although you can’t read their full names in this screen shot. It’s hard to fit all the stuff into a screen shot that fits nicely 370 Chapter8•ETLTechniques in a book. The subtabs are Control Flow, Data Flow, Event Handlers, and Package Explorer. Currently, the Control Flow subtab is selected. The control flow contains Tasks and Precedence Constraints. The tasks are things you need to do as part of the SSIS package’s execution. In the preceding example, the tasks are Backup the Database, Export Data, Send Success Email, and Send Error Email. The red and green colored lines are called Precedence Constraints. Precedence Constraints place requirements on the execution of the task to which they point. For example, the green line that points from the Backup the Database to the Export Data task implies that the Export Data task is not allowed to execute unless the Backup the Database task runs successfully. The colors of a Precedence Constraint indicate the required result of the previous task. Green indicates that the previous task has to have succeeded, red indicates failure, and blue indicates completion, regardless of success or failure. If you look at the Toolbox in Figure 8.4, you see a number of icons that represent type kinds of tasks. Here is a description of some of those tasks: Bulk Insert Task Performs bulk loads similar to BCP or BULK INSERT, but from within as SSIS Package Execute SQL Task Runs a SQL statement (like a BACKUP, RESTORE, or whatever you need) against a database FTP Task Could be used to send or receive files from a server using FTP Script Task Allows developers to write their own code in Visual Basic. Net or C# as part of an SSIS package Data Flow Task The Data Flow Task is the true ETL task. It allows you to specify any source, any destination, and any set of transformations that you need as part of a data movement. The Data Flow Task can require a lot of configuration. It could have multiple sources, multiple destinations, multiple paths for the data to flow through, multiple transformations to perform on the data, and so on. The task could have such a com- plex definition that it has its own design surface in the Package Designer. Figure 8.5 shows an example of the “Export Data” Data Flow Task show in Figure 8.4. ETLTechniques•Chapter8 371 In Figure 8.5, you can see the data connections that represent the source and destination of your data flow, and the green arrow represents the set of “good” rows from the source that make it into the destination. You can also take rows that have errors (not shown in the screen shot) and send them down their own paths to either log the errors or possibly even fix them and route them back into the destination. The “Toolbox” in Figure 8.5 shows a sampling of the transformations that can be performed on the data as it moves from the source to the destination. The screen shot doesn’t show any transformations in use, but the following list describes some of the common transformations that can be applied: Character Map Used to convert strings to all upper or lower case, convert between character and Unicode formats, and so on. Conditional Split Could direct rows down different paths based on the data in the records. Data Conversion Converts data to different types. For example, Data Conversion could convert a datetime to an nvarchar. Figure 8.5 BIDS Package Designer—Data Flow . place requirements on the execution of the task to which they point. For example, the green line that points from the Backup the Database to the Export Data task implies that the Export Data task. errors (not shown in the screen shot) and send them down their own paths to either log the errors or possibly even fix them and route them back into the destination. The “Toolbox” in Figure. editing. The various boxes and lines in the package designer represent tasks and how they are sequenced. The window at the top right is the Solution Explorer. It displays all the projects in the