3 – The migratory data 80 Guaranteeing unique data Regardless of the data comparison tool you use, you will need to guarantee uniqueness in your source and target data sources, in order for these techniques to work. Heretofore, in a frenzy to load data as fast as possible, I have not been concerned with duplicate data rows or indexes. For this next section, however, I need to start with a clean slate and refactor the SQL_Conn and SQL_Conn_Archive tables in such a way that we can guarantee uniqueness of data. The SISS package that populates the SQL_Conn table runs every 15 minutes, and captures the date and time that the package ran as a field value, so this "run time" would be a good candidate for a unique column. However, to truly guarantee uniqueness, the clustered index will be a composite of the package run date and several other fields. I will also add an identity column ( ID) to the tables. Since the tables currently contain duplicate rows, in this example, I will first need to clean up a bit by truncating both the SQL_Conn and SQL_Conn_Archive. I can then run the package to populate the SQL_Conn table with current process information. Figure 3.11 shows the new record count for the SQL_Conn table, a mere 32 records. Figure 3.11: Repopulated SQL_Conn table ready for merging data. The SQL_Conn_Archive table currently holds zero records, as we would expect, clean slate and all. Figure 3.12 shows the empty table, in preparation for the prestidigitation to follow. 3 – The migratory data 81 Figure 3.12: Empty SQL_Conn_Archive table. So, now I am going to add the clustered index and an identity column ( ID) to both tables. Figure 3.13 shows the 5 fields from the SQL_Conn table that I used to guarantee uniqueness. I am heartened by the fact that, as this table fills up over time, the clustered index will also benefit me when running interrogative queries for reports of sever activity. Figure 3.13: Building clustered index for SQL_Conn table. 3 – The migratory data 82 With the newly built index and identity column on both source (SQL_Conn) and target ( SQL_Conn_Archive), it is time to introduce Tablediff.exe, which we will use to keep the two tables in sync. Tablediff.exe Tablediff.exe is a little known and free tool that comes with SQL Server, and has done so for many releases. It was designed to assist with comparing replication sets for native SQL Server replication. However, even if you are not using SQL replication, it can be put to good use in comparing and synchronizing source and target tables. It compares one table at a time and displays the differences between the two tables. Further, it can generate scripts that will sync the two tables, if there are differences. For SQL Server 2005, Tablediff.exe can be found in the C:\Program Files\Microsoft SQL Server\90\COM\ folder. It has options that allow you to define the source and destination servers, as well the required databases and tables. Listing 3.6 shows the command line execution that will compare the freshly loaded SQL_Conn source table and the destination SQL_Conn_Archive. The options -q and -t 200, respectively, tell tablediff to do a simple record count rather than a full blown compare, and to timeout after 200 seconds. You also have the ability to lock the source and target tables, as I am doing here with -sourcelocked and –destinationlocked. C:\Program Files\Microsoft SQL Server\90\COM\tablediff.exe" - sourceserver MW4HD1 -sourcedatabase DBA_Rep -sourcetable SQL_Conn -sourcelocked -destinationlocked -destinationserver MW4HD1 -destinationdatabase DBA_Rep -destinationtable SQL_Conn_Archive -q -t 200 Listing 3.6: Tablediff with source and destination options. Figure 3.14 shows the 32 records that are different in SQL_Conn and SQL_Conn_Archive tables, by doing a simple row count. It took less than a tenth of a second to deliver the results. 3 – The migratory data 83 Figure 3.14: Tablediff.exe row count differences. Without the -q option, you will get a list of differences, by ID column, as shown in Figure 3.15. This comparison took 0.15 seconds for the 32 records. Figure 3.15: Tablediff showing detailed differences. 3 – The migratory data 84 The best thing about Tablediff.exe is that it will generate a script that will bring the two tables in sync. That option is -f which takes a filename and path, as shown in Listing 3.7. C:\Program Files\Microsoft SQL Server\90\COM\tablediff.exe" - sourceserver MW4HD1 -sourcedatabase DBA_Rep -sourcetable SQL_Conn -sourcelocked -destinationlocked -destinationserver MW4HD1 -destinationdatabase DBA_Rep -destinationtable SQL_Conn_Archive -q -t 200 –f C:\Output\SQL_Conn.sql Listing 3.7: The tablediff comand to synch the two tables. Running this tablediff command, with the -f option, generates a file containing all of the T-SQL statements to make the two tables identical. Figure 3.16 show the SQL_Conn.sql file that the command created. Figure 3.16: Output of fix file for Tablediff.exe Executing the script against the SQL_Conn_Archive table and then re-running the tablediff.exe will show that the two tables are now identical. "High Availability" tools Finally, we move on to a discussion of what I have termed "high availability" data migration tools, simply because these tools generally form part of an organizations strategy in minimizing downtime and maximizing the availability of its data. Up to now we have been working with single source tables, whereas these techniques are more applicable when whole databases need to be migrated, and . comes with SQL Server, and has done so for many releases. It was designed to assist with comparing replication sets for native SQL Server replication. However, even if you are not using SQL replication,. differences. For SQL Server 2005, Tablediff.exe can be found in the C:Program FilesMicrosoft SQL Server 90COM folder. It has options that allow you to define the source and destination servers,. C:Program FilesMicrosoft SQL Server 90COM ablediff.exe" - sourceserver MW4HD1 -sourcedatabase DBA_Rep -sourcetable SQL_ Conn -sourcelocked -destinationlocked -destinationserver MW4HD1 -destinationdatabase