Nielsen c17.tex V4 - 07/21/2009 12:57pm Page 432 Part III Beyond Relational The following script adds the level to the WHERE cause of the previous script and finds only those mem- bers of the Information Servies Department who report directly to Jean: DECLARE @ManagerNode HierarchyID, @ManagerLevel INT SELECT @ManagerNode = OrganizationNode, @ManagerLevel = OrganizationLevel FROM HumanResources.Employee WHERE OrganizationNode = ‘/5/’ Jean Trenary - IT Manager SELECT E.BusinessEntityID, P.FirstName + ‘ ‘ + P.LastName as ‘Name’, OrganizationNode, OrganizationNode.ToString() as ‘HierarchyID.ToString()’, OrganizationLevel FROM HumanResources.Employee E JOIN Person.Person P ON E.BusinessEntityID = P.BusinessEntityID WHERE OrganizationNode.IsDescendantOf(@ManagerNode) = 1 AND OrganizationLevel = @ManagerLevel + 1 Result: BusinessEntityID Name HierarchyID.ToString() OrganizationLevel 264 Stephanie Conroy /5/1/ 2 267 Karen Berg /5/2/ 2 268 Ramesh Meyyappan /5/3/ 2 269 Dan Bacon /5/4/ 2 270 Fran ¸ cois Ajenstat /5/5/ 2 271 Dan Wilson /5/6/ 2 272 Janaina Bueno /5/7/ 2 Inserting new nodes Inserting the root node into a HierarchyID hierarchy must start with the binary root node. The getroot() method returns a root node value. Inserting additional nodes into HierarchyID’s relative node position hierarchy first requires determin- ing which nodes are already there. The GetDescendant() method can be used to generate the next node position. You can download additional hierarchical sample code from www.sqlserverbible.com. Performance The HierarchyID data type is a compact data type optimized for storing relative node postion. As such, it takes less space than a character-based materialized path. The HierarchyID node column should be indexed, which will aid subtree-type searches. There are specific Query Optimizer optimizations for indexing and HierarchyID, but I have found that materi- alized path is still slightly faster than HierarchyID. 432 www.getcoolebook.com Nielsen c17.tex V4 - 07/21/2009 12:57pm Page 433 Traversing Hierarchies 17 If the searches are primarily along a specific level, then another index keyed by level column and the HierarchyID node column will be of use. However, I’ve found that this type of search is rarely used in the production hierarchies I’ve worked with. HierarchyID pros and cons The new HierarchyID is not without controversy. It’s new and gets plenty of press and demo time, but I’m not sure it’s a problem that needed another solution. On the pro side, HierarchyID has the following advantage: ■ It’s faster than the adjacency list pattern. On the con side: ■ HierarchyID embeds data within a binary data type so it’s more difficult to navigate and diagnose. ■ Storing only the relative node position is risky. If a node is moved or deleted without very careful coding, it can be nearly impossible to correct the orphaned nodes. Adjacency list and materialized path both store the actual primary keys values, so they are both more robust than HierarchyID. Although Microsoft intends the HierarchyID to use a relative node postion, the contents of the data type accept any numeric values up to 45 bytes per node position (space between the /n/.) There’s no reason why you can’t insert a primary key instead of the relative node postion. It won’t be as compact, but it will be more robust. Summary Hierarchical data can present a challenge if you’re not armed with patterns and solutions, but knowing the possible patterns and having several navigational methods in your toolset will increase the odds that your next hierarchical data project will be successful. Adjacency pairs is the most common hierarchy solution, as it’s easy to work with and well understood. Unless your requirement is very high performance when searching for all ancestors, stick with the known solution. If the hierarchy is the crux of the database and most functions use the hierarchy in some way, then go with the materialized-path solution. My Nordic software navigates the class hierarchy in nearly every query, so I tested the three patterns with a million-node hierarchy. For searching a subtree of descen- dents, materialized path was slightly faster than HierarchyID. For finding all ancestors, materialized path was significantly faster than HierarchyID. Both easily beat the adjacency list pattern. The next chapter continues with the theme of working with non-relational data and looks at SQL Server’s new spatial data types. 433 www.getcoolebook.com Nielsen c17.tex V4 - 07/21/2009 12:57pm Page 434 www.getcoolebook.com Nielsen c18.tex V4 - 07/21/2009 1:01pm Page 435 Manipulating XML Data IN THIS CHAPTER Understanding the XML data type Creating and using XML columns and variables Loading XML documents from disk files Understanding XPath and XQuery Generating XML documents using FOR XML Performing XML data modification Handling XML namespaces Shredding XML using OPENXML Creating and using XML Schema Collections Working with XML indexes XML best practices T here has been a significant increase in the popularity and usage of XML in the past few years. XML is becoming widely accepted as the preferred format for exchanging and publishing information. Almost all modern applications seem to be touching XML in one way or another. They either generate XML for consumption by other applications or components, or they consume the XML produced by others. This chapter closely examines the XML capabilities of SQL Server 2008 XML is seen everywhere today. Knowingly or unknowingly, every application deals with XML in one way or another. For example, .NET applications use XML files to store configuration information. ASP.NET web pages are XML documents. Almost all modern websites generate and publish information as XML feeds (RDF, RSS, ATOM, OPML, etc.). Changing application a rchitectures in recent years contributed to the widespread acceptance of XML. Service-oriented architecture (SOA) promotes the usage of XML Web Services. AJAX (Asynchronous JavaScript and XML) has become a basic requirement of every web application today. XHTML has become the default stan- dard for every web page. To support the changing data processing requirements, Microsoft has added XML processing capabilities to SQL Server starting from SQL Server 2000. The XML support in SQL Server 2000 was very much limited. SQL Server 2005 added significant improvements to the XML processing capabilities, and SQL Server 2008 enhanced it further. SQL Seconr’s own usage of XML in a number of core components shows the level of acceptance XML is getting inside SQL Server. A few examples are given here: ■ SSRS (SQL Server Reporting Services) stores report definitions as XML documents. ■ SSSB (SQL Server Service Broker) exchanges messages in XML format. 435 www.getcoolebook.com Nielsen c18.tex V4 - 07/21/2009 1:01pm Page 436 Part III Beyond Relational ■ SSIS (SQL Server Integration Services) stores the SSIS packages as XML documents and supports XML configuration files. ■ DDL triggers use XML format for exchanging event information. This chapter discusses the XML processing capabilities of SQL Server 2008 in detail. What’s New for XML in SQL Server 2008? S QL Server 2008 adds a number of enhancements to the XML processing capabilities of the previous versions. Improved Schema Validation Capabilities ■ Added support for ‘‘lax’’ validation ■ Added full support for date, time,anddateTime data types, including preservation of time zone information ■ Improved support for union and list types. Allows creating unions of list types and lists of union types. Enhancements to XQuery Support ■ Added support for the ‘‘let’’ clause in FLWOR operations DML Language Enhancements ■ XML Data Type’s modify() method now accepts XML data type variables in the XML DML expressions: insert and replace value of. XML Processing in SQL Server 2008 The XML processing capabilities of SQL Server 2008 may be classified into three broad categories: ■ Capability to generate XML documents ■ Capability to query XML documents ■ Capability to validate XML documents 436 www.getcoolebook.com Nielsen c18.tex V4 - 07/21/2009 1:01pm Page 437 Manipulating XML Data 18 Generating XML documents Today, a large number of applications publish and exchange information in XML format. One of the most common examples is a website that publishes RSS or ATOM feeds. Applications that expose or consume XML Web Services need to generate XML documents containing the information to be exchanged. Web applications that support the AJAX programming model need to generate and serve information in XML format. Most of the time, the data being exchanged in XML format by these applications comes from one or more databases. A major part of the data may come from one or more relational tables and the rest may come from some unstructured data storage. Some component or module has to take the responsibility for building the required XML document from the source data. In the past, the responsibility for building the XML documents was handled by the application layer. The a pplication might execute a stored procedure that returns one or more result sets, and it could build the desired XML document using a set of XML API functions exposed by the application programming language. The first version of SQL Server shipped with XML generation capabilities is SQL Server 2000. SQL Server 2000 introduced a new T-SQL clause FOR XML, which transforms the results of a SELECT query to an XML stream. This new feature removed the burden of building XML documents at the application layer. Instead of returning tabular result sets to the client application, SQL Server could serve XML streams using the FOR XML clause. SQL Server 2000’s FOR XML clause offers a different level of control over the structure of the output XML document through the additional directives AUTO, RAW,andEXPLICIT. When used with AUTO and RAW, FOR XML generates an XML document that contains an XML element for each row in the result set. While they have a simpler syntax and are quite easy to use, they provide very little control over the structure of the output XML document. EXPLICIT, on the contrary, pro- vides a great extent of control over the output XML structure, but has a much more complex syntax and most people find it very hard to use. SQL Server 2005 introduced a new directive, PATH, that has a simpler syntax and powerful customiza- tion capabilities. Almost all customization requirements that were possible only with EXPLICIT in SQL Server 2000 are now possible with PATH. SQL Server’s FOR XML clause, along with the RAW, AUTO, PATH,andEXPLICIT directives can generate XML documents to serve almost all XML generation requirements. A detailed explanation of FOR XML, along with several examples, is given in the section ‘‘FOR XML’’ later in this chapter. Querying XML documents Almost all applications that exchange information in XML format need to have a module or component capable of extracting information from the XML markup. For example, an RSS reader desktop application might need to extract channel a nd item information from the XML document before it can be displayed in the UI element. A client application that calls a Web Service to retrieve a weather f orecast or stock price needs to parse the XML result document and extract the required pieces of information. 437 www.getcoolebook.com Nielsen c18.tex V4 - 07/21/2009 1:01pm Page 438 Part III Beyond Relational Many of these applications might decide to store the information received in one or more relational tables. A Web Service that accepts sales order information from customers is one such example. The Web Service might send the order information to a stored procedure that stores the information in a set of relational tables. Applications that run on SQL Server versions prior to 2000 do the XML parsing at the application layer. They parse XML documents and extract the required values, which are then sent to the database as parameters. Applications that deal with a variable number of values (like sales orders) either call the stored procedure once for each line (which is very bad for performance) or pass all the information in a single stored procedure call as a delimited string (which adds considerable string parsing overhead at the stored procedure level). Microsoft shipped SQL Server 2000 with a very powerful XML processing function, OPENXML(),that is capable of shredding an XML document into a result set. Applications that run on SQL Server 2000 could pass entire XML documents to a SQL Server stored procedure without bothering to parse the XML document at the application layer. Applications that deal with a variable number of parameters, like sales or purchase orders, could make use of this XML processing capability of SQL Server and send all the information in a n XML document with a single stored procedure call. A typical call to OPENXML() includes a system stored procedure call to prepare an XML document han- dle, a call to OPENXML() to retrieve the result set, and another call to a system stored procedure to release the document handle. Because of this specific requirement — a system stored procedure call before and after the OPENXML() call — OPENXML() can neither be called f rom a function nor used in a set-based operation. For example, to read information from an XML formatted string stored in a SQL Server column, a WHILE loop is required, because OPENXML() can process only one XML document at a time. Before and after processing each document, calls to system stored procedures to initialize and release the document handle are necessary. SQL Server 2005 introduced the native XML data type. An XML data type can store a complete XML document or a fragment. The XML data type supports XQuery, a language designed to query XML doc- uments. The introduction of XQuery made querying information from XML documents much easier. XQuery is lightweight, very powerful, much easier to use compared to OPENXML(), and it can be used in a set-based operation or can be called from a function. SQL Server is equipped with powerful XML querying capabilities. A database developer can perform almost every XML parsing/shredding requirement using XQuery or OPENXML(), both of which are discussed in detail in this chapter. Validating XML documents One of the basic requirements of any serious application is a set of robust validation processes applied on information b eing exchanged. A well-written piece of code would ideally validate all the input parameters before performing the actual processing for which it was created. Usually, correct validation of parameters protects the application and database from logical and data errors. It also helps to protect the data from malicious attempts such as SQL injection. When information is exchanged in XML format, a more rigid validation is needed because the chances of invalid values are greater in the case of an XML document. An application passing the value ‘‘thirty’’ to the @age parameter of a stored procedure (@age INT) would receive a conversion error right away, as SQL Server will perform an implicit data type validation. The developer writing the stored procedure 438 www.getcoolebook.com Nielsen c18.tex V4 - 07/21/2009 1:01pm Page 439 Manipulating XML Data 18 does not need to perform validations such as ensuring that all integer parameters are valid integer values. The developer might only need to perform validations specific to the business logic, such as a range validation on the age of an employee to ensure that it is within the accepted range (e.g., 18 to 65). The story will be different i n the case of an XML parameter. SQL Server may not be able to detect a possible error in an XML document such as ‘‘ <Employee age=˝who knows˝/>’’ because the attribute @age is not associated with a data type and SQL Server does not know how to validate it. This indicates that XML documents might need much more validation than non-XML parameters. For example: ■ There are times when the elements in the XML document should follow a certain order. An application that parses the XML document sequentially might require that the OrderDate element should precede the ExpectedShippingDate element to validate the shipping date based on the order date. ■ Structure of the XML should be validated. For example, an application might expect the employee address under the employee element, and the zip code as an attribute of the address element. ■ Certain elements may be optional and certain elements mandatory. ■ Certain elements may appear more than once and others may be allowed only once. ■ Data type validations may be needed. For example, age should be an integer value. ■ Restrictions might b e needed on the accepted range of values, such as age of an employee should be between 18 and 65. ■ Certain values may follow specific formats. For example, an application might require phone numbers to be in the format of (999) 999-9999, or social security numbers must be 999-99-9999. XSD (XML Schema Definition Language) is a W3C recommended language used for describing and vali- dating XML documents. Based on the structure, format, and validation rules required by an application, an XSD schema can be created and a schema validator can then validate XML documents against the rules defined in the schema document. Microsoft added XSD validation support in SQL Server 2005 and extended it further in SQL Server 2008. SQL Server stores an XML schema as XML SCHEMA COLLECTION objects. XML schema collec- tions are SQL Server objects just like tables, views, constraints, and so on. An XML schema collection can be created from the definition of an XML schema. An XML column, variable, parameter, or return value of a function can be associated with a specific XML schema collection. XML data type values that are bound to an XML schema collection are called typed XML. The other flavor of XML (not bound to an XML schema collection) is called untyped XML. SQL Server validates typed XML values against the rules defined in the XML schema collection. When a new value is assigned to a typed XML variable or column, or the value is modified by a DML oper- ation (insert, update, or delete), SQL Server will validate the new value against the schema and the operation will succeed only if the new value passes the validation. Support of XSD schemas is a powerful addition to the XML processing capabilities of SQL Server. A good combination of the capabilities to generate, query, and validate XML documents makes SQL Server the right platform for building applications that handle structured and unstructured data. 439 www.getcoolebook.com Nielsen c18.tex V4 - 07/21/2009 1:01pm Page 440 Part III Beyond Relational Sample Tables and Data To explain and demonstrate the XML processing capabilities of SQL Server 2008, this chapter presents a number of code snippets. To make the examples and the underlying data more comprehensible, a fixed set of sample tables and sample data is used throughout the examples presented in this chapter. The sample tables include tables for storing Customer, Item, and Order information. Here’s a script to create sample tables: IF OBJECT_ID(’Customers’,’U’) IS NOT NULL DROP TABLE Customers CREATE TABLE Customers ( CustomerID INT IDENTITY PRIMARY KEY, CustomerNumber VARCHAR(20), Name VARCHAR(40), Phone VARCHAR(15), Street VARCHAR(40), City VARCHAR(20), State VARCHAR(10), Zip VARCHAR(10) ) GO IF OBJECT_ID(’Items’,’U’) IS NOT NULL DROP TABLE Items CREATE TABLE Items ( ItemID INT IDENTITY PRIMARY KEY, ItemNumber VARCHAR(20), ItemDescription VARCHAR(40) ) GO IF OBJECT_ID(’OrderHeader’,’U’) IS NOT NULL DROP TABLE OrderHeader CREATE TABLE OrderHeader ( OrderID INT IDENTITY PRIMARY KEY, OrderNumber VARCHAR(20), OrderDate DATETIME, CustomerID INT ) GO IF OBJECT_ID(’OrderDetails’,’U’) IS NOT NULL DROP TABLE OrderDetails CREATE TABLE OrderDetails ( OrderDetailID INT IDENTITY PRIMARY KEY, OrderID INT, ItemID INT, Quantity INT, Price MONEY ) IF OBJECT_ID(’OrderXML’,’U’) IS NOT NULL DROP TABLE OrderXML CREATE TABLE OrderXML (OrderID INT, ItemData XML) GO Here’s a script to insert sample data: Populate Customer Table INSERT INTO Customers ( CustomerNumber, Name, Phone, Street, 440 www.getcoolebook.com Nielsen c18.tex V4 - 07/21/2009 1:01pm Page 441 Manipulating XML Data 18 City, State, Zip ) SELECT ‘J001’, ‘Jacob Sebastian’, ‘(999) 999-9999’, ‘401, Jacobs Street’, ‘New York’, ‘NY’, ‘12345’ Populate Items Table INSERT INTO Items (ItemNumber, ItemDescription) SELECT ‘D001’,’DELL XPS 1130 Laptop’ UNION ALL SELECT ‘Z001’,’XBOX 360 Console’ Create order "SO101" INSERT INTO OrderHeader( OrderNumber, OrderDate, CustomerID ) SELECT ‘SO101’,’2009-01-23’,1 Add Line Items INSERT INTO OrderDetails (OrderID, ItemID, Quantity, Price) SELECT 1, 1, 1, 900 UNION ALL SELECT 1, 2, 1, 200 Create order "SO102" INSERT INTO OrderHeader( OrderNumber, OrderDate, CustomerID ) SELECT ‘SO102’,’2009-01-24’,1 Add Line Items INSERT INTO OrderDetails (OrderID, ItemID, Quantity, Price) SELECT 2, 1, 1, 900 And finally, some XML data INSERT INTO OrderXML (OrderID, ItemData) SELECT 1, ‘ <Order OrderID="1"> <Item ItemNumber="D001" Quantity="1" Price="900"/> <Item ItemNumber="Z001" Quantity="1" Price="200"/> </Order>’ INSERT INTO OrderXML (OrderID, ItemData) SELECT 2, ‘ <Order OrderID="2"> <Item ItemNumber="D001" Quantity="1" Price="900"/> </Order>’ These sample tables and sample data will be used for the examples presented in this chapter. You can download them from www.sqlserverbible.com. XMLDataType The most significant addition to the XML capabilities of SQL Server is the introduction of the XML native data type released with SQL Server 2005. XML data can store a complete XML document or a fragment. The XML data type can be used like other native SQL Server data types. SQL Server supports 441 www.getcoolebook.com . processing requirements, Microsoft has added XML processing capabilities to SQL Server starting from SQL Server 2000. The XML support in SQL Server 2000 was very much limited. SQL Server 2005 added. information. This chapter discusses the XML processing capabilities of SQL Server 2008 in detail. What’s New for XML in SQL Server 2008? S QL Server 2008 adds a number of enhancements to the XML processing. against the rules defined in the schema document. Microsoft added XSD validation support in SQL Server 2005 and extended it further in SQL Server 2008. SQL Server stores an XML schema as XML SCHEMA COLLECTION