CHAPTER 10 Storing: XML and Databases
10.2 The Need for Persistence
A great deal of the XML data most people encounter today are stored somewhere;
that is, they are persistent . Storing XML data persistently makes a great deal of sense for data that may be used many times, especially when that data have a high value and may have been expensive, even diffi cult, to create.
Examples of such XML abound: Our movie collection is documented in an XML document; corporations are increasingly likely to store business data like purchase orders in an XML form; many technical books are being produced from XML sources; the W3C ’ s specifi cations themselves are all coded in XML; even computer applications ’ initialization and scripting information are increasingly represented in XML. Of course, different types of information present different requirements for persistent storage. Some sorts, such as the books owned by a publisher, prob- ably need to be retained for lengthy periods of time, while others (e.g., messaging data) might have a lifetime measured in seconds or minutes. The various mecha- nisms discussed in the remainder of this section easily support the wide variety of requirements for storing XML.
10.2.1 Databases
A database , according to Wikipedia, 1 is “ an information set with a regular struc- ture. ” A database system, or database management system (DBMS), is thus (for our purposes, at least) a computer system that manages a computerized database.
While it ’ s not unknown for some people to apply the term database management system to extremely primitive data management products, the term is most often used to describe systems that provide a number of important characteristics for data integrity. Among these characteristics are:
■ Query tools, such as a query language like SQL or XQuery.
■ Transaction capabilities that include the so-called ACID properties: a tomicity of operations, c onsistency of the database as a whole, i solation from other concur- rent users ’ operations, and d urability of operations even across system crashes.
■ Scalability and robustness.
■ Management of security and performance, including registration and manage- ment of users and their privileges, creation of indices on the data, and provision hints for the optimization of operations.
1 Wikipedia, The Free Encyclopedia; available at www.en.wikipedia.org .
Several types of database management systems are in wide use by enterprises of all sorts, but we believe that only three are commonly employed to store and manage XML data: relational, object-oriented, and “ pure XML. ” All of these types of database inherently provide the ability not only to store and retrieve XML documents but also to search that data through the use of query languages of some sort. Querying XML data in a DBMS is probably more effective than querying XML data stored in other media, if for no other reason than the existence of various performance-enhancing features of a DBMS, such as indices.
It is worth noting one important consideration when storing XML in a database system: XML, by defi nition, is based on the Unicode character set. 2 Not all database systems support Unicode, and some support Unicode only when that character set was chosen when the database system was installed or when the specifi c database was created. Increasingly, however, we see that all of the major relational database systems are being updated to employ Unicode internally, implying that this may no longer be a serious issue in a few years. We have not investigated the status of Unicode in object-oriented DBMSs, but the fact that many of them have Java interfaces suggests that they may use Unicode internally. Naturally, pure-XML databases will always use Unicode internally.
Relational Databases
You won ’ t be surprised to hear that a very large fraction of persistent XML is found in relational databases, right along with other data vital to an enterprise ’ s business.
Most large businesses today — and an increasing percentage of smaller businesses — depend on relational databases to store and protect their data.
Relational database management systems (RDBMSs) have been on the scene since the early 1980s and have arguably become the most widely used form of DBMS. The billions of dollars that have been invested into commercial relational database systems (such as Oracle ’ s Oracle database, IBM ’ s DB2, and Microsoft ’ s SQL Server) have given them formidable strengths in the data management envi- ronment. Such systems are tremendously scalable, often able to handle thousands of concurrent users accessing many terabytes — even petabytes — of data.
Some say that the relational database systems — because of the two decades and billions of dollars invested in their infrastructure and code, their proven ability to adapt to new types of data, and their entrenchment in so many organiza- tions — might never be superseded in the marketplace by other, more specialized database products. Whether this is mere hubris or a realistic view of the world, we see that the vendors of RDBMS products are adapting very quickly to a world in which XML support is a major requirement.
Starting in roughly 2001, most commercial relational database vendors began adding support for XML data into their products. Initially, the focus was on merely storing XML documents and retrieving them in whole, without the ability to
2 The Unicode Standard, Version 4.1.0 (Mountain View, CA: The Unicode Consortium, 2005). Avail- able at www.unicode.org/versions/Unicode4.1.0/ .
perform any signifi cant operations on the content of those documents. Some systems merely stored serialized XML data in character string columns or CLOB (character large object) columns, while others explored ways of breaking the XML data down into component elements, attributes, and other nodes for storage into columns in various tables. (This latter mechanism, commonly called shredding the XML, is discussed in Section 10.2.3 .)
As the vendors ’ experience with — and customers ’ requirements for — XML grew, the products gained more direct support for XML as a true data type of its own. A native XML type (see Section 10.3 ) was defi ned for the use of database designers and application authors. New built-in functions were developed to transform ordinary relational data into XML structures of the users ’ choice. And a variety of ways were invented to query within XML stored in that native XML type, including the ability to invoke XPath and XQuery on that XML. In addition, these products have been given the ability to support XML metadata, largely in the form of XML schema.
Of course, we may be biased by our years of participation in the relational database world, but we believe that RDBMS products are rapidly becoming as fully capable of managing XML data as they are of managing ordinary business data.
Object-Oriented Databases
In the late 1980s and early 1990s, a new form of DBMS was introduced to the data management marketplace, the object-oriented database management system (OODBMS). Unlike the RDBMS products, OODBMS products suffered from not having a formal data model on which their design was based. As a result, the meaning of the term OODBMS varied widely between implementations. What they all had in common, of course, was that they managed objects instead of tuples of attributes or rows of columns .
Arguably, the real world is better represented as a collection of objects, each having a state (data about the individual object) and behaviors (functions that implement common semantics of classes of objects). Object-oriented program- ming languages (OOPLs) were coming into prominence (and have since tended to dominate some application domains), and it was natural to want to persistently store the objects being manipulated in OOPL programs. Some OODBMSs took the approach of allowing individual objects (or classes of objects) handled by a par- ticular OOPL program to be “ marked ” with a fl ag that indicated whether or not the object (or members of the class) were to be automatically placed into persis- tent storage, without any specifi c action (e.g., a store command) taken by the program. Others made the OODBMS an integral part of the OOPL so that storing and retrieving objects was done completely seamlessly without any application code involved. Still others required that the OOPL programs explicitly store and retrieve objects when the program made the decision to do so.
What was generally missing from all of these OODBMS products was a common query language that allowed applications to locate objects based on their states
and to retrieve information about specifi c objects. The RDBMS world had standard- ized on the database language SQL, so the OODBMS community 3 decided to adapt SQL for use as a query language in their world; the result of that adaptation is a language called OQL, which is a search and retrieval – only language without built- in update capabilities.
A signifi cant portion of the XML community views XML as naturally object- oriented; for example, every node in an XML document has a unique identity, as do objects in all object-oriented systems. Consequently, when XML became a signifi cant market force, we expected that Object Data Management Group (ODMG) would quickly move to incorporate this new type of data, if only by adapting an XML data model like the Document Object Model (DOM) 4 for use in the context of ODMG. While the owners of the ODMG standard have not yet published a new version with explicit XML support, a group of academics did just that in a system they called Ozone. 5 Subsequently, an open-source effort providing an Ozone database system 6 was established. The documentation of this effort states that “ ozone [ sic ] includes a fully W3C-compliant DOM implementation that allows you to store XML data. ”
We are unaware of any signifi cant presence in the marketplace of OODBMS products that incorporate explicit support of XML as a data type (in the sense that the Ozone system does, at least). This may be due to the fact that OODBMSs in general have found secure niches in the data management community and that those niches have little need for XML except as a data interchange format.
It may also be due to the fact that many (but not all) RDBMSs have embraced object technology and are popularly known as object-relational database management systems (ORDBMSs). In any case, we do not perceive a near-term movement toward the use of OODBMS products for large-scale management of XML data.
Native XML Databases
We were not surprised that a number of start-up companies as well as some estab- lished data management companies determined that XML data would be best managed by a DBMS that was designed specifi cally to deal with semistructured data — that is, a native XML database.
But what, exactly, is a native XML database? One resource we found 7 defi nes it in terms of the following three principle characteristics.
3 R.G.G. Cattell, et al. (eds.). The Object Data Standard (ODBM 3.0) . Morgan Kaufmann, 2000.
4 Document Object Model (DOM) Level 3 Core Specifi cation Version 1.0. Cambridge, MA: World Wide Web Consortium, 2004. Available at www.w3.org/TR/DOM-Level-3-Core .
5 Serge Abiteboul, Jennifer Widom, and Tirthankar Lahiri. A Unifi ed Approach for Querying Struc- tured Data and XML, 1998. Available at www.w3.org/TandS/QL/QL98/pp/serge.html .
6 The Ozone Database Project. Available at www.ozone-db.org .
7 Kimbro Staken. Introduction to Native XML Databases, 2001. Available at www.xml.com/pub/
a/2001/10/31/nativexmldb.html .
■ Defi nes a (logical) model for an XML document.
■ Has an XML document as its fundamental unit of (logical) storage.
■ Is not required to have any particular underlying physical storage model.
Undoubtedly, the most important of those three criteria is the fi rst one: the defi ni- tion of a model for XML documents. A number of data models for XML are in current use. The specifi c model chosen for a native XML database system is less important than the requirement that it support arbitrarily deep levels of nesting and complexity, document order, unique identity of nodes, mixed content, semi- structured data, and so on.
Unfortunately for companies that invested heavily in the development of what we call pure-XML database systems, the widely accepted defi nition of “ native XML ” database systems doesn ’ t exclude other existing technologies. The defi ni- tion cited earlier makes it clear that relational database systems can provide all of the required characteristics of a native XML database. This can be done either by building an XML-centric layer atop a relational system or by incorporating new XML-specifi c facilities directly into relational engines. Of course, that doesn ’ t mean that there is no marketplace for pure-XML DBMSs. However, we suspect that, like OODBMSs before them, pure-XML DBMSs will fi nd small but secure niches for themselves where they satisfy very specifi c needs that are not targeted by RDBMS (or ORDBMS) products.
10.2.2 Other Persistent Media
While a great proportion of enterprise XML data is managed by explicit DBMSs, we believe that a large majority of XML in the world today does not get stored in DBMSs at all. Instead, XML documents are found in ordinary operating system fi les and on Web pages. A quick search of just one of our computers found several thousand XML documents, most of which we didn ’ t even realize were there, since they were created as part of the installation of several software products.
The advantage of storing XML documents in ordinary fi les on your own com- puter is, of course, that everybody with a computer has a fi le system, while most of us don ’ t (yet) have formal DBMSs installed on our computers or even unre- stricted access to our organizations ’ DBMSs. Better yet, those fi les are completely under your control and not governed by some database administrator somewhere in your organization. Of course, there are disadvantages as well: You ’ re usually responsible for backing up your own fi les, lack of transactional control makes data loss more likely, and the problems of keeping track of perhaps thousands of XML fi les are quite tedious. Perhaps more importantly, there is usually no way to enforce any consistent relationships among those thousands of XML fi les — those documents that specify confi guration information for software products might defi ne the same operating system environment variable in multiple, incompatible ways.
Some people argue that a single XML document can be a sort of “ database in a fi le. ” If you take this type of approach, you would just mark up your data “ on
the fl y, ” making up tag names as you go. Unfortunately, unless you write a good XML schema to validate that document, it ’ s awfully diffi cult to keep that data internally consistent, because you might use different “ spellings ” of tags to repre- sent the same conceptual entity; for example, < SerialNumber > one time, < SerNum >
another time, < Serial-num > a third time, all to represent the serial numbers of the products that you own. We recommend strongly against such an approach to storing your data, although the concept might be very useful for transporting data from one environment to another — that is, as a data exchange representation.
XML documents that are found across the World Wide Web (WWW) probably don ’ t outnumber those found in ordinary fi le systems, but you are personally likely to fi nd more Web-available XML documents than there are XML documents on your personal fi le system. The problem with those Web documents is that a given website may or may not be “ reachable ” at any given time, making access to those documents somewhat less dependable at any moment than access to your own documents.
That, of course, has implications on querying those XML documents. A query facility that accesses fi les stored in your local fi le system always has access to those fi les (subject only to the availability of your fi le system), whereas a query facility that searches data on the WWW may sometimes fi nd a given document and other times not fi nd it because of websites going offl ine temporarily (or permanently).
Nonetheless, we believe there is a market for XML querying tools that don ’ t depend on the existence of a DBMS but that search XML documents in local fi le systems and across the WWW. Many of these tools will implement XQuery, while others may provide some other query language.
10.2.3 Shredding Your Data
In the “ Relational Databases ” section we mention that some relational database vendors provided a way for XML documents to be broken down into their com- ponent elements, attributes, and other nodes for storage into columns in one or more tables. It can be argued that such shredding of XML documents does not preserve the integrity — the “ XML-ness ” — of those documents. While that argu- ment is probably valid for some shredding implementations, others manage to preserve the documents ’ XML-ness. In fact, such implementations usually provide options that allow the user to control what level of XML-ness must be preserved.
Vendors of those products typically provide a variety of ways of reconstructing the XML documents from the shredded fragments. What many of the shredding implementations do not do particularly well is to allow queries to be written that depend heavily on complex structures in some XML documents or that search for data located at arbitrarily deep levels of nesting.
The purpose of shredding is to improve (relative to character string or CLOB representations, that is) the effi ciency of access to the data found in XML docu-
ments. When XML serves the same purposes as its ancestor SGML — that is, rep- resentation of documents , such as books and technical reports — the data represented in the XML are semistructured by nature. However, XML is also used to represent much more regular, or structured, data, such as purchase orders and personnel records. Most people would not consider shredding an appropriate way of handling books or magazine articles marked up in XML. Instead, it is much more likely to be used for dealing with data-oriented XML.
Shredding can be done in a very naive manner, such as defi ning an SQL table for each element type (at least those that are allowed to have mixed content) in a document, with columns for each attribute, the nonelement content of those elements, and the content of child elements that are not allowed to have element content themselves. For simple documents, the naive approach might not be completely inappropriate, as illustrated in Example 10.1 and Table 10.1 .
EXAMPLE 10.1
Shredding an XML Document into a Relational Database First, the XML to be shredded:
< movies >
< movie runtime = “ 99 ” >
< title > What About Bob? < / title >
< MPAArating > PG < / MPAArating >
< yearReleased > 1991 < / yearReleased >
< director >
< givenName > Frank < / givenName >
< familyName > Oz < / familyName >
< / director >
< / movie >
< movie runtime = “ 108 ” >
< title > A Fish Called Wanda < / title >
< MPAArating > R < / MPAArating >
< yearReleased > 1988 < / yearReleased >
< director >
< givenName > Charles < / givenName >
< familyName > Chrichton < / familyName >
< / director >
< / movie >
< movie runtime = “ 90 ” >
< title > Best in Show < / title >
< MPAArating > PG-13 < / MPAArating >
< yearReleased > 2000 < / yearReleased >
< director >
< givenName > Christopher < / givenName >
< familyName > Guest < / familyName >