Oracle® Database Data Warehousing Guide 10g Release 1 (10.1) Part No. B10736-01 December 2003 Oracle Database Data Warehousing Guide, 10g Release 1 (10.1) Part No. B10736-01 Copyright © 2001, 2003 Oracle Corporation. All rights reserved. Primary Author: Paul Lane Contributing Authors: Viv Schupmann, Ingrid Stuart (Change Data Capture) Contributors: Patrick Amor, Hermann Baer, Mark Bauer, Subhransu Basu, Srikanth Bellamkonda, Randy Bello, Tolga Bozkaya, Lucy Burgess, Rushan Chen, Benoit Dageville, John Haydu, Lilian Hobbs, Hakan Jakobsson, George Lumpkin, Alex Melidis, Valarie Moore, Cetin Ozbutun, Ananth Raghavan, Jack Raitto, Ray Roccaforte, Sankar Subramanian, Gregory Smith, Murali Thiyagarajan, Ashish Thusoo, Thomas Tong, Jean-Francois Verrier, Gary Vincent, Andreas Walter, Andy Witkowski, Min Xiao, Tsae-Feng Yu The Programs (which include both the software and documentation) contain proprietary information of Oracle Corporation; they are provided under a license agreement containing restrictions on use and disclosure and are also protected by copyright, patent and other intellectual and industrial property laws. Reverse engineering, disassembly or decompilation of the Programs, except to the extent required to obtain interoperability with other independently created software or as specified by law, is prohibited. The information contained in this document is subject to change without notice. If you find any problems in the documentation, please report them to us in writing. Oracle Corporation does not warrant that this document is error-free. Except as may be expressly permitted in your license agreement for these Programs, no part of these Programs may be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without the express written permission of Oracle Corporation. If the Programs are delivered to the U.S. Government or anyone licensing or using the programs on behalf of the U.S. Government, the following notice is applicable: Restricted Rights Notice Programs delivered subject to the DOD FAR Supplement are "commercial computer software" and use, duplication, and disclosure of the Programs, including documentation, shall be subject to the licensing restrictions set forth in the applicable Oracle license agreement. Otherwise, Programs delivered subject to the Federal Acquisition Regulations are "restricted computer software" and use, duplication, and disclosure of the Programs shall be subject to the restrictions in FAR 52.227-19, Commercial Computer Software - Restricted Rights (June, 1987). Oracle Corporation, 500 Oracle Parkway, Redwood City, CA 94065. The Programs are not intended for use in any nuclear, aviation, mass transit, medical, or other inherently dangerous applications. It shall be the licensee's responsibility to take all appropriate fail-safe, backup, redundancy, and other measures to ensure the safe use of such applications if the Programs are used for such purposes, and Oracle Corporation disclaims liability for any damages caused by such use of the Programs. Oracle is a registered trademark, and Express, Oracle8i, Oracle9i, Oracle Store, PL/SQL, Pro*C, and SQL*Plus are trademarks or registered trademarks of Oracle Corporation. Other names may be trademarks of their respective owners. iii Contents Send Us Your Comments xxix Preface xxxi Audience xxxii Organization xxxii Related Documentation xxxiv Conventions xxxv Documentation Accessibility xxxviii What's New in Oracle Database? xxxix Oracle Database 10g Release 1 (10.1) New Features in Data Warehousing xl Volume 1 Part I Concepts 1 Data Warehousing Concepts What is a Data Warehouse? 1-2 Subject Oriented 1-2 Integrated 1-2 Nonvolatile 1-3 Time Variant 1-3 Contrasting OLTP and Data Warehousing Environments 1-3 Data Warehouse Architectures 1-5 iv Data Warehouse Architecture (Basic) 1-5 Data Warehouse Architecture (with a Staging Area) 1-6 Data Warehouse Architecture (with a Staging Area and Data Marts) 1-6 Part II Logical Design 2 Logical Design in Data Warehouses Logical Versus Physical Design in Data Warehouses 2-2 Creating a Logical Design 2-2 Data Warehousing Schemas 2-3 Star Schemas 2-4 Other Schemas 2-4 Data Warehousing Objects 2-5 Fact Tables 2-5 Creating a New Fact Table 2-5 Dimension Tables 2-6 Hierarchies 2-6 Typical Dimension Hierarchy 2-7 Unique Identifiers 2-7 Relationships 2-7 Example of Data Warehousing Objects and Their Relationships 2-8 Part III Physical Design 3 Physical Design in Data Warehouses Moving from Logical to Physical Design 3-2 Physical Design 3-2 Physical Design Structures 3-3 Tablespaces 3-4 Tables and Partitioned Tables 3-4 Table Compression 3-5 Views 3-5 Integrity Constraints 3-5 Indexes and Partitioned Indexes 3-6 v Materialized Views 3-6 Dimensions 3-6 4 Hardware and I/O Considerations in Data Warehouses Overview of Hardware and I/O Considerations in Data Warehouses 4-2 Configure I/O for Bandwidth not Capacity 4-2 Stripe Far and Wide 4-3 Use Redundancy 4-3 Test the I/O System Before Building the Database 4-4 Plan for Growth 4-4 Storage Management 4-4 5 Parallelism and Partitioning in Data Warehouses Overview of Parallel Execution 5-2 When to Implement Parallel Execution 5-2 Granules of Parallelism 5-3 Block Range Granules 5-3 Partition Granules 5-4 Partitioning Design Considerations 5-4 Types of Partitioning 5-4 Partitioning Methods 5-5 Index Partitioning 5-9 Performance Issues for Range, List, Hash, and Composite Partitioning 5-9 Partitioning and Table Compression 5-16 Table Compression and Bitmap Indexes 5-17 Example of Table Compression and Partitioning 5-18 Partition Pruning 5-19 Pruning Using DATE Columns 5-20 Avoiding I/O Bottlenecks 5-20 Partition-Wise Joins 5-20 Full Partition-Wise Joins 5-20 Partial Partition-wise Joins 5-26 Benefits of Partition-Wise Joins 5-28 Performance Considerations for Parallel Partition-Wise Joins 5-29 Partitioning and Subpartitioning Columns and Keys 5-30 vi Partition Bounds for Range Partitioning 5-31 Comparing Partitioning Keys with Partition Bounds 5-31 MAXVALUE 5-31 Nulls 5-32 DATE Datatypes 5-32 Multicolumn Partitioning Keys 5-33 Implicit Constraints Imposed by Partition Bounds 5-33 Index Partitioning 5-33 Local Partitioned Indexes 5-34 Global Partitioned Indexes 5-37 Summary of Partitioned Index Types 5-39 The Importance of Nonprefixed Indexes 5-40 Performance Implications of Prefixed and Nonprefixed Indexes 5-40 Guidelines for Partitioning Indexes 5-41 Physical Attributes of Index Partitions 5-42 6 Indexes Using Bitmap Indexes in Data Warehouses 6-2 Benefits for Data Warehousing Applications 6-2 Cardinality 6-3 Bitmap Indexes and Nulls 6-5 Bitmap Indexes on Partitioned Tables 6-6 Using Bitmap Join Indexes in Data Warehouses 6-6 Four Join Models for Bitmap Join Indexes 6-6 Bitmap Join Index Restrictions and Requirements 6-9 Using B-Tree Indexes in Data Warehouses 6-10 Using Index Compression 6-10 Choosing Between Local Indexes and Global Indexes 6-11 7 Integrity Constraints Why Integrity Constraints are Useful in a Data Warehouse 7-2 Overview of Constraint States 7-3 Typical Data Warehouse Integrity Constraints 7-3 UNIQUE Constraints in a Data Warehouse 7-4 FOREIGN KEY Constraints in a Data Warehouse 7-5 vii RELY Constraints 7-6 Integrity Constraints and Parallelism 7-6 Integrity Constraints and Partitioning 7-7 View Constraints 7-7 8 Basic Materialized Views Overview of Data Warehousing with Materialized Views 8-2 Materialized Views for Data Warehouses 8-2 Materialized Views for Distributed Computing 8-3 Materialized Views for Mobile Computing 8-3 The Need for Materialized Views 8-3 Components of Summary Management 8-5 Data Warehousing Terminology 8-7 Materialized View Schema Design 8-8 Schemas and Dimension Tables 8-8 Materialized View Schema Design Guidelines 8-9 Loading Data into Data Warehouses 8-10 Overview of Materialized View Management Tasks 8-11 Types of Materialized Views 8-12 Materialized Views with Aggregates 8-12 Requirements for Using Materialized Views with Aggregates 8-15 Materialized Views Containing Only Joins 8-15 Materialized Join Views FROM Clause Considerations 8-16 Nested Materialized Views 8-17 Why Use Nested Materialized Views? 8-17 Nesting Materialized Views with Joins and Aggregates 8-19 Nested Materialized View Usage Guidelines 8-19 Restrictions When Using Nested Materialized Views 8-20 Creating Materialized Views 8-20 Creating Materialized Views with Column Alias Lists 8-21 Naming Materialized Views 8-22 Storage And Table Compression 8-22 Build Methods 8-23 Enabling Query Rewrite 8-24 Query Rewrite Restrictions 8-24 viii Materialized View Restrictions 8-24 General Query Rewrite Restrictions 8-25 Refresh Options 8-25 General Restrictions on Fast Refresh 8-27 Restrictions on Fast Refresh on Materialized Views with Joins Only 8-27 Restrictions on Fast Refresh on Materialized Views with Aggregates 8-27 Restrictions on Fast Refresh on Materialized Views with UNION ALL 8-29 Achieving Refresh Goals 8-30 Refreshing Nested Materialized Views 8-30 ORDER BY Clause 8-31 Materialized View Logs 8-31 Using the FORCE Option with Materialized View Logs 8-33 Using Oracle Enterprise Manager 8-33 Using Materialized Views with NLS Parameters 8-33 Adding Comments to Materialized Views 8-33 Registering Existing Materialized Views 8-34 Choosing Indexes for Materialized Views 8-36 Dropping Materialized Views 8-37 Analyzing Materialized View Capabilities 8-37 Using the DBMS_MVIEW.EXPLAIN_MVIEW Procedure 8-37 DBMS_MVIEW.EXPLAIN_MVIEW Declarations 8-38 Using MV_CAPABILITIES_TABLE 8-38 MV_CAPABILITIES_TABLE.CAPABILITY_NAME Details 8-40 MV_CAPABILITIES_TABLE Column Details 8-42 9 Advanced Materialized Views Partitioning and Materialized Views 9-2 Partition Change Tracking 9-2 Partition Key 9-3 Join Dependent Expression 9-4 Partition Marker 9-5 Partial Rewrite 9-6 Partitioning a Materialized View 9-7 Partitioning a Prebuilt Table 9-7 Benefits of Partitioning a Materialized View 9-8 ix Rolling Materialized Views 9-9 Materialized Views in OLAP Environments 9-9 OLAP Cubes 9-9 Partitioning Materialized Views for OLAP 9-10 Compressing Materialized Views for OLAP 9-11 Materialized Views with Set Operators 9-11 Examples of Materialized Views Using UNION ALL 9-11 Materialized Views and Models 9-13 Invalidating Materialized Views 9-14 Security Issues with Materialized Views 9-14 Querying Materialized Views with Virtual Private Database 9-15 Using Query Rewrite with Virtual Private Database 9-16 Restrictions with Materialized Views and Virtual Private Database 9-16 Altering Materialized Views 9-17 10 Dimensions What are Dimensions? 10-2 Creating Dimensions 10-4 Dropping and Creating Attributes with Columns 10-8 Multiple Hierarchies 10-9 Using Normalized Dimension Tables 10-10 Viewing Dimensions 10-11 Using Oracle Enterprise Manager 10-11 Using the DESCRIBE_DIMENSION Procedure 10-11 Using Dimensions with Constraints 10-12 Validating Dimensions 10-12 Altering Dimensions 10-14 Deleting Dimensions 10-14 Part IV Managing the Data Warehouse Environment 11 Overview of Extraction, Transformation, and Loading Overview of ETL in Data Warehouses 11-2 ETL Tools for Data Warehouses 11-3 x Daily Operations in Data Warehouses 11-3 Evolution of the Data Warehouse 11-4 12 Extraction in Data Warehouses Overview of Extraction in Data Warehouses 12-2 Introduction to Extraction Methods in Data Warehouses 12-2 Logical Extraction Methods 12-3 Full Extraction 12-3 Incremental Extraction 12-3 Physical Extraction Methods 12-4 Online Extraction 12-4 Offline Extraction 12-4 Change Data Capture 12-5 Timestamps 12-6 Partitioning 12-6 Triggers 12-6 Data Warehousing Extraction Examples 12-7 Extraction Using Data Files 12-7 Extracting into Flat Files Using SQL*Plus 12-8 Extracting into Flat Files Using OCI or Pro*C Programs 12-9 Exporting into Export Files Using the Export Utility 12-10 Extracting into Export Files Using External Tables 12-10 Extraction Through Distributed Operations 12-11 13 Transportation in Data Warehouses Overview of Transportation in Data Warehouses 13-2 Introduction to Transportation Mechanisms in Data Warehouses 13-2 Transportation Using Flat Files 13-2 Transportation Through Distributed Operations 13-2 Transportation Using Transportable Tablespaces 13-3 Transportable Tablespaces Example 13-3 Other Uses of Transportable Tablespaces 13-6 [...]... Oracle Database Enterprise Edition or Oracle Database Personal Edition, and some of these are optional For example, to create partitioned tables and indexes, you must have the Oracle Database Enterprise Edition or Oracle Database Personal Edition xxxi Audience This guide is intended for database administrators, system administrators, and database application developers who design, maintain, and use data. .. information about Oracle's data warehousing capabilities This preface contains these topics: s Audience s Organization s Related Documentation s Conventions s Documentation Accessibility Note: The Oracle Data Warehousing Guide contains information that describes the features and functionality of the Oracle Database Standard Edition, Oracle Database Enterprise Edition, and Oracle Database Personal Edition... relational database concepts, basic Oracle server concepts, and the operating system environment under which you are running Oracle Organization This document contains: Part 1: Concepts Chapter 1, "Data Warehousing Concepts" This chapter contains an overview of data warehousing concepts Part 2: Logical Design Chapter 2, "Logical Design in Data Warehouses" This chapter discusses the logical design of a data. .. describes using analytic services and data mining in combination with Oracle Database1 0g Chapter 24, "Using Parallel Execution" This chapter describes how to tune data warehouses using parallel execution Glossary The glossary defines important terms used in this guide Related Documentation For more information, see these Oracle resources: s Oracle Database Performance Tuning Guide Many of the examples in this... Loading and Transformation in Data Warehouses Transformation Flow Multistage Data Transformation Pipelined Data Transformation Loading Mechanisms Loading a Data Warehouse with SQL*Loader Loading a Data Warehouse with External Tables Loading a Data Warehouse with OCI and Direct-Path APIs Loading a Data Warehouse with Export/Import... chapter describes transforming and loading data in data warehouses Chapter 15, "Maintaining the Data Warehouse" This chapter describes how to refresh a data warehouse Chapter 16, "Change Data Capture" This chapter describes how to use Change Data Capture capabilities Chapter 17, "SQLAccess Advisor" This chapter describes how to use the SQLAccess Advisor xxxiii Part 5: Data Warehouse Performance Chapter 18,... 15-29 15-29 15-29 15-31 15-32 15-33 Change Data Capture Overview of Change Data Capture 16-2 Capturing Change Data Without Change Data Capture 16-2 Capturing Change Data with Change Data Capture 16-4 Publish and Subscribe Model 16-5 Publisher 16-6 Subscribers 16-7 Change Sources and Modes of Data Capture 16-9 Synchronous ... 22-45 22-46 22-46 22-47 22-47 23-2 23-2 23-2 23-3 Manageability Backup and Recovery Security Oracle Data Mining Overview Enabling Data Mining Applications Data Mining in the Database Data Preparation Model Building Model Evaluation Model Apply (Scoring) ODM Programmatic... useful in data warehousing environments Chapter 20, "SQL for Aggregation in Data Warehouses" This chapter explains how to use SQL aggregation in data warehouses Chapter 21, "SQL for Analysis and Reporting" This chapter explains how to use analytic functions in data warehouses Chapter 22, "SQL for Modeling" This chapter explains how to use the spreadsheet clause for SQL modeling Chapter 23, "OLAP and Data. .. how to use dimensions in data warehouses Part 4: Managing the Data Warehouse Environment Chapter 11, "Overview of Extraction, Transformation, and Loading" This chapter is an overview of the ETL process Chapter 12, "Extraction in Data Warehouses" This chapter describes extraction issues Chapter 13, "Transportation in Data Warehouses" This chapter describes transporting data in data warehouses Chapter . Oracle® Database Data Warehousing Guide 10g Release 1 (10.1) Part No. B10736-01 December 2003 Oracle Database Data Warehousing Guide, 10g Release. xxxviii What's New in Oracle Database? xxxix Oracle Database 10g Release 1 (10.1) New Features in Data Warehousing xl Volume 1 Part I Concepts 1 Data Warehousing Concepts What