RELATIONAL MANAGEMENT and DISPLAY of SITE ENVIRONMENTAL DATA © 2002 by CRC Press LLC LEWIS PUBLISHERS A CRC Press Company Boca Raton London New York Washington, D.C. RELATIONAL MANAGEMENT and DISPLAY of SITE ENVIRONMENTAL DATA David W. Rich, Ph.D. © 2002 by CRC Press LLC This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author and the publisher cannot assume responsibility for the validity of all materials or for the consequences of their use. Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage or retrieval system, without prior permission in writing from the publisher. The consent of CRC Press LLC does not extend to copying for general distribution, for promotion, for creating new works, or for resale. Specific permission must be obtained in writing from CRC Press LLC for such copying. Direct all inquiries to CRC Press LLC, 2000 N.W. Corporate Blvd., Boca Raton, Florida 33431. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation, without intent to infringe. Visit the CRC Press Web site at www.crcpress.com © 2002 by CRC Press LLC Lewis Publishers is an imprint of CRC Press LLC No claim to original U.S. Government works International Standard Book Number 1-56670-591-6 Library of Congress Card Number 2002019441 Printed in the United States of America 1 2 3 4 5 6 7 8 9 0 Printed on acid-free paper Library of Congress Cataloging-in-Publication Data Rich, David William, 1952- Relational management and display of site environmental data / David W. Rich. p. cm. Includes bibliographical references and index. ISBN 1-56670-591-6 (alk. paper) 1. Pollution—Measurement—Data processing. 2. Environmental monitoring—Data processing. 3. Database management. I. Title. TD193 .R53 2002 628.5′028′ 7—dc21 2002019441 © 2002 by CRC Press LLC PREFACE The environmental industry is changing, along with the way it manages data. Many projects are making a transition from investigation through remediation to ongoing monitoring. Data management is evolving from individual custom systems for each project to standardized, centralized databases, and many organizations are starting to realize the cost savings of this approach. The objective of Relational Management and Display of Site Environmental Data is to bring together in one place the information necessary to manage the data well, so everyone, from students to project managers, can learn how to benefit from better data management. This book has come from many sources. It started out as a set of course notes to help transfer knowledge about earth science computing and especially environmental data management to our clients as part of our software and consulting practice. While it is still used for that purpose, it has evolved into a synthesis of theory and a relation of experience in working with site environmental data. It is not intended to be the last word on the way things are or should be done, but rather to help people learn from the experience of others, and avoid mistakes whenever possible. The book has six main sections plus appendices. Part One provides an overview of the subject and some general concepts, including a discussion of system data content. Part Two covers system design and implementation, including database elements, user interface issues, and implementation and operation of the system. Part Three addresses gathering the data, starting with an overview of site investigation and remediation, progressing through gathering samples in the field, and ending with laboratory analysis. Part Four covers the data management process, including importing, editing, maintaining data quality, and managing multiple projects. Part Five is about using the data once it is in the database. It starts with selecting data, and then covers various aspects of data output and analysis including reporting and display; graphs; cross sections and similar displays; a large chapter on mapping and GIS; statistical analysis; and integration with other programs. Section Six discusses problems, benefits, and successes with implementing a site environmental data management system, along with an attempt to look into the future of data management and environmental projects. Appendices include examples of a needs assessment, a data model, a data transfer standard, typical constituent parameters, some exercises, a glossary, and a bibliography. A number of people have contributed directly and indirectly to this book, including my parents, Dr. Robert and Audrey Rich; Dr. William Fairley, my uncle and professor of geology at the University of Notre Dame; and Dr. Albert Carozzi, my advisor and friend at the University of Illinois. Numerous coworkers and friends at Texaco, Inc., Shell Oil Company, Sabine Corporation, Grant Environmental, and Geotech Computer Systems, Inc. helped bring me to the point professionally where I could write this book. These include Larry Ratliff, Jim Thomson, Dr. James L. Grant, Neil Geitner, Steve Wampler, Jim Quin, Cathryn Stewart, Bill Thoen, Judy Mitchell, Dr. Mike Wiley, and other Geotech staff members who helped with the book in various ways. Friends © 2002 by CRC Press LLC in other organizations have also helped me greatly in this process, including Jim Reed of RockWare, Tom Bresnahan of Golden Software, and other early members of the Computer Oriented Geological Society. Thanks also go to Dr. William Ganus, Roy Widmann, Sherron Hendricks, and Frank Schultz of Kerr-McGee for their guidance. I would also like to specifically thank those who reviewed all or part of the book, including Cathryn Stewart (AquAeTer), Bill Thoen (GISNet), Mike Keester (Oklahoma State University), Bill Ganus and Roy Widmann (Kerr-McGee), Mike Wiley (The Consulting Operation), and Sue Stefanosky and Steve Clough (Roy. F. Weston). The improvements are theirs. The errors are still mine. Finally, my wife, business partner, and best friend, Toni Rich, has supported me throughout my career, hanging in there through the good times and bad, and has always done what she could to make our enterprise successful. She’s also a great proofreader. Throughout this book a number of trademarks and registered trademarks are used. The registered trademarks are registered in the United States, and may be registered in other countries. Any omissions are unintentional and will be remedied in later editions. Enviro Data and Spase are registered trademarks of Geotech Computer Systems, Incorporated. Microsoft, Office, Windows, NT, Access, SQL Server, Visual Basic, Excel, and FoxPro are trademarks or registered trademarks of Microsoft Corporation. Oracle is a registered trademark of Oracle Corporation. Paradox and dBase are registered trademarks of Borland International, Incorporated. IBM and DB2 are registered trademarks of International Business Machines Corporation. AutoCAD and AutoCAD Map are registered trademarks of Autodesk, Incorporated. ArcView is a registered trademark of Environmental Systems Research Institute, Incorporated. Norton Ghost is a trademark of Symantec Corporation. Apple and Macintosh are registered trademarks of Apple Computer, Incorporated. Sun is a registered trademark and Sparcstation is a trademark of Sun Microsystems. Capability Maturity Model and CMM are registered trademarks of The Software Engineering Institute of Carnegie Mellon University. Adobe and Acrobat are registered trademarks of Adobe Systems. Grapher is a trademark and Surfer is a registered trademark of Golden Software, Inc. RockWare is a registered trademark and RockWorks and Gridzo are trademarks of RockWare, Inc. Intergraph and GeoMedia are trademarks of Intergraph Corporation. Corel is a trademark and Corel Draw is a registered trademark of Corel Corporation. UNIX is a registered trademark of The Open Group. Linux is a trademark of Linus Torvalds. Use of these products is for illustration only, and does not signify endorsement by the author. A Web site has been established for updates, exercises, and other information related to this book. It is located at www.geotech.com/relman. I welcome your comments and questions. I can be reached by email at drdave@geotech.com. David W. Rich © 2002 by CRC Press LLC AUTHOR David W. Rich is founder and president of Geotech Computer Systems, Inc. in Englewood, CO. Geotech provides off-the-shelf and custom software and consulting services for environmental data management, GIS, and other technical computing projects. Dr. Rich received his B.S. in Geology from the University of Notre Dame in 1974, and his M.S. and Ph.D. in Geology from the University of Illinois in 1977 and 1979, with his dissertation on “Porosity in Oolitic Limestones.” He worked for Texaco, Inc. in Tulsa, OK and Shell Oil Company in Houston, TX, exploring for oil and gas in Illinois and Oklahoma. He then moved to Sabine Corporation in Denver, CO as part of a team that successfully explored for oil in the Minnelusa Formation in the Powder River Basin of Wyoming. He directed the data management and graphics groups at Grant Environmental in Englewood, CO where he worked on several projects involving soil and groundwater contaminated with metals, organics, and radiologic constituents. His team created automated systems for mapping and cross section generation directly from a database. In 1986 he founded Geotech Computer Systems, Inc., where he has developed and supervised the development of custom and commercial software for data management, GIS, statistics, and Web data access. Environmental projects with which Dr. Rich has been directly involved include two Superfund wood treating sites, three radioactive material processing facilities, two hazardous waste disposal facilities, many municipal solid waste landfills, two petroleum refineries, and several mining and petroleum production and transportation projects. He has been the lead developer on three public health projects involving blood lead and related data, including detailed residential environmental measurements. In addition he has been involved in many projects outside of the environmental field, including a real-time Web-based weather mapping system, an agricultural GIS analysis tool, and database systems for petroleum exploration and production data, paleontological data, land ownership, health care tracking, parts inventory and invoice printing, and GPS data capture. Dr. Rich has been using computers since 1970, and has been applying them to earth science problems since 1975. He was a co-founder and president of the Computer Oriented Geological Society in the early 1980s, and has authored or co-authored more than a dozen technical papers, book chapters, and journal articles on environmental and petroleum data management, geology, and computer applications. He has taught many short courses on geological and environmental computing in several countries, and has given dozens of talks at various industry conventions and other events. When he is not working, Dr. Rich enjoys spending time with his family and riding his motorcycle in the mountains, and often both at the same time. © 2002 by CRC Press LLC CONTENTS PART ONE - OVERVIEW AND CONCEPTS CHAPTER 1 - OVERVIEW OF ENVIRONMENTAL DATA MANAGEMENT Concern for the environment The computer revolution Convergence - Environmental data management Concept of data vs. information EMS vs. EMIS vs. EDMS CHAPTER 2 - SITE DATA MANAGEMENT CONCEPTS Purpose of data management Types of data storage Responsibility for data management Understanding the data CHAPTER 3 - RELATIONAL DATA MANAGEMENT THEORY What is relational data management? History of relational data management Data normalization Structured Query Language Benefits of normalization Automated normalization CHAPTER 4 - DATA CONTENT Data content overview Project technical data Project administrative data Project document data Reference data Document management PART TWO - SYSTEM DESIGN AND IMPLEMENTATION CHAPTER 5 - GENERAL DESIGN ISSUES Database management software © 2002 by CRC Press LLC Database location options Distributed vs. centralized databases The data model Data access requirements Government EDMS systems Other issues CHAPTER 6 - DATABASE ELEMENTS Hardware and software components Units of data storage Databases and files Tables (“databases”) Fields (columns) Records (rows) Queries (views) Other database objects CHAPTER 7 - THE USER INTERFACE General user interface issues Conceptual guidelines Guidelines for specific elements Documentation CHAPTER 8 - IMPLEMENTING THE DATABASE SYSTEM Designing the system Buy or build? Implementing the system Managing the system CHAPTER 9 - ONGOING DATA MANAGEMENT ACTIVITIES Managing the workflow Managing the data Administering the system PART THREE - GATHERING ENVIRONMENTAL DATA CHAPTER 10 - SITE INVESTIGATION AND REMEDIATION Overview of environmental regulations The investigation and remediation process Environmental Assessments and Environmental Impact Statements CHAPTER 11 - GATHERING SAMPLES AND DATA IN THE FIELD General sampling issues Soil Sediment Groundwater Surface water Decontamination of equipment Shipping of samples Air © 2002 by CRC Press LLC Other media Overview of parameters CHAPTER 12 - ENVIRONMENTAL LABORATORY ANALYSIS Laboratory workflow Sample preparation Analytical methods Other analysis issues PART FOUR - MAINTAINING THE DATA CHAPTER 13 - IMPORTING DATA Manual entry Electronic import Tracking imports Undoing an import Tracking quality CHAPTER 14 - EDITING DATA Manual editing Automated editing CHAPTER 15 - MAINTAINING AND TRACKING DATA QUALITY QA vs. QC The QAPP QC samples and analyses Data quality procedures Database support for data quality and usability Precision vs. accuracy Protection from loss CHAPTER 16 - DATA VERIFICATION AND VALIDATION Types of data review Meaning of verification Meaning of validation The verification and validation process Verification and validation checks Software assistance with verification and validation CHAPTER 17 - MANAGING MULTIPLE PROJECTS AND DATABASES One file or many? Sharing data elements Moving between databases Limiting site access PART FIVE - USING THE DATA CHAPTER 18 - DATA SELECTION Text-based queries Graphical selection Query-by-form CHAPTER 19 - REPORTING AND DISPLAY © 2002 by CRC Press LLC Text output Formatted reports Formatting the result Interactive output Electronic distribution of data CHAPTER 20 - GRAPHS Graph overview General concepts Types of graphs Graph examples Curve fitting Graph theory CHAPTER 21 - CROSS SECTIONS, FENCE DIAGRAMS, AND 3-D DISPLAYS Lithologic and wireline logs Cross sections Profiles Fence diagrams and stick displays Block Diagrams and 3-D displays CHAPTER 22 - MAPPING AND GIS Mapping concepts Mapping software Displaying data Contouring and modeling Specialized displays CHAPTER 23 - STATISTICS AND ENVIRONMENTAL DATA Statistical concepts Types of statistical analyses Outliers and comparison with limits Toxicology and risk assessment CHAPTER 24 - INTEGRATION WITH OTHER PROGRAMS Export-import Digital output Export-import advantages and disadvantages Direct connection Data warehousing and data mining Data integration PART SIX - PROBLEMS, BENEFITS, AND SUCCESSES CHAPTER 25 - AVOIDING PROBLEMS Manage expectations Use the right tool Prepare for problems with the data Plan project administration Increasing the chance of a positive outcome © 2002 by CRC Press LLC [...]... LLC Well B -1 B -1 B -1 B -1 B -1 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-3 B-3 B-3 B-3 B-3 Elev 725 725 725 725 725 706 706 706 706 706 706 706 706 706 714 714 714 714 714 X 10 50 10 50 10 50 10 50 10 50 342 342 342 342 342 342 342 342 342 785 785 785 785 785 Y 6 81 6 81 6 81 6 81 6 81 880 880 880 880 880 880 880 880 880 11 01 110 1 11 01 110 1 11 01 SampDate 2/3/96 2/3/96 5/8/96 5/8/96 5/8/96 11 /4/95 11 /4/95 11 /4/95 2/3/96... Stations Well B -1 B-2 B-3 Elev 725 706 714 Analyses X 10 50 342 785 Y 6 81 880 11 01 Samples Well B -1 B -1 B-2 B-2 B-2 B-3 B-3 SampDate 2/3/96 5/8/96 11 /4/95 2/3/96 5/8/96 2/3/96 5/8/96 Sampler JLG DWR JAM JLG DWR JLG CRS Well B -1 B -1 B -1 B -1 B -1 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-2 B-3 B-3 B-3 B-3 B-3 SampDate 2/3/96 2/3/96 5/8/96 5/8/96 5/8/96 11 /4/95 11 /4/95 11 /4/95 2/3/96 2/3/96 2/3/96 5/8/96 5/8/96 5/8/96... Each column contains the same kind of data An example of a flat file of environmental data is shown in the following table: Well B -1 B -1 B-2 B-2 B-2 B-3 B-3 Elev 725 725 706 706 706 714 714 X 10 50 10 50 342 342 342 785 785 Y 6 81 6 81 880 880 880 11 01 110 1 SampDate 2/3/96 5/8/96 11 /4/95 2/3/96 5/8/96 2/3/96 5/8/96 Sampler JLG DWR JAM JLG DWR JLG CRS As 05 05 3.7 2 .1 1.4 05 05 AsFlag not det not det detected... Parameter 1 As 2 Cl 3 pH Y 6 81 880 11 01 Order 1 2 3 Sampler JLG DWR JAM JLG DWR JLG CRS Analyses SampleID Param 1 1 1 3 2 1 2 2 2 3 3 1 3 2 3 3 4 1 4 2 4 3 5 1 5 2 5 3 6 1 6 3 7 1 7 2 7 3 Value 05 6.8 05 05 6.7 3.7 9 .1 5.2 2 .1 8.4 5.3 1. 4 7.2 5.8 05 8 .1 05 05 7.9 Flag u v u u v v v v v v v v v v u v u u v Figure 8 - Environmental data in fifth normal form with simple keys and coded values Data retrieval... facilities and orphan sites, is growing as well The volume of data often exceeds the capacity of simple tools like paper reports and spreadsheets When that happens it is appropriate to implement a more powerful data management system and often the system of choice is a relational database manager This section provides a top-down discussion of management of environmental data It focuses on the purpose and. .. the data be in First Normal Form (rows and columns, no repeating groups) before they can work with it In this section we will go through the process of normalizing a data set We will start with a flat file of site analytical data in a format similar to the way it would be received from a laboratory This is shown in Figure 4 Well B -1 B -1 B-2 B-2 B-2 B-3 B-3 Elev 725 725 706 706 706 714 714 X 10 50 10 50... sidebar), the advent of personal computers in the 19 80s made it possible to use them effectively on environmental projects For more information on the history of computers, see Augarten (19 84) and Evans (19 81) Discussions of the history of geological use of computers are contained in Merriam (19 83 ,19 85) With the advent of Windows-based, consumer-oriented database management programs in the 19 90s, the tools... 342 785 785 Y 6 81 6 81 880 880 880 11 01 110 1 SampDate 2/3/96 5/8/96 11 /4/95 2/3/96 5/8/96 2/3/96 5/8/96 Sampler JLG DWR JAM JLG DWR JLG CRS As 05 05 3.7 2 .1 1.4 05 05 AsFlag not det not det detected detected detected not det not det Cl ClFlag 05 9 .1 8.4 7.2 not det detected detected detected 05 not det pH 6.8 6.7 5.2 5.3 5.8 8 .1 7.9 Figure 4 - Environmental data prior to normalization - Problems: Repeating... in relational databases, while the rest is in flat files, audio, video, pre -relational, and unstructured formats Flat file A flat file is a two-dimensional array of data organized in rows and columns similar to a spreadsheet This is the simplest type of database manager All of the data for a particular type of object is stored in a single file or table, and each record can have one instance of data. .. limit to the number of rows and columns that they contain, and these limits can easily be exceeded by a large data set For example, Lotus 12 3 has a limit of about 16 ,000 rows of data, and Excel 97 has a limit of 65,536 rows Spreadsheets do have their place in working with environmental data They are particularly useful for statistical analysis of data and for graphing in a variety of ways Spreadsheets . David William, 19 5 2- Relational management and display of site environmental data / David W. Rich. p. cm. Includes bibliographical references and index. ISBN 1- 5 667 0-5 9 1- 6 (alk. paper) 1. Pollution—Measurement Data. of data vs. information EMS vs. EMIS vs. EDMS CHAPTER 2 - SITE DATA MANAGEMENT CONCEPTS Purpose of data management Types of data storage Responsibility for data management Understanding the data CHAPTER. CONTENTS PART ONE - OVERVIEW AND CONCEPTS CHAPTER 1 - OVERVIEW OF ENVIRONMENTAL DATA MANAGEMENT Concern for the environment The computer revolution Convergence - Environmental data management Concept