Refactoring SQL Applications Other resources from O’Reilly Related titles oreilly.com The Art of SQL Learning SQL Making Things Happen SQL in a Nutshell SQL Pocket Guide oreilly.com is more than a complete catalog of O’Reilly books You’ll also find links to news, events, articles, weblogs, sample chapters, and code examples oreillynet.com is the essential portal for developers interested in open and emerging technologies, including new platforms, programming languages, and operating systems Conferences O’Reilly brings diverse innovators together to nurture the ideas that spark revolutionary industries We specialize in documenting the latest tools and systems, translating the innovator’s knowledge into useful skills for those in the trenches Visit conferences.oreilly.com for our upcoming events Safari Bookshelf (safari.oreilly.com) is the premier online reference library for programmers and IT professionals Conduct searches across more than 1,000 books Subscribers can zero in on answers to time-critical questions in a matter of seconds Read the books on your Bookshelf from cover to cover or simply flip to the page you need Try it today for free Refactoring SQL Applications Stéphane Faroult with Pascal L’Hermite Beijing • Cambridge • Farnham • Kưln • Sebastopol • Taipei • Tokyo Refactoring SQL Applications by Stéphane Faroult with Pascal L’Hermite Copyright © 2008 Stéphane Faroult and Pascal L’Hermite All rights reserved Printed in the United States of America Published by O’Reilly Media, Inc 1005 Gravenstein Highway North, Sebastopol, CA 95472 O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (safari.oreilly.com) For more information, contact our corporate/institutional sales department: (800) 998-9938 or corporate@oreilly.com Editor: Mary Treseler Cover Designer: Mark Paglietti Production Editor: Rachel Monaghan Interior Designer: Marcia Friedman Copyeditor: Audrey Doyle Illustrator: Robert Romano Indexer: Lucie Haskins Printing History: August 2008: First Edition The O’Reilly logo is a registered trademark of O’Reilly Media, Inc Refactoring SQL Applications and related trade dress are trademarks of O’Reilly Media, Inc Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks Where those designations appear in this book, and O’Reilly Media, Inc was aware of a trademark claim, the designations have been printed in caps or initial caps Java™ is a trademark of Sun Microsystems, Inc While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein This book uses RepKover™, a durable and flexible lay-flat binding ISBN: 978-0-596-51497-6 [M] CONTENTS PREFACE ASSESSMENT A Simple Example Assessing Possible Gains 19 SANITY CHECKS 37 Statistics and Data Skewness Indexing Review Parsing and Bind Variables Bulk Operations Transaction Management 38 44 55 70 73 USER FUNCTIONS AND VIEWS 75 User-Defined Functions Views vii 76 103 TESTING FRAMEWORK 115 Generating Test Data Comparing Alternative Versions 116 132 STATEMENT REFACTORING 147 Execution Plans and Optimizer Directives Analyzing a Slow Query Refactoring the Query Core Rebuilding the Initial Query 148 152 158 176 TASK REFACTORING 179 The SQL Mindset Restructuring the Code 180 185 REFACTORING FLOWS AND DATABASES 211 Reorganizing Processing Shaking Foundations 212 233 v vi C O N T E N T S HOW IT WORKS: REFACTORING IN PRACTICE 243 Can You Look at the Database? Queries of Death All These Fast Queries No Obvious Very Wrong Query Time to Conclude 243 245 247 248 249 A SCRIPTS AND SAMPLE PROGRAMS 251 B TOOLS 261 INDEX 269 Chapter Preface Ma, sendo l’intento mio scrivere cosa utile a chi la intende, mi è parso più conveniente andare drieto alla verità effettuale della cosa, che alla immaginazione di essa But, it being my intention to write a thing which shall be useful to him who apprehends it, it appears to me more appropriate to follow up the real truth of a matter than the imagination of it —Niccolò Machiavelli Il Principe, XV T HERE IS A STORY BEHIND THIS BOOK I HAD HARDLY FINISHED T HE A RT OF SQL, WHICH WASN ’ T ON sale yet, when my then editor, Jonathan Gennick, raised the idea of writing a book about SQL refactoring SQL, I knew But I had never heard about refactoring I Googled the word In a famous play by Molière, a wealthy but little-educated man who takes lessons in his mature years marvels when he discovers that he has been speaking “prose” for all his life Like Monsieur Jourdain, I discovered that I had been refactoring SQL code for years without even knowing it—performance analysis for my customers led quite naturally to improving code through small, incremental changes that didn’t alter program behavior It is one thing to try to design a database as best as you can, and to lay out an architecture and programs that access this database efficiently It is another matter to try to get the best performance from systems that were not necessarily well designed from the start, or which have grown out of control over the years but that you have to live with And there was something appealing in the idea of presenting SQL from a point of view that is so often mine in my professional life The last thing you want to when you are done with a book is to start writing another one But the idea had caught my fancy I discussed it with a number of friends, one of whom is one of the most redoubtable SQL specialists I know This friend burst into righteous vii indignation against buzzwords For once, I begged to differ with him It is true that the idea first popularized by Martin Fowler* of improving code by small, almost insignificant, localized changes may look like a fad—the stuff that fills reports by corporate consultants who have just graduated from university But for me, the true significance of refactoring lies in the fact that code that has made it to production is no longer considered sacred, and in the recognition that a lot of mediocre systems could, with a little effort, much better Refactoring is also the acknowledgment that the fault for unsatisfactory performance is in ourselves, not in our stars—and this is quite a revelation in the corporate world I have seen too many sites where IT managers had an almost tragic attitude toward performance, people who felt crushed by fate and were putting their last hope into “tuning.” If the efforts of database and system administrators failed, the only remaining option in their view was to sign and send the purchase order for more powerful machines I have read too many audit reports by self-styled database experts who, after reformatting the output of system utilities, concluded that a few parameters should be bumped up and that more memory should be added To be fair, some of these reports mentioned that a couple of terrible queries “should be tuned,” without being much more explicit than pasting execution plans as appendixes I haven’t touched database parameters for years (the technical teams of my customers are usually competent) But I have improved many programs, fearlessly digging into them, and I have tried as much as I could to work with developers, rather than stay in my ivory tower and prescribe from far above I have mostly met people who were eager to learn and understand, who needed little encouragement when put on the right tracks, who enjoyed developing their SQL skills, and who soon began to set performance targets for themselves When the passing of time wiped from my memory the pains of book writing, I took the plunge and began to write again, with the intent to expand the ideas I usually try to transmit when I work with developers Database accesses are probably one of the areas where there is the most to gain by improving the code My purpose in writing this book has been to give not recipes, but a framework to try to improve the less-than-ideal SQL applications that surround us without rewriting them from scratch (in spite of a very strong temptation sometimes) Why Refactor? Most applications bump, sooner or later, into performance issues In the best of cases, the success of some old and venerable application has led it to handle, over time, volumes of data for which it had never been designed, and the old programs need to be given a new lease on life until a replacement application is rolled out in production In the worst of cases, performance tests conducted before switching to production may reveal a dismal failure to meet service-level requirements Somewhere in between, data volume * Fowler, M et al Refactoring: Improving the Design of Existing Code Boston: Addison-Wesley Professional viii PREFACE increases, new functionalities, software upgrades, or configuration changes sometimes reveal flaws that had so far remained hidden, and backtracking isn’t always an option All of those cases share extremely tight deadlines to improve performance, and high pressure levels The first rescue expedition is usually mounted by system engineers and database administrators who are asked to perform the magical parameter dance Unless some very big mistake has been overlooked (it happens), database and system tuning often improves performance only marginally At this point, the traditional next step has long been to throw more hardware at the application This is a very costly option, because the price of hardware will probably be compounded by the higher cost of software licenses It will interrupt business operations It requires planning Worryingly, there is no real guarantee of return on investment More than one massive hardware upgrade has failed to live up to expectations It may seem counterintuitive, but there are horror stories of massive hardware upgrades that actually led to performance degradation There are cases when adding more processors to a machine simply increased contention among competing processes The concept of refactoring introduces a much-needed intermediate stage between tuning and massive hardware injection Martin Fowler’s seminal book on the topic focuses on object technologies But the context of databases is significantly different from the context of application programs written in an object or procedural language, and the differences bring some particular twists to refactoring efforts For instance: Small changes are not always what they appear to be Due to the declarative nature of SQL, a small change to the code often brings a massive upheaval in what the SQL engine executes, which leads to massive performance changes—for better or for worse Testing the validity of a change may be difficult If it is reasonably easy to check that a value returned by a function is the same in all cases before and after a code change, it is a different matter to check that the contents of a large table are still the same after a major update statement rewrite The context is often critical Database applications may work satisfactorily for years before problems emerge; it’s often when volumes or loads cross some thresholds, or when a software upgrade changes the behavior of the optimizer, that performance suddenly becomes unacceptable Performance improvement work on database applications usually takes place in a crisis Database applications are therefore a difficult ground for refactoring, but at the same time the endeavor can also be, and often is, highly rewarding PREFACE ix .. .Refactoring SQL Applications Other resources from O’Reilly Related titles oreilly. com The Art of SQL Learning SQL Making Things Happen SQL in a Nutshell SQL Pocket Guide oreilly. com... Try it today for free Refactoring SQL Applications Stéphane Faroult with Pascal L’Hermite Beijing • Cambridge • Farnham • Kưln • Sebastopol • Taipei • Tokyo Refactoring SQL Applications by Stéphane... Indexer: Lucie Haskins Printing History: August 2008: First Edition The O’Reilly logo is a registered trademark of O’Reilly Media, Inc Refactoring SQL Applications and related trade dress are