MySQL High Availability, Second Edition by Charles Bell, Mats Kindahl, and Lars Thalmann Copyright © 2014 Charles Bell, Mats Kindahl, Lars Thalmann All rights reserved Printed in the United States of America Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://my.safaribooksonline.com) For more information, contact our corporate/ institutional sales department: 800-998-9938 or corporate@oreilly.com Editor: Andy Oram Production Editor: Nicole Shelby Copyeditor: Jasmine Kwityn Proofreader: Linley Dolby June 2010: First Edition April 2014: Second Edition Indexer: Lucie Haskins Cover Designer: Karen Montgomery Interior Designer: David Futato Illustrator: Rebecca Demarest Revision History for the Second Edition: 2014-04-09: First release See http://oreilly.com/catalog/errata.csp?isbn=9781449339586 for release details Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc MySQL High Availability, the image of an American robin, and related trade dress are trademarks of O’Reilly Media, Inc Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks Where those designations appear in this book, and O’Reilly Media, Inc was aware of a trademark claim, the designations have been printed in caps or initial caps While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein ISBN: 978-1-449-33958-6 [LSI] Table of Contents Foreword for the Second Edition xv Foreword for the First Edition xix Preface xxi Part I High Availability and Scalability Introduction What’s This Replication Stuff, Anyway? So, Backups Are Not Needed Then? What’s With All the Monitoring? Is There Anything Else I Can Read? Conclusion 7 MySQL Replicant Library 11 Basic Classes and Functions Supporting Different Operating Systems Servers Server Roles Conclusion 15 16 17 19 21 MySQL Replication Fundamentals 23 Basic Steps in Replication Configuring the Master Configuring the Slave Connecting the Master and Slave A Brief Introduction to the Binary Log What’s Recorded in the Binary Log Watching Replication in Action The Binary Log’s Structure and Content 24 25 27 28 29 30 30 33 iii Adding Slaves Cloning the Master Cloning a Slave Scripting the Clone Operation Performing Common Tasks with Replication Reporting Conclusion 35 37 39 41 42 43 49 The Binary Log 51 Structure of the Binary Log Binlog Event Structure Event Checksums Logging Statements Logging Data Manipulation Language Statements Logging Data Definition Language Statements Logging Queries LOAD DATA INFILE Statements Binary Log Filters Triggers, Events, and Stored Routines Stored Procedures Stored Functions Events Special Constructions Nontransactional Changes and Error Handling Logging Transactions Transaction Cache Distributed Transaction Processing Using XA Binary Log Group Commit Row-Based Replication Enabling Row-based Replication Using Mixed Mode Binary Log Management The Binary Log and Crash Safety Binlog File Rotation Incidents Purging the Binlog File The mysqlbinlog Utility Basic Usage Interpreting Events Binary Log Options and Variables Options for Row-Based Replication iv | Table of Contents 52 54 56 58 58 59 59 65 67 70 75 78 81 82 83 86 87 91 94 97 98 99 100 100 101 103 104 105 106 113 118 120 Conclusion 121 Replication for High Availability 123 Redundancy Planning Slave Failures Master Failures Relay Failures Disaster Recovery Procedures Hot Standby Dual Masters Slave Promotion Circular Replication Conclusion 124 126 127 127 127 127 128 130 135 144 149 151 MySQL Replication for Scale-Out 153 Scaling Out Reads, Not Writes The Value of Asynchronous Replication Managing the Replication Topology Application-Level Load Balancing Hierarchical Replication Setting Up a Relay Server Adding a Relay in Python Specialized Slaves Filtering Replication Events Using Filtering to Partition Events to Slaves Managing Consistency of Data Consistency in a Nonhierarchical Deployment Consistency in a Hierarchical Deployment Conclusion 155 156 158 162 170 171 172 173 174 176 177 178 180 187 Data Sharding 189 What Is Sharding? Why Should You Shard? Limitations of Sharding Elements of a Sharding Solution High-Level Sharding Architecture Partitioning the Data Shard Allocation Mapping the Sharding Key Sharding Scheme 190 191 192 194 196 197 202 206 206 Table of Contents | v Shard Mapping Functions Processing Queries and Dispatching Transactions Handling Transactions Dispatching Queries Shard Management Moving a Shard to a Different Node Splitting Shards Conclusion 210 215 216 218 220 220 225 225 Replication Deep Dive 227 Replication Architecture Basics The Structure of the Relay Log The Replication Threads Starting and Stopping the Slave Threads Running Replication over the Internet Setting Up Secure Replication Using Built-in Support Setting Up Secure Replication Using Stunnel Finer-Grained Control Over Replication Information About Replication Status Options for Handling Broken Connections How the Slave Processes Events Housekeeping in the I/O Thread SQL Thread Processing Semisynchronous Replication Configuring Semisynchronous Replication Monitoring Semisynchronous Replication Global Transaction Identifiers Setting Up Replication Using GTIDs Failover Using GTIDs Slave Promotion Using GTIDs Replication of GTIDs Slave Safety and Recovery Syncing, Transactions, and Problems with Database Crashes Transactional Replication Rules for Protecting Nontransactional Statements Multisource Replication Details of Row-Based Replication Table_map Events The Structure of Row Events Execution of Row Event Events and Triggers Filtering in Row-Based Replication vi | Table of Contents 228 229 233 234 235 237 238 239 239 248 249 249 250 257 258 259 260 261 263 264 266 268 268 270 274 275 278 280 282 283 284 286 Partial Row Replication Conclusion 288 289 MySQL Cluster 291 What Is MySQL Cluster? Terminology and Components How Does MySQL Cluster Differ from MySQL? Typical Configuration Features of MySQL Cluster Local and Global Redundancy Log Handling Redundancy and Distributed Data Architecture of MySQL Cluster How Data Is Stored Partitioning Transaction Management Online Operations Example Configuration Getting Started Starting a MySQL Cluster Testing the Cluster Shutting Down the Cluster Achieving High Availability System Recovery Node Recovery Replication Achieving High Performance Considerations for High Performance High Performance Best Practices Conclusion Part II 292 292 293 293 294 296 297 297 298 300 303 304 304 306 306 308 313 314 314 317 318 319 324 325 326 328 Monitoring and Managing 10 Getting Started with Monitoring 333 Ways of Monitoring Benefits of Monitoring System Components to Monitor Processor Memory Disk Network Subsystem 334 335 335 336 337 338 339 Table of Contents | vii Monitoring Solutions Linux and Unix Monitoring Process Activity Memory Usage Disk Usage Network Activity General System Statistics Automated Monitoring with cron Mac OS X Monitoring System Profiler Console Activity Monitor Microsoft Windows Monitoring The Windows Experience The System Health Report The Event Viewer The Reliability Monitor The Task Manager The Performance Monitor Monitoring as Preventive Maintenance Conclusion 340 341 342 347 350 353 355 356 356 357 359 361 365 366 367 369 372 374 375 377 377 11 Monitoring MySQL 379 What Is Performance? MySQL Server Monitoring How MySQL Communicates Performance Performance Monitoring SQL Commands The mysqladmin Utility MySQL Workbench Third-Party Tools The MySQL Benchmark Suite Server Logs Performance Schema Concepts Getting Started Using Performance Schema to Diagnose Performance Problems MySQL Monitoring Taxonomy Database Performance Measuring Database Performance Best Practices for Database Optimization Best Practices for Improving Performance viii | Table of Contents 380 381 381 382 383 389 391 402 405 407 409 410 412 420 421 423 423 435 444 About the Authors Dr Charles A Bell is a Senior Software Engineer at Oracle He is currently the lead developer for backup and a member of the MySQL Backup and Replication team He lives in a small town in rural Virginia with his loving wife He received his Doctor of Philosophy in Engineering from Virginia Commonwealth University in 2005 His re‐ search interests include database systems, versioning systems, semantic web, and agile software development Dr Mats Kindahl is principal senior software developer in the MySQL team at Oracle He is the main architect and implementor of MySQL’s row-based replication and several other replication features and is currently architect and project lead for the MySQL High Availability team that is developing MySQL Fabric Before starting at MySQL, he did research in formal methods, program analysis, and distributed systems, the area where he earned his doctoral degree in computer science He has also spent several years de‐ veloping C/C++ compilers Dr Lars Thalmann is the development manager for MySQL replication and backup He is responsible for the strategy and development of these features and leads the cor‐ responding engineering teams Thalmann has worked with MySQL development since 2001, when he was a software developer working on MySQL Cluster More recently, he has driven the creation and development of the MySQL backup feature, has guided the evolution of MySQL replication since 2004, and has been a key player in the development of MySQL Cluster replication Thalmann holds a doctorate in computer science from Uppsala University, Sweden Colophon The animal on the cover of MySQL High Availability is an American robin (Turdus migratorius) Instantly recognizable by its distinctive appearance–dark head, reddishorange breast, and brown back–this member of the thrush family is among the most common American birds (Though it shares its name with the European robin, which also has a reddish breast, the two species are not closely related.) The American robin inhabits a range of six million square miles in North America and is resident year-round through much of the United States Commonly considered a harbinger of spring, robins are early to sing in the morning and among the last birds singing at night Their diets consist of invertebrates (often earthworms) and fruit and berries Robins favor open ground and short grass, so they are frequent backyard visi‐ tors, and they are often found in parks, in gardens, and on lawns The cover image is from Johnson’s Natural History, Volume II The cover fonts are URW Typewriter and Guardian Sans The text font is Adobe Minion Pro; the heading font is Adobe Myriad Condensed; and the code font is Dalton Maag’s Ubuntu Mono ... 82 83 86 87 91 94 97 98 99 10 0 10 0 10 1 10 3 10 4 10 5 10 6 11 3 11 8 12 0 Conclusion 12 1 Replication for High Availability 12 3 Redundancy Planning Slave Failures... MySQL High Availability, Second Edition by Charles Bell, Mats Kindahl, and Lars Thalmann Copyright © 2 014 Charles Bell, Mats Kindahl, Lars Thalmann All rights reserved Printed in the United... Conclusion 15 5 15 6 15 8 16 2 17 0 17 1 17 2 17 3 17 4 17 6 17 7 17 8 18 0 18 7 Data Sharding 18 9 What Is Sharding? Why Should You Shard? Limitations