ptg5994185 ptg5994185 Praise for the The Art of Scalability “This book is much more than you may think it is. Scale is not just about designing Web sites that don’t crash when lots of users show up. It is about designing your company so that it doesn’t crash when your business needs to grow. These guys have been there on the front lines of some of the most successful Internet companies of our time, and they share the good, the bad, and the ugly about how to not just survive, but thrive.” —Marty Cagan, Founder, Silicon Valley Product Group “A must read for anyone building a Web service for the mass market.” —Dana Stalder, General Partner, Matrix Partners “Abbott and Fisher have deep experiences with scale in both large and small enter- prises. What’s unique about their approach to scalability is they start by focusing on the true foundation: people and process, without which true scalability cannot be built. Abbott and Fisher leverage their years of experience in a very accessible and practical approach to scalability that has been proven over time with their significant success.” —Geoffrey Weber, VP of Internet Operations/IT, Shutterfly “If I wanted the best diagnoses for my health I would go to the Mayo Clinic. If I wanted the best diagnoses for my portfolio companies’ performance and scalability I would call Martin and Michael. They have recommended solutions to performance and scalability issues that have saved some of my companies from a total rewrite of the system.” —Warren M. Weiss, General Partner, Foundation Capital “As a manager who worked under Michael Fisher and Marty Abbott during my time at PayPal/eBay, the opportunity to directly absorb the lessons and experiences pre- sented in this book are invaluable to me now working at Facebook.” —Yishan Wong, Director of Engineering, Facebook ptg5994185 “The Art of Scalability is by far the best book on scalability on the market today. The authors tackle the issues of scalability from processes, to people, to performance, to the highly technical. Whether your organization is just starting out and is defining processes as you go, or you are a mature organization, this is the ideal book to help you deal with scalability issues before, during, or after an incident. Having built sev- eral projects, programs, and companies from small to significant scale, I can honestly say I wish I had this book one, five, and ten years ago.” —Jeremy Wright, CEO, b5media, Inc. “Only a handful of people in the world have experienced the kind of growth-related challenges that Fisher and Abbott have seen at eBay, PayPal, and the other companies they’ve helped to build. Fewer still have successfully overcome such challenges. The Art of Scalability provides a great summary of lessons learned while scaling two of the largest internet companies in the history of the space, and it’s a must-read for any executive at a hyper-growth company. What’s more, it’s well-written and highly entertaining. I couldn’t put it down.” —Kevin Fortuna, Partner, AKF Consulting “Marty and Mike’s book covers all the bases, from understanding how to build a scalable organization to the processes and technology necessary to run a highly scal- able architecture. They have packed in a ton of great practical solutions from real world experiences. This book is a must-read for anyone having difficulty managing the scale of a hyper-growth company or a startup hoping to achieve hyper growth.” —Tom Keeven, Partner, AKF Consulting “The Art of Scalability is remarkable in its wealth of information and clarity; the authors provide novel, practical, and demystifying approaches to identify, predict, and resolve scalability problems before they surface. Marty Abbott and Michael Fisher use their rich experience and vision, providing unique and groundbreaking tools to assist small and hyper-growth organizations as they maneuver in today’s demanding technological environments.” —Joseph M. Potenza, Attorney, Banner & Witcoff, Ltd. ptg5994185 The Art of Scalability ptg5994185 This page intentionally left blank ptg5994185 The Art of Scalability Scalable Web Architecture, Processes, and Organizations for the Modern Enterprise Martin L. Abbott Michael T. Fisher Upper Saddle River, NJ • Boston • Indianapolis • San Francisco New York • Toronto • Montreal • London • Munich • Paris • Madrid Capetown • Sydney • Tokyo • Singapore • Mexico City ptg5994185 Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the pub- lisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals. The authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein. The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include electronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact: U.S. Corporate and Government Sales (800) 382-3419 corpsales@pearsontechgroup.com For sales outside the United States please contact: International Sales international@pearson.com Visit us on the Web: informit.com/aw Library of Congress Cataloging-in-Publication Data Abbott, Martin L. The art of scalability : scalable web architecture, processes, and organizations for the modern enterprise / Martin L. Abbott, Michael T. Fisher. p. cm. Includes index. ISBN-13: 978-0-13-703042-2 (pbk. : alk. paper) ISBN-10: 0-13-703042-8 (pbk. : alk. paper) 1. Web site development. 2. Computer networks—Scalability. 3. Business enterprises— Computer networks. I. Fisher, Michael T. II. Title. TK5105.888.A2178 2010 658.4'06—dc22 2009040124 Copyright © 2010 Pearson Education, Inc. All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding permissions, write to: Pearson Education, Inc. Rights and Contracts Department 501 Boylston Street, Suite 900 Boston, MA 02116 Fax: (617) 671-3447 ISBN-13: 978-0-13-703042-2 ISBN-10: 0-13-703042-8 Text printed in the United States on recycled paper at RR Donnelley in Crawfordsville, Indiana. First printing, December 2009 Editor-in-Chief Mark Taub Acquisitions Editor Trina MacDonald Development Editor Songlin Qiu Managing Editor John Fuller Project Editor Anna Popick Copy Editor Kelli Brooks Indexer Richard Evans Proofreader Debbie Liehs Technical Reviewers Jason Bloomberg Robert Guild Robert Hines Jeremy Wright Cover Designer Chuti Prasertsith Compositor Rob Mauhar ptg5994185 To my father for teaching me how to succeed, and to Heather for teaching me how to have fun. —Marty Abbott To my parents for their guidance, and to my wife and son for their unflagging support. —Michael Fisher ptg5994185 This page intentionally left blank ptg5994185 ix Contents Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiii About the Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxv Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Part I: Staffing a Scalable Organization. . . . . . . . . . . . . . . 7 Chapter 1: The Impact of People and Leadership on Scalability . . . . . . . . . . . . . . . 9 Introducing AllScale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Why People . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Why Organizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Why Management and Leadership . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Key Points. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Chapter 2: Roles for the Scalable Technology Organization . . . . . . . . . . . . . . . . . 21 The Effects of Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Defining Roles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Executive Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 CEO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 CFO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Business Unit Owners, General Managers, and P&L Owners . . . . . . . . 27 CTO/CIO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Organizational Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Architecture Responsibilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Engineering Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Production Operations Responsibilities. . . . . . . . . . . . . . . . . . . . . . . . . 30 Infrastructure Responsibilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Quality Assurance Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Capacity Planning Responsibilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Individual Contributor Responsibilities and Characteristics . . . . . . . . . . . 32 Architect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 [...]... 11 1 Setting the Example 11 1 Educating Other Executives 11 1 Using the RASCI Model 11 2 Speaking in Business Terms 11 2 Getting Them Involved 11 3 Scaring the Executive Team with Facts 11 3 The Business Case for. .. thought of such computing complexity is overwhelming This is exactly why you need to read The Art of Scalability Two of the brightest minds of the information age have come together to share their knowledge and experience in delivering peak performance with the precision and detail that their West Point education mandates Marty Abbott and Mike Fisher have fought some of the most challenging enterprise architecture. .. must-reading in the winning organization Bob Davis Managing Partner, Highland Capital Partners, and Founder/Former CEO, Lycos Acknowledgments The authors would like to recognize, first and foremost, the experience and advice of our partner and cofounder Tom Keeven The process and technology portions of this book were built over time with the help of Tom and his many years of experience Tom started the business... level of scale within a platform, organization, or process Rather, the interactions between platforms, organizations, and processes have profound impacts on the adaptation of any specific and highly structured approach The approach to scaling must be crafted around the ecosystem created by the intersection of the current technology platform, the characteristics of the organization, and the maturity and. .. became AKF Partners We often joke that Tom has forgotten more about architecting highly available and scalable sites than most of us will ever learn We further would like to recognize our colleagues and teams at Quigo, eBay, and PayPal These are the companies at which we really started to build and test many of the approaches mentioned in the technology and process sections of this book The list of names... by Merriam-Webster are “skill acquired by experience, study, or observation” and “an occupation requiring knowledge or skill.” All of these are true of the nature of scaling platforms, processes, and organizations But perhaps more important in our choice of art here are the images the word conjures up of being more fluid versus the view of science, which is more structured and 2 Merriam-Webster Online... is the result of careful planning and great execution; this is certainly so with today’s incredibly complex networks and databases The book walks you through the steps necessary to think straight and succeed in the most challenging of circumstances Marty and Mike have danced in boardrooms and executed on the frontlines with many of the nation’s top businesses These two are the best of the best With The. .. designed in the laboratory nor are they based on unapplied theory Rather, these lessons have been designed and implemented by engineers, technology leaders, and organizations through years of struggling to keep their dreams, businesses, and systems a float The authors have had the great fortune to be a small part of many of these teams in many different roles— sometimes as active participants, other times... organization, your processes, and (as a result) the scalability of your product, platform, or systems They are at the heart of everything you do and the core of everything you need to scale a company and a platform Paradoxically, they are the things we overlook most often when attempting to scale large systems: Our people are overlooked and underappreciated; organization structure is a once-ayear, check -the- box... development, and client services Prior to Quigo, Marty spent nearly six years at eBay, most recently as SVP of technology and CTO and member of the executive staff Prior to eBay, Marty held domestic and international engineering, management, and executive positions at Gateway and Motorola Marty serves on the boards of directors for OnForce, LodgeNet Interactive (NASD:LNET), and Bullhorn; and is on the advisory . Ltd. ptg599 418 5 The Art of Scalability ptg599 418 5 This page intentionally left blank ptg599 418 5 The Art of Scalability Scalable Web Architecture, Processes, and Organizations for the Modern Enterprise Martin. on the Web: informit.com/aw Library of Congress Cataloging-in-Publication Data Abbott, Martin L. The art of scalability : scalable web architecture, processes, and organizations for the modern. Wong, Director of Engineering, Facebook ptg599 418 5 The Art of Scalability is by far the best book on scalability on the market today. The authors tackle the issues of scalability from processes,