AWS System Administration Mike Ryan AWS System Administration by Mike Ryan Copyright © 2010 Mike Ryan All rights reserved Printed in the United States of America Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://safaribooksonline.com) For more information, contact our corporate/ institutional sales department: 800-998-9938 or corporate@oreilly.com Editors: Andy Oram and Mike Hendrickson Production Editor: FIX ME! Copyeditor: FIX ME! Proofreader: FIX ME! January -4712: Indexer: FIX ME! Cover Designer: Karen Montgomery Interior Designer: David Futato Illustrator: Rebecca Demarest First Edition Revision History for the First Edition: 2014-10-07: Early release revision 2015-05-05: Early release revision See http://oreilly.com/catalog/errata.csp?isbn=0636920027638 for release details Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc !!FILL THIS IN!! and related trade dress are trademarks of O’Reilly Media, Inc Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks Where those designations appear in this book, and O’Reilly Media, Inc was aware of a trademark claim, the designations have been printed in caps or initial caps While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein ISBN: 063-6-920-02763-8 [?] Table of Contents Preface vii Setting Up AWS Tools Preparing Your Tools Installing the AWS Command Line Interface Parsing JSON Output with jq Installing the Earlier AWS Command-Line Tools 2 First Steps with EC2 and CloudFormation What Is an Instance? Instance Types Processing Power Storage Networking Launching Instances Launching from the Management Console Launching with Command-Line Tools Launching from Your Own Programs and Scripts Introducing CloudFormation Working with CloudFormation Stacks Creating the Stack Updating the Stack Looking Before You Leap Deleting the Stack Which Method Should I Use? Amazon Machine Images Building Your Own AMI Deregistering AMIs Recap 10 10 11 12 12 19 23 26 28 28 29 32 32 33 35 37 39 40 iii Access Management and Security Groups 43 Identity and Access Management Amazon Resource Names IAM Policies IAM Users and Groups IAM Roles Using IAM Roles from Other AWS Accounts Using IAM in CloudFormation Stacks Security Groups Protecting Instances with SSH Whitelists Virtual Private Networks and Security Groups Recap 43 44 44 53 55 62 62 67 69 71 78 Configuration Management 79 Why Use Configuration Management? OpsWorks Choosing a Configuration Management Package Puppet on AWS A Quick Introduction to Puppet Puppet and CloudFormation User Data and Tags Executing Tasks with Fabric Master-less Puppet Building AMIs with Packer 79 80 81 83 83 89 101 103 106 110 An Example Application Stack 115 Overview of Application Components The Web Application Database and Caching Background Task Processing Installing the Web Application Preparing Puppet and CloudFormation Puppet Files CloudFormation Files Creating an RDS Database RDS: Updating Puppet and CloudFormation Creating an ElastiCache Node ElastiCache: Updating Puppet and CloudFormation Installing Celery with Simple Queueing Service Celery: Updating Puppet and CloudFormation Building the AMIs Creating the Stack with CloudFormation iv | Table of Contents 115 116 116 116 117 121 121 127 128 133 138 143 145 152 156 158 Recap 159 Auto Scaling and Elastic Load Balancing 161 What Is Auto Scaling? Static Auto Scaling Groups Notifications of Scaling Activities Scaling Policies Scaling on CloudWatch Metrics Elastic Load Balancing Elastic Load Balancer and Auto Scaling Groups Recap 161 163 167 169 169 174 175 178 Deployment Strategies 179 Instance-Based Deployments Executing Code on Running Instances with Fabric Updating Instances at Launch Time AMI-Based Deployments Deploying AMIs with CloudFormation Deploying AMIs with the EC2 API Recap 179 180 184 185 185 186 187 Building Reusable Components 189 Role-Based AMIs Mapping Instances to Roles Patterns for Configuration Management Tools Modular CloudFormation Stacks 189 191 192 195 Log Management 199 Central Logging Logstash Configuration Logging to S3 AWS Service Logs S3 Life Cycle Management 199 201 205 208 210 10 DNS with Route 53 213 Why Use Route 53? Failure Is an Option: Using Route 53 to Handle Service Failover Ramping Up Traffic Surviving ELB and Application Outages with Route 53 Recap 213 214 218 219 223 Table of Contents | v 11 Monitoring 225 Why Are You Monitoring? CloudWatch Auto Scaling and Custom Metrics Old Tools, New Tricks 225 226 227 230 12 Backups 235 Backing Up Static Files from EC2 Instances to S3 Rolling Backups with S3 and Glacier PostgreSQL and Other Databases pg_dump Snapshots and Continuous Archiving Off-Site Backups vi | Table of Contents 237 238 241 241 242 246 Preface System administration is a complicated topic that requires practitioners to be familiar with an ever-expanding range of applications and services In some ways, Amazon Web Services (AWS) is just another tool to add to your toolkit, yet it can also be considered a discipline in and of itself Successfully building and deploying infrastructure on AWS involves a thorough understanding of the underlying operating system concerns, soft‐ ware architecture, and delivery practices, as well as the myriad components that make up Amazon Web Services I run a DevOps consultancy, helping startups and small businesses reap the benefits of tools and processes that were previously available only to organizations with large teams of systems administrators Many of these businesses not have a dedicated systems administrator, and the development team is responsible for deploying and maintaining the architecture In working with these clients, I noticed patterns in how people were working with AWS Those who came from a pure development background (without sysadmin experience) would often build an infrastructure that left out many of the things sysadmins would take for granted, such as monitoring and logging The lack of monitoring and logging would then make it difficult to track down issues, leading to more downtime than was necessary At the other end of the spectrum were those with a lot of sysadmin experience, but less or no development experience This group was more likely to treat AWS as nothing more than a virtual machine hosting provider, simply using EC2 to run a fleet of static instances without taking advantage of any high-availability features such as Auto Scaling and Elastic Load Balancers This is akin to buying a Ferrari and then using it only to buy groceries once per week: fun, but not cost-effective Using AWS requires a fundamentally different mindset than when deploying groups of static servers You not simply set up some servers and then periodically perform maintenance Instead, you use the AWS toolset (automatic instance replacement, scaling vii up and down in response to demand, etc.) to build a system In this sense, it is more like programming than traditional system administration The aim of this book is to help you reach a compromise between these two approaches, and help you make the right choice for your application’s specific hosting requirements If you are a developer, this book will give you enough system administration knowledge to ensure that you are using AWS effectively, and help you build a robust and resilient application infrastructure For systems administrators, it will show you how you can keep your favorite tools and processes while working with AWS, and hopefully save you from reinventing some wheels along the way AWS is a collection of cloud computing services that can be combined to build scalable and reliable applications and services It comprises a number of components, each with their own names and configuration options, which are offered under the AWS umbrella Some of these—such as EC2 and S3—are extremely popular and well-known Others, such as Kinesis and CloudFormation, are less well-known Because covering each of these services in detail would result in a multivolume tome of formidable size, this book focuses on the more commonly used services and provides a jumping-off point for learning about the others If you are familiar with AWS, feel free to hop between chapters to find the information that is most interesting or relevant to your current project Beginners to AWS should work through the book sequentially, as each chapter builds on information presented in the previous chapters Chapter helps you get set up with the tools you will need to interact with AWS and build the example infrastructure Chapter introduces what is perhaps the most well-known of all AWS services, EC2 This chapter also introduces my personal favorite AWS service, CloudFormation In Chapter 3, we look at some of the security features offered by AWS Chapter introduces configuration management tools, a common requirement when automating a cloud infrastructure Using these tools, Chapters and demonstrate the process of deploying an example application to AWS Chapter looks at some of the methods of deploying application and infrastructure updates to your environment Chapter builds on this and discusses the creation of reusable components to save time Log management, a more traditional sysadmin task that has some interesting implica‐ tions in the cloud, is the topic of Chapter Chapter 10 covers another traditional sysadmin task: DNS with Amazon’s Route 53 service viii | Preface Monitoring with Amazon’s CloudWatch service and other monitoring tools is discussed in Chapter 11 Finally, Chapter 12 looks at some of the ways of backing up your data both in and outside the Amazon cloud Audience This book is written for system administrators and developers I assume you are com‐ fortable with the basic tools used to administer the operating system and common services such as DNS If you plan to use Puppet or Chef for automation, you need to learn basic information about those tools elsewhere You should have a working knowl‐ edge of Git or another source code management system I not expect you to have prior knowledge of AWS or other virtualization and cloud products Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, data types, and environment variables Constant width bold Shows commands or other text that should be typed literally by the user Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context This icon signifies a general note This icon signifies a tip or suggestion Preface | ix ... EC2_HOME=/usr/local /aws/ ec2 AWS_ IAM_HOME=/usr/local /aws/ iam AWS_ RDS_HOME=/usr/local /aws/ rds AWS_ ELB_HOME=/usr/local /aws/ elb AWS_ CLOUDFORMATION_HOME=/usr/local /aws/ cfn AWS_ AUTO_SCALING_HOME=/usr/local /aws/ as... CS_HOME=/usr/local /aws/ cloudsearch AWS_ CLOUDWATCH_HOME=/usr/local /aws/ cloudwatch Preparing Your Tools | export AWS_ ELASTICACHE_HOME=/usr/local /aws/ elasticache export AWS_ SNS_HOME=/usr/local /aws/ sns export AWS_ ROUTE53_HOME=/usr/local /aws/ route53.. .AWS System Administration Mike Ryan AWS System Administration by Mike Ryan Copyright © 2010 Mike Ryan All rights reserved