Chef Essentials Table of Contents Chef Essentials Credits About the Author About the Reviewers www.PacktPub.com Support files, eBooks, discount offers and more Why Subscribe? Free Access for Packt account holders Preface What this book covers What you need for this book Who this book is for Conventions Reader feedback Customer support Errata Piracy Questions Installing Chef Terminology Working with Chef Installing chef-solo The Ruby gem Managing gems Verifying that chef-solo works Installing a Chef server Requirements and recent changes Installation requirements What you will be installing Getting the installer Installation outline Installing on Ubuntu Downloading the package Installing the package Installing on Red Hat Enterprise Linux Downloading the package Configuring a Chef server Understanding how chef-server-ctl works What’s happening on my server? Verifying that the services are running Validating that your service is working Ensuring that your knife configuration works Summary Modeling Your Infrastructure Getting to know Chef Modeling your infrastructure Roles Defining roles A web application service role An image-processing role An image search role A PostgreSQL service role A Solr service role An OpenSSH service role Implementing a role Determining which recipes you need Installing a cookbook Applying recipes to roles Mapping your roles to nodes Converging a node Environments Organizing your configuration data Example attribute data Data bags Knowing when to use data bags Large-scale infrastructure Summary Integrating with the Cloud Leveraging the cloud Amazon EC2 Installing the EC2 knife plugin Setting up EC2 authentication Provisioning an instance Bootstrapping the instance Terminating the instance Removing the Chef node Rackspace Cloud Provisioning an instance Terminating an instance Removing the Chef node Summary Working with Cookbooks Attributes Multiple attribute files Supporting multiple platforms Loading external attributes Using attributes Metadata Recipes Resources Using resources Overriding a default behavior Templates Why use templates? A quick ERB primer Executing Ruby Variable replacement The template resource The template variables Passing variables to a template Accessing computed configurations Searching for templates Definitions Recipes Developing recipes Writing recipes Starting out small Installing a simple service Getting more advanced Summary Testing Your Recipes Testing recipes RSpec RSpec and ChefSpec Testing basics Comparing RSpec with other testing libraries Using ChefSpec Getting started with ChefSpec Installing ChefSpec Locking your dependencies in Ruby Creating a simple recipe and a matching ChefSpec test Writing a ChefSpec test Building your recipe Executing tests Understanding failures Expanding your tests Multiple examples in a spec test Testing for multiple platforms Summary From Development to Deployment Describing the setup Deploying software with Chef Configuring your local environment Modeling a simple Python application Managing the cookbooks Downloading cookbooks Looking at the database recipe Looking at your application deployment cookbook Preparing the directories Constructing your Python virtual environment Checking the source code Installing any extra dependencies Managing dependencies in Chef Managing dependencies elsewhere Using Python’s requirements file Configuring your application Keeping your application running Defining roles Creating the base server role Creating the database server role Creating the web server role Adding users Provisioning EC2 instances Configuring the database host Configuring the web server Deploying your software Manually deploying updates Automating deployment Summary Beyond Basic Recipes and Cookbooks Managing users Evolution of a shell user recipe Storing data in data bags Creating a data bag for users Searching for data Searching inside recipes Enhancing your user cookbook Distributing SSH keys Templating the authorized keys Adding deployment keys Writing custom extensions Developing a custom definition Organizing your code Writing a definition for using PIP Defining a full application template Building a resource Defining the resource Implementing the provider Modifying resources Loading an existing resource Declaring that a resource was updated Working with data bags Securing your data bags Secret keys Encrypting your data Decrypting your data Storing keys on nodes Searching your data Searching your data bags with knife Searching your data bags from a recipe Querying your data Managing multiple machines with search queries Summary Extras You Need to Know Vagrant and Chef-solo Installing Vagrant Provisioning a new host with Vagrant Booting your Vagrant image Combining Vagrant with Chef-solo Understanding the limitations of Chef-solo Configuring Chef-solo Telling Chef-solo what to run Using roles and data bags with Chef-solo Injecting custom JSON data Providing a custom node name Getting to know the Chef shell Using the Chef shell The standalone mode The solo mode The client mode Interacting with the Chef server using the shell Interacting with data Searching your data Editing your data Transforming data K knife-rackspace plugin / Provisioning an instance knife node list / Interacting with the Chef server using the shell knife plugin, Amazon EC2 installing / Installing the EC2 knife plugin knife search node… / Interacting with the Chef server using the shell knife tool / Validating that your service is working used, for searching databags / Searching your data bags with knife L large-scale infrastructure / Large-scale infrastructure load_current_resource method / Loading an existing resource local environment configuring / Configuring your local environment M Mash, Chef / Using attributes matching ChefSpec test creating / Creating a simple recipe and a matching ChefSpec test metadata about / Metadata mock methods / RSpec and ChefSpec multiple attribute files about / Multiple attribute files multiple platforms, supporting / Supporting multiple platforms external attributes, loading / Loading external attributes multiple machines managing, with search queries / Managing multiple machines with search queries mysql / Looking at the database recipe N *name parameter / Writing a definition for using PIP new host provisioning, with Vagrant / Provisioning a new host with Vagrant node / Terminology about / Getting to know Chef role, mapping to / Mapping your roles to nodes converging / Converging a node node attributes defining / Defining node attributes setting / Setting attributes accessing / Accessing attributes node delete command / Removing the Chef node node hash / Using attributes O Ohai plugin writing / Writing an Ohai plugin, A note about writing Ohai plugins attributes / Writing an Ohai plugin omnibus installation package, Chef / Requirements and recent changes omnibus installer, Chef server obtaining / Getting the installer outline / Installation outline steps / Installation outline OpenSSH service role defining / An OpenSSH service role P PIP used, for writing definition / Writing a definition for using PIP postgresql / Looking at the database recipe PostgreSQL service role defining / A PostgreSQL service role provider / Getting to know Chef about / Building a resource implementing / Implementing the provider Python application modelling / Modeling a simple Python application Python requirements file using / Using Python’s requirements file Python virtual environment constructing / Constructing your Python virtual environment Python’s requirements file using / Using Python’s requirements file R Rackspace Cloud about / Rackspace Cloud instance, provisioning / Provisioning an instance instance, terminating / Terminating an instance Chef node, removing / Removing the Chef node read-eval-print-loop (REPL) / Getting to know the Chef shell recipe / Terminology about / Getting to know Chef determining / Determining which recipes you need need for / Determining which recipes you need cookbook, installing / Installing a cookbook applying, to role / Applying recipes to roles executing / Starting out small testing / Testing recipes testing, with ChefSpec / Testing recipes building / Building your recipe tests, executing / Executing tests failures / Understanding failures creating, in Chef shell / Creating a recipe in the shell recipes about / Recipes, Recipes starting states / Recipes developing / Developing recipes writing / Writing recipes simple service, installing / Installing a simple service complicated recipe, actions / Getting more advanced enhancing, with search method / Searching inside recipes databags, searching from / Searching your data bags from a recipe recipes, executing with Chef shell about / Executing recipes with Chef shell, Interactively executing recipes recipe mode / Creating a recipe in the shell attribute mode / Creating a recipe in the shell node attributes, defining / Defining node attributes attributes, setting / Setting attributes attributes, accessing / Accessing attributes configuration blocks, using / Using configuration blocks Red Hat Enterprise Linux Chef server, installing on / Installing on Red Hat Enterprise Linux resource / Getting to know Chef building / Building a resource package resource / Building a resource defining / Defining the resource provider, implementing / Implementing the provider modifying / Modifying resources existing resource, loading / Loading an existing resource updated status, declaring / Declaring that a resource was updated resources about / Resources built-in resources / Resources using / Using resources default behavior, overriding / Overriding a default behavior reusable resources, defining in Chef benefits / Defining a full application template role about / Getting to know Chef, Roles defining / Defining roles implementing / Implementing a role applying, to recipes / Applying recipes to roles mapping, to node / Mapping your roles to nodes role, defining about / Defining roles web application service role / A web application service role image-processing role / An image-processing role image search role / An image search role PostgreSQL service role / A PostgreSQL service role Solr service role / A Solr service role OpenSSH service role / An OpenSSH service role roles defining / Defining roles base server role, creating / Creating the base server role database server role, creating / Creating the database server role web server role, creating / Creating the web server role roles, using with Chef-solo about / Using roles and data bags with Chef-solo custom JSON data, injecting / Injecting custom JSON data custom node name, providing / Providing a custom node name RSpec about / RSpec and ChefSpec / RSpec and ChefSpec testing levels / RSpec and ChefSpec comparing, with testing libraries / Comparing RSpec with other testing libraries failures / Understanding failures Ruby dependencies, locking / Locking your dependencies in Ruby Ruby gem mechanism / The Ruby gem Ruby Version Manager (RVM) about / The Ruby gem URL / Managing gems run list / Getting to know Chef runner / Testing for multiple platforms S search method used, for enhancing recipes / Searching inside recipes search queries used, for managing multiple machines / Managing multiple machines with search queries secret keys, databags managing, with search queries / Secret keys server list subcommand / Terminating an instance service-oriented architecture (SOA) / Vagrant and Chef-solo setup describing / Describing the setup shell user recipe evolution / Evolution of a shell user recipe software deploying, with Chef / Deploying software with Chef deploying / Deploying your software software deployment updates, deploying manually / Manually deploying updates automating / Automating deployment solo mode, Chef shell / The solo mode Solr service role defining / A Solr service role source code checking / Checking the source code SSH keys distributing / Distributing SSH keys authorized keys, templating / Templating the authorized keys deployment keys, adding / Adding deployment keys standalone mode, Chef shell / The standalone mode supervisord service URL / Keeping your application running T templates about / Templates need for / Why use templates? ERB primer / A quick ERB primer template resource / The template resource variables / The template variables variables, passing to / Passing variables to a template computed configuration, accessing / Accessing computed configurations searching for / Searching for templates search order / Searching for templates terminology, Chef node / Terminology Chef service / Terminology workstation / Terminology recipe / Terminology cookbook / Terminology test-driven-development (TDD) / Testing basics Test Kitchen using / Using Test Kitchen installing / Installing Test Kitchen used, for testing / Testing with Test Kitchen tasks / Testing with Test Kitchen cookbook, building / Building a simple cookbook cookbook, preparing for / Preparing your cookbook for the kitchen cookbook, testing / Testing your new cookbook U Ubuntu Chef server, installing on / Installing on Ubuntu updated_by_last_action method / Declaring that a resource was updated user cookbook enhancing / Enhancing your user cookbook user management Chef / Managing users shell user recipe, evolution / Evolution of a shell user recipe data, storing in data bags / Storing data in data bags recipe, enhancing with search method / Searching inside recipes user cookbook, enhancing / Enhancing your user cookbook SSH keys, distributing / Distributing SSH keys users adding / Adding users managing / Managing users V Vagrant about / Vagrant and Chef-solo installing / Installing Vagrant URL / Installing Vagrant used, for provisioning new host / Provisioning a new host with Vagrant image, booting / Booting your Vagrant image combining, with Chef-solo / Combining Vagrant with Chef-solo, Understanding the limitations of Chef-solo, Telling Chef-solo what to run Vagrant Cloud URL / Provisioning a new host with Vagrant Vagrant image booting / Booting your Vagrant image W web application service role defining / A web application service role web server configuring / Configuring the web server web server role creating / Creating the web server role why-run mechanism / Implementing the provider workstation / Terminology, Getting to know Chef Y YAML / Deploying software with Chef [...]... Provisioning the instance Converging the newly created instance Writing a simple test Combining all the steps Extending Chef Writing an Ohai plugin A note about writing Ohai plugins Chef with Capistrano Automation and integration Automated updates and deployments Summary Index Chef Essentials Chef Essentials Copyright © 2014 Packt Publishing All rights reserved No part of this book may be reproduced, stored in a retrieval system,... This scenario, and many others like it, is where Chef is indispensable Having the ability to describe your hosts, configuration data, and roles, and then apply that across as many hosts as you like means that you can manage large fleets of hosts just as easily as you can manage one or two What this book covers Chapter 1, Installing Chef, introduces you to the architecture of Chef, various installation methods, and a guide to setting up Chef (solo and self-hosted)... methods, and a guide to setting up Chef (solo and self-hosted) It includes information on using hosted Chef (and what that means for your team) and Vagrant with Chef- solo Chapter 2, Modeling Your Infrastructure, introduces how to model your infrastructure with Chef using your newly installed system This chapter will cover modeling environments, small and large, as well as how to integrate with cloud technologies using Chef (AWS, Rackspace Cloud, and so on) Chapter 3, Integrating with the Cloud, covers how Chef helps you scale your... Chapter 3, Integrating with the Cloud, covers how Chef helps you scale your infrastructure using any combination of physical, virtual, and cloud-hosted systems This chapter discusses how to use Chef to provision and manage hosts using cloud providers as easily as your local systems including AWS and Rackspace Cloud Chapter 4, Working with Cookbooks, covers how every Chef needs cookbooks—once your systems are part of your Chef- managed fleet, you can begin collecting, developing, and... Chapter 7, Beyond Basic Recipes and Cookbooks, delves into developing extensions to Chef through advanced concepts, including custom providers and resource types, using the Chef search engine, advanced scripting, and more Chapter 8, Extras You Need to Know, expands your knowledge of how to leverage Chef for infrastructure automation, complex systems integration, and securely storing and distributing sensitive data with Chef What you need for this book... vended by the API service One of the most attractive features of Chef is that you can leverage its API to easily integrate existing tools, or you can develop new tools to meet specific needs Any organization with a moderate number of developer resources can harness the power of Chef to manage their systems For example, one can easily build software to import data from Chef into a reporting tool of some form and dynamically reconfigure infrastructure... This is incredibly valuable to anybody who has an existing infrastructure because it provides a convenient path to integrate Chef into their environment There are a number of ways to access Chef The quickest way for a single user to manage his/her infrastructure (virtual machines, a handful of hosts, and so on) is to use Chef- solo, a product geared towards single-user environments In a small environment, setting up a hosted server is a good way to manage infrastructure automation among team members... However, Packt Publishing cannot guarantee the accuracy of this information First published: September 2014 Production reference: 11 9091 4 Published by Packt Publishing Ltd Livery Place 35 Livery Street Birmingham B3 2PB, UK ISBN 978-1-78398-304-9 www.packtpub.com Cover image by Prashant Timappa Shetty () Credits Author John Ewart Reviewers Joshua Black Lauren Malhoit Eric Maxey Commissioning Editor...Executing recipes with Chef shell Creating a recipe in the shell Defining node attributes Setting attributes Accessing attributes Using configuration blocks Interactively executing recipes Debugging with the Chef shell Using the breakpoint resource Integration testing Using Test Kitchen Installing Test Kitchen Testing with Test Kitchen... Chapter 5, Testing Your Recipes, focuses on one compelling reason to use Chef to configure your infrastructure, that is, recipes are written in Ruby code and can be tested as any program would be tested Here, you will learn how to test your recipes through a variety of testing mechanisms Chapter 6, From Development to Deployment, covers how to take a custom application from development to a production deployment with Chef It contains a complete example ... Extending Chef Writing an Ohai plugin A note about writing Ohai plugins Chef with Capistrano Automation and integration Automated updates and deployments Summary Index Chef Essentials Chef Essentials. .. Errata Piracy Questions Installing Chef Terminology Working with Chef Installing chef- solo The Ruby gem Managing gems Verifying that chef- solo works Installing a Chef server Requirements and recent changes... Testing Your Recipes Testing recipes RSpec RSpec and ChefSpec Testing basics Comparing RSpec with other testing libraries Using ChefSpec Getting started with ChefSpec Installing ChefSpec Locking your dependencies in Ruby