Catchpoint-eBook-OReilly-Devops-for-Finance

Co m pl im en ts Reducing Risk Through Continuous Delivery updated for 2017 Jim Bird of DevOps for Finance DevOps for Finance Jim Bird DevOps for Finance by Jim Bird Copyright © 2015 O’Reilly Media, Inc All rights reserved Printed in the United States of America Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://oreilly.com/safari) For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com Editor: Brian Anderson September 2015: Production Editor: Kristen Brown Proofreader: Rachel Head Interior Designer: David Futato Cover Designer: Karen Montgomery First Edition Revision History for the First Edition 2015-09-16: 2017-03-27: First Release Second Release The O’Reilly logo is a registered trademark of O’Reilly Media, Inc DevOps for Finance, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limi‐ tation responsibility for damages resulting from the use of or reliance on this work Use of the information and instructions contained in this work is at your own risk If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsi‐ bility to ensure that your use thereof complies with such licenses and/or rights 978-1-491-93822-5 [LSI] Table of Contents Introduction ix Challenges in Adopting DevOps Is DevOps Ready for the Enterprise? The High Cost of Failure System Complexity and Interdependency Weighed Down by Legacy The Costs of Compliance Security Threats to the Finance Industry 11 16 Adopting DevOps in Financial Systems 19 Entering the Cloud Containers in Continuous Delivery Introducing DevOps: Building on Agile From Continuous Integration to Continuous Delivery Changing Without Failing DevOpsSec: Security as Code Compliance as Code Continuous Delivery or Continuous Deployment DevOps for Legacy Systems Implementing DevOps in Financial Markets 19 21 22 23 32 42 51 55 58 60 vii Introduction Disclaimer: The views expressed in this book are those of the author, and not reflect those of his employer or the publisher DevOps, until recently, has been a story about unicorns: innovative, engineering-driven online tech companies like Flickr, Etsy, Twitter, Facebook, and Google Netflix and its Chaos Monkey Amazon deploying thousands of changes per day DevOps was originally all about WebOps at cloud providers and online Internet startups It started at these companies because they had to find some way to succeed in Silicon Valley’s high-stakes, build fast, scale fast, or fail fast business environment They found new, simple, and collaborative ways of working that allowed them to innovate and learn faster and at a lower cost, and to scale much more effectively than organizations had done before But other enterprises, which we think of as “horses” in contrast to the internet unicorns, are under the same pressure to innovate and deliver new customer experiences, and to find better and more effi‐ cient ways to scale—especially in the financial services industry At the same time, these organizations have to deal with complex legacy issues and expensive compliance and governance obligations They are looking at if and how they can take advantage of DevOps ideas and tools, and how they need to adapt them This short book assumes that you have heard about DevOps and want to understand how DevOps practices like Continuous Delivery and Infrastructure as Code can be used to solve problems in finan‐ cial systems at a trading firm, or a big bank or stock exchange or ix some other financial institution We’ll look at the following key ideas in DevOps, and how they fit into the world of financial systems: Breaking down the “wall of confusion” between development and operations, and extending Agile practices and values from development to operations—and to security and compliance too Using automated configuration management tools like Chef, Puppet, and Ansible to programmatically provision and config‐ ure systems (Infrastructure as Code) Building Continuous Integration and Continuous Delivery (CI/CD) pipelines to automatically build, test, and push out changes, and wiring security and compliance into these pipe‐ lines Using containerization and virtualization technologies like Docker and Vagrant, and infrastructure automation platforms like Terraform and CloudFormation, to create scalable Infra‐ structure, Platform, and Software as a Service (IaaS, PaaS, and SaaS) clouds Running experiments, creating fast feedback loops, and learning from failure—without causing failures To follow this book you need to understand a little about these ideas and practices There is a lot of good stuff about DevOps out there, amid the hype A good place to start is by watching John Allspaw and Paul Hammond’s presentation at Velocity 2009, “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr”, which introduced DevOps ideas to the public IT Revolution’s free “DevOps Guide” will also help you to get started with DevOps, and point you to other good resources The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win by Gene Kim, Kevin Behr, and George Spafford (also from IT Revolution) is another great introduction, and surprisingly fun to read If you want to understand the technical practices behind DevOps, you should also take the time to read Continuous Delivery (AddisonWesley), by Dave Farley and Jez Humble Finally, DevOps in Practice is a free ebook from O’Reilly that explains how DevOps can be applied in large organizations, walking through DevOps initiatives at Nordstrom and Texas.gov x | Introduction Challenges in Common From small trading firms to big banks and exchanges, financial industry players are looking at the success of Facebook and Amazon for ideas on how to improve speed of delivery in IT, how to innovate faster, how to reduce operations costs, and how to solve online scal‐ ing problems Financial services, cloud services providers, and other Internet tech companies share many common technology and business chal‐ lenges They all deal with problems of scale They run farms of thousands or tens of thousands of servers, and thousands of applications No bank—even the biggest too-big-to-fail bank—can compete with the number of users that an online company like Facebook or Twitter supports On the other hand, the volume and value of transactions that a major stock exchange or clearinghouse handles in a trading day dwarfs that of online sites like Amazon or Etsy While Netflix deals with massive amounts of streaming video traffic, financial trading firms must be able to keep up with streaming low-latency market data feeds that can peak at several millions of messages per second, where nanosecond precision is necessary These Big Data worlds are coming closer together, as more financial firms such as Morgan Stanley, Credit Suisse, and Bank of America adopt data analytics platforms like Hadoop Google, in partnership with SunGard, was one of the shortlisted providers bidding on the Securities and Exchange Commission’s (SEC’s) new Consolidated Audit Trail (CAT), a massively scaled surveillance and reporting platform that will record every order, quote, and trade in the US equities and equities options markets CAT will be one of the world’s largest data warehouses, handling more than 50 billion records per day from over 2,000 trading firms and exchanges The financial services industry, like the online tech world, is viciously competitive, and there is a premium on continuous growth and meeting short-term quarterly targets Businesses (and IT) are under constantly increasing pressure to deliver new services faster, and with greater efficiency—but not at the expense of reliability of service or security Financial services can look to DevOps for ways to introduce new products and services faster, but at the same time they need to work within constraints to meet strict uptime and per‐ Introduction | xi formance service-level agreements (SLAs) and compliance and gov‐ ernance requirements DevOps Tools in the Finance Industry DevOps is about changing culture and improving collaboration between development and operations But it is also about automat‐ ing as many of the common jobs in delivering software and main‐ taining operating systems as possible: testing, compliance and secu‐ rity checks, software packaging and configuration management, and deployment This strong basis in automation and tooling explains why so many vendors are so excited about DevOps A common DevOps toolchain1 includes: • Version control and artifact repositories • Continuous Integration/Continuous Delivery servers like Jen‐ kins, Bamboo, TeamCity, and Go • Automated testing tools (including static analysis checkers and automated test frameworks) • Automated release/deployment tools • Infrastructure as Code: software-defined configuration manage‐ ment tools like Ansible, Chef, CFEngine, and Puppet • Virtualization and containerization technologies such as Docker and Vagrant Build management tools like Maven and Continuous Integration servers like Jenkins are already well established across the industry through Agile development programs Using static analysis tools to test for security vulnerabilities and common coding bugs and imple‐ menting automated system testing are common practices in devel‐ oping financial systems But as we’ll see, popular test frameworks like JUnit and Selenium aren’t a lot of help in solving some of the hard test automation problems for financial systems: integration testing, security testing, and performance testing Log management and analysis tools such as Splunk are being used effectively at financial services organizations like BNP Paribas, Credit Suisse, ING, and the Financial Industry Regulatory Authority (FINRA) for operational and security event monitoring, fraud anal‐ Xebia Labs publishes a cool “Periodic Table” of tools for solving DevOps problems xii | Introduction