Co m pl im en ts of DevOps for Media and Entertainment Accelerate Your Digital Transformation Through Automation, Monitoring, and the Cloud Zubin Irani & Brandon Cipes DevOps for Media and Entertainment Accelerate Your Digital Transformation through Automation, Monitoring, and the Cloud Zubin Irani and Brandon Cipes Beijing Boston Farnham Sebastopol Tokyo DevOps for Media and Entertainment by Zubin Irani and Brandon Cipes Copyright © 2017 O’Reilly Media, Inc All rights reserved Printed in the United States of America Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://oreilly.com/safari) For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com Editors: Brian Anderson and Virginia Wilson Production Editor: Colleen Cole July 2017: Copyeditor: Octal Publishing, Inc Interior Designer: David Futato Cover Designer: Karen Montgomery First Edition Revision History for the First Edition 2017-07-10: First Release The O’Reilly logo is a registered trademark of O’Reilly Media, Inc DevOps for Media and Entertainment, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work Use of the information and instructions contained in this work is at your own risk If any code samples or other technology this work contains or describes is sub‐ ject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights 978-1-491-98745-2 [LSI] Table of Contents DevOps in Media and Entertainment New Media Was First in DevOps Digital Disruption in Media and Entertainment DevOps Defined: Six Key Competencies Advancing Beyond Agile Benefits and Limits of DevOps CI/CD Dynamic Cloud Infrastructure Automated Testing Security Automation/DevSecOps Seven DevSecOps Imperatives Monitoring Beginning Your DevOps Journey DevOps and the Next Media and Entertainment Disruptor 10 14 17 18 20 22 24 26 iii DevOps in Media and Entertainment Traditional media and entertainment businesses are struggling with digital transformation on a more dramatic scale than those in any other industry Media and entertainment executives who have been around long enough have experienced the move from running entirely on analog data and devices to becoming wholly digital com‐ panies More recently, they’ve made the rapid leap from embracing software products as a tool to realizing that software—operating in the cloud like a living, breathing organism needing constant care and feeding—is their product And they’ve seen the fundamental definition of what a media and entertainment business is expand as methods of content creation and consumption have exploded And the changes are still coming More than 70 percent of media and entertainment industry executives who responded to a poll last year said they anticipated “moderate or massive digital disruption” in the next 12 months Whatever their sector—movies, music, gam‐ ing, television, radio, or publishing—the leaders of these businesses know that speed and agility are crucial to keeping up with the pace of technological change, consumer expectations, and their competi‐ tors We believe that the key to enabling this level of agility is a highly col‐ laborative and communicative relationship between Development and IT Operations The merged term DevOps symbolizes the inte‐ gration that needs to occur between the two departments New media leaders such as Facebook and Google are built upon DevOps And among traditional media and entertainment businesses, the most progressive have already adopted DevOps In this report, we focus on a few of them as case studies, including Netflix, Disney, and Sony Across all industries, 80 percent of “global 1,000” organizations are expected to implement DevOps by 2019 And yet, as many as 60 percent of leaders in organizations of between 500 and 10,000 employees don’t know what DevOps is or are unsure if their compa‐ nies practice it In our view, the first step to “doing” DevOps is to understand the disciplines that comprise a DevOps journey In this report, we aim to enlighten media and entertainment executives about the compo‐ nents of a DevOps transformation and to show the ways that adopt‐ ing DevOps practices will enable their businesses to evolve with their industry’s ever-changing landscape We also aim to convince readers who don’t already agree with us on this point: media and entertainment enterprises that don’t implement a DevOps approach will inevitably fall far behind their industry’s leaders New Media Was First in DevOps It’s no surprise that the earliest example of DevOps in action came from a new media business that was challenged to house and serve massive banks of digital files In 2009, Flickr stored more than billion photos that its users accessed at a rate of 40,000 per second And that year its operations manager and a software engineer made history giving the first known public talk about the importance of bridging the divide between Development and Operations The evidence for their argu‐ ment wowed the audience at O’Reilly’s Velocity 2009 conference: Flickr was deploying new features to its website 10 times a day Back then, that rate of deployment was a big deal O’Reilly’s confer‐ ence program noted: Flickr takes the idea of “release early, release often” to an extreme— on a normal day there are 10 full deployments of the site to our servers This session discusses why this rate of change works so well, and the culture and technology needed to make it possible In their presentation, “10+ Deploys Per Day: Dev and Ops Coopera‐ tion at Flickr”, colleagues John Allspaw and Paul Hammond argued that communications and cooperation between the traditionally adversarial development and operations teams was the key to their | DevOps in Media and Entertainment business’s ability to deploy new code, plan projects, and manage cri‐ ses quickly and effectively Allspaw and Hammond agreed at the time that only frequent, con‐ tinual improvements could keep a business like theirs from being overtaken by the likes of Facebook and Twitter But change wasn’t the norm in their profession at the time In fact, many viewed every change as a site outage risk The archetypal “ops person” would argue against frequent software changes in favor of site stability, so Allspaw and Hammond built tools and a culture of cooperation between their two departments that allowed change to happen as often as possible DevOps and continuous delivery/continuous deployment took the risk out of change by making it normal “As web infrastructures grow,” they said, “the line between systems and software become quite blurred Operations and development are disciplines that historically have been limited to a predictable list of responsibilities and have suffered from a culture of fingerpointiness.”1 To overcome that, they recommended simply that the humans in development and the humans in operations try to learn to think more like each other At the time, few traditional media and entertainment business lead‐ ers would have thought their development and IT operations teams had something to learn from a five-year-old new-media startup But, thanks to its DevOps approach, Flickr has scaled to store 13 billion digital photos and videos, and process millions per day for its online customers, while the structure and function of traditional media and entertainment businesses have evolved to have much in common with them To be sure, even today, releases as frequent as what Flickr cited in 2009 are not the norm in most industries Few application development and IT operations managers claim to release applica‐ tions more frequently than once per month But as the digital revo‐ lution has forced all varieties of content providers to reinvent themselves as technology businesses, Flickr’s approach to keeping pace with consumer demand for device-agnostic online media access has become a model for media businesses—old and new “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr” New Media Was First in DevOps | Digital Disruption in Media and Entertainment It’s no secret that top DevOps performers are taking the lead in the media and entertainment industry Netflix, which reported earlier this year that its DevOps team facilitates 4,000 software deployments per day, now considers its biggest competition not another video streaming platform or traditional broadcaster, but its customers’ time asleep.2, A scan of the job openings at publishers including The New York Times, Dow Jones, and even 94-year-old book publisher W.W Nor‐ ton reveal that DevOps engineers are also in demand in old media, where print has given way to electronic content and websites are competing with mobile apps for eyeballs Digital delivery has also completely transformed the businesses of film and music producers and distributors Mobile technology has revolutionized gaming And AI, machine learning, virtual and augmented reality, Internet of Things, and Living Services are poised to transform them all again.4, 5, 6, 7, Meanwhile, accessible technologies have lowered barriers to entry in all areas of the industry, so that content and competition has explo‐ ded, and innovative new entrants present serious threats to longtime leaders Every sector of the industry is shapeshifting: Print media looks more like broadcast media, video streaming platforms are venturing into film and television production, and social media platforms have morphed into providers of news and video The entire sector is melting into a single category of competitors The media & entertainment businesses that are thriving are also rebuilding their infrastructures—namely, they’re applying the practi‐ ces, methods, and cultural principles of DevOps to empower their “Making multi-cloud deployment a reality at Netflix with Spinnaker” “Netflix’s Biggest Competition Is Sleep, Says CEO Reed Hastings” “Artificial Intelligence Will Revolutionize the Media Business Here’s How.” “Leveraging cloud-based predictive analytics to strengthen audience engagement” “What a venture capitalist sees in the virtual and augmented reality market” “Four digital trends reshaping the media industry” “Living Services: The next Wave in the Digitization of Everything” | DevOps in Media and Entertainment Dynamic Cloud Infrastructure Every production, test, and development “instance” of software needs to reside somewhere To build and configure new server envi‐ ronments, IT organizations have long relied on golden copies, oneoff scripts, and manual configuration, which has led to slow and error-prone deployments When your teams accelerate development through CI and CD, you’ll be faced with major bottlenecks if you don’t expand your server space and power DevOps organizations see multiple develop‐ ers, be they or 500, releasing and testing on the same day To keep pace without bankrupting the business, dynamic cloud infrastruc‐ ture is the DevOps answer Dynamic cloud infrastructure is the key DevOps-enabling tool Especially in the media and entertainment industry, for which pro‐ duction traffic ebbs and flows throughout the day and in response to a variety of predictable and unpredictable triggers, the DevOps con‐ cept wasn’t practical until the Amazon Web Services public cloud came along Even Netflix, with its 4,000 daily deploys, relies on AWS The company’s video streaming usage swells to occupy twothirds of all internet traffic in the United States at night, which is only possibly with AWS In fact, deciding whether to rely on the public cloud or set up a pri‐ vate one is a dilemma for many companies The fact is, on its own servers—a private cloud—no company can possibly scale to the level that one on a public cloud can We won’t say that you can’t DevOps on a private cloud, but the truth is, you can’t as much And for media and entertainment companies whose work is predi‐ cated on massive data and files, private cloud is rarely the best choice One exception is the giant Disney The company invested heavily to build a private cloud for its Disney Interactive video gaming and web properties.12 Then, two years ago it began “making history as the first major combo broadcaster and cable programmer to begin to build a global distribution company entirely on its own global broadcast cloud” Disney is big enough to afford it But the capital 12 “Disney Builds Private Cloud for Videogame Empire” 14 | DevOps in Media and Entertainment expenditure might look questionable in comparison to features available in five years from cloud providers like Amazon, Google, or Microsoft With IaaS, DevOps teams can configure the environments they need, when they need them Your team needs additional servers for deploying software? They’re available When they don’t, you’re not paying for those unused servers Conversely, the desire to move to the cloud can be a back door to DevOps Asked what the secret to cloud success is, CloudExpo 2016 attendees overwhelmingly responded with a mix of Agile and DevOps Indeed, DevOps deployment gets the most from the cloud Most organizations that deliver applications monthly or more fre‐ quently rely on cloud-based environments Amazon itself reports: “Media and entertainment companies are increasingly taking advan‐ tage of the scalability, elasticity, and security of the AWS Cloud to enable their businesses in new ways.” Still, plenty of media and entertainment companies don’t It’s a tough move for businesses that already have spent tens of millions of dollars on in-house hardware Dynamic Cloud Case Study: Sony Digital Media Group Within Sony Pictures Technologies, the Digital Media Group (DMG) was developing a system to manage entertainment assets But it was hamstrung by manual processes, and software delivery was taking too long Months could go by before the group shipped a piece of completely developed software to users To enable speedier delivery and simultaneously reduce infrastruc‐ ture and delivery costs, Sony DMG created an automated cloud delivery system running the Amazon Web Services infrastructure With outside help, the group stood up a complete CD system on seven AWS components for managing and provisioning resources, managing application stacks, securely isolating cloud resources, compute instances, storage, scalable DNS, and secure control of access to AWS services and resources DMG executive director Charles Cole said that the integrated approach enabled his group to be “more adaptive and responsive to customers by releasing new features and changes based on business needs.” Other reported outcomes included “happier customers, happier developers, and a significant cost reduction as releasing [became] a Dynamic Cloud Infrastructure | 15 ‘non-event’ that happens with the click of a button.” And instead of months to ship software to users, now it takes minutes Dynamic Cloud Case Study: Discovery Communications With 155 worldwide television networks including Discovery Channel, TLC, and Animal Planet, Discovery Communications has 1.8 billion cumulative subscribers in 218 countries and territories When the company wanted to upgrade and consolidate its website infrastructure and gain the ability to scale quickly to meet demand, but avoid a costly upfront one-time expense for updating its hard‐ ware, it turned to Amazon Web Services Between January 2012 and June 2013, Discovery migrated more than 40 sites to the cloud Now it hosts all digital media there and expands or shrinks its infrastructure as business requirements change by automatically scaling each part of the stacks horizontally Discovery IT executives say that relying on the cloud infrastructure enables them to focus on business initiatives without having to manage hardware and infrastructure, and, meanwhile, administra‐ tors in the Digital Media division have grown their skills to be sys‐ tem engineers, which provided more benefit to the company “Without using the AWS API and services, we wouldn’t be able to provide our staff with the tools we Our entire continuous deliv‐ ery system and our development platform are built around using [them],” says senior systems engineer/architect Shawn Stratton.13 Dynamic Cloud Case Study: NewsCorp The media conglomerate NewsCorp, with 400 million global view‐ ers and multiple lines of business, set some ambitious goals for a cloud transformation it launched in FY2014 at its largest unit, Dow Jones: Save $100 million annually by moving 75 percent of the com‐ pany’s computational power to the cloud and consolidating 50 global datacenters to just In a recent webcast, Dow Jones CIO Shaown Nandi says that might seem normal in 2017, but in 2013, “people thought it was crazy.” 13 “AWS Case Study: Discovery Communications” 16 | DevOps in Media and Entertainment The a 130-year-old public company was facing the challenges that many large enterprises on their DevOps journeys Organiza‐ tional silos stood in the way of forming a cohesive cloud deploy‐ ment strategy; legacy apps required manual deployment; compliance bottlenecks in production slowed releases, and slow release cycles slowed time to market; and people thought they should be working on security but didn’t know how In 2016, with implementation help from an outside team, the DevOps transformation began delivering results In 18 months, the company jettisoned 2,500 physical servers, and dropped power con‐ sumption to less than its 2005 rates Instead of randomly, develop‐ ers began committing code hundreds of times a day Integration rates jumped from monthly to three times a day, deployment to QA went from monthly to every other week, and deployment to pro‐ duction went from quarterly to every other week As for security, after developers no longer needed to request access to go into production to debug, they could fix and redeploy soft‐ ware on their own Security became much less of an issue, Nandi says Automated Testing In the world of DevOps and highly integrated systems, where doz‐ ens or hundreds of applications are talking to each other, automated testing is crucial And it’s not just that automation itself is difficult, but there’s so much more to be automated The sheer volume increase in applications and tools contributes to a heavier QA work‐ load Whether it’s end-user or black-box testing, unit testing, inte‐ gration testing, regression testing, or performance testing, doing it manually is no longer an option for keeping up with code across a dynamically changing infrastructure in a DevOps practice Auto‐ mating those tasks is the only way to maintain the accelerated release pace in a DevOps environment To automate some types of tests is more straightforward than others, but an automation strategy that covers every type of test is critical to your team’s ability to validate the quality and functionality of your software produced by CI/CD methods A team that accelerates CI/CD without ramping up testing simultaneously is asking for trouble Automated Testing | 17 DevOps practice executes tests as part of a software build Incorpo‐ rating testing into CI/CD lets your developers improve applications as they work It also enables teams to more punctually discover problems brought on by separate build and test processes, it ensures that critical tests are always run, and it provides better test coverage Testing whole networked systems is also essential to make sure that properly functioning independent systems also work correctly in concert Another benefit of automated testing is the ability to segment test cases Automated testing can set your servers spinning for 36 hours running a full test, especially for businesses that have larger net‐ works of systems and upward of 50,000 test cases to run By segmenting test cases, organizations can stay on the “release early and often” track Of course, understanding what kinds of changes necessitate which levels of testing is crucial to avoid paralysis If you fail to get in front of testing, you’ll be buried dealing with bugs that must be addressed immediately DevOps philosophy emphasizes that testing is as much an organiza‐ tional effort as it is technical It touches so many moving parts that it can’t be just the job of code developers or Q/A You need a test case management suite that can capture the entire ecosystem of test cases, results, and roles—the stronger your testing, the more robust your CI/CD The biggest obstacles to implementing automated testing are time, budget, and culture Companies don’t like to hear that integrating these steps into the process will make it 20 percent more expensive or take that much longer But DevOps is a continuous cycle of ideat‐ ing, building, operating, and changing software that isn’t sustainable without integrating this kind of testing into your software builds, where Dev and Ops have mutual responsibility for testing Security Automation/DevSecOps Just as automated functional testing is critical to ensuring that soft‐ ware operates as intended, automated security testing ensures that all known security concerns are being checked as part of the soft‐ ware delivery process Ultimately, by building automated security tests into your development process, you will produce higherquality applications 18 | DevOps in Media and Entertainment To be sure, most security teams report having a tough time keeping up with the rapid pace of feature releases that DevOps teams pro‐ duce.14 Making matters more complicated is the fact that in a fastpaced development environment, typical applications are assembled largely from open-source components and frameworks.15 Security flaws seep into code unnoticed The Heartbleed bug, for instance, that enabled users to expose private security keys in some situations, was coded into 70 percent of publicly available applications Adding automated scanning and other security testing to CI/CD will catch more vulnerabilities earlier It also means that you can trace and audit what tests were done and when In a DevOps environment, security cannot be an afterthought addressed at audit stage As systems have become more complex, elaborate, and technical, audits have become expensive and inade‐ quate DevSecOps offers the advantage of checking for common vul‐ nerabilities as software is being built—not after it’s deployed into production By baking basic security checks into the build process, you’re checking systems before they go online; you’re preventing problems from even being allowed to exist Open ports, user access, unpatched libraries are all secured For instance, if users can get into an open port they could get into your system, and an unpatched library could allow malware into your environment Media and entertainment DevOps teams are tasked with responsi‐ bility for protecting customer privacy, intellectual property, and dig‐ ital rights management/Digital Millennium Copyright Act compliance, as well as compliance with Payment Card Industry Data Security Standard Organizations want to know, “Is all of our con‐ tent encrypted in a format that can’t be read by anyone not using my iTunes player?” Or, “If we have a license to sell content in the US, but not in China, have we built in regional protections?” Adopting DevSecOps will also minimize the pain of audits: the method pro‐ duces version-controlled lists of all security checks performed with every deployment By making security part of the build process instead of enforcing checks after-the-fact, DevSecOps drastically reduces the time team members spend remediating issues The approach ensures that safe practices are being followed with every 14 “DevSecOps Transformation: The New DNA of Agile Business” 15 “2017 DevSecOps Community Survey” Security Automation/DevSecOps | 19 release It also simplifies security audits, and assures everyone on the team that every release undergoes the same security testing It pre‐ vents problems from ever being allowed to exist And yet, one recent IT pro survey found that even among selfprofessed mature DevOps teams, only 58 percent automate security testing within the CI/CD practice The same survey also revealed that half of developers don’t have time to spend on security Seven‐ teen percent consider it someone else’s job, yet at most organiza‐ tions, developers outnumber security professionals by 100:1 The DevSecOps mindset makes security everyone’s responsibility Seven DevSecOps Imperatives The SANS Institute, a private cybersecurity training organization, offers this advice for building security into DevOps environments: Embed automated tests and validation of controls into the deployment cycle Inventory and analyze reusable code to avoid reintroducing flaws Monitor code and results continuously in production Create triggered responses that can roll controls back to a known good state if there’s a problem Evaluate AppSec tools for DevOps capabilities and automation; replace them as needed Align and coordinate with Dev, Sec, and IT Ops teams and keep communication constant between them Commit to a culture of process descriptions, automation, and continuous monitoring and remediation DevSecOps Case Study: Sony Pictures The devastating cyberattack experienced by Sony Pictures in 2014 is a case that demonstrates the costs of not doing DevSecOps Whether the massive global network infiltration and subsequent public dump of Sony documents, emails, and IP was perpetrated by North Korean hackers, disgruntled employees, or another actor is, 20 | DevOps in Media and Entertainment to this day, debated.16 The sophistication of the hack has been deba‐ ted, too Some said it would have gone undetected by all but a few private industry network security systems Others argued that Sony’s security was like most organizations'—weak and out of date Either way, the absence of basic safeguards, such as two-factor iden‐ tification for emails, makes clear Sony was not doing DevOps For‐ tune reports a “litany of laxity,” including use of “email for longterm storage of business records, contracts, and documents saved in case of litigation.” More than 40,000 social security numbers were stored in unprotected, unencrypted files Hacked emails included a recent audit that warned security monitoring was not equipped to effectively detect or resolve incidents affecting some of the compa‐ ny’s network or infrastructure devices The audit noted 193 security incidents over 10 months And any security monitoring methods that might have been in place failed to notice the months-long 2014 breach.17 In a CNBC commentary, Scott Borg, director and chief economist of the U.S Cyber Consequences Unit, said the attackers “were pok‐ ing into everything, identifying and mapping everything on the corporate network, opening huge numbers of documents, running many applications, pushing the CPUs to very high utilization levels, and moving many terabytes of data around for months without being detected.” The fact that fewer than a third of cyberattacked companies detect breaches themselves is “evidence that most companies haven’t embraced the right precautions,” Fortune reported “With few exceptions, [Sony] didn’t segregate or provide extra security for even its most precious secrets In effect, once the invaders made it past the network gates they could go anywhere they wanted because Sony hadn’t locked any doors.” A Sony security chief, now famously, told CIO magazine in 2005 that he wouldn’t spend $10 million to protect against a $1 million loss.18 But a 2011 PlayStation Network hack cost the company a reported $171 million, and Sony estimated out-of-pocket costs of 16 “A Breakdown and Analysis of the December, 2014 Sony Hack” 17 “Sony Pictures: Inside the Hack of the Century, Part 1” 18 “Sony Pictures’ security chief once thought data breaches weren’t a big deal” (Note, the quote has disappeared from the archived CIO story, but it is quoted in numerous places online.) Seven DevSecOps Imperatives | 21 the 2014 hack at $41 million, not including pending litigation and costs to its reputation Monitoring Monitoring is among the most critical pieces of a healthy software services practice, and it’s where Development and Operations truly meet Unfortunately, it’s also the task most often overlooked and most likely to be cut from IT budgets Today, DevOps teams deal with so much more in the way of auto‐ mation, test cases, processes, services, and activity than in the old days (OK, last month) when 20 or 30 developers with one or two apps online could keep up If an app went down for an hour, your handful of users wouldn’t mind Now, every application is expected to be up 24/7, working on five continents for a billion users If you don’t incorporate a monitoring strategy that keeps you in the know about whether your service is up or down, if it’s fast or slow, and whether all 2,700 features are working, you will quickly find yourself in a bad situation When things break, you won’t know where, for how long, or for what portion of your user base Take a 2010 catastrophe at the Zappos sister site 6pm.com, for instance, as an unfortunate example of a monitoring fail The com‐ pany’s business development director explained, a “big mistake” in the pricing engine capped everything at $49.95 starting at midnight one day Shoppers snapped up bargains for six hours before its oper‐ ations team noticed the error, which cost the company more than $1.6 million in lost revenue Monitoring is not just a nice-to-have It’s about protecting your rev‐ enue, customer perceptions of your brand, and the security of your users and business Monitoring will find issues before customers (or your bosses) Building it into your automated processes will reduce manual labor and tracking efforts And it’s a way to build better feedback loops and stability; monitoring brings development teams closer to the details of how their code operates Monitoring is also critical to successful innovation and driving development from a minimum viable product, because it’s invalua‐ ble for gathering information about usage and usability Monitoring is the route to collecting A/B experimentation metrics, getting real 22 | DevOps in Media and Entertainment feedback from users in production, and understanding how users interact with your system, what works, and what doesn’t Monitoring in a DevOps environment involves automating the deployment and configuration of monitoring tools When CI/CD and dynamic cloud infrastructure combine to speed up your team’s activity and spread it out over many servers, monitoring is the way to ensure that everything is being tracked and operating as expected It also imposes development practices to ensure that your software is relaying the most critical information and validating proper opera‐ tions The DevOps approach is to start monitoring any new service from the moment it goes online We put monitoring into two categories Application Performance Monitoring (APM) tracks resources like memory, storage, and processing power It tells you when your raw resources are saturated Machine Data Monitoring looks for anoma‐ lies in application logs and machine information details such as IP addresses that are accessing the app And it can keep up with user test fails, order processes, or customer reviews Every one of your apps is kicking out information, and machine data monitoring tells you what triggers each event How you monitor is relatively simple What you with the infor‐ mation your monitoring systems collect is not A lot of organiza‐ tions collect monitoring data but ignore it (that’s one reason it’s often cut from the budget) Deriving crucial insights is difficult The key is to develop your apps to announce the events you want to be aware of If people on the Dev side aren’t building in the monitoring of granular details, those on the Ops side aren’t going to be able to avoid problems Bridging the DevOps divide produces stronger monitoring initiatives when you develop an integrated strategy for measuring results and acting on them Monitoring Case Study: Netflix Netflix is widely considered among the world’s top DevOps per‐ formers Case studies abound about how the organization’s DevOps excellence contributes to minimal downtime, robust resiliency, and continued profits Indeed, there’s no better illustration of the power of test automation and monitoring in a DevOps environment than the Netflix Simian Army Monitoring | 23 The “army” comprises several “monkey” troops and one “gorilla,” who are sent in to deliberately sabotage the Netflix system The army, available as open source software, keeps the Netflix DevOps team on its toes the way Cato Fong did to Inspector Clouseau The DevOps team automates failure to practice recovering from it In a 2011 blog post, Netflix described its army as a way to “con‐ stantly test our ability to survive once-in-a-blue-moon failures.” Among the monkeys, one known as Chaos Monkey randomly disa‐ bles production instances during the business day so that the DevOps team can build automatic recovery systems that would deal with such instances in the middle of the night Latency Monkey cre‐ ates artificial delays to simulate service degradation or entire service downtime Doctor Monkey monitors health signs, such as CPU load Conformity Monkey finds and shuts down nonconforming instances Janitor Monkey searches for and disposes of unused resources And so on Chaos Gorilla simulates an outage of the entire Amazon Web Serv‐ ices Availability Zone on which Netflix runs Netflix explains, “We want to verify that our services automatically rebalance to the func‐ tional Availability Zones without user-visible impact or manual intervention.” A Carnegie Mellon case study notes, “Thus, while writing code, Netflix developers are constantly operating in an environment of unreliable services and unexpected outages This chaos not only gives developers a unique opportunity to test their software in unexpected failure conditions, but incentivizes them to build fault tolerant systems to make their day-to-day job as developers less frustrating This is DevOps at its finest.” Beginning Your DevOps Journey Overwhelmed by the prospect of implementing DevOps at your organization? Baby steps is the answer When we work with a client to initiate a DevOps transformation from the ground up, we start small We’ll begin our work with one team working on a single app After they are collaborating and using new tools in synch in a way that proves the concept, we stage a “show and tell” to other groups This begins to generate interest for the approach throughout the company 24 | DevOps in Media and Entertainment This land-and-expand approach is the very nature of Agile Never assume that you know how to roll out DevOps across 1,000 people Get it working with one team, and when others see how successful it can be, aversion melts away People who at first said, “No thanks,” now say, “We want to start yesterday.” Here’s another tip: begin with your most adventurous group We ask clients to identify the team in their organization that always wants to experiment or try new things That’s the group most likely to suc‐ ceed It won’t fight change and it will be able to serve as an example to those in your organization who are held up by momentum and who would be resistant to a third party telling them how to change, but open to their own colleagues showing them how Here, in a nutshell, is our process for undertaking a DevOps trans‐ formation: Get the lay of land: Seek to understand how an organization operates today Identify the pain points: Is the organizational structure a mess? Are people lacking the right tools? Are teams disconnected, noncommunicative, or set up in a way that’s counterproductive? Define the plan for improvement: Determine and describe the steps for implementing the DevOps best practices model and how it will address the biggest weaknesses Apply the plan to a small group: Choose the least change-averse team as your test case and go for it Do a road show: After the test team is effectively doing DevOps, share the proof of concept with other teams and start expanding the plan to other groups As technical and technologically focused as DevOps is, at its heart it’s a people problem How people work together, get along, and hand off to each other? There’s a lot of talk about empathy in the DevOps world Remember Allspaw and Hammond at Flickr? They were right when they said that the humans in development and the humans in operations need to learn to think a bit more like each other No question, doing DevOps requires a major change in mindset and getting going demands a bit of psychology Don’t expect to tell your Beginning Your DevOps Journey | 25 teams that it will work; you will need to show them Demonstrate the value, benchmark the progress, and celebrate the wins And there’s no reason to reinvent the wheel Other organizations have survived through and thrived after DevOps transformations Take their advice There’s a ton of documentation and thousands of programs out there for DIYers But there’s also in-person, hands-on help available from experienced DevOps coaches to make your tran‐ sition time effective and cost efficient Your DevOps transformation will be easier when you work with someone who has done it before And the ultimate payoffs of that investment will be tangible, includ‐ ing better-quality innovations, scalability, and speed to market DevOps and the Next Media and Entertainment Disruptor The recent pace of change in media and entertainment technology goes to show why no organization in this industry can afford to wait to make the kind of changes we’re talking about Consider cinema: not much changed in the sector between the time it began in the 1800s until the end of the 20th century But in the last two decades, change has been constant and quickening, outstripping anything we’ve ever seen We’ve gone from 100 years of watching films in the‐ aters to watching digital movies at home, streaming them to our TVs, and then to our phones It won’t be long before media is deliv‐ ered directly to our retinas, by DevOps organizations, of course Even so, a lot of traditional media and entertainment businesses are still far more focused on producing content than on managing it The old-time businesses in the sector that are relying on legacy sys‐ tems and siloed tech teams are dealing with increasingly large data‐ sets, expanding numbers of applications, and looming security breaches DevOps is the way these businesses can survive, even thrive, in a complex future We won’t pretend that doing DevOps is easy It’s dif‐ ficult A DevOps transformation takes commitment, patience, and determination But the ability to adopt DevOps methods is not out of reach to any organization Media and entertainment businesses face the same issues in adopting DevOps that those in any industry do, and every industry is moving in the DevOps direction 26 | DevOps in Media and Entertainment The rewards are great If your business has a traditional IT depart‐ ment, you’ve got people trying to make 100 changes at once, hitting their performance caps, and generally stressing out and losing sleep As you progress down the DevOps path, life will become easier Your developers and operations teams will be able to focus on inno‐ vating in IT, not just solving problems And that’s good for every‐ one In our digital future, only the businesses that rise to the challenge of adopting this new mindset will be able to keep up with the expecta‐ tions of the workforce and consumers to thrive in the marketplace In coming years, the race for media and entertainment market share will be won not by the biggest budgets or the most established brands, but by the organizations that have established the most sophisticated DevOps practices DevOps and the Next Media and Entertainment Disruptor | 27 About the Authors Zubin Irani is a cofounder and CEO of cPrime, a 300-person full‐ service Agile+DevOps consultancy that helps Fortune 1000 firms and many of Silicon Valley’s biggest employers, including some of the leading companies in the media and entertainment industry, to implement Agile transformations, implement automation, embrace new forms of collaboration, integrate security, and adopt cloud serv‐ ices on the way to total DevOps transformations Irani’s previous experience at Deloitte and DirecTV included man‐ agement of a wide variety of application deployments with complex backend integration on projects ranging from to 400 team mem‐ bers and $100,000 to $120 million budgets He holds an Executive MBA from Columbia University and University of California, Berkeley, and a Bachelor’s degree from University of Southern Cali‐ fornia He grew up in England and California Brandon Cipes, vice president of DevOps at cPrime, has spent his career overhauling and refactoring teams, systems, software, and processes to bring the best solutions possible to all variety of techni‐ cal and business challenges He began in research, has been a part of multiple startups (one was even successful!), moved on to corporate America, and now consults to organizations looking to modernize their approach to technology Cipes excels at conceiving and deliver‐ ing large-scale projects on time, he’s a specialist in utilizing the latest in cloud technology and Agile and DevOps methodologies, and he has a people-focused talent for building teams, nurturing a healthy and productive culture, and developing sustainable solutions across multiple industries ... DevOps in Media and Entertainment tion We call the benefits “itys”—velocity, stability, and agility, which all contribute to profitability In fact, high-performing DevOps-enabled organizations... DevOps practice It automates security testing so that all known security concerns are being proactively audited as part of the software delivery process Result: Simpli‐ fied security audits and peace... year that its DevOps team facilitates 4,000 software deployments per day, now considers its biggest competition not another video streaming platform or traditional broadcaster, but its customers’