Co m pl im en ts of Modernizing Cybersecurity Operations with Machine Intelligence Advanced Threat Detection, Hunting, and Analysis Peter Guerra & Paul Tamburello Modernizing Cybersecurity Operations with Machine Intelligence Advanced Threat Detection, Hunting, and Analysis Peter Guerra and Paul Tamburello Beijing Boston Farnham Sebastopol Tokyo Modernizing Cybersecurity Operations with Machine Intelligence by Peter Guerra and Paul Tamburello Copyright © 2018 O’Reilly Media, Inc All rights reserved Printed in the United States of America Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 O’Reilly books may be purchased for educational, business, or sales promotional use Online editions are also available for most titles (http://oreilly.com/safari) For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com Editor: Courtney Allen Production Editor: Colleen Cole Copyeditor: Octal Publishing, Inc Interior Designer: David Futato March 2018: Cover Designer: Randy Comer Illustrator: Rebecca Demarest Technical Contributors: Aaron Sant-Miller and Brian Behe First Edition Revision History for the First Edition 2018-03-08: First Release The O’Reilly logo is a registered trademark of O’Reilly Media, Inc Modernizing Cybersecurity Operations with Machine Intelligence, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work Use of the information and instructions contained in this work is at your own risk If any code samples or other technology this work contains or describes is sub‐ ject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights This work is part of a collaboration between O’Reilly and Booz Allen Hamilton See our statement of editorial independence 978-1-492-03596-1 [LSI] Table of Contents Introduction The Benefits of Applying Machine Intelligence to Cybersecurity Machine Intelligence Defined The Current Threat Landscape Common Challenges Why Machine Intelligence Offers a Better Solution than Current Approaches 11 The Capabilities of Machine Intelligence Today 13 Current Capabilities of Machine Intelligence Current Limitations of Machine Intelligence Recommendations for Successful Machine Intelligence Implementations 13 15 17 Real-World Security Applications for Machine Intelligence 19 Hunting for Advanced Threats Detecting and Classifying Malware Scoring Risk in a Network 19 24 30 Addressing Readiness and Maturity for Machine Intelligence in an Organization 37 First Steps to Applying Machine Intelligence to Security Overcoming Common Challenges 37 39 Conclusion 43 iii CHAPTER Introduction It’s no secret that adversaries and hackers have gained significant and distinct advantages over the good guys in modern cyber war‐ fare The good guys in this case include government organizations and businesses of all sizes who are constantly under a barrage of cyber-attacks from a growing set of adversaries Their cyber defen‐ sive tools have also become less effective as attack campaigns have evolved to circumvent the current signature and behavior-based paradigm In 2017 alone, cyber criminals breached major credit bureaus, tele‐ com providers, government entities, mobile applications, shipping companies, U.S voting institutions, and countless individuals.1 Data stolen from these groups contains personally identifiable informa‐ tion, financial records, and even classified intelligence; each of which attackers can use toward harmful means One such major breach occurred in May 2017, in which the Wanna‐ Cry ransomware attacked tens of thousands of PCs, forcing 16 hos‐ pitals in the United Kingdom to close According to The Guardian, the attack “resulted in operations being canceled, ambulances being diverted, and documents such as patient records made unavailable in England and Scotland.”2 Unfortunately, the barrier to entry for Selena Larson, “The hacks that left us exposed in 2017”, CNN, Dec 18, 2017 Damien Gayle et al, “NHS seeks to recover from global cyber-attack as security con‐ cerns resurface”, The Guardian, 13 May 2017 cyber criminals is shrinking and the capabilities are no longer exclu‐ sive to sophisticated spy agencies In many cases, the technologies utilized to execute these massive-scale attacks are available through open source means and exploit pervasive vulnerabilities, such as those in common software libraries or within the operating system The realities of the impacts on cybercrime on organizations include a number of worrisome facts: • Ransomware such as WannaCry, NotPetya, CryptoWall, and others now earn attackers more than $1 billion in annual reve‐ nue.3 • On average, it can take 240 days to detect an intrusion.4 • By 2020, organizations worldwide will spend more than $100 billion annually on cybersecurity software, hardware, and serv‐ ices.5 • In recent years, the global cost of cybercrime exceeded $400 bil‐ lion in funds stolen and costs to clean up the damage.6 • By 2022, there will be a shortfall of 1.8 million cybersecurity workers globally.7 The ecosystem of advanced persistent threats is growing in scale and complexity, evolving more rapidly than our capabilities to respond New attack surfaces and threat vectors emerge daily, creating vulner‐ abilities in even the most hardened and secure environments Through our experience developing governance and compliance programs, we have seen typical organizations more than double their information system records when they have been instructed to Anthony Cuthbertson, “Ransomware Attacks Rise 250 Percent in 2017, Hitting U.S Hardest”, Newsweek, 5/23/17 Beverly Mowery Cooper, “Resiliency and Recovery Offset Cybersecurity Detection Limits”, SIGNAL Magazine, May 27, 2014 “Worldwide Revenue for Security Technology Forecast to Surpass $100 Billion in 2020, According to the New IDC Worldwide Semiannual Security Spending Guide”, Interna‐ tional Data Corporation, October 12, 2016 Trish Rimo and Monica Walth, “McAfee and CSIS: Stopping Cybercrime Can Posi‐ tively Impact World Economies”, McAfee, June 9, 2014 “2017 Global Information Security Workforce Study”, Frost & Sullivan, May 2017 | Chapter 1: Introduction start including assessments of Industrial Control Systems and other Operational Technologies Attackers are creative, fast, and opportunistic, and new cyber threats take many forms that can evade sophisticated defenses To keep pace with the exponential escalation of these threats, organizations need to modernize their security operations by integrating machine intel‐ ligence into their technology systems, business practices, and mis‐ sion operations Introduction | Scoring Risk in a Network Cyber risk scoring uses context-defined predictive analytics to pro‐ vide quantitative, data-driven outputs, allowing organizations to pri‐ oritize and focus remediation activities on network areas that are exposed to the greatest risk As information systems increase in number and connectivity, the attack surfaces in need of strategic and informed cyber defense grow exponentially Organizations not have the time or resources to allocate remedia‐ tion action to all known and existing vulnerabilities By providing a predictive risk score, these techniques allow organizations to priori‐ tize defense efforts and focus resources on areas of greatest danger to their overall cybersecurity As a comprehensive metric, risk is composed of the following: Attack likelihood The impact of the attack, assuming it is successful The context of the defined vulnerability, given that different network environments can have the same vulnerabilities, but different degrees of exploitability based on their configuration Accordingly, proactive remediation action is guided by a compre‐ hensive understanding of the intersection of all three criteria, a cyber risk score How Risk Scoring Poses a Challenge for Cyber Operations The growing connectivity among information systems creates increased opportunities for adversaries to take advantage of cyber vulnerabilities, disrupting strategic missions, key systems, and criti‐ cal infrastructure Not only are there more ways to enter and exploit an organization’s systems, but adversaries are becoming increasingly creative and innovative in their attack design Cyber risk scoring’s importance is formed by two core components: Attack prevention Now, more than ever, organizations need cutting-edge defensive techniques to counter rapidly evolving cybersecurity threats with the hope of preventing intrusions before they can occur As organizations depend more and more on information systems, 30 | Chapter 4: Real-World Security Applications for Machine Intelligence housing secure data and essential systems in linked environ‐ ments, preventing intrusions and attacks becomes increasingly important No longer can organizations focus their time responding to existing breaches and mitigating impacts of known attacks with a treatment-centric approach Rather, they must work to prevent attacks before those attacks can occur, immunizing their systems of risk before adversaries can exploit them By deploying a risk-informed, predictive, and immunization-centric technique, organizations can move toward strategic cyber resilience and information security Risk scoring also informs the cyber operational analysts what reme‐ diations and tasks are a priority for your enterprise By focusing on preventing attacks on your mission-critical systems, breaches will become less impactful and response time to them will be reduced until the entire network can be secured to the best point possible while still being able to support mission needs Scope of remediation activity Organizations cannot remediate all their vulnerabilities and cyber risks With growing IT dependence and interconnectivity creating broader attack surfaces, vulnerabilities are appearing faster than organizations can respond to them To maintain the highest level of cybersecurity, organizations must intelligently prioritize their resources and time to mitigate vulnerabilities that expose them to the greatest risk By providing a predictive ranking metric, cyber risk scoring offers a means to decrease organizational cyber vulnerability in the most strategic and effi‐ cient manner Cyber risk scoring has been performed by domain experts for many years, but subjectively defined metrics often fail to provide the requisite granularity for effective prioritization schemes and are time intensive to generate How Machine Intelligence Applies to Risk Scoring By driving cyber risk assessments with machine learning instead of domain expert interpretation, scores are entirely data-driven and quantitative These scores can offer both precise point estimates of scaled risk as well as data-driven uncertainty bounds around these scores to better inform decision makers Scoring Risk in a Network | 31 Additionally, models can score vulnerabilities and exploit opportu‐ nities at scale and efficiently, covering the landscape of known risk in a matter of hours, rather than days, weeks, and months That said, machine learning–curated cyber risk scores are not agnostic of domain expertise and subjective input As naturallanguage processing (NLP) and data-mining techniques have become more advanced, models have been increasingly capable of structuring and ingesting unstructured expert-curated assessments This allows for expert-informed machine learning models that can ingest institutional knowledge in a variety of data formats In fact, cyber risk scores blend objective characteristics, historical context, and subjective input into one comprehensive machine learning suite for remediation prioritization In implementation, machine learning–driven techniques function within proven risk management frameworks such as NIST RMF.8 Here, machine learning models can digest characteristics of vulnera‐ bilities, known historical exploits, attacks, and threat intelligence as well as subjective input By using feature engineering, feature generation, and artificial intel‐ ligence–driven techniques, these models can return point estimates for all components of cyber risk (i.e., likelihood and impact) framed in the known configuration context Although these risk scores live at the vulnerability level, they can be rolled up at the system level by marrying historically curated risk scores with known incident data in a predictive manner, prior to the incident This allows organizations to develop mature, predictive, systemlevel scores to better identify systems at the greatest risk, as well as the vulnerabilities that are driving this risk With increased risk awareness, organizations can operationalize a framework to efficiently and strategically decrease risk Here, resources can be mapped to remediation strategies that carry defined financial and labor-time costs and predictively measured risk reductions “Risk Management Framework (RMF) Overview”, National Institute of Standards and Technology: Computer Security Resource Center, February 9, 2018 32 | Chapter 4: Real-World Security Applications for Machine Intelligence By optimizing resource deployment to activities that maximize the decreased risk, both in magnitude and time, organizations can pro‐ actively immunize themselves to the most salient threats in the most efficient manner How to Build a Machine Intelligence Capability to Support Risk Scoring Building a machine intelligence capability for cyber risk scoring isn’t about the recipe for machine learning as much as it’s about the ingredients you put into the pot Machine intelligence can be incredibly impactful in a predictive framework, allowing organiza‐ tions to operate proactively rather than reactively For machine learning algorithms to function in this manner, they require a com‐ plete representation of the environment, including the threat land‐ scape, vulnerability characteristics, known impacts, and system configurations prior to an exploit This allows algorithms to asso‐ ciate these pre-exploit characteristics with whether an exploit occur‐ red and the outcomes of that exploit In doing so, organizations can develop tailored subscores for cyber risk as well as comprehensive measures for risk As Figure 4-3 illustrates, these inputs are piped through NLP engines and feature-generation techniques and passed to the machine learning engine that outputs data-driven risk scor‐ ing Scoring Risk in a Network | 33 Figure 4-3 Workflow for machine learning model development to enable cyber risk scoring Utilizing This Machine Intelligence Technique in the Real World You can utilize risk scoring approaches with machine intelligence to accurately predict the likelihood of a threat actor exploiting a soft‐ ware vulnerability, such as a Common Vulnerability and Exposure (CVE) from the National Vulnerability Database (NVD).9 It’s possi‐ ble to derive impact scores through close partnership with the orga‐ nization and an intimate understanding of the mission Additionally, environment context can be measured and defined “National Vulnerability Database”, National Institute of Standards and Technology, February 15, 2018 34 | Chapter 4: Real-World Security Applications for Machine Intelligence only by understanding both an organization’s current and historic system configurations Generally speaking, the likelihood of an exploit reflects the ease of exploit By applying a predictive modeling scheme with machine learning, it’s possible to prioritize known vulnerabilities that have yet to be exploited and flag existing vulnerabilities that have a known exploit available This allows organizations to immediately harden their defenses with machine intelligence–curated and opti‐ mized vulnerability risk scoring, moving from reactive techniques toward proactive techniques By blending machine learning scores with organizationally informed impact assessments, framed in the context of environmen‐ tal configuration, organizations can quickly stand up a cyber risk scoring capability This moves organizations from treatment-centric cyber defense to immunization-centric cyber defense, where reme‐ diation activities can be strategically allocated and precisely targeted to prevent attacks before they can occur Tips and Best Practices Following are some tips and best practices for assessing vulnerabili‐ ties and building compliance programs utilizing machine intelli‐ gence • Organizations need to proactively begin collecting data To maintain a predictive capability, machine learning algorithms require both a snapshot of the cyber environment before and after an exploit This includes data on vulnerabilities, system configurations at the time of exploit, and known threat intelli‐ gence at that time • Organizations need to develop tailored impact scores for known exploits These scores should consider the data one can access with an exploit, the degree to which that exploit can waterfall into subsequent exploits, and the organizational damage that exploit can cause • Don’t forget unstructured data and SME analysis Robust NLP algorithms can consolidate this unstructured data into more granular and quantitative measures that are easy for machine learning algorithms to digest Oftentimes, it is the expert- Scoring Risk in a Network | 35 curated insight captured in the text data that differentiates pow‐ erful machine intelligence from passable machine intelligence • Organizations need to rigorously document both known exploits in their environment, and log all system-level, exploittriggered incidents This allows machine intelligence algorithms to function in a fully supervised manner, which can thus allow for highly predictive and powerful risk scoring 36 | Chapter 4: Real-World Security Applications for Machine Intelligence CHAPTER Addressing Readiness and Maturity for Machine Intelligence in an Organization First Steps to Applying Machine Intelligence to Security Because of its inherent complexity, machine intelligence cybersecur‐ ity is not a “one size fits all” solution for most organizations Many organizations, however, can benefit greatly from the careful imple‐ mentation of specific machine intelligence capabilities The following five-question list is designed to help organizations assess their readiness to move ahead with machine intelligence cybersecurity initiatives.1 Are all of the devices attached to my network compliant with my poli‐ cies? This is challenging because you need 100% visibility (i.e., no shadow IT), you need a way to have the sensors “know” what your policy is (i.e., desired state), you need a way to assess the current state of the device (i.e., actual state), and you need a way to measure the delta between desired and actual Booz Allen Hamilton 37 Which of my systems are most vulnerable? This is also challenging to answer because very few organiza‐ tions have correlated “devices” with “systems” in an automated manner Thus, while it’s easy to understand which devices are vulnerable, it’s much more challenging to know which systems those devices belong to What are the biggest cyber threats facing my organization? Answering this means correlating vulnerabilities and risk with external threat actor activity, and perhaps finding latent risk/ vulnerabilities (or worse) in the system (i.e., through red team‐ ing) before it becomes an issue How effective are my cyber efforts? Effectiveness is a tricky thing to define but being able to this for cyber tools is important to see if they are working Example metrics could be “mean time to detect” or “mean time to respond.” Improving in these dimensions would indicate the tools or processes are effective What areas can I make more efficient, and what is the return on investment? You need to build out models that define and measure this on an ongoing basis And have the courage to end a project that isn’t going well or might not really be needed For example, the use of automation and machine learning are awesome new capabilities that would benefit many organizations But it’s diffi‐ cult for them to know “where to start.” One issue that you must address is the substantial gap between the need for trained cybersecurity workers and the availability of quali‐ fied applicants It’s imperative for organizations to take inventory of their in-house skills and determine whether the best course of action is outsourcing talent or developing it from within Either course entails difficulties and obstacles Outsourcing is expensive and it’s usually not the best long-term solution When you outsource technical talent, you lose both intellectual property and institutional knowledge when the engagement ends In-house training is also expensive and requires competencies that are beyond the reach of many organizations The benefit of in-house training is that the people you train tend to stay longer and contrib‐ ute more knowledge to the organization 38 | Chapter 5: Addressing Readiness and Maturity for Machine Intelligence in an Organization Some organizations have experimented with training data scientists for roles in cybersecurity and providing cybersecurity workers with training in data science A new field of cybersecurity data science is slowly emerging, which is already showing some promise in import‐ ing machine intelligence techniques from other domains However, due to the unique scale of cyber data, constantly evolving threats, and real-time nature of operations machine intelligence applications require special tuning and deep cyber domain knowledge to be effective Overcoming Common Challenges Because the machine intelligence ecosystem itself is continually expanding and evolving, knowing where to begin can be difficult There is no reason, however, for that inherent difficulty to prevent or delay implementations of machine intelligence cybersecurity projects and strategies Here are several steps for overcoming initial difficulties and challenges: Define goals Investments in machine intelligence cybersecurity capabilities should be part of an overall strategy Develop a multiyear plan and state clearly the goals you intend to achieve Machine intel‐ ligence capabilities are should be viewed as strategic invest‐ ments; resist the urge to implement machine intelligence solutions for tactical purposes Determine tolerance for risk Machine intelligence doesn’t have a long history or track record As a result, it poses risks that are both unknown and unknowa‐ ble It’s important to prepare the organization for the inevitable challenges of implementing new and complex technology solu‐ tions If your organization has a very low appetite for risk, machine intelligence might not be a good choice until the field further matures Assess current state of data assets Machine intelligence and the techniques supporting it (e.g., machine learning, deep learning, reinforcement learning, cogni‐ tive learning, NLP etc.) all require large amounts of data This includes quality labeled data, as we have discussed Although much has been written about the ability of machine intelligence to use vast quantities of unstructured data, it still takes effort to Overcoming Common Challenges | 39 prepare data for machine intelligence systems Determine first whether the MI solutions you are considering can use your data Assess current state of talent assets The global shortage of data scientists has been widely docu‐ mented Working with machine intelligence systems requires a certain type of mindset and an unusual blend of skills Make sure that you have enough people with the skills and mindset to implement your machine intelligence projects and keep it run‐ ning long enough to generate suitable returns on your invest‐ ments Start small, and then scale up Develop interest and support for machine intelligence initiatives by identifying low-hanging fruit and opportunities for easy vic‐ tories It’s usually better to start with a small pilot project and then gradually grow it, rather than beginning with a huge project and being forced to scale it back if it fails to meet its pro‐ jected goals We recommend utilizing an Agile process, such as one that can create an analytics life cycle composed of a Research Phase, Minimum Viable Experiment (MVE) Phase, Minimum Viable Product (MVP) Phase, and Refinement Phase Table 5-1 looks at each of these phases Table 5-1 Agile process to integrate machine intelligence Research Phase • Identify data requirements (What are the features we need? What sources will they come from?) • Explore and aggregate research papers relevant to the model approaches available • Asses model approach viability given technology environment MVE Phase • Build prototype code implementing model approaches identified in research phase • Assess practicality across dimensions to include: performance (speed, memory usage, etc.), data sufficiency, and model accuracy MVP Phase • Ensure analytic meets technology architecture requirements • Ensure script ingest methods and output are appropriately modular • Deploy script to run as a part of the larger analytic workflow/system • Refactor MVE for deployment standard 40 | Chapter 5: Addressing Readiness and Maturity for Machine Intelligence in an Organization Refinement Phase • Collect performance measures (true/false positives, accuracy) in production • Solicit feedback from analysts and operators on usefulness • Update and retrain models in production over time to improve performance Develop a communications plan Machine intelligence is deeply mystifying to most people You will need a detailed communications plan to make sure that all stakeholders in your machine intelligence strategy understand its purpose and understand their roles in nurturing its success The communications plan should consist of more than a single document or email Ideally, the plan will include meetings, blog posts, online tutorials, internal webinars, podcasts, and teambuilding activities such as hack-a-thons and competitions Use a framework Using frameworks to evaluate and guide investments in new technologies and complex systems can be greatly beneficial Table 5-2 presents a reference framework that your organization can use as a guideline for matching its goals with available machine intelligence capabilities Table 5-2 Example framework for machine intelligence capabilities Types of Machine Intelligence Goal(s) Simple task execution Pattern recognition Contextual reasoning • Automate simple processes • Augment staff in low- to medium-complexity tasks • Free staff from rote work • Generate new business/ operational insights • Create an industryleading technology/ tool • Create efficiencies/ save money • Transform internal operations Risk tolerance Low to weeks Time investment • Create an enduring competitive • Create a new client-facing advantage capability • Enhance corporate brand Moderate – High High to 12 months 12 to 36 months Overcoming Common Challenges | 41 Types of Machine Intelligence Simple task execution Pattern recognition Data asset requirements • Few data assets • Data assets siloed/ unorganized • Massive data assets or access to data • Data labeled, organized Talent requirements New in-house MI talent generally not required —implementation typically through vendors, consultants MI talent required, either inhouse or accessed via partnerships (e.g., with academic institutions) Solutions • Robotic process automation • Core machine learning software • Expert systems • Computer vision • Simple automation • NLP • Automate threat reporting • Create recommendation engines • Automate incident response • Predict threats from disparate data Examples Contextual reasoning Variable data assets —generally large Leading machine intelligence talent required, typically accessed through partnerships (e.g., in engagements with large tech vendors) Contextual machine learning (“semantic” or “cognitive computing”) • Develop autonomous systems • Understand motivations and reasoning 42 | Chapter 5: Addressing Readiness and Maturity for Machine Intelligence in an Organization CHAPTER Conclusion The capabilities of cyber attackers are growing faster than the capa‐ bilities of organizations to defend against them Because the cyber threat landscape is both complex and highly dynamic, machine intelligence solutions offer unique strengths and benefits to organi‐ zations tasked with protecting critical data and information systems From an operational perspective, machine intelligence provides greater efficiency and a wider range of options for dealing with per‐ sistent threats, persistent malware, and pervasive software exploits The emergence of massively parallelized processing on commodity hardware, together with the availability of big data and the rise of data science, has rapidly transformed machine intelligence from a dream into hard reality Operationally, machine intelligence is fast becoming an essential part of the basic information technology portfolio Machine intelligence is no longer a theory confined to university laboratories Its essential capabilities have been abstracted into usa‐ ble tools and solutions In other words, machine intelligence is now ready for duty Now is the time for organizations to begin the process of acquiring the knowledge, experience and expertise that will be absolutely nec‐ essary to develop, implement and manage machine intelligence capabilities to create substantive advantages in your efforts to counter cyber attackers and protect critical systems and data 43 About the Authors Peter Guerra is a chief data scientist and vice president at Booz Allen Hamilton, where he helps transform cybersecurity for clients within the federal, commercial, and Department of Defense mar‐ kets Peter leads a team of cybersecurity professionals—including threat hunters, reverse engineers, and data scientists—focused on employing sophisticated analytics tools to protect the nation and organizations from cyber attacks Peter also helped build Booz Allen Hamilton’s data science capability, increasing the team from a few dozen to more than 600 people—now the largest data science team in the U.S federal realm Paul Tamburello is a chief technologist at Booz Allen Hamilton, where he leads teams of experts in cybersecurity, software engineer‐ ing, and machine intelligence He delivers high-end data analytic capabilities for clients in Government and Commercial organiza‐ tions, with an emphasis on big data and machine learning He archi‐ tected and developed several of the largest data systems in the Federal Government that support real-time defensive and offensive cyberspace operations Paul holds a bachelor’s degree in Computer Science from the University of North Carolina at Chapel Hill as well as a master’s degree in Information Systems Engineering from Johns Hopkins University ... algorithms or technologies 12 | Chapter 2: The Benefits of Applying Machine Intelligence to Cybersecurity CHAPTER The Capabilities of Machine Intelligence Today Current Capabilities of Machine Intelligence. .. concept of machine intelligence new in the context of cyberspace operations Here is a brief definition of machine intelligence: Machine intelligence is a field concerned with producing machines... practices, and mis‐ sion operations Introduction | CHAPTER The Benefits of Applying Machine Intelligence to Cybersecurity Machine Intelligence Defined Improving cybersecurity operations over the short