Measuring the effectiveness of your ISMS implementations based on ISO/IEC 27001 Information Security Management Systems Guidance series The Information Security Management Systems (ISMS) series of books are designed to provide users with assistance on establishing, implementing, maintaining, checking and auditing their ISMS in order to prepare for certification Titles in this Information Security Management Systems Guidance Series include: • Guidelines on requirements and preparation for ISMS certification based on ISO/IEC 27001 (ref.: BIP 0071 ) • Are you ready for an ISMS audit based on ISO/IEC 27001 ? (ref.: BIP 0072) • Guide to the implementation and auditing of ISMS controls based on ISO/IEC 27001 (ref.: BIP 0073) • Measuring the effectiveness of your ISMS implementations based on ISO/IEC 27001 (ref.: BIP 0074) Measuring the effectiveness of your ISMS implementations based on ISO/IEC 27001 Ted Humphreys and Angelika Plate First published in the UK in 2006 by BSI 389 Chiswick High Road London W4 4AL © British Standards Institution 2006 All rights reserved Except as permitted under the Copyright, Designs and Patents Act 1988, no part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means – electronic, photocopying, recording or otherwise – without prior permission in writing from the publisher Whilst every care has been taken in developing and compiling this publication, BSI accepts no liability for any loss or damage caused, arising directly or indirectly in connection with reliance on its contents except to the extent that such liability may not be excluded in law Typeset in Frutiger by Monolith Printed in Great Britain by Hobbs the Printers Ltd, Totton, Hampshire British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library ISBN 580 4601 Contents Introduction General ix 1 Scope 1 Definitions 1 Related documents 2.1 What are metrics, measures and measurements? 2.2 Why are measurements necessary? 2.2.1 General reasons and benefits 2.2.2 Requirements in ISO/IEC 27001 2.2.3 PLAN, DO, CHECK and ACT (PDCA) Model 2.2.4 Other benefits of using metrics About metrics and measurements Classes of ISMS metrics and measurements 3.1 Introduction 3.2 Management controls 3.2.1 Introduction 3.2.2 Examples 3.3 3.4 3.5 3.6 Business processes 11 3.3.1 Introduction 11 3.3.2 Examples 11 Operational controls 12 3.4.1 Introduction 12 3.4.2 Examples 12 Technical controls 14 3.5.1 Introduction 14 3.5.2 Examples 14 Audits, reviews and testing 17 3.6.1 Introduction 17 3.6.2 Examples 17 Example methods and approaches 4.1 19 Management controls 19 4.1 19 Compliance with best practice v Measuring the effectiveness of your ISMS implementations 4.2 4.3 Management cost–benefit, impact and performance reviews 21 4.1 Management reviews 23 4.1 Training and awareness measures 25 4.1 Asset management (Control ISO/IEC 7799:2005, 7.1 ) 27 ISMS processes 30 4.2.1 30 Measures for the assessment and reassessment processes Examples of operational control metrics and measurement 33 4.3.1 General operating procedures 33 4.3.2 Back-up 34 4.4 Examples of physical control metrics and measurement 36 4.5 Examples of technical control metrics and measurement 39 4.5.1 Firewalls, security gateways and intrusion detection 40 4.5.2 Patch management (SANS) 42 4.5.3 Metric for cryptographic controls 44 Developing a metrics and measurements approach 49 5.1 PLAN phase 49 5.1 49 5.2 5.3 vi 4.1 Define business policy and objectives DO phase 50 5.2.1 Defining suitable metrics and measurements 50 5.2.2 Generating metrics to measure ISMS effectiveness 50 5.2.3 Generating metrics for controls or groups of controls 51 5.2.4 Indicators, performance targets, and frequency of reviews 52 5.2.5 Implement and deploy metrics and measures 54 5.2.6 Integrating the control measurements 54 5.2.7 Integrating the ISMS effectiveness measurements 55 5.2.8 Responsibilities and resources 55 5.2.9 Documentation 56 5.2.1 Reporting 56 CHECK phase 57 5.3.1 Evaluate the results 57 5.3.2 Analyse the results 57 5.3.3 Identify corrective and preventive actions 57 Contents 5.4 ACT phase 58 5.4.1 Implementing corrective and preventive actions 58 5.4.2 Adjusting the metric and measures 58 5.4.3 Improvements in the metrics and measurement scheme 58 vii Introduction Information is one of your organization’s most valuable assets The objectives of information security are to protect the confidentiality, integrity and availability of information These basics elements of information security help to ensure that an organization can protect against: • sensitive or confidential information being given away, leaked or disclosed both accidentally or in an unauthorized way; • critical information being accidentally or intentionally modified without your knowledge; • any important business information being lost without trace or hope of recovery; • any important business information being rendered unavailable when needed It should be the responsibility of all managers, information system owners or custodians and users in general to ensure that their information is properly managed and protected from a variety of risks and threats faced by every organization The two standards ISO/IEC 7799:2005, Code of practice for information security management and ISO/IEC 27001 :2005 (revised version of BS 7799 Part 2:2002 ) Information security management systems — Requirements together provide a basis for organizations to develop an effective information security management framework for managing and protecting their important business assets whilst minimizing their risks, maximizing the investments and business opportunities of the organization and ensuring their information systems continue to be available and operational The standard ISO/IEC 7799:2005 provides a comprehensive set of best practice for information security, which organizations can adopt and implement to address the risks that they face using the risk management approach specified in the standard ISO/IEC 27001 :2005 In addition, ISO/IEC 27001 :2005 is the base requirements standard for accredited third-party ISMS (information security management system) certification based on this risk management approach Organizations applying these standards, especially those going through the accredited certification route to obtain an ISMS certificate, will need mechanisms in place to enable them to determine the effectiveness of the overall ISMS as well as of the controls that have been implemented to reduce the identified risks This is the revised version of ISO/IEC 7799:2000, which was previously BS 7799-1 :1 999 With the publication of ISO/IEC 27001 :2005, the current version of BS 7799 Part will be withdrawn and will no longer be a valid standard for third-party accredited certification Any such certification work will be carried out against the requirements specified in ISO/IEC 27001 :2005 Accreditation Bodies are responsible for issuing (see 4.2.1 of this guide) a ‘Transition Statement’ that provides details of the period during which organization’s and Certification Bodies (see 4.2.1 of this guide) involved in the ISMS certification process need to make the transition from BS 7799-2:2002 to ISO/IEC 27001 :2005 The accredited certification process also employs the accreditation and certification guides and standards ISO Guide 62/EN 4501 and EA 7/03 ix Measuring the effectiveness of your ISMS implementations Metric 3a: Adding interpretation by including time Additional information about how well and timely the patch management is working can be obtained by adding a metric that considers the timeliness aspect For example, it can be measured how fast patches are installed compared with malicious software exploiting the known vulnerability by identifying the percentage of the system patched before malicious software exploiting the vulnerability is released This could be, for example, measured on a daily basis, giving a better picture of the timeliness of the organization’s reaction The formula for this measurement is: Percentage of systems where the patch is installed prior to release of malicious software = [Systems_With_Patch/(All_Systems + Systems_With_Patch)] * 00 Metric 3b: Percentage of patch coverage for the latest critical patch By slightly modifying and combining Metrics and 3a and measuring the timeliness of the installation of the latest critical patch and measuring each day, this can give a good indication of how well the systems are covered Other metrics There are also other metrics that can be used to measure how well, effective and timely the patch management is working in an organization, such as: • the number of patches missing from each system or each PC; • the number of systems or PCs having problems after patching; • the number of applications having problems after patching Presenting results to management Applying the different metrics for patch management can easily create a lot of fairly technical data, which is, in its raw form, not suitable to be presented to management It is therefore useful to define indicators, analyse and summarize the results and perhaps choose a graphical representation for the results 4.5.3 Metric for cryptographic controls The objective of the NIST paper ‘Cryptographic Algorithm Metric’ by Norman Jorstad is to introduce a metric that allows the measurement and specification of the relative cryptographic strength of cryptographic controls This paper only considers a selected sample of symmetric block cipher algorithms and one asymmetric key algorithm, but most of the types of metrics introduced in this paper can of course also be applied to other algorithms, as the general questions, such as strength or key length also apply to them The paper considers the algorithms: • DES – the Data Encryption Standard specified in FIPS-PUB 46-2; this standard has been chosen for consideration as it has been used for a long time; • Triple DES – the DES standard executed three times and this, together with using two (or even three keys), makes the algorithm harder to break; ibid 44 Example methods and approaches • SKIPJACK – this algorithm has been chosen because its use is required by the NIST Escrow Encryption Standard; • RC5 – the algorithm named by its inventor Ron Rivest as Ron’s Code; this algorithm has been chosen because it is a fast, symmetric block cipher and the information necessary to apply the metrics were available; • RSA – the algorithm named after its inventors R.L Rivest, A Shamir and L.M Adleman; this algorithm has been chosen because it is popular and has been extensively analysed The different metrics for these cryptographic algorithms that have been considered are as follows Key length in bits The key length is one of the important factors in the security of an algorithm; the longer the key is, the more resistant the algorithm is to brute force attacks Therefore, key length was chosen as one of the metrics for cryptographic algorithms For some algorithms, there is a standard key length (such as for DES), for others (such as RSA) the key length can be increased and therefore the algorithm’s resistance against brute force attacks Please note: this consideration works for brute force attacks but if other, more sophisticated, attacks are used, the key length might have far less influence on how easily the algorithm can be broken Attack time in years The ‘attack time’ is the time required to perform the fastest known attack To make this really meaningful, the processor for which this assessment is made needs to be specified, as all this is obviously highly dependent on the computing power Of course, the use of the same processor was assumed here and all this metric can really give is a relative measure, especially taking into account the very fast evolving technological developments in this area What can also be considered in such a metric is the cost involved in running such attacks and again, if such metrics are used, they need adaptation over time to be up-todate with the latest developments Attack steps The ‘attack steps’ mean the number of steps required to perform the best known attack The number of steps helps determine the time that might be required for a successful attack, using a particular processor, without actually having to run the attack on the algorithm, which may not be feasible Rounds The number of rounds is important for some of the algorithms (not for all, e.g this is not applicable to RSA at all); for those algorithms where rounds influence the security, a larger number of rounds can generate some more confusion or diffusion and therefore more security Depending on the algorithms considered, this might or might not be a valuable metric; the organization should decide this on a case-by-case basis 45 Measuring the effectiveness of your ISMS implementations Algorithm strength This is the most interesting, yet also the most interpretable and subjective type of metric to be considered, as obviously ‘strength’ depends on the key length considered and the types of attacks assumed This type of metric is only meaningful if a comparable key length is assumed for all the algorithms considered The idea of this more subjective metric is to have a measurement for the strength of the algorithm; an algorithm is considered ‘Computationally Strong’ (meaning something close to being unbreakable) if the resources required for timely cryptanalysis are either unavailable or prohibitively expensive When determining the strength of an algorithm, consideration of the best known methods of attack and the length of time required to carry out those attacks using current technology is necessary The NIST paper ‘Cryptographic Algorithm Metric’ suggests a set of evaluation criteria to support the judgement of the algorithm strength This guide does not cover this in that amount of detail – for more information, consult the NIST paper Based on these criteria, the following graduation is suggested to measure the strength of cryptographic algorithms • US – an algorithm is U nconditionally S ecure if, no matter how much ciphertext is intercepted, there is not enough information in the ciphertext to determine the plaintext uniquely (This definition excludes algorithms that are subject to a plaintextciphertext attack and algorithms that permit the attacker to reduce the possible plaintext message to one of two values.) • CS – an algorithm is Computationally S ecure, or strong, if it cannot be broken by systematic analysis with available resources in a short enough time to permit exploitation • CCS – an algorithm is Conditionally Computationally S ecure if the algorithm could be implemented with keys that are not quite ‘long enough’ or with not quite ‘enough’ rounds to warrant a CS rating • W – a Weak algorithm is one that can be broken by a brute force attack, i.e the key can be recovered in an acceptable time (24 hours) with an ‘affordable’ investment ($200k) in cryptanalytic resources by searching every possible key An algorithm would also be weak if its structure permitted a short-cut method of attack such as differential cryptanalysis • VW – a Very Weak algorithm is one that can be broken by determining the key systematically in a short period (8 hours) with a small investment ($20k) in cryptanalysis resources The details of the assessment are fairly technical and are not the purpose of this guide but they can all be found in the NIST paper ‘Cryptographic Algorithm Metrics’ In summary, it can be said that the algorithms DES, Triple DES and SKIPJACK are rated as ’Computationally Secure’ and RC5 and RSA as ‘Conditionally Computationally Secure’ Obviously, the results are based on the computing power assumed, the key lengths used and several other aspects, so every organization should go through their own assessment of algorithms, where they can take account of their particular requirements (it might be that only asymmetric algorithms can be used anyway, due to the business purpose of the application) 46 ibid Example methods and approaches The metrics that are used in the NIST paper highlight the different aspects of cryptographic algorithms that it can be helpful to consider when selecting suitable algorithm(s) for the organization It is of course necessary, as explained in Chapter 2, to weigh the advantages of the different algorithms against the objectives of the organization and the intended business use of these algorithms 47 Developing a metrics and measurements approach As described in 2.2 above, organizations need to define metrics to measure the performance of the ISMS processes, policies and procedures and for ISMS controls or groups of controls These metrics should be suitable and tailored for the organization’s requirements, so there is no point in prescribing particular metrics (but the examples given in Chapters and hopefully help to illustrate what can be measured and how to construct metrics) To support the development of suitable metrics, this guide suggests a PDCA approach (to be applied together with the ISMS PDCA cycle described above) the organization can follow in order to have a structured approach for developing the necessary metrics These steps are suitable for metrics of any kind, irrespective of whether they might aim at ISMS processes or specific controls The recommended steps are as follows 5.1 PLAN phase 5.1.1 Define business policy and objectives Developing, implementing and applying metrics and measurements will take time and resources, so the organization should make sure that it gets the most benefit from the metrics and achieves the right balance in measuring what needs to be measured but does not generate a lot of data that are not useful or needed In order to achieve the most benefit from using metrics, the goals, aims and objectives of these metrics need to be clearly identified When identifying these, the following factors should be taken into account Input from stakeholders and interested parties The views, needs and requirements of key stakeholders and interested parties involved in the ISMS process should be identified This might not be necessary for each detail of a metric but more to identify particular goals important for these stakeholders and interested parties that should be achieved by the metrics So a manager might be interested in the overall ISMS performance and the person responsible for the communications services might want to ensure that the success of the firewalls is regularly measured The organization should identify these inputs and the best way to this is to use the responsibilities and people involved in the establishment and implementation of the ISMS There might also be some high-level requirements for metrics, such as ‘easy to use and easy to understand’ and ‘communicate the metrics and the results of measurements’ that should be taken into account when developing metrics Identification of organizational requirements The organization might have requirements to comply with legislation and regulations, might have defined business objectives that should be achieved, might wish to apply corporate governance, have existing information security or ISMS policies and security objectives in place – these and any other issues of that kind should be taken into account when developing metrics, because the results of applying the metrics can help to achieve these goals Senior management might also be interested in the success of the ISMS using some relative measures, such as comparing: 49 Measuring the effectiveness of your ISMS implementations • industry standards, best practices and benchmarks; • measuring the progress over time, e.g how much the risks have been reduced, how incidents have been managed, or how the expenses are managed; • comparing with competitors, if there are publicly available and commonly used benchmarks; • comparing the progress and achievements of different parts of the organization, different sites, or sites in different countries 5.2 DO phase 5.2.1 Defining suitable metrics and measurements The first decision that needs to be made when defining metrics is the selection of the processes, policies, procedures and controls or groups of controls that should be measured Given the different requirements in ISO/IEC 27001 :2005, it is useful to distinguish the situations where the performance and effectiveness of the ISMS processes or controls should be measured In addition, the goals and objectives that have been identified in the PLAN phase of this process should be taken into account It should be possible to derive the necessary information from the metrics to satisfy the goals and objectives that have been identified in 5.1 of this metrics-generating process There are also some general properties that are expected of a good metric, for example: • alignment with the overall organization’s principles, strategies and culture and supporting the information security objectives; • the metric should be objective, repeatable and verifiable The results of the measurement activities need to have a certain quality; should be independent of who carries out the measurements; it should be possible to repeat the measurements and arrive at comparable results; and it should be possible that others (e.g third parties) can verify the results of the measurements; • provide all information, complete, correct and timely that has been identified as necessary to support the goals and objectives of carrying out the measurements 5.2.2 Generating metrics to measure ISMS effectiveness Start with looking at the processes in clauses 4–8 of ISO/IEC 27001 :2005 Due to the requirements in ISO/IEC 27001 , it is obvious that the performance and effectiveness of these processes should be measured in some way The level of granularity of this measurement is where the objectives and goals from 5.1 above play a role and should be used to tailor the metrics used to the most suitable ones for the organization As an example, consider the risk assessment processes described in clauses 4.2.1 d) and e) of ISO/IEC 27001 :2005 If an organization has no particular goals and objectives relating to this part of the ISMS processes (except for managing the risks successfully), it might be sufficient that the organization measures the success of the risk assessment over time, i.e how much the risks are reduced over time, what are the incidents, and the results of the risk assessment match the reality of working life If, on the other hand, the organization has a specific requirement to have an always up-to-date, complete and correct asset register then the organization can decide to introduce an additional metric that helps to assess how well the asset register is developed and maintained (see also 4.1 above) 50 Developing a metrics and measurements approach Other forms of metrics can be the results of audits that regularly take place, indications of compliance or non-compliance with standards or policies, or how the organization manages incidents, resolves problems or implements improvements Overall, it is important to note that the decisions on the metrics to apply, what to measure, how often to measure, etc needs to be made by the organization Further information and examples about metrics that can be suitable to measure the effectiveness of ISMS processes can be found in Chapters and 5.2.3 Generating metrics for controls or groups of controls It might well be excessive for most, if not all organizations to define a specific metric for each of the controls that have been selected to reduce the identified risks Therefore, the organization needs to decide which of the controls can be summarized in a group that is measured together and for which controls a specific, unique metric needs to be developed This decision should be based on the risk assessment (controls that have been selected to reduce critical risks might need more attention than controls that have been selected to deal with less critical risks) and on the overall security objectives that the controls are supposed to achieve It might, for example, make sense to measure controls that achieve the same goal all with one metric, such as the different firewalls, routers and gateways that are controlling the access of a particular network If this network has a more sensitive part that is segregated from the rest, then it might also make sense to introduce a different metric, or at least a different frequency of measuring, for the access controls of that part of the network The decision of combining controls in a group for which the same measurement is applied remains entirely with the organization, all the organization should be prepared for is to explain why the decision has been made in that way and how this relates to the identified risks and security objectives When designing the metrics to be applied to controls or groups of controls, there is also a need to identify what type of metric is more suitable There are two different types of metrics that can be applied • Effectiveness – metrics that measure the performance of controls or groups of controls, how effective they are and what they achieve This is useful if a control (or group of controls) has been completely implemented and it is just a question of how it is performing • Implementation – metrics that measure the progress that a control or its implementation might make This is useful if the organization decides on a staged approach to a control or group of controls, e.g starting with a simple form and enhancing it over time These types of metrics might also make use of other standards, such as the SSE-CMM (‘System Security Engineering – Capability Maturity Model’ described in ISO/IEC 21 827:2002) that can be used to measure maturity of an implementation 9 If you wonder why this type of metric has not been discussed for the ISMS processes – the reason is simple! The ISMS processes need to be in place anyway, as they are mandatory requirements of ISO/IEC 27001 :2005 If improvements are applied then these are achieved following the continual improvement processes in the PDCA cycle Continual improvement is a different approach to using stages of implementation; therefore, these concepts cannot be applied to the ISMS processes 51 Measuring the effectiveness of your ISMS implementations More information and examples about metrics for controls or groups of controls, the different types of measurements that can be applied and how all this can be usefully implemented is contained in Chapter 5.2.4 Indicators, performance targets, and frequency of reviews For each of the metrics that have been identified above in 5.2.1 –5.2.3 (for the ones addressing the ISMS processes as well as for the ones dealing with controls or groups of controls), there are several aspects that need to be established These are as follows Frequency of measurement The frequency indicates how often the measurement should be performed There might be cases where an almost continuous measurement is needed (e.g related to network intrusions and the functioning of the controls against that) and there might be others where once a month is sufficient However, frequent measures only make sense if things are changing or new data are produced frequently and in any case they generate data that need to be evaluated The frequency should be determined based on what exactly is measured and what information is necessary to support the organization’s objectives when applying the metric Data source and data collection procedure Identify where the measurements should take place and where the results should be collected, e.g from questionnaires, internal or external audits, reports, automatic outputs, or whatever else might be applicable In addition, the procedure of data collection should be audited to ensure that no information gets lost, destroyed or modified without approval Indicators The objectives that the metric is supposed to achieve might be difficult to measure directly; they might be formulated at too high a level to allow detailed measurements Therefore, indicators should be specified, which can then be assessed and evaluated to identify whether and how far the objectives have been met Considering these indicators can help to evaluate the results of the measurements and to identify trends Identifying the suitable indicators for a metric can be difficult, especially if a metric has never been applied before In such cases, it can be helpful to look at information previously assessed (incident reports, audit reports, logs, etc.), or to a few measurements before finally setting the indicators Indicators should also be adjusted over time Examples of indicators are as follows • Impacts and consequences – this describes the potential impacts and consequences if particular targets are not met An example of such an indicator is the result that a failure of a particular network service has on the work that is carried out, which might range from ‘negligible’ over ‘disturbing’ to ‘extremely damaging’, depending on how long the service is not available • Targets – this defines the levels of performance or implementation (depending on the type of metric that is applied, see also 5.2.2 and 5.2.3 above) that are to be achieved and how far the results of the measurements are still away from these, such as ‘good’, 52 Developing a metrics and measurements approach ‘sufficient’, ‘insufficient’, or ‘not at all’ The appropriate levels of performance might be difficult to determine and should be chosen carefully to ensure that realistic targets are aimed for For implementation metrics the situation is easier, as normally complete implementation should be aimed for An example of such indicators could be the level of awareness of personnel, which is checked through a test Depending on the results of this test, the awareness of the person completing the test can be rated • Representation – the interpretation of indicators can also be supported by a graphical representation, such as ‘green’, ‘orange’ and ‘red’ traffic lights, bar charts, or whatever else is used within the organization Whilst this does not seem to be the most important issue, it is always good to capture the attention and to make information security relevant facts known and visible to management At the end of this process, the following sheet should be filled in for each of the metrics that have been identified in the previous parts of this process The dark grey highlighted aspects are those resulting from the actions in 5.2.1 –5.2.3 and the light grey ones are those resulting from 5.2.4 of this metrics-generation process; the white rows at the end are to be filled in as the result of a particular measurement that is carried out Table 13 — Measurement sheet for a metric Metric: This field should include the name of the metric and a description of the scales that are used in this metric Scope of the metric: This field should describe what should be measured with help of the metric, i.e the ISMS process or control(s) and what property or part of that process or control(s) Purpose and objectives: This field should define the purpose of the metric, which goals and objectives should be achieved through the application of the metric Measurement method: This field should describe how the measurement should take place and how the information should be linked together, e.g using calculation, formulae, or percentages Measurement frequency: This field should describe the frequency of the measurement, e.g whether it takes place monthly, weekly, daily, etc Data source and data collection procedures: This field should define where the data is collected from and which methods are used for data collection Indicators: This field should contain the indicators that are used to enhance the metric, define their purpose and how they are supposed to be applied Date of measurement and person responsible: This field should identify the date when the measurement took place and the person(s) responsible for that action 53 Measuring the effectiveness of your ISMS implementations Level of effectiveness achieved (or level of maturity, in case of a maturity metric for controls): This field should contain the results of the measurement and the date of the measurements (if that is not identified elsewhere) Causes for non-achievement: This field should contain the causes of nonachievement of targets, indicators, etc It might not always be possible to identify these causes but wherever it is, this should be recorded to help future improvements 5.2.5 Implement and deploy metrics and measures Once the metrics and all related information has been established, the metrics should be put in place in the organization and the first measurements should be carried out In order to achieve the most benefit from the efforts and resources going into the measurement activities, the implementation of the metrics should be a well-defined and ordered process The metrics should also be aligned with, and supportive of, the overall ISMS processes by providing the necessary information to identify essential improvements and should in turn be supported by the ISMS through the establishment of the policies and procedures that are necessary to operate the metrics 5.2.6 Integrating the control measurements Requirement 4.2.2 d) ‘ Define how to measure the effectiveness of the selected controls or groups of controls and specify how these measurements are to be used to assess control effectiveness to produce comparable and reproducible results (see 4.2.3 c)) ’ explicitly states that it should be defined how the effectiveness of the selected controls or groups of controls should be measured This means that the previous steps of the metrics development and implementation process described above should be carried out and should have been completed to give the input into fulfilling this requirement In order to address requirement 4.2.2 d) completely, metrics for all of the controls or groups of controls listed in the Statement of Applicability should have been developed and should be implemented together with these controls The implementation of the metrics is the aim of the DO phase in this process and, as the implementation requires resources, possibly also training and should be linked into the overall operations, the implementation of the metrics should be linked into the other elements of the DO phase in ISO/IEC 27001 :2005, as applicable The next requirement in ISO/IEC 27001 :2005 that addresses metrics and measurements for controls is 4.3.2 c) ‘ Measure the effectiveness of controls to verify that security requirements have been met.’ in the CHECK phase of the standard This requires that the metrics that have been defined are now used and that the results are evaluated, as 5.3 below describes The results of the measurements that have been made and the subsequent analysis should now be looked at to measure the effectiveness of the controls Depending on the results of the measurement analysis, improvements need to be identified and corrective and preventive action taken, which then supports the CHECK activities in ISO/IEC 27001 :2005 This last action related to metrics is not explicitly stated in ISO/IEC 27001 but it is obvious that the identification of necessary improvements is one of the main aims of applying metrics 54 Developing a metrics and measurements approach 5.2.7 Integrating the ISMS effectiveness measurements The requirements for the ISMS measurement are not as clearly detailed or stated as those for the ISMS controls – the only requirement that explicitly addresses the measurement of the ISMS effectiveness is 4.3.2 b) ‘ Undertake regular reviews of the effectiveness of the ISMS (including meeting ISMS policy and objectives, and review of security controls) taking into account results of security audits, incidents, effectiveness measurements, suggestions and feedback from all interested parties.’ When looking at this requirement, there are implicit statements in there: in order to be able to undertake effectiveness measurements (as required), the metrics for these measurements need to be developed and implemented, otherwise it is not possible to fulfil this requirement This means that the organization should go through the metric development and implementation programme described above in the PLAN and DO phases of the ISMS process in order to have these metrics readily available for the measurements required in requirement 4.3.2 b) of the CHECK phase The results of these measurements should feed into the CHECK phase, in the same way as it does for the ISMS controls, and should be the basis of identifying the necessary improvements of the ISMS In addition to what is described above, the ISMS should support the metric and measurement activities by making the necessary policies and procedures available, including: • policies and procedures to monitor the implementation and efficiency status of controls and the efficiency of the ISMS and to provide information into the measurements process; • procedures and controls to detect and respond to incidents and include these into the measurement process, as necessary; • procedures to carry out audits and reviews; • providing the necessary management framework, as described in 5.2.8–5.2.1 below, such as for: – responsibilities and resources; – training; – documentation 5.2.8 Responsibilities and resources There are a lot of different activities involved in running the metrics programme, ranging from defining the metrics to carrying out measurements, reporting data and analysing the results In order to ensure that these activities are taken seriously and carried out correctly, responsibilities need to be assigned Therefore, for each metric, the ‘owner’ of the metric, i.e the person responsible for the development of the metric, analysis of results and maintaining and updating the metric should be nominated In addition, the measurement activities need to be supported in the organization, and this support takes different forms, depending on the role in the organization • Senior management – should be supportive of the ISMS and the overall information security activities and therefore of the metrics and measurement programme that is implemented This means that they need to provide the necessary resources (see also below) and important feedback about the business objectives and requirements 55 Measuring the effectiveness of your ISMS implementations that should be addressed with the help of metrics and the targets that should be achieved • Security officers – people that have the responsibility to lead the metrics and measurement programme, to assign specific responsibilities for the implementation of specific metrics, to act as the communicator between the results of the measurement activities and senior management, to ensure that adequate resources are provided and to evaluate the results of the measurement activities and integrate them into the overall ISMS process • Asset owners, system managers, etc – all those able to provide valuable input into the measurement process should provide this input in all phases of the measurement Once the responsibilities have been defined, it should be ensured that all these people have enough time available to fulfil their tasks in the measurement process and that all other resources, such as particular software to support the measurement activities, can be put in place as planned The security officers responsible for the metrics should identify the necessary resources and communicate these needs to management and management should ensure that these resources are provided The requirements for training, e.g if additional competence is necessary to carry out the activities in the measurement process, should be identified and this training should be provided 5.2.9 Documentation Similar to all other ISMS processes, the process of measuring the effectiveness of ISMS processes and the process of measuring the implementation or effectiveness of controls should be documented This documentation should ensure that all relevant measurement results are captured and can be used to support the improvement process The documentation should also describe how the results of the measurement process are influencing the existing processes, procedures, policies and controls and need to be referenced as reasons for changes, if applicable The documented definition of the metrics needs to be available to provide evidence for the fulfilment of the requirement stated in 4.2.2 d) of ISO/IEC 27001 :2005 and the final results of the measurement processes (for the ISMS processes as well as for ISMS controls or groups of controls) should also be available as documented evidence that the requirements stated in 4.2.3 b) and c) of ISO/IEC 27001 :2005 have been correctly addressed 5.2.10 Reporting Reporting the results of the measurement process is important to ensure that the right action can be taken as a result of the measurement process and to achieve sufficient visibility at management level In accordance with the reporting and data-collection procedures and formats, the results from the measurement process should be reported to the person responsible, likely to be the ‘owner’ of the metric It should be ensured that all required information is reported in a complete and timely way to facilitate well-informed decision making There might also be a need to ensure the correctness of the information reported, as well as some arrangements to ensure that only authorized persons can view these reports, as they might highlight problem areas or vulnerabilities that might be exploited, if they become known to the wrong people 56 Developing a metrics and measurements approach 5.3 CHECK phase 5.3.1 Evaluate the results The results of the measurement processes need to be evaluated to be able to identify improvements and to learn lessons from things that are not running well so far Therefore, it is necessary to analyse the data received from the measurement activities to identify the reasons for any insufficient performance or implementation and to define the corrective action It might also be necessary to adjust the metric itself, if the feedback has shown that it is not working as intended when it was planned These activities are described in more detail below 5.3.2 Analyse the results The first action when receiving the results from a measurement activity is to compare the collected measurement results with the target indicators that have been defined, i.e to identify the gaps between the desired performance or implementation and the real situation The next step is to identify the impacts that this might have (which can be very easy if indicators are available supporting this) and an attempt to identify the causes of the poor performance and/or bad implementation should be made The metrics form contains a field where the person carrying out the measurement can note their ideas of what the causes for non-achievement of the intended performance or implementation are These entries can help to identify the causes but might also need to be further evaluated to get to the root of the problem To identify causes of poor performance, it can be useful to consider the measurement results of more than one metric, as this can help to highlight the reasons In addition, it can be beneficial to consider the results of a metric over time, to identify trends and to get an idea of whether the situation is improving or getting worse The following are some examples that can be the reason for insufficient performance or implementation; it might also be the case that combinations of these reasons might occur: • insufficient or inefficient processes; • insufficient or non-existent policies, procedures and guidelines; • lack of resources (human, time, money, etc.); • lack of training and awareness; • lack of management and control of information and IT systems 5.3.3 Identify corrective and preventive actions Based on the findings of the analysis, corrective action should take place This means that the performance of ISMS processes and/or controls, or the implementation status of the controls should be improved The implementation of these actions should be integrated into the ACT phase of the ISMS processes (see also 5.4 below) and should follow a planned procedure, such as: • determine the scope and range of corrective action and, based on the results, the impacts and the causes – this should be identified and applied for each metric; • prioritize the corrective action based on the identified impacts In addition, the ISMS processes should be performing well anyway (as required by ISO/IEC 27001 :2005), so 57 Measuring the effectiveness of your ISMS implementations any of these improvements should have priority and the priorities for controls should be decided based on the risks that they have been selected to reduce; • identify preventive action based on the results of trends analysis 5.4 ACT phase 5.4.1 Implementing corrective and preventive actions Based on the corrective and preventive actions that have been identified and the plan for their implementation that has been developed (see 5.3.3 above), these actions should be implemented to improve the existing security controls and processes, where necessary Although metrics and measurements are often primarily used to identify performance and effectiveness of the ISMS and its controls, the results of the metrics process can play an important part in improving the situation These corrective and preventive actions and the changes resulting from their implementation should be communicated to all parties involved and the agreement of all of these parties should be sought to ensure that the corrective and preventive implementations are successful 5.4.2 Adjusting the metric and measures When the measurement processes have been running for a while and the metrics have been used in practice, the analysis of results might show that some of the indicators, the measurement frequency, the data-collection procedures, or any other element of the metric might need adjustment The results of the measurements should be monitored for a while and the results compared to identify any changes that might be necessary to make the metric more suitable to the practical application It should also be validated that the metric is useful in practice for the organization, providing sufficient information to fulfil the originally assessed goals and objectives (see also 5.1 above) To assess the usefulness of a particular metric, the following questions can be asked • Does the metric give useful and valid information? • Do the actions carried out in reaction to applying the metric improve the situation? • Are the costs of applying the metric reasonable, compared with what is gained? Finally, there might also be other changes in the organization that should trigger a review, and possibly a change, of the metrics applied This could be changes in business processes, priorities and objectives, changes in the risks or in the systems that are considered – or any other changes that influence the goals and objectives that have been identified in the PLAN phase 5.4.3 Improvements in the metrics and measurement scheme Developing and implementing a process for generating suitable metrics to measure the effectiveness of the ISMS and the effectiveness or implementation of controls, or groups of controls, is a very important part of the overall ISMS PDCA lifecycle, as described in ISO/IEC 27001 :2005 Therefore, it is important that this process of developing, implementing and applying metrics is integrated with the overall ISMS process to be most beneficial for the overall aim of continual improvement of the ISMS 58