Forensic systems engineering

Forensic Systems Engineering Wiley Series in Systems Engineering and Management William Rouse, Editor ALPHONSE CHAPANIS Andrew P Sage, Founding Editor Human Factors in Systems Engineering ANDREW P SAGE and JAMES D PALMER YACOV Y HAIMES Software Systems Engineering WILLIAM B ROUSE Risk Modeling, Assessment, and Management, Third Edition Design for Success: A Human‐Centered Approach to Designing Successful Products and Systems DENNIS M SUEDE LEONARD ADELMAN ANDREW P SAGE and JAMES E ARMSTRONG, Jr Evaluating Decision Support and Expert System Technology The Engineering Design of Systems: Models and Methods, Second Edition Introduction to Systems Engineering WILLIAM B ROUSE ANDREW P SAGE Decision Support Systems Engineering Essential Challenges of Strategic Management YEFIM FASSER and DONALD BRETINER YEFIM FASSER and DONALD BRETTNER Process Improvement in the Electronics Industry, Second Edition WILLIAM B ROUSE Strategies for Innovation ANDREW P SAGE Systems Engineering HORST TEMPELMEIER and HEINRICH KUHN Flexible Manufacturing Systems: Decision Support for Design and Operation WILLIAM B ROUSE Catalysts for Change: Concepts and Principles for Enabling Innovation UPING FANG, KEITH W HIPEL, and D MARC KILGOUR Interactive Decision Making: The Graph Model for Conflict Resolution DAVID A SCHUM Evidential Foundations of Probabilistic Reasoning JENS RASMUSSEN, ANNELISE MARK PEJTERSEN, and LEONARD P GOODSTEIN Cognitive Systems Engineering ANDREW P SAGE Systems Management for Information Technology and Software Engineering Management for Quality in High‐ Technology Enterprises THOMAS B SHERIDAN Humans and Automation: System Design and Research Issues ALEXANDER KOSSIAKOFF and WILLIAM N SWEET Systems Engineering Principles and Practice HAROLD R BOOHER Handbook of Human Systems Integration JEFFREY T POLLOCK and RALPH HODGSON Adaptive Information: Improving Business Through Semantic Interoperability, Grid Computing, and Enterprise Integration ALAN L PORTER and SCOTT W CUNNINGHAM Tech Mining: Exploiting New Technologies for Competitive Advantage REX BROWN Rational Choice and Judgment: Decision Analysis for the Decider WILLIAM B ROUSE and KENNETH R BOFF (Editors) Organizational Simulation HOWARD EISNER Managing Complex Systems: Thinking Outside the Box STEVE BELL Lean Enterprise Systems: Using IT for Continuous Improvement J JERRY KAUFMAN and ROY WOODHEAD YORAM KOREN The Global Manufacturing Revolution: Product‐Process‐Business Integration and Reconfigurable Systems Stimulating Innovation in Products and Services: With Function Analysis and Mapping AVNER ENGEL WILLIAM B ROUSE WILLIAM B ROUSE (Editor) Enterprise Transformation: Understanding and Enabling Fundamental Change JOHN E GIBSON, WILLIAM T SCHERER, and WILLAM F GIBSON How to Do Systems Analysis WILLIAM F CHRISTOPHER Holistic Management: Managing What Matters for Company Success WILLIAM B ROUSE People and Organizations: Explorations of Human‐Centered Design MOJAMSHIDI System of Systems Engineering: Innovations for the Twenty‐First Century ANDREW P SAGE and WILLIAM B ROUSE Handbook of Systems Engineering and Management, Second Edition JOHN R CLYMER Simulation‐Based Engineering of Complex Systems, Second Edition KRAG BROTBY Information Security Governance: A Practical Development and Implementation Approach JULIAN TALBOT and MILES JAKEMAN Security Risk Management Body of Knowledge SCOTT JACKSON Architecting Resilient Systems: Accident Avoidance and Survival and Recovery from Disruptions JAMES A GEORGE and JAMES A RODGER Smart Data: Enterprise Performance Optimization Strategy Verification, Validation, and Testing of Engineered Systems The Economics of Human Systems Integration: Valuation of Investments in People’s Training and Education, Safety and Health, and Work Productivity ALEXANDER KOSSIAKOFF, WILLIAM N SWEET, SAM SEYMOUR, and STEVEN M BIEMER Systems Engineering Principles and Practice, Second Edition GREGORY S PARNELL, PATRICK J DRISCOLL, and DALE L HENDERSON (Editors) Decision Making in Systems Engineering and Management, Second Edition ANDREW P SAGE and WILLIAM B ROUSE Economic Systems Analysis and Assessment: Intensive Systems, Organizations, and Enterprises BOHDAN W OPPENHEIM Lean for Systems Engineering with Lean Enablers for Systems Engineering LEV M KLYATIS Accelerated Reliability and Durability Testing Technology BJOERN BARTELS, ULRICH ERMEL, MICHAEL PECHT, and PETER SANDBORN Strategies to the Prediction, Mitigation, and Management of Product Obsolescence LEVANT YILMAS and TUNCER OREN Agent‐Directed Simulation and Systems Engineering ELSAYED A ELSAYED Reliability Engineering, Second Edition BEHNAM MALAKOOTI Operations and Production Systems with Multipme Objectives MENG‐LI SHIU, JUI‐CHIN JIANG, and MAO‐HSIUNG TU Quality Strategy for Systems Engineering and Management ANDREAS OPELT, BORIS GLOGER, WOLFGANG PFARL, and RALF MITTERMAYR Agile Contracts: Creating and Managing Successful Projects with Scrum KINJI MORI Concept‐Oriented Research and Development in Information Technology KAILASH C KAPUR and MICHAEL PECHT Reliability Engineering MICHAEL TORTORELLA Reliability, Maintainability, and Supportability: Best Practices for Systems Engineers DENNIS M BUEDE and WILLIAM D MILLER The Engineering Design of Systems: Models and Methods, Third Edition JOHN E GIBSON, WILLIAM T SCHERER, WILLIAM F GIBSON, and MICHAEL C SMITH How to Do Systems Analysis: Primer and Casebook GREGORY S PARNELL Trade‐off Analytics: Creating and Exploring the System Tradespace CHARLES S WASSON Systems Engineering Analysis, Design and Development Forensic Systems Engineering Evaluating Operations by Discovery William A Stimson This edition first published 2018 © 2018 John Wiley & Sons, Inc All rights reserved No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions The right of William A Stimson to be identified as the author of this work has been asserted in accordance with law Registered Offices John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA Editorial Office 111 River Street, Hoboken, NJ 07030, USA For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com Wiley also publishes its books in a variety of electronic formats and by print‐on‐demand Some content that appears in standard print versions of this book may not be available in other formats Limit of Liability/Disclaimer of Warranty The publisher and the authors make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties; including without limitation any implied warranties of fitness for a particular purpose This work is sold with the understanding that the publisher is not engaged in rendering professional services The advice and strategies contained herein may not be suitable for every situation In view of on‐going research, equipment modifications, changes in governmental regulations, and the constant flow of information relating to the use of experimental reagents, equipment, and devices, the reader is urged to review and evaluate the information provided in the package insert or instructions for each chemical, piece of equipment, reagent, or device for, among other things, any changes in the instructions or indication of usage and for added warnings and precautions The fact that an organization or website is referred to in this work as a citation and/or potential source of further information does not mean that the author or the publisher endorses the information the organization or website may provide or recommendations it may make Further, readers should be aware that websites listed in this work may have changed or disappeared between when this works was written and when it is read No warranty may be created or extended by any promotional statements for this work Neither the publisher nor the author shall be liable for any damages arising here from Library of Congress Cataloging‐in‐Publication Data Names: Stimson, William A., author Title: Forensic systems engineering : evaluating operations by discovery / William A Stimson Description: Hoboken, NJ : Wiley, 2018 | Series: Wiley series in systems engineering and management | Includes bibliographical references and index | Identifiers: LCCN 2017039503 (print) | LCCN 2017042410 (ebook) | ISBN 9781119422761 (pdf ) | ISBN 9781119422785 (epub) | ISBN 9781119422754 (hardback) Subjects: LCSH: Failure analysis (Engineering) | System failures (Engineering) | Forensic sciences | Evidence, Expert | BISAC: TECHNOLOGY & ENGINEERING / Electronics / General Classification: LCC TA169.5 (ebook) | LCC TA169.5 S755 2018 (print) | DDC 620/.00452–dc23 LC record available at https://lccn.loc.gov/2017039503 Cover Design: Wiley Cover Image: © Digital Vision./Gettyimages Set in 10/12pt Warnock by SPi Global, Pondicherry, India Printed in the United States of America 10 9 8 7 6 5 4 3 2 1 To Josette, my love, my wife, my friend, my life ix Contents Preface  xix What Is Forensic Systems Engineering?  1.1 Systems and Systems Engineering  1.2 ­Forensic Systems Engineering  References  Contracts, Specifications, and Standards  2.1 ­General  2.2 ­The Contract  2.2.1 Considerations 9 2.2.2 Contract Review  10 2.3 ­Specifications  12 2.4 ­Standards  14 Credits  16 References  16 Management Systems  17 3.1 ­Management Standards  18 3.1.1 Operations and Good Business Practices  18 3.1.2 Attributes of Management Standards  18 3.2 ­Effective Management Systems  19 3.2.1 Malcolm Baldrige  19 3.2.2 Total Quality Management  20 3.2.3 Six Sigma  20 3.2.4 Lean 21 3.2.5 Production Part Approval Process  22 3.3 ­Performance and Performance 23 3.4 ­Addendum  23 ­Credits  24 References  24 x Contents Performance Management: ISO 9001  25 4.1 ­Background of ISO 9000  26 4.1.1 ISO 9001 in the United States  27 4.1.2 Structure of ISO 9000:2005  27 4.1.3 The Process Approach  28 Form and Substance  32 4.2 4.2.1 Reference Performance Standards  33 4.2.2 Forensics and the Paper Trail  34 ­Credits  35 References  35 The Materiality of Operations  37 5.1 Rationale for Financial Metrics  38 5.1.1 Sarbanes–Oxley  38 5.1.1.1 Title III: Corporate Responsibility  38 5.1.1.2 Title IV: Enhanced Financial Disclosures  39 5.1.2 Internal Control  39 5.1.3 The Materiality of Quality  41 5.2 Mapping Operations to Finance  41 5.2.1 The Liability of Quality  43 5.2.2 The Forensic View  44 ­Credits  44 References  44 Process Liability  47 6.1 6.1.1 6.1.2 6.2 Theory of Process Liability  48 Operations and Process Liability  50 Process Liability and Misfeasance  51 Process Liability and the Law  52 ­Credits  52 References  52 Forensic Analysis of Process Liability  55 7.1 7.1.1 7.1.1.1 7.1.1.2 7.1.1.3 7.1.1.4 7.1.2 7.1.2.1 7.1.2.2 7.1.2.3 Improper Manufacturing Operations  57 Verification and Validation  57 Nonstandard Design Procedures  57 Unverified or Unvalidated Design  58 Tests Waived by Management  58 Altered Test Procedures and Results  58 Resource Management  59 Unmonitored Outsourcing  59 Substandard Purchased Parts  60 Ghost Inventory  60 Contents 7.1.2.4 7.1.3 7.1.3.1 7.1.3.2 7.2 7.2.1 7.2.2 7.2.3 7.2.4 7.2.5 7.2.6 Ineffective Flow Down  61 Process Management  61 Forced Production  61 Abuse and Threats by Management  62 Management Responsibility  62 Effective Internal Controls  62 Business Standards of Care  63 Liability Risk Management  64 Employee Empowerment  65 Effective Management Review  65 Closed‐Loop Processes  66 References  67 Legal Trends to Process Liability  71 8.1 8.2 8.2.1 8.2.2 9.1 9.1.1 9.1.2 9.1.3 9.1.4 9.2 9.2.1 9.2.2 9.3 9.3.1 9.3.2 9.4 10 10.1 10.2 10.2.1 10.2.2 10.2.3 10.2.4 10.2.5 An Idea Whose Time Has Come  71 Some Court Actions Thus Far  72 QMS Certified Organizations  73 QMS Noncertified Organizations  74 References  75 Process Stability and Capability  77 Process Stability  77 Stability and Stationarity  78 Stability Conditions  79 Stable Processes  80 Measuring Process Stability  82 Process Capability  83 Measuring Capability  83 A Limit of Process Capability  85 The Rare Event  85 Instability and the Rare Event  85 Identifying the Rare Event  86 Attribute Testing  87 References  88 Forensic Issues in Product Reliability  91 Background in Product Reliability  91 Legal Issues in the Design of Reliability  94 Good Design Practices  95 Design Is Intrinsic to Manufacturing and Service  95 Intended Use  95 Paper Trail of Evidence  96 Reliability Is an Implied Design Requirement  97 xi 326 Appendix F  Nonstatistical Sampling Plans Table F.1  Selecting audit confidence level as a function of the ADR Acceptable deviation rate Assessment Kelly Guy Substantial confidence 2–7% ≤5% Moderate confidence 6–12% ≤10% Little confidence 11–20% ≤20% No confidence Omit test Omit test However, Guy (1981c) advises against using ADRs greater than 10% if the purpose of the inquiry is about the reliance of internal controls, as opposed to some other of their characteristics As reliance of a control is a major issue in systems operation, then 10% can be considered an upper limit for ADRs for purposes of forensic systems examination As in statistical sampling, in a nonstatistical plan you use knowledge of past system performance to estimate the control deviation rate in order to arrive at a sample size Then having chosen the sample size and conducted the test, you evaluate the control based on the difference between the sample and the ADRs F.1.6  The Effect of Sample Size on Beta Error Many analysts consider Beta error as the more grave sampling error, as Alpha error tends to be self‐correcting If an analyst makes an Alpha error, the decision will be challenged because the analyst is challenging a control that the performer believes to be good The performer will insist on additional testing with larger sample size However, a Beta error will not be challenged; the analyst reports good news to the performer, although the good news is false In addition, additional testing increases cost Apostolou (1991b) classifies an Alpha Error as an efficiency indicator; a Beta error as an effectiveness indicator Nonstatistical sampling plans not measure Beta error, but you know that it exists You must make some effort to reduce it, and sample size has much to with it Sample size is affected by the error rate as shown in Table E.2, in which changes in sample size affect key system parameters and conversely For example, suppose that you want to choose a smaller sample size By doing so, you increase the chance of Beta error—the risk of assessing a control as effective when, in fact, it is not, because sample size and Beta error are inversely related As another example, if you want to increase the ADR, then you can decrease the sample size But in doing so, you increase the chance of Beta error The reason should be clear A larger ADR offers a less effective F.2  Nonstatistical Estimations control, or a less accurate sample of a good control But if you want to tighten the system assessment by decreasing the ADR, then a larger sample population is needed, again to err on the fail‐safe side As a final example, if you expect a low SDR, then you can use a lower sample size, understanding that the lesser accuracy is unimportant for a very good system But there, again, the chance of Beta error increases On the other hand, if you expect a high system error rate, possibly above the acceptance rate, then you need a larger sample size to be sure of the margins These considerations require judgment, experience, and knowledge of the system, but they enhance the validity of a nonstatistical plan F.1.7  Evaluating Sample Results Because nonstatistical sampling does not provide a reliable estimate of sampling risk, the analyst must make a judgment of whether the difference between the ADR and the measured deviation rate is an adequate allowance for sampling error For example, suppose the analyst expects a SDR of about 1% and will accept a deviation rate (ADR) of 7% From Table E.1, a sample of 83 is taken Suppose further that in this sample, two errors are found Then the measured deviation rate is 2/83 = 2.4% The analyst must then decide whether the 4.6% difference between the measured rate and the ADR is sufficient to cover sampling errors Here is where the more you know about the process the better will be your judgment A sample of 2/83 is about as likely to derive from a distribution centered at 5% as from a distribution centered at 1% In this case, the measured deviation rate (2.4%) was more than double the expected rate (1%), so a test of the 95% confidence level is called for A BETAINV test of 2/83 shows the result to have a 95% confidence that the true SDR is less than 7.4% (Apostolou, 1991c) As this limit is just over the ADR, the control should be deemed effective in order to avoid an unwinnable challenge in litigation Thus, the bounds and confidence levels of a test of controls are critical and must be justified, with generous allowance made for uncertainty For example, if the confidence level were set at 90%, the upper limit of the SDR at 2/83 would have been 6.8%, thus below the ADR If the expected SDR were 1.5% instead of 1%, the upper limit of the SDR at 2/109 would have been 5.7%, well below the ADR F.2 ­Nonstatistical Estimations Sampling conclusions are always estimations, irrespective of the type of sampling plan that is used Given that caution, statistical sampling plans offer a means to estimate the errors in comparing the sample to the population Nonstatistical sampling plans cannot estimate measurement errors and require 327 328 Appendix F  Nonstatistical Sampling Plans extraordinary understanding of the processes being monitored and larger margins between ADRs and estimated deviation rates Nevertheless, numerical results from nonstatistical sampling are useful when framed to answer the right questions: (i) “What is the highest SDR that is likely to yield the sample?” (ii) “What is the smallest Beta risk that is likely, given the sample?” Still, forensic systems engineering is about preparing engineering data for trial All the conclusions based on the analysis of evidence can be subject to challenge and rebuttal Therefore, when using a nonstatistical sampling plan the benefit of doubt in close margins of deviation rates goes to the defense Chapter 17 addresses this dilemma and recommends a forensic focus on systemic failure when statistical conclusions derived from the evidence contain controversial margins in probable sampling errors ­References Apostolou, B and Alleman, F (1991a) Internal Audit Sampling Altamonte Springs, FL: The Institute of Internal Auditors, p 11 Apostolou (1991b), p Apostolou (1991c), p 60 Grant, E L and Leavenworth, R S (1988) Statistical Quality Control New York: McGraw‐Hill, p 201 Guy, D M (1981a) Introduction to Statistical Sampling in Auditing New York: John Wiley & Sons, Inc., p Guy (1981b), p 140 Guy (1981c), p 46 Institute of Internal Auditors (2013) Practice advisory 2320‐3: Audit sampling, p. 2 https://www.iia.nl/SiteFiles/PA_2320‐3%20(1).pdf Accessed September 13, 2017 International Accounting Standard Board (2001) Framework for the Preparation and Presentation of Financial Statements, p 83 http://mca.gov.in/XBRL/pdf/ framework_fin_statements.pdf Accessed September 13, 2017 Kelly, J W (1986) How to Use Statistical Sampling in Your Audit Practice New York: Matthew Bender 329 Index All references are with respect to page numbers If the reference is a figure, the page number will be in italics If the reference is made to a table, the page number will be in bold print Three entries are giants in American industry: Henry Ford, Steve Jobs, and Dave Packard They are so listed in the index The standards: ANSI/ISO/ASQ 9000 (2005), 9001 (2015), and 9004 (2009) are fundamental to this book and appear frequently in many chapters Therefore, they are listed in this index only when they are used as a reference in the narrative a Abernathy, R B.  262 Abraham, B.  79, 160 Accelerated product testing see Reliability testing Acceptable deviation rate (ADR) see Sampling Acceptable performance level  292 Accountability, as a control  122 Advanced Product Quality Planning (APQP) 85 Alleman, F.  289, 290, 299, 306, 308, 321 Alpha error  295 (see also Type I error) risks related to  208 American Bar Association  63 ANAB (American National Accreditation Board)  27, 28 Anderson, E W.  185 ANSI (American National Standards Institute) 27 standards and strict liability  74 standards as good business practices 95 ANSI/ASQC (1987) ANSI/ASQC Q91‐1987 American National Standard: Quality systems—Model for Quality Assurance on Design/Development, Production Installation and Servicing  187 ANSI/ISO/ASQ (2009) ANSI/ISO/ASQ Q9004‐2009: Quality Management Systems—managing for Sustained Success of an Organization component of the set of standards ISO 9000 27 guidelines for performance excellence  recognition in duty of care  72 ANSI/ISO/ASQ (2005) ANSI/ISO/ASQ Q9000‐2005: Quality Management Systems—Fundamentals and Vocabulary background 26 first component of the set of standards ISO 9000  27 international recognition in court  23, 64, 72 performance standard  14 Forensic Systems Engineering: Evaluating Operations by Discovery, First Edition William A. Stimson © 2018 John Wiley & Sons, Inc Published 2018 by John Wiley & Sons, Inc 330 Index ANSI/ISO/ASQ (2015) ASQ/ANSI/ISO 9001‐2015: Quality Management systems—Requirements compared to other similar standards  26, 40 component of the set of standards ISO 9000 27 contractual 28 design  57, 94, 247 intended use  72, 94 outsource controls  59, 216 Apostolou, B.  289, 290, 299, 306, 308, 321 Arbib, M.A.  Aris, A.  165 Arnold, K.L.  223 Arora, J.S.  57, 115, 247 Arter, D.  231 AS9100 aerospace standard  169 AS9102 aerospace standard  23, 80, 193 ASQ (American Society for Quality)  27, 85 permission and credit  44 Assessment, elements of  238 Attribute, measure of quality  153 sampling  202–209, 291 Authority, as a control  118 Autocorrelation coefficient  140 Autocovariance generating function (AGF) 79 Automotive Industry Action Group  22 Autoregressive process (AR) model  170 nonstationarity 171 Average, mean value  274 median and mode  275 b Baird, H.  162 Bass, L.  150 Bathtub model of reliability  257 Bayesian estimate of product life  98 Becker, D.V.  150, 157 Bensinger, K.  51 Berk, J.  150 Bernoulli process  298 mutually exclusive events  322 Beta distribution  285 highest likely value  312 inverse 285 Beta error  296 (see also Type II error) relationship to SDR  311 Bhote, K.R.  84 Binomial distribution  280 defining equation  314 Blum, B.  177 Body armor protection, Type II  224 Boehm, B.W.  58 Boehm, T C.  51, 71, 72, 102, 150 Bower, J.L.  116 Bowerman, B.L.  79 Box, G.E.P.  79, 80, 140, 155, 167, 171, 292 Boyer, K E.  176 B‐percentile see Weibull analysis Bradford Hill criteria  164 Broomfield, J.R.  63, 95 Brumm, E.K.  245 c Calabrese, A.  185 Carrano, J.  160 Case International Harvester litigation  73 Cause and effect  163 transition 147 Ceglarek, D.  177 Central tendency of measurements  274 Challenger spacecraft  104 Chamberland, J F.  175 Characteristic equation  114 Charki, A.  175 Chesterton, G.K.  107 Christofol, H.  175 Cianfrani, C.  36 Claes, F.  194 Class action lawsuits  162 Close, C.M.  88, 121 Closed loop control system  108 CobIT  14, 39, 62 compared to ISO 9001  40 Cohn, M E., superior court judge  126 Committee of Sponsoring Organizations of the Treadway commission (COSO)  33, 108 Index definition of internal control  39, 170 framework 34 Common and special causes  72 as disturbances  163 Compliance xxiv Confidence, substantive  33 coefficient 284 finding an existing defect  314 interval 300 level  208, 284 and sample size  325 Conformance xxv Consumer and Producer risks  208, 294 (see also alpha and beta risks) Contract 7 content 10 contracting 10, 11 review 12 Control abandoned 61 change  94 charting 192 corporate activity  34 corporate environment  34 design (see Design and Development) documentation  31, 243 external provisions  95 financial 38 “in control”  82, 192 internal (see Internal control) limits 81 objectives for Sarbanes–Oxley  14 operations  30 pollution 48 process (see Process control) quality  58, 181 supplier (see supplier control) verification and validation  57 Control law  66 Control risk  207 control risk defined  294 correlation 154 Controllability  2, xxiv Cook, R.I.  150 Corbett, M.F.  214 Corporate Fraud Task Force  56 Correlated defect rate  142 Cost function  122 Cost of quality  43 Costa, O.L.V.  166 Cox, J.  23, 61 Cpk, capability index  84 industrial average  84 PPAP requirement  85 Crosby, Philip  183 Customer property  219 d Dahlberg, J.E.  64 Damages, in tort law  47 and compliance to ISO 9001  73 and false claims  56, 67, 156 liability in contract requirements  150 and process liability  50 and rare event  86 and reliability  100, 103 and systemic failure  206 Daniels Fund Ethics Initiative  51 Datta, A.  175 Dave Packard  184 Davis, J.  150 Dawson, C.  162 Declaration of Independence  33 Deming, W.E., and employee empowerment 65 and fear  62 management responsibility  17, 51, 163 process stability  72, 163 productivity  183, 189 Demri, A.  167 Dependence, serial  140 causal correlation  140–141 non‐causal correlation  147 of process, control and serial  153 properties of  142–146 and randomness  297 Derivative control see Internal control DeRusso, P.M.  79, 111 Design and development  248 design process  248 interactive 249 intermediate testing  249–251 331 332 Index Deviation see Process deviation Dietrich, F H.  192, 207, 274, 310 Differential and difference equations  111 Discovery 149 Dispersions, variance and range  276 Distributions continuous 278 correlated attributes  144 discrete 279 location 274 shape 278 Distributions: Disturbances 163 and special cause  110 Dlugopolski, T.  44, 184 Documentation 239 controls 243 Domain knowledge and causes  167 Dougherty, E R.  175 Drucker, P.  213, 214 Dryer, N.A.  164, 166 Due diligence  103 Durability 255 Dysfunction 162 and off line design  167 and open loop processes  190 and process liability  and response time  121 e Eagle Group  43 Eaton corporation  84 Eckner, A.  165, 167 Elsayed, E.A.  106, 124, 177, 317 Employee empowerment  65 Engineering, forensic systems  adaption to legal strategy  216 analysis of processes  147, 240 and decision making  236 and legal strategy  149 operations performance  186, 199 purpose 3 Escapes  238, 292 f Failure graphing  266, 267, 268 Failure rate of product  256 Falb, P.L.  False alarms  82, 83, 170 False claims  56 in advertising  225 and liability  44, 55 and malfeasance  58, 72 and negligence  56 and reliability  92, 93 False Claims Act  56, 156 Faryabi, B.  166 Federal acquisition regulations (FAR)  60, 64, 95, 96, 103, and ISO 9001 Feigenbaum, A.V.  183, 189 Feldman, R M.  166 Findings 237 First article inspection  193 First pass yield  192 Fixed size attribute sampling  203, 204 choosing 209 strategy 306 Flow down  61 and outsourcing  212 path of  218 FMEA (failure mode and effects analysis)  96, 263 Forced production  61 Ford/Firestone litigation  51 reliability 103 Ford, Henry  184 Forensic inquiry into design  252 Forensic science  157 National Research Council  150 and serial dependence  146 157 Forensic systems engineering see Engineering, forensic systems Form 33 Fragoso, M D.  175 Fraud 56 allegations in payment  56 malfeasant supply systems  226 and operations  55 and product reliability  92 and professional skepticism  210 undetected by auditors  62 Freelance 23 Fuerman, R.D.  203 Index g Garrett, B.L.  158 Garvin, D.A.  162, 181 GATT (General Agreement on Tariffs and Trade) 26 General Electric Fanuc  20 General ledger  41 and ghost inventory  61 Ghost inventory  60 Gibson, G A.  266 Gibson, J.E.  2, 118, 201 Gibson‐Dunn Law Firm  162 Gilchrist, W.  80 Goldratt, E.  21, 23, 61 Gooden, R.  19, 150 Gould, J.B.  158 Grandfathering suppliers  218 Grant, E.L.  82, 137, 141, 168, 198, 256, 290, 292, 317 Greenman v Yuba Power Products, Inc. 48 Guerin, F.  175 Guidance documents  28 Guy, D M.  290, 295, 321, 326 h Hall, A.D.  143 Harry, M.J.  85 Hashim, M.  180 Hausman, D M.  166 Hayes, R H.  181, 183, 184 Hendricks, K.  37 Hendricks, R.C.  106, 266 Hernandez, C.  158 Hidden causes  164 Hillier, F.S.  Hitchcock, C.  165 Hoerl, R.  20 Höfler, M.  166 Hogg, R V.  276 Homogeneity 199 and attributes  204, 307 defined 290 Homogeneous nonstationarity  79 Honoré, A.  165 Hoye, R.W.  214 Hsiang, T.  106, 124, 177, 317 Hume, D.  163 Hunter, J.S.  140, 155 Hunter, W.G.  140, 155 Hybert, P.R.  10 Hypergeometric distribution  298 Hypotheses tests  281–284 i Illari, P M.  166 Imai, Masaaki  61, 168, 184 Infant mortality  257 beta value  259 causes 258 control objectives (see CobIT) and financial disclosures  39 Information technology  11 Information Technology Governance Institute (ITGI)  33 model of performance  40 Inspection 187 as a control of stability  115 cost of quality  43 history of  180 inadequate process  51, 61 ineffective 165 of supplies  60, 95 as a work station  142 Institute of Internal Auditors  201, 321 Integral control` see Internal control Intended use  57, 94 dysfunctional 163 and reliability  72, 95 and suppliers  219 and validation  187 Internal control  109 and COSO  108, 170 derivative control  121 integral control  120 and ISO 9000  131, 152 major nonconformity  230 PID (proportional, integral, derivative) 116 rate control  121 responses, natural and transient  111 sampling of  199 system response  116 International Automotive Task Force  23 333 334 Index International Federation of Accountants 210 International Standard of Auditing (ISA) 294 International Standards Accounting Board (IASB)  41, 324 Interventions 163 Inventory turnover rate  120 Ishikawa (cause and effect) diagram  264 ISO (International Organization for Standardization) 26 ISO 9000  27 adapted as auto industry standard  23 definition of organization  66 history 26 as performance standard  14 set of good business practices  72 as standard for the book  structure 27 ISO 9001  27 communications 61 and controversy  25, 30 core requirements  30 and FAR  101 and good design  94 intended use  72, 94 and internal controls  64, 66 as performance standard  33 and planning phases  57 and reliability  97 and service industries  86 similarity to CobIT  40 in the U.S.  27 ISO 9004 acceptance in litigation  63 in duty of care  72 as guidance document  28, 64 ISO 9004  27 j Jank, W.  174 Jenkins, G.M.  79, 80, 167, 171, 292 Job description see Work instruction Jobs, Steve  184 Judgment in control risk  324 Juran, J.M.  17, 163, 201 k Kalman, R.E.  1, 77, 169 Kaner, C.  42 Karan, M.  166 Kaya, C Y.  176 Keller, P A.  189 Kelly, J W.  325 Kelton, W D.  285 Key characteristic  80 Khan, M.  180 King, R.  247 Kolka, J.W.  28 due diligence and negligence  64, 72 good business practices  73 ISO 9000  31 relevant evidence  95 Kruger, R.N.  245 Kurasawa, A.  232 Kuszewski, J.  41 l Labor productivity  182, 183 Lach, A.  214 Lahti, B.P.  110 111 Lamprecht, J.L.  191 LaPlace transform  112 Law, A M.  285 Lean 21 misreading of policies  91, 115 nonvalue adding  189 and Six Sigma  48 Toyota Production System  22 Leavenworth, R.S.  82, 137, 141, 168, 198, 256, 290, 292, 317 Ledolter, J.  79 Lee, J.H.  166 Leo, R.  160 Lester, J C.  176 Li, J and Shi, J.  166 Liability 47 cost of  156, 157 exposure to  73 reduction of  152, 221 risk management  64 strict liability  48, 74 Index Liability, process  47 and control effectiveness  170 and false claims  56 and law  52, 71 limit to  85 and misfeasance  51–52 and operations  50–51, 55 and systemic failure  72, 154 theory of  48–50 Liability, product  47–48 and class action  162 reduction of  150 Liebeck, Stella v McDonald’s Restaurants, Inc. 48 Lieberman, G.J.  Life testing  269 Liker, J.K.  22, 189 Livny, M.  146 Lochner, R H.  256 Lucas, R.M.  166 Luenberger, D.G.  113, 167 Lütkepohl, H.  79, 80 m MacKay, J.  160 Madison, J.  190 Malcolm Baldrige National Quality Award (MBNQA) 19 performance excellence program  20 Malfeasance 57–58 indifference and negligence  56 relation to nonconformity  239 substandard parts  60 supply systems  226 Management misfeasance see misfeasance Management responsibility  29, 62, 72 control environment  34 damages 48 employee empowerment  65 internal controls  62 misfeasance 51 process stability  66, 72, 77 product quality and reliability  51 reviews of operations  65 risk management  64 Sarbanes Oxley  38, 56 standards of care  63 system problems  17 Management standards  18 attributes 18–19 Management systems  19 control of costs  38 defined 3 ISO 9000  25–26, 29 Manicas, P.T.  245 Marginal stability  80 Marques, R P.  175 Masanao, A.  79 Mastrangelo, C.M.  146, 168 Matar, J E.  256 Materiality 41 consideration in sampling  324 liability 43 and quality  42 rule of thumb  41 Maximum likelihood estimator  299 McClave, J T.  192, 207, 274, 310 McLinn, J.  259 McMichael, A.J.  166 Mean value of correlated attributes  145 Measurement  29 of attributes  87 of capability  83–85 and confidence level  208 of conformity  58 cost of quality  38, 43 defined 191 in forced production  61 ISO 9001 requirement  30 legal issues in reliability  97–100 as objective evidence  236 purpose 100 statistical process control  81–83 Melamed, B.  146 Methods of sample selection  307 Metro Machine Shipyard  66 Military Standard 662F, armor ballistic testing 224 Miller, L.A.  4, 167 theory applied to operations  50–51, 150 theory of process liability  48–50, 85 Mills, C.A.  153, 205, 290, 292, 294, 307, 314 335 336 Index Mills, D.  43 Mil‐Q‐45208 inspection system requirements 26 Mil‐Q‐9858 quality program requirements  26, 65 Minka, T P.  177 Misfeasance 57 allegation of  92 reduction with QMS  72 and reliability  92 and sampling  201 substandard parts  60 and systemic failure  209 Mitchell, C M.  166 Modern quality assurance  181 Monitor and measure  188 MTBF, MTTF  256 description 265 Murphy, G.J.  113, 114, 116 Musk, E.  104 n National Institute of Standards and Technology 257 National Research Council  150 National Tooling and Machining Association 190 National Transportation Safety Board  234 Naval Sea Systems Command  Naval Surface Warfare Center, Cardrock MD 27 Nave, D.  22 Net income  184 Nonconformity  xxv, 229 causal consideration  235 classes of nonconformity  152 correlation of control and unit nonconformity 154–156 description 238–240 as a deviation  199 as a finding  237 identifying nonconformity  231–234 industrial by‐product  167 large scale class action  71 and misfeasance  75, 92 QMS requirements  30 rare event  86 reporting of  232 stable and unstable systems  85 sustained 210 Nonstatistical sampling  202 admissible and viable  201, 288, 321 confidence level  326 jury acceptance  203 sampling format  322 sampling risk  209, 325 subjective judgment  288 Normal distribution  277–278 o Objective evidence  236 Observability  2, xxiv, 162 Observation 140 analysis  236 considerations 235 potential nonconformity  231 O’Connell, R.T.  79 Office of management and budget (OMB)  15, 152 Offshore outsourcing  214 Okes, D.  231 Olson, W.  178 100 percent inspection  151, 204 effectiveness  201, 316 necessity 317 Open loop control system  67, 114 instability  115, 190 lack of control  114, 213 transfer function  113 Oppenheim, B.  20 Ordered statistics  284 Outsourcing 213 advantages 214 contrast with supply chain management 215 core competencies  214, 217 flow down  61 offshore outsourcing  214 Owen, D.G.  48, 87 p Packard, V.  181 Paperless documentation  240 Pearl, J.  164, 167 Index Philipps, K.  178 Picara, R W.  177 PID (proportional, integral, derivative) control see internal control Pine, M.  214 Plaisant, C.  174 Planned obsolescence  181 Points of inflection  278 Population 288 discovery 198 heterogeneous  276, 290, 294 parent 288 sample 289 size 290 strata 153 Powell, J H.  181, 182 Power and corruption  64 Power of a test  296 Probability 273 distributions (see Distributions) models 277 nonstatistical sampling  324 rare events  82 Probability of failure on demand (PDF)  256 Process 1–2 deviation 289 equilibrium state  78 responses, natural and transient (see Internal control) Process approach  28 Process capability  83 Process control  81, 143 Process deviation  83, 199, 207, 209, 289, 292 Process liability see Liability, process Product reliability  91 Production Part Approval Process (PPAP)  22, 85 Productivity  182, 183 and cost  184, 189 and GDP  181 and quality  185 Professional judgment and skepticism  210 Przasnyski, Z.  20 Puik, E.  167 Purchasing  212–213 Pyzdek, T.  189 q Qi, Y.  167 Quality  180, 255, xxv critical characteristics  13, 80 materiality (see Materiality) and productivity  182 and reliability  95, 97 stability 81 supplier 221 Quality Management System  aerospace AS 9100  23 automotive IATF 16949  23 ISO 9000  14, 25 in law  151 sufficient measures  235 supplier QMS  212, 217 Quality manual  241–242 Qui Tam  65 r Rare event  82, 147, 169 identifying 86 plea 31 and process liability  50 Rashomon effect  232 Rate control see Internal control Reachability xxiv Record (documentation)  244–245 Regrade of product  205 Reiter, J P.  166 Reliability  91, 256 dead on arrival  258 design 261–265 failure rate  237, 266 legal issues in design  94–97 legal issues in measuring  97–100 legal issues in testing  100–101 measuring 265–268 misfeasance 92–93 MTBF 256 MTTF 256 noncontractual 102–104 Probability of failure on demand (PFD) 356 testing 269–271 and warranty  104–105 337 338 Index Responsibility, as a control  122 Risk  207, 294 alpha and beta  208 assessment  40 consumer risk  294 control  207, 294 corrective action response  156 correlation to nonconformity  156, 231 and disturbance  171 estimation 154–156 and ISO 9000  30, 35 liability in negligence  51 liability management  64 management 96 management options for consumer and producer  38, 105 and process dysfunction  121, 174 producer risk  295 QMS subsystems  155 sampling  207, 209, 294 and specifications  13, 172 weakest link principle  154 Robust process  110 Root cause  148 cause and effect  164 and complex systems  150, 215 and duration of instability  154 and forensic analysis  3, 198, 222 and system level approach  Ross, S.M.  78 Roy, R.J.  88, 121 Russell, J.P.  150, 156, 236, 275 Russo, F.  176 Rust, R T.  194 s Sampling 288 acceptable deviation rate (ADR)  292, 313 alpha and beta errors  295–296 attribute  204–209, 291–292 blocking  290 (see also stratified sampling) confidence interval  300 deviation 289 distributions 298 estimating the SDR  299 inference 297 noise 289 population size versus sample size  309 populations 289 relation of beta error and deviation rates 311 risks (see Risk, sampling) size  299, 310, 314 stratified 200 system deviation rate (SDR)  293 Sampling plans, nonstatistical  201, 202, 321 estimation of likeliest SDR  327 relation of ADR to confidence level  326 relation of beta error to sample size  326 sample selection  324 Sampling plans, statistical  201, 202, 305 fixed size  201, 306–312 relation of SDR to confidence level  302 sample sizes  315 stop or go  201, 313–316 Sanborn, P.  150, 157 Sarbanes–Oxley  56, 150 title III, corporate responsibility  38 title IV, enhanced financial disclosures 39 title IX, white collar crime penalty enhancements 32 Savant, C.J.  124 Schorn, T.J.  317 Schroeder, B.  288 Schubert, D.  162 Schultheiss, P.M.  116 Scott, W.  138 Securities and Exchange Commission  109 Self assessment  23 Self release supplier  59 Serial dependence  140 Set point  113 Shewhart, W.A.  85, 161 control chart  81, 82 early research  26 economic control of quality of manufactured product  80 process in control  82 stable system of chance causes  80, 86, 139 statistical process control  81 Index Shi, P.  178 Shifted process  296–297, 310 Shmueli, G.  174 Shneiderman, B.  174 Siljak, D.D.  78, 111, 118 Sims, R.R.  48 Singhal, V.  37 Six Sigma  20–21, 85, 161, 247 Soditus, S.M.  106 Specification limits  12, 83 Spencer, M.P.  48 Stability 77 conditions 79 correlation model  171 correlation with conformity  169 dysfunction 171–173 measuring with control chart  82, 169 and reliability  264, 265 and stationarity  78 Stable failure mode  98 Standard of care  63 Standard Specifications for Ship Repair and Alteration Committee  15 Standards 14 admissibility in court  95 guidance 72 management 18 performance  14, 33 protective value  48 voluntary 10 Stationarity (weak sense)  78 condition 79 equilibrium 80 Steady state error  116 Steiner, S.  155 Steyvers, M.  166 Stratified sampling  200, 290 Substance 33 Superposition 110 Supplier control  59 principles 216 Supply chain management  215 Sutherland, J.  178 System 2 closed loop control  67 deviation rate  207, 293 equivalence to process  xxiv inherent system noise  72, 289 Kalman definition  nonconformity 231 reduced capability  155 stability 77 supply  216, 226 t Taguchi, G.  93, 115, 167, 317 Tanis, E A.  276 Telgen, D.  177 Tenenbaum, J B.  177 Thomson, W.  179 Tooling 190 Tort law  149 applications 102 and manufacturing  49 product liability  150 types of injuries  47 Total quality management  20 Toyota  51, 162 Toyota Production system  161 task values  22, 189 Traceability 223 Tracking parts  218 Transfer function  113 Tsay, R S.  166 Tsiakais, J.  36 Tsiolis, A.  146 Type I error see alpha error Type II ballistic testing  224, 225 muzzle velocity  225 Type II error  209, 283, 288, 295, 313 (see also Beta error) u Ulmer, J.M.  51, 71, 72, 102, 150 U.S Code 31: Money and Finance  56, 156 U.S Congress  38, 108, 169 U.S Department of Justice (DOJ)  32, 92 civil division  56, 162 U.S District Court, Eastern District of Arkansas  73, 95 U.S District Court, Eastern District of Pennsylvania 65 US Eighth Circuit Court of Appeals  73, 74 339 340 Index U.S Naval Sea Systems Command (NAVSEA) 9 U.S Supreme Court  73 v Vadrevu, S.  167, 172 Vahedi, G.  175 Valdez‐Flores, C.  166 Valujet aircraft crash  234 Van Moergestel, L.  177 Vartabedian, R.  51 Verification and validation  186 change of processes  186 control activity  34 design 94 negative correlation to productivity 185 non‐standard design  57 non‐value adding activity  22, 174, 185 open loop  66, 115, 190 paper trail  96, 97 risk factor  240 test waiver  58 Vincins, R.A.  152, 170 Vitasek, K.  213 w Wagenmakers, E J.  177 Warranty 104–106 field data for reliability  256, 269 operational limits  265 Web Finance, Inc.  103 Webster, J.G.  Weibull, W.  259 Weibull analysis  97, 98, 259 B‐percentile 262 distribution 99, 260 graph of failure data  267 scale 262 shape 259 slope 267 West, J.  28 Wheeler, D J.  301 Wheelwright, S C.  181, 183, 184 Whistleblowers see Qui tam Wiener, J.L.  104 Williamson, J.  176 Wilson point estimate  301 Wong, W.C.  166 Woodard, J.  166 Work instruction  242, 243 compared to job instruction  242 Work station  142 Wright, G.  181 y Yielded cost  157 Young, J.  160 Young, R B.  266 z Zaretsky, E.V.  98, 266 ... friend, my life ix Contents Preface  xix What Is Forensic Systems Engineering?   1.1 Systems and Systems Engineering 1.2 Forensic Systems Engineering References  Contracts, Specifications,... Nonlinear Systems: Parameter Analysis and Design New York: John Wiley & Sons, Inc., pp 445–446 xxvii 1 What Is Forensic Systems Engineering? CHAPTER MENU 1.1  Systems and Systems Engineering,   1.2  Forensic. .. and Systems Engineering,   1.2  Forensic Systems Engineering,   References, 4 Forensic systems engineering can be defined as the preparation of systems engineering data for trial This snapshot