Forensic Systems Engineering Wiley Series in Systems Engineering and Management William Rouse, Editor ALPHONSE CHAPANIS Andrew P Sage, Founding Editor Human Factors in Systems Engineering ANDREW P SAGE and JAMES D PALMER YACOV Y HAIMES Software Systems Engineering WILLIAM B ROUSE Risk Modeling, Assessment, and Management, Third Edition Design for Success: A Human‐Centered Approach to Designing Successful Products and Systems DENNIS M SUEDE LEONARD ADELMAN ANDREW P SAGE and JAMES E ARMSTRONG, Jr Evaluating Decision Support and Expert System Technology The Engineering Design of Systems: Models and Methods, Second Edition Introduction to Systems Engineering WILLIAM B ROUSE ANDREW P SAGE Decision Support Systems Engineering Essential Challenges of Strategic Management YEFIM FASSER and DONALD BRETINER YEFIM FASSER and DONALD BRETTNER Process Improvement in the Electronics Industry, Second Edition WILLIAM B ROUSE Strategies for Innovation ANDREW P SAGE Systems Engineering HORST TEMPELMEIER and HEINRICH KUHN Flexible Manufacturing Systems: Decision Support for Design and Operation WILLIAM B ROUSE Catalysts for Change: Concepts and Principles for Enabling Innovation UPING FANG, KEITH W HIPEL, and D MARC KILGOUR Interactive Decision Making: The Graph Model for Conflict Resolution DAVID A SCHUM Evidential Foundations of Probabilistic Reasoning JENS RASMUSSEN, ANNELISE MARK PEJTERSEN, and LEONARD P GOODSTEIN Cognitive Systems Engineering ANDREW P SAGE Systems Management for Information Technology and Software Engineering Management for Quality in High‐ Technology Enterprises THOMAS B SHERIDAN Humans and Automation: System Design and Research Issues ALEXANDER KOSSIAKOFF and WILLIAM N SWEET Systems Engineering Principles and Practice HAROLD R BOOHER Handbook of Human Systems Integration JEFFREY T POLLOCK and RALPH HODGSON Adaptive Information: Improving Business Through Semantic Interoperability, Grid Computing, and Enterprise Integration ALAN L PORTER and SCOTT W CUNNINGHAM Tech Mining: Exploiting New Technologies for Competitive Advantage REX BROWN Rational Choice and Judgment: Decision Analysis for the Decider WILLIAM B ROUSE and KENNETH R BOFF (Editors) Organizational Simulation HOWARD EISNER Managing Complex Systems: Thinking Outside the Box STEVE BELL Lean Enterprise Systems: Using IT for Continuous Improvement J JERRY KAUFMAN and ROY WOODHEAD YORAM KOREN The Global Manufacturing Revolution: Product‐Process‐Business Integration and Reconfigurable Systems Stimulating Innovation in Products and Services: With Function Analysis and Mapping AVNER ENGEL WILLIAM B ROUSE WILLIAM B ROUSE (Editor) Enterprise Transformation: Understanding and Enabling Fundamental Change JOHN E GIBSON, WILLIAM T SCHERER, and WILLAM F GIBSON How to Do Systems Analysis WILLIAM F CHRISTOPHER Holistic Management: Managing What Matters for Company Success WILLIAM B ROUSE People and Organizations: Explorations of Human‐Centered Design MOJAMSHIDI System of Systems Engineering: Innovations for the Twenty‐First Century ANDREW P SAGE and WILLIAM B ROUSE Handbook of Systems Engineering and Management, Second Edition JOHN R CLYMER Simulation‐Based Engineering of Complex Systems, Second Edition KRAG BROTBY Information Security Governance: A Practical Development and Implementation Approach JULIAN TALBOT and MILES JAKEMAN Security Risk Management Body of Knowledge SCOTT JACKSON Architecting Resilient Systems: Accident Avoidance and Survival and Recovery from Disruptions JAMES A GEORGE and JAMES A RODGER Smart Data: Enterprise Performance Optimization Strategy Verification, Validation, and Testing of Engineered Systems The Economics of Human Systems Integration: Valuation of Investments in People’s Training and Education, Safety and Health, and Work Productivity ALEXANDER KOSSIAKOFF, WILLIAM N SWEET, SAM SEYMOUR, and STEVEN M BIEMER Systems Engineering Principles and Practice, Second Edition GREGORY S PARNELL, PATRICK J DRISCOLL, and DALE L HENDERSON (Editors) Decision Making in Systems Engineering and Management, Second Edition ANDREW P SAGE and WILLIAM B ROUSE Economic Systems Analysis and Assessment: Intensive Systems, Organizations, and Enterprises BOHDAN W OPPENHEIM Lean for Systems Engineering with Lean Enablers for Systems Engineering LEV M KLYATIS Accelerated Reliability and Durability Testing Technology BJOERN BARTELS, ULRICH ERMEL, MICHAEL PECHT, and PETER SANDBORN Strategies to the Prediction, Mitigation, and Management of Product Obsolescence LEVANT YILMAS and TUNCER OREN Agent‐Directed Simulation and Systems Engineering ELSAYED A ELSAYED Reliability Engineering, Second Edition BEHNAM MALAKOOTI Operations and Production Systems with Multipme Objectives MENG‐LI SHIU, JUI‐CHIN JIANG, and MAO‐HSIUNG TU Quality Strategy for Systems Engineering and Management ANDREAS OPELT, BORIS GLOGER, WOLFGANG PFARL, and RALF MITTERMAYR Agile Contracts: Creating and Managing Successful Projects with Scrum KINJI MORI Concept‐Oriented Research and Development in Information Technology KAILASH C KAPUR and MICHAEL PECHT Reliability Engineering MICHAEL TORTORELLA Reliability, Maintainability, and Supportability: Best Practices for Systems Engineers DENNIS M BUEDE and WILLIAM D MILLER The Engineering Design of Systems: Models and Methods, Third Edition JOHN E GIBSON, WILLIAM T SCHERER, WILLIAM F GIBSON, and MICHAEL C SMITH How to Do Systems Analysis: Primer and Casebook GREGORY S PARNELL Trade‐off Analytics: Creating and Exploring the System Tradespace CHARLES S WASSON Systems Engineering Analysis, Design and Development Forensic Systems Engineering Evaluating Operations by Discovery William A Stimson This edition first published 2018 © 2018 John Wiley & Sons, Inc All rights reserved No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by law Advice on how to obtain permission to reuse material from this title is available at http://www.wiley.com/go/permissions The right of William A Stimson to be identified as the author of this work has been asserted in accordance with law Registered Offices John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, USA Editorial Office 111 River Street, Hoboken, NJ 07030, USA For details of our global editorial offices, customer services, and more information about Wiley products visit us at www.wiley.com Wiley also publishes its books in a variety of electronic formats and by print‐on‐demand Some content that appears in standard print versions of this book may not be available in other formats Limit of Liability/Disclaimer of Warranty The publisher and the authors make no representations or warranties with respect to the accuracy or completeness of the contents of this work and specifically disclaim all warranties; including without limitation any implied warranties of fitness for a particular purpose This work is sold with the understanding that the publisher is not engaged in rendering professional services The advice and strategies contained herein may not be suitable for every situation In view of on‐going research, equipment modifications, changes in governmental regulations, and the constant flow of information relating to the use of experimental reagents, equipment, and devices, the reader is urged to review and evaluate the information provided in the package insert or instructions for each chemical, piece of equipment, reagent, or device for, among other things, any changes in the instructions or indication of usage and for added warnings and precautions The fact that an organization or website is referred to in this work as a citation and/or potential source of further information does not mean that the author or the publisher endorses the information the organization or website may provide or recommendations it may make Further, readers should be aware that websites listed in this work may have changed or disappeared between when this works was written and when it is read No warranty may be created or extended by any promotional statements for this work Neither the publisher nor the author shall be liable for any damages arising here from Library of Congress Cataloging‐in‐Publication Data Names: Stimson, William A., author Title: Forensic systems engineering : evaluating operations by discovery / William A Stimson Description: Hoboken, NJ : Wiley, 2018 | Series: Wiley series in systems engineering and management | Includes bibliographical references and index | Identifiers: LCCN 2017039503 (print) | LCCN 2017042410 (ebook) | ISBN 9781119422761 (pdf ) | ISBN 9781119422785 (epub) | ISBN 9781119422754 (hardback) Subjects: LCSH: Failure analysis (Engineering) | System failures (Engineering) | Forensic sciences | Evidence, Expert | BISAC: TECHNOLOGY & ENGINEERING / Electronics / General Classification: LCC TA169.5 (ebook) | LCC TA169.5 S755 2018 (print) | DDC 620/.00452–dc23 LC record available at https://lccn.loc.gov/2017039503 Cover Design: Wiley Cover Image: © Digital Vision./Gettyimages Set in 10/12pt Warnock by SPi Global, Pondicherry, India Printed in the United States of America 10 9 8 7 6 5 4 3 2 1 To Josette, my love, my wife, my friend, my life ix Contents Preface xix What Is Forensic Systems Engineering? 1.1 Systems and Systems Engineering 1.2 Forensic Systems Engineering References Contracts, Specifications, and Standards 2.1 General 2.2 The Contract 2.2.1 Considerations 9 2.2.2 Contract Review 10 2.3 Specifications 12 2.4 Standards 14 Credits 16 References 16 Management Systems 17 3.1 Management Standards 18 3.1.1 Operations and Good Business Practices 18 3.1.2 Attributes of Management Standards 18 3.2 Effective Management Systems 19 3.2.1 Malcolm Baldrige 19 3.2.2 Total Quality Management 20 3.2.3 Six Sigma 20 3.2.4 Lean 21 3.2.5 Production Part Approval Process 22 3.3 Performance and Performance 23 3.4 Addendum 23 Credits 24 References 24 x Contents Performance Management: ISO 9001 25 4.1 Background of ISO 9000 26 4.1.1 ISO 9001 in the United States 27 4.1.2 Structure of ISO 9000:2005 27 4.1.3 The Process Approach 28 Form and Substance 32 4.2 4.2.1 Reference Performance Standards 33 4.2.2 Forensics and the Paper Trail 34 Credits 35 References 35 The Materiality of Operations 37 5.1 Rationale for Financial Metrics 38 5.1.1 Sarbanes–Oxley 38 5.1.1.1 Title III: Corporate Responsibility 38 5.1.1.2 Title IV: Enhanced Financial Disclosures 39 5.1.2 Internal Control 39 5.1.3 The Materiality of Quality 41 5.2 Mapping Operations to Finance 41 5.2.1 The Liability of Quality 43 5.2.2 The Forensic View 44 Credits 44 References 44 Process Liability 47 6.1 6.1.1 6.1.2 6.2 Theory of Process Liability 48 Operations and Process Liability 50 Process Liability and Misfeasance 51 Process Liability and the Law 52 Credits 52 References 52 Forensic Analysis of Process Liability 55 7.1 7.1.1 7.1.1.1 7.1.1.2 7.1.1.3 7.1.1.4 7.1.2 7.1.2.1 7.1.2.2 7.1.2.3 Improper Manufacturing Operations 57 Verification and Validation 57 Nonstandard Design Procedures 57 Unverified or Unvalidated Design 58 Tests Waived by Management 58 Altered Test Procedures and Results 58 Resource Management 59 Unmonitored Outsourcing 59 Substandard Purchased Parts 60 Ghost Inventory 60 Contents 7.1.2.4 7.1.3 7.1.3.1 7.1.3.2 7.2 7.2.1 7.2.2 7.2.3 7.2.4 7.2.5 7.2.6 Ineffective Flow Down 61 Process Management 61 Forced Production 61 Abuse and Threats by Management 62 Management Responsibility 62 Effective Internal Controls 62 Business Standards of Care 63 Liability Risk Management 64 Employee Empowerment 65 Effective Management Review 65 Closed‐Loop Processes 66 References 67 Legal Trends to Process Liability 71 8.1 8.2 8.2.1 8.2.2 9.1 9.1.1 9.1.2 9.1.3 9.1.4 9.2 9.2.1 9.2.2 9.3 9.3.1 9.3.2 9.4 10 10.1 10.2 10.2.1 10.2.2 10.2.3 10.2.4 10.2.5 An Idea Whose Time Has Come 71 Some Court Actions Thus Far 72 QMS Certified Organizations 73 QMS Noncertified Organizations 74 References 75 Process Stability and Capability 77 Process Stability 77 Stability and Stationarity 78 Stability Conditions 79 Stable Processes 80 Measuring Process Stability 82 Process Capability 83 Measuring Capability 83 A Limit of Process Capability 85 The Rare Event 85 Instability and the Rare Event 85 Identifying the Rare Event 86 Attribute Testing 87 References 88 Forensic Issues in Product Reliability 91 Background in Product Reliability 91 Legal Issues in the Design of Reliability 94 Good Design Practices 95 Design Is Intrinsic to Manufacturing and Service 95 Intended Use 95 Paper Trail of Evidence 96 Reliability Is an Implied Design Requirement 97 xi 326 Appendix F Nonstatistical Sampling Plans Table F.1 Selecting audit confidence level as a function of the ADR Acceptable deviation rate Assessment Kelly Guy Substantial confidence 2–7% ≤5% Moderate confidence 6–12% ≤10% Little confidence 11–20% ≤20% No confidence Omit test Omit test However, Guy (1981c) advises against using ADRs greater than 10% if the purpose of the inquiry is about the reliance of internal controls, as opposed to some other of their characteristics As reliance of a control is a major issue in systems operation, then 10% can be considered an upper limit for ADRs for purposes of forensic systems examination As in statistical sampling, in a nonstatistical plan you use knowledge of past system performance to estimate the control deviation rate in order to arrive at a sample size Then having chosen the sample size and conducted the test, you evaluate the control based on the difference between the sample and the ADRs F.1.6 The Effect of Sample Size on Beta Error Many analysts consider Beta error as the more grave sampling error, as Alpha error tends to be self‐correcting If an analyst makes an Alpha error, the decision will be challenged because the analyst is challenging a control that the performer believes to be good The performer will insist on additional testing with larger sample size However, a Beta error will not be challenged; the analyst reports good news to the performer, although the good news is false In addition, additional testing increases cost Apostolou (1991b) classifies an Alpha Error as an efficiency indicator; a Beta error as an effectiveness indicator Nonstatistical sampling plans not measure Beta error, but you know that it exists You must make some effort to reduce it, and sample size has much to with it Sample size is affected by the error rate as shown in Table E.2, in which changes in sample size affect key system parameters and conversely For example, suppose that you want to choose a smaller sample size By doing so, you increase the chance of Beta error—the risk of assessing a control as effective when, in fact, it is not, because sample size and Beta error are inversely related As another example, if you want to increase the ADR, then you can decrease the sample size But in doing so, you increase the chance of Beta error The reason should be clear A larger ADR offers a less effective F.2 Nonstatistical Estimations control, or a less accurate sample of a good control But if you want to tighten the system assessment by decreasing the ADR, then a larger sample population is needed, again to err on the fail‐safe side As a final example, if you expect a low SDR, then you can use a lower sample size, understanding that the lesser accuracy is unimportant for a very good system But there, again, the chance of Beta error increases On the other hand, if you expect a high system error rate, possibly above the acceptance rate, then you need a larger sample size to be sure of the margins These considerations require judgment, experience, and knowledge of the system, but they enhance the validity of a nonstatistical plan F.1.7 Evaluating Sample Results Because nonstatistical sampling does not provide a reliable estimate of sampling risk, the analyst must make a judgment of whether the difference between the ADR and the measured deviation rate is an adequate allowance for sampling error For example, suppose the analyst expects a SDR of about 1% and will accept a deviation rate (ADR) of 7% From Table E.1, a sample of 83 is taken Suppose further that in this sample, two errors are found Then the measured deviation rate is 2/83 = 2.4% The analyst must then decide whether the 4.6% difference between the measured rate and the ADR is sufficient to cover sampling errors Here is where the more you know about the process the better will be your judgment A sample of 2/83 is about as likely to derive from a distribution centered at 5% as from a distribution centered at 1% In this case, the measured deviation rate (2.4%) was more than double the expected rate (1%), so a test of the 95% confidence level is called for A BETAINV test of 2/83 shows the result to have a 95% confidence that the true SDR is less than 7.4% (Apostolou, 1991c) As this limit is just over the ADR, the control should be deemed effective in order to avoid an unwinnable challenge in litigation Thus, the bounds and confidence levels of a test of controls are critical and must be justified, with generous allowance made for uncertainty For example, if the confidence level were set at 90%, the upper limit of the SDR at 2/83 would have been 6.8%, thus below the ADR If the expected SDR were 1.5% instead of 1%, the upper limit of the SDR at 2/109 would have been 5.7%, well below the ADR F.2 Nonstatistical Estimations Sampling conclusions are always estimations, irrespective of the type of sampling plan that is used Given that caution, statistical sampling plans offer a means to estimate the errors in comparing the sample to the population Nonstatistical sampling plans cannot estimate measurement errors and require 327 328 Appendix F Nonstatistical Sampling Plans extraordinary understanding of the processes being monitored and larger margins between ADRs and estimated deviation rates Nevertheless, numerical results from nonstatistical sampling are useful when framed to answer the right questions: (i) “What is the highest SDR that is likely to yield the sample?” (ii) “What is the smallest Beta risk that is likely, given the sample?” Still, forensic systems engineering is about preparing engineering data for trial All the conclusions based on the analysis of evidence can be subject to challenge and rebuttal Therefore, when using a nonstatistical sampling plan the benefit of doubt in close margins of deviation rates goes to the defense Chapter 17 addresses this dilemma and recommends a forensic focus on systemic failure when statistical conclusions derived from the evidence contain controversial margins in probable sampling errors References Apostolou, B and Alleman, F (1991a) Internal Audit Sampling Altamonte Springs, FL: The Institute of Internal Auditors, p 11 Apostolou (1991b), p Apostolou (1991c), p 60 Grant, E L and Leavenworth, R S (1988) Statistical Quality Control New York: McGraw‐Hill, p 201 Guy, D M (1981a) Introduction to Statistical Sampling in Auditing New York: John Wiley & Sons, Inc., p Guy (1981b), p 140 Guy (1981c), p 46 Institute of Internal Auditors (2013) Practice advisory 2320‐3: Audit sampling, p. 2 https://www.iia.nl/SiteFiles/PA_2320‐3%20(1).pdf Accessed September 13, 2017 International Accounting Standard Board (2001) Framework for the Preparation and Presentation of Financial Statements, p 83 http://mca.gov.in/XBRL/pdf/ framework_fin_statements.pdf Accessed September 13, 2017 Kelly, J W (1986) How to Use Statistical Sampling in Your Audit Practice New York: Matthew Bender 329 Index All references are with respect to page numbers If the reference is a figure, the page number will be in italics If the reference is made to a table, the page number will be in bold print Three entries are giants in American industry: Henry Ford, Steve Jobs, and Dave Packard They are so listed in the index The standards: ANSI/ISO/ASQ 9000 (2005), 9001 (2015), and 9004 (2009) are fundamental to this book and appear frequently in many chapters Therefore, they are listed in this index only when they are used as a reference in the narrative a Abernathy, R B. 262 Abraham, B. 79, 160 Accelerated product testing see Reliability testing Acceptable deviation rate (ADR) see Sampling Acceptable performance level 292 Accountability, as a control 122 Advanced Product Quality Planning (APQP) 85 Alleman, F. 289, 290, 299, 306, 308, 321 Alpha error 295 (see also Type I error) risks related to 208 American Bar Association 63 ANAB (American National Accreditation Board) 27, 28 Anderson, E W. 185 ANSI (American National Standards Institute) 27 standards and strict liability 74 standards as good business practices 95 ANSI/ASQC (1987) ANSI/ASQC Q91‐1987 American National Standard: Quality systems—Model for Quality Assurance on Design/Development, Production Installation and Servicing 187 ANSI/ISO/ASQ (2009) ANSI/ISO/ASQ Q9004‐2009: Quality Management Systems—managing for Sustained Success of an Organization component of the set of standards ISO 9000 27 guidelines for performance excellence recognition in duty of care 72 ANSI/ISO/ASQ (2005) ANSI/ISO/ASQ Q9000‐2005: Quality Management Systems—Fundamentals and Vocabulary background 26 first component of the set of standards ISO 9000 27 international recognition in court 23, 64, 72 performance standard 14 Forensic Systems Engineering: Evaluating Operations by Discovery, First Edition William A. Stimson © 2018 John Wiley & Sons, Inc Published 2018 by John Wiley & Sons, Inc 330 Index ANSI/ISO/ASQ (2015) ASQ/ANSI/ISO 9001‐2015: Quality Management systems—Requirements compared to other similar standards 26, 40 component of the set of standards ISO 9000 27 contractual 28 design 57, 94, 247 intended use 72, 94 outsource controls 59, 216 Apostolou, B. 289, 290, 299, 306, 308, 321 Arbib, M.A. Aris, A. 165 Arnold, K.L. 223 Arora, J.S. 57, 115, 247 Arter, D. 231 AS9100 aerospace standard 169 AS9102 aerospace standard 23, 80, 193 ASQ (American Society for Quality) 27, 85 permission and credit 44 Assessment, elements of 238 Attribute, measure of quality 153 sampling 202–209, 291 Authority, as a control 118 Autocorrelation coefficient 140 Autocovariance generating function (AGF) 79 Automotive Industry Action Group 22 Autoregressive process (AR) model 170 nonstationarity 171 Average, mean value 274 median and mode 275 b Baird, H. 162 Bass, L. 150 Bathtub model of reliability 257 Bayesian estimate of product life 98 Becker, D.V. 150, 157 Bensinger, K. 51 Berk, J. 150 Bernoulli process 298 mutually exclusive events 322 Beta distribution 285 highest likely value 312 inverse 285 Beta error 296 (see also Type II error) relationship to SDR 311 Bhote, K.R. 84 Binomial distribution 280 defining equation 314 Blum, B. 177 Body armor protection, Type II 224 Boehm, B.W. 58 Boehm, T C. 51, 71, 72, 102, 150 Bower, J.L. 116 Bowerman, B.L. 79 Box, G.E.P. 79, 80, 140, 155, 167, 171, 292 Boyer, K E. 176 B‐percentile see Weibull analysis Bradford Hill criteria 164 Broomfield, J.R. 63, 95 Brumm, E.K. 245 c Calabrese, A. 185 Carrano, J. 160 Case International Harvester litigation 73 Cause and effect 163 transition 147 Ceglarek, D. 177 Central tendency of measurements 274 Challenger spacecraft 104 Chamberland, J F. 175 Characteristic equation 114 Charki, A. 175 Chesterton, G.K. 107 Christofol, H. 175 Cianfrani, C. 36 Claes, F. 194 Class action lawsuits 162 Close, C.M. 88, 121 Closed loop control system 108 CobIT 14, 39, 62 compared to ISO 9001 40 Cohn, M E., superior court judge 126 Committee of Sponsoring Organizations of the Treadway commission (COSO) 33, 108 Index definition of internal control 39, 170 framework 34 Common and special causes 72 as disturbances 163 Compliance xxiv Confidence, substantive 33 coefficient 284 finding an existing defect 314 interval 300 level 208, 284 and sample size 325 Conformance xxv Consumer and Producer risks 208, 294 (see also alpha and beta risks) Contract 7 content 10 contracting 10, 11 review 12 Control abandoned 61 change 94 charting 192 corporate activity 34 corporate environment 34 design (see Design and Development) documentation 31, 243 external provisions 95 financial 38 “in control” 82, 192 internal (see Internal control) limits 81 objectives for Sarbanes–Oxley 14 operations 30 pollution 48 process (see Process control) quality 58, 181 supplier (see supplier control) verification and validation 57 Control law 66 Control risk 207 control risk defined 294 correlation 154 Controllability 2, xxiv Cook, R.I. 150 Corbett, M.F. 214 Corporate Fraud Task Force 56 Correlated defect rate 142 Cost function 122 Cost of quality 43 Costa, O.L.V. 166 Cox, J. 23, 61 Cpk, capability index 84 industrial average 84 PPAP requirement 85 Crosby, Philip 183 Customer property 219 d Dahlberg, J.E. 64 Damages, in tort law 47 and compliance to ISO 9001 73 and false claims 56, 67, 156 liability in contract requirements 150 and process liability 50 and rare event 86 and reliability 100, 103 and systemic failure 206 Daniels Fund Ethics Initiative 51 Datta, A. 175 Dave Packard 184 Davis, J. 150 Dawson, C. 162 Declaration of Independence 33 Deming, W.E., and employee empowerment 65 and fear 62 management responsibility 17, 51, 163 process stability 72, 163 productivity 183, 189 Demri, A. 167 Dependence, serial 140 causal correlation 140–141 non‐causal correlation 147 of process, control and serial 153 properties of 142–146 and randomness 297 Derivative control see Internal control DeRusso, P.M. 79, 111 Design and development 248 design process 248 interactive 249 intermediate testing 249–251 331 332 Index Deviation see Process deviation Dietrich, F H. 192, 207, 274, 310 Differential and difference equations 111 Discovery 149 Dispersions, variance and range 276 Distributions continuous 278 correlated attributes 144 discrete 279 location 274 shape 278 Distributions: Disturbances 163 and special cause 110 Dlugopolski, T. 44, 184 Documentation 239 controls 243 Domain knowledge and causes 167 Dougherty, E R. 175 Drucker, P. 213, 214 Dryer, N.A. 164, 166 Due diligence 103 Durability 255 Dysfunction 162 and off line design 167 and open loop processes 190 and process liability and response time 121 e Eagle Group 43 Eaton corporation 84 Eckner, A. 165, 167 Elsayed, E.A. 106, 124, 177, 317 Employee empowerment 65 Engineering, forensic systems adaption to legal strategy 216 analysis of processes 147, 240 and decision making 236 and legal strategy 149 operations performance 186, 199 purpose 3 Escapes 238, 292 f Failure graphing 266, 267, 268 Failure rate of product 256 Falb, P.L. False alarms 82, 83, 170 False claims 56 in advertising 225 and liability 44, 55 and malfeasance 58, 72 and negligence 56 and reliability 92, 93 False Claims Act 56, 156 Faryabi, B. 166 Federal acquisition regulations (FAR) 60, 64, 95, 96, 103, and ISO 9001 Feigenbaum, A.V. 183, 189 Feldman, R M. 166 Findings 237 First article inspection 193 First pass yield 192 Fixed size attribute sampling 203, 204 choosing 209 strategy 306 Flow down 61 and outsourcing 212 path of 218 FMEA (failure mode and effects analysis) 96, 263 Forced production 61 Ford/Firestone litigation 51 reliability 103 Ford, Henry 184 Forensic inquiry into design 252 Forensic science 157 National Research Council 150 and serial dependence 146 157 Forensic systems engineering see Engineering, forensic systems Form 33 Fragoso, M D. 175 Fraud 56 allegations in payment 56 malfeasant supply systems 226 and operations 55 and product reliability 92 and professional skepticism 210 undetected by auditors 62 Freelance 23 Fuerman, R.D. 203 Index g Garrett, B.L. 158 Garvin, D.A. 162, 181 GATT (General Agreement on Tariffs and Trade) 26 General Electric Fanuc 20 General ledger 41 and ghost inventory 61 Ghost inventory 60 Gibson, G A. 266 Gibson, J.E. 2, 118, 201 Gibson‐Dunn Law Firm 162 Gilchrist, W. 80 Goldratt, E. 21, 23, 61 Gooden, R. 19, 150 Gould, J.B. 158 Grandfathering suppliers 218 Grant, E.L. 82, 137, 141, 168, 198, 256, 290, 292, 317 Greenman v Yuba Power Products, Inc. 48 Guerin, F. 175 Guidance documents 28 Guy, D M. 290, 295, 321, 326 h Hall, A.D. 143 Harry, M.J. 85 Hashim, M. 180 Hausman, D M. 166 Hayes, R H. 181, 183, 184 Hendricks, K. 37 Hendricks, R.C. 106, 266 Hernandez, C. 158 Hidden causes 164 Hillier, F.S. Hitchcock, C. 165 Hoerl, R. 20 Höfler, M. 166 Hogg, R V. 276 Homogeneity 199 and attributes 204, 307 defined 290 Homogeneous nonstationarity 79 Honoré, A. 165 Hoye, R.W. 214 Hsiang, T. 106, 124, 177, 317 Hume, D. 163 Hunter, J.S. 140, 155 Hunter, W.G. 140, 155 Hybert, P.R. 10 Hypergeometric distribution 298 Hypotheses tests 281–284 i Illari, P M. 166 Imai, Masaaki 61, 168, 184 Infant mortality 257 beta value 259 causes 258 control objectives (see CobIT) and financial disclosures 39 Information technology 11 Information Technology Governance Institute (ITGI) 33 model of performance 40 Inspection 187 as a control of stability 115 cost of quality 43 history of 180 inadequate process 51, 61 ineffective 165 of supplies 60, 95 as a work station 142 Institute of Internal Auditors 201, 321 Integral control` see Internal control Intended use 57, 94 dysfunctional 163 and reliability 72, 95 and suppliers 219 and validation 187 Internal control 109 and COSO 108, 170 derivative control 121 integral control 120 and ISO 9000 131, 152 major nonconformity 230 PID (proportional, integral, derivative) 116 rate control 121 responses, natural and transient 111 sampling of 199 system response 116 International Automotive Task Force 23 333 334 Index International Federation of Accountants 210 International Standard of Auditing (ISA) 294 International Standards Accounting Board (IASB) 41, 324 Interventions 163 Inventory turnover rate 120 Ishikawa (cause and effect) diagram 264 ISO (International Organization for Standardization) 26 ISO 9000 27 adapted as auto industry standard 23 definition of organization 66 history 26 as performance standard 14 set of good business practices 72 as standard for the book structure 27 ISO 9001 27 communications 61 and controversy 25, 30 core requirements 30 and FAR 101 and good design 94 intended use 72, 94 and internal controls 64, 66 as performance standard 33 and planning phases 57 and reliability 97 and service industries 86 similarity to CobIT 40 in the U.S. 27 ISO 9004 acceptance in litigation 63 in duty of care 72 as guidance document 28, 64 ISO 9004 27 j Jank, W. 174 Jenkins, G.M. 79, 80, 167, 171, 292 Job description see Work instruction Jobs, Steve 184 Judgment in control risk 324 Juran, J.M. 17, 163, 201 k Kalman, R.E. 1, 77, 169 Kaner, C. 42 Karan, M. 166 Kaya, C Y. 176 Keller, P A. 189 Kelly, J W. 325 Kelton, W D. 285 Key characteristic 80 Khan, M. 180 King, R. 247 Kolka, J.W. 28 due diligence and negligence 64, 72 good business practices 73 ISO 9000 31 relevant evidence 95 Kruger, R.N. 245 Kurasawa, A. 232 Kuszewski, J. 41 l Labor productivity 182, 183 Lach, A. 214 Lahti, B.P. 110 111 Lamprecht, J.L. 191 LaPlace transform 112 Law, A M. 285 Lean 21 misreading of policies 91, 115 nonvalue adding 189 and Six Sigma 48 Toyota Production System 22 Leavenworth, R.S. 82, 137, 141, 168, 198, 256, 290, 292, 317 Ledolter, J. 79 Lee, J.H. 166 Leo, R. 160 Lester, J C. 176 Li, J and Shi, J. 166 Liability 47 cost of 156, 157 exposure to 73 reduction of 152, 221 risk management 64 strict liability 48, 74 Index Liability, process 47 and control effectiveness 170 and false claims 56 and law 52, 71 limit to 85 and misfeasance 51–52 and operations 50–51, 55 and systemic failure 72, 154 theory of 48–50 Liability, product 47–48 and class action 162 reduction of 150 Liebeck, Stella v McDonald’s Restaurants, Inc. 48 Lieberman, G.J. Life testing 269 Liker, J.K. 22, 189 Livny, M. 146 Lochner, R H. 256 Lucas, R.M. 166 Luenberger, D.G. 113, 167 Lütkepohl, H. 79, 80 m MacKay, J. 160 Madison, J. 190 Malcolm Baldrige National Quality Award (MBNQA) 19 performance excellence program 20 Malfeasance 57–58 indifference and negligence 56 relation to nonconformity 239 substandard parts 60 supply systems 226 Management misfeasance see misfeasance Management responsibility 29, 62, 72 control environment 34 damages 48 employee empowerment 65 internal controls 62 misfeasance 51 process stability 66, 72, 77 product quality and reliability 51 reviews of operations 65 risk management 64 Sarbanes Oxley 38, 56 standards of care 63 system problems 17 Management standards 18 attributes 18–19 Management systems 19 control of costs 38 defined 3 ISO 9000 25–26, 29 Manicas, P.T. 245 Marginal stability 80 Marques, R P. 175 Masanao, A. 79 Mastrangelo, C.M. 146, 168 Matar, J E. 256 Materiality 41 consideration in sampling 324 liability 43 and quality 42 rule of thumb 41 Maximum likelihood estimator 299 McClave, J T. 192, 207, 274, 310 McLinn, J. 259 McMichael, A.J. 166 Mean value of correlated attributes 145 Measurement 29 of attributes 87 of capability 83–85 and confidence level 208 of conformity 58 cost of quality 38, 43 defined 191 in forced production 61 ISO 9001 requirement 30 legal issues in reliability 97–100 as objective evidence 236 purpose 100 statistical process control 81–83 Melamed, B. 146 Methods of sample selection 307 Metro Machine Shipyard 66 Military Standard 662F, armor ballistic testing 224 Miller, L.A. 4, 167 theory applied to operations 50–51, 150 theory of process liability 48–50, 85 Mills, C.A. 153, 205, 290, 292, 294, 307, 314 335 336 Index Mills, D. 43 Mil‐Q‐45208 inspection system requirements 26 Mil‐Q‐9858 quality program requirements 26, 65 Minka, T P. 177 Misfeasance 57 allegation of 92 reduction with QMS 72 and reliability 92 and sampling 201 substandard parts 60 and systemic failure 209 Mitchell, C M. 166 Modern quality assurance 181 Monitor and measure 188 MTBF, MTTF 256 description 265 Murphy, G.J. 113, 114, 116 Musk, E. 104 n National Institute of Standards and Technology 257 National Research Council 150 National Tooling and Machining Association 190 National Transportation Safety Board 234 Naval Sea Systems Command Naval Surface Warfare Center, Cardrock MD 27 Nave, D. 22 Net income 184 Nonconformity xxv, 229 causal consideration 235 classes of nonconformity 152 correlation of control and unit nonconformity 154–156 description 238–240 as a deviation 199 as a finding 237 identifying nonconformity 231–234 industrial by‐product 167 large scale class action 71 and misfeasance 75, 92 QMS requirements 30 rare event 86 reporting of 232 stable and unstable systems 85 sustained 210 Nonstatistical sampling 202 admissible and viable 201, 288, 321 confidence level 326 jury acceptance 203 sampling format 322 sampling risk 209, 325 subjective judgment 288 Normal distribution 277–278 o Objective evidence 236 Observability 2, xxiv, 162 Observation 140 analysis 236 considerations 235 potential nonconformity 231 O’Connell, R.T. 79 Office of management and budget (OMB) 15, 152 Offshore outsourcing 214 Okes, D. 231 Olson, W. 178 100 percent inspection 151, 204 effectiveness 201, 316 necessity 317 Open loop control system 67, 114 instability 115, 190 lack of control 114, 213 transfer function 113 Oppenheim, B. 20 Ordered statistics 284 Outsourcing 213 advantages 214 contrast with supply chain management 215 core competencies 214, 217 flow down 61 offshore outsourcing 214 Owen, D.G. 48, 87 p Packard, V. 181 Paperless documentation 240 Pearl, J. 164, 167 Index Philipps, K. 178 Picara, R W. 177 PID (proportional, integral, derivative) control see internal control Pine, M. 214 Plaisant, C. 174 Planned obsolescence 181 Points of inflection 278 Population 288 discovery 198 heterogeneous 276, 290, 294 parent 288 sample 289 size 290 strata 153 Powell, J H. 181, 182 Power and corruption 64 Power of a test 296 Probability 273 distributions (see Distributions) models 277 nonstatistical sampling 324 rare events 82 Probability of failure on demand (PDF) 256 Process 1–2 deviation 289 equilibrium state 78 responses, natural and transient (see Internal control) Process approach 28 Process capability 83 Process control 81, 143 Process deviation 83, 199, 207, 209, 289, 292 Process liability see Liability, process Product reliability 91 Production Part Approval Process (PPAP) 22, 85 Productivity 182, 183 and cost 184, 189 and GDP 181 and quality 185 Professional judgment and skepticism 210 Przasnyski, Z. 20 Puik, E. 167 Purchasing 212–213 Pyzdek, T. 189 q Qi, Y. 167 Quality 180, 255, xxv critical characteristics 13, 80 materiality (see Materiality) and productivity 182 and reliability 95, 97 stability 81 supplier 221 Quality Management System aerospace AS 9100 23 automotive IATF 16949 23 ISO 9000 14, 25 in law 151 sufficient measures 235 supplier QMS 212, 217 Quality manual 241–242 Qui Tam 65 r Rare event 82, 147, 169 identifying 86 plea 31 and process liability 50 Rashomon effect 232 Rate control see Internal control Reachability xxiv Record (documentation) 244–245 Regrade of product 205 Reiter, J P. 166 Reliability 91, 256 dead on arrival 258 design 261–265 failure rate 237, 266 legal issues in design 94–97 legal issues in measuring 97–100 legal issues in testing 100–101 measuring 265–268 misfeasance 92–93 MTBF 256 MTTF 256 noncontractual 102–104 Probability of failure on demand (PFD) 356 testing 269–271 and warranty 104–105 337 338 Index Responsibility, as a control 122 Risk 207, 294 alpha and beta 208 assessment 40 consumer risk 294 control 207, 294 corrective action response 156 correlation to nonconformity 156, 231 and disturbance 171 estimation 154–156 and ISO 9000 30, 35 liability in negligence 51 liability management 64 management 96 management options for consumer and producer 38, 105 and process dysfunction 121, 174 producer risk 295 QMS subsystems 155 sampling 207, 209, 294 and specifications 13, 172 weakest link principle 154 Robust process 110 Root cause 148 cause and effect 164 and complex systems 150, 215 and duration of instability 154 and forensic analysis 3, 198, 222 and system level approach Ross, S.M. 78 Roy, R.J. 88, 121 Russell, J.P. 150, 156, 236, 275 Russo, F. 176 Rust, R T. 194 s Sampling 288 acceptable deviation rate (ADR) 292, 313 alpha and beta errors 295–296 attribute 204–209, 291–292 blocking 290 (see also stratified sampling) confidence interval 300 deviation 289 distributions 298 estimating the SDR 299 inference 297 noise 289 population size versus sample size 309 populations 289 relation of beta error and deviation rates 311 risks (see Risk, sampling) size 299, 310, 314 stratified 200 system deviation rate (SDR) 293 Sampling plans, nonstatistical 201, 202, 321 estimation of likeliest SDR 327 relation of ADR to confidence level 326 relation of beta error to sample size 326 sample selection 324 Sampling plans, statistical 201, 202, 305 fixed size 201, 306–312 relation of SDR to confidence level 302 sample sizes 315 stop or go 201, 313–316 Sanborn, P. 150, 157 Sarbanes–Oxley 56, 150 title III, corporate responsibility 38 title IV, enhanced financial disclosures 39 title IX, white collar crime penalty enhancements 32 Savant, C.J. 124 Schorn, T.J. 317 Schroeder, B. 288 Schubert, D. 162 Schultheiss, P.M. 116 Scott, W. 138 Securities and Exchange Commission 109 Self assessment 23 Self release supplier 59 Serial dependence 140 Set point 113 Shewhart, W.A. 85, 161 control chart 81, 82 early research 26 economic control of quality of manufactured product 80 process in control 82 stable system of chance causes 80, 86, 139 statistical process control 81 Index Shi, P. 178 Shifted process 296–297, 310 Shmueli, G. 174 Shneiderman, B. 174 Siljak, D.D. 78, 111, 118 Sims, R.R. 48 Singhal, V. 37 Six Sigma 20–21, 85, 161, 247 Soditus, S.M. 106 Specification limits 12, 83 Spencer, M.P. 48 Stability 77 conditions 79 correlation model 171 correlation with conformity 169 dysfunction 171–173 measuring with control chart 82, 169 and reliability 264, 265 and stationarity 78 Stable failure mode 98 Standard of care 63 Standard Specifications for Ship Repair and Alteration Committee 15 Standards 14 admissibility in court 95 guidance 72 management 18 performance 14, 33 protective value 48 voluntary 10 Stationarity (weak sense) 78 condition 79 equilibrium 80 Steady state error 116 Steiner, S. 155 Steyvers, M. 166 Stratified sampling 200, 290 Substance 33 Superposition 110 Supplier control 59 principles 216 Supply chain management 215 Sutherland, J. 178 System 2 closed loop control 67 deviation rate 207, 293 equivalence to process xxiv inherent system noise 72, 289 Kalman definition nonconformity 231 reduced capability 155 stability 77 supply 216, 226 t Taguchi, G. 93, 115, 167, 317 Tanis, E A. 276 Telgen, D. 177 Tenenbaum, J B. 177 Thomson, W. 179 Tooling 190 Tort law 149 applications 102 and manufacturing 49 product liability 150 types of injuries 47 Total quality management 20 Toyota 51, 162 Toyota Production system 161 task values 22, 189 Traceability 223 Tracking parts 218 Transfer function 113 Tsay, R S. 166 Tsiakais, J. 36 Tsiolis, A. 146 Type I error see alpha error Type II ballistic testing 224, 225 muzzle velocity 225 Type II error 209, 283, 288, 295, 313 (see also Beta error) u Ulmer, J.M. 51, 71, 72, 102, 150 U.S Code 31: Money and Finance 56, 156 U.S Congress 38, 108, 169 U.S Department of Justice (DOJ) 32, 92 civil division 56, 162 U.S District Court, Eastern District of Arkansas 73, 95 U.S District Court, Eastern District of Pennsylvania 65 US Eighth Circuit Court of Appeals 73, 74 339 340 Index U.S Naval Sea Systems Command (NAVSEA) 9 U.S Supreme Court 73 v Vadrevu, S. 167, 172 Vahedi, G. 175 Valdez‐Flores, C. 166 Valujet aircraft crash 234 Van Moergestel, L. 177 Vartabedian, R. 51 Verification and validation 186 change of processes 186 control activity 34 design 94 negative correlation to productivity 185 non‐standard design 57 non‐value adding activity 22, 174, 185 open loop 66, 115, 190 paper trail 96, 97 risk factor 240 test waiver 58 Vincins, R.A. 152, 170 Vitasek, K. 213 w Wagenmakers, E J. 177 Warranty 104–106 field data for reliability 256, 269 operational limits 265 Web Finance, Inc. 103 Webster, J.G. Weibull, W. 259 Weibull analysis 97, 98, 259 B‐percentile 262 distribution 99, 260 graph of failure data 267 scale 262 shape 259 slope 267 West, J. 28 Wheeler, D J. 301 Wheelwright, S C. 181, 183, 184 Whistleblowers see Qui tam Wiener, J.L. 104 Williamson, J. 176 Wilson point estimate 301 Wong, W.C. 166 Woodard, J. 166 Work instruction 242, 243 compared to job instruction 242 Work station 142 Wright, G. 181 y Yielded cost 157 Young, J. 160 Young, R B. 266 z Zaretsky, E.V. 98, 266 ... friend, my life ix Contents Preface xix What Is Forensic Systems Engineering? 1.1 Systems and Systems Engineering 1.2 Forensic Systems Engineering References Contracts, Specifications,... Nonlinear Systems: Parameter Analysis and Design New York: John Wiley & Sons, Inc., pp 445–446 xxvii 1 What Is Forensic Systems Engineering? CHAPTER MENU 1.1 Systems and Systems Engineering, 1.2 Forensic. .. and Systems Engineering, 1.2 Forensic Systems Engineering, References, 4 Forensic systems engineering can be defined as the preparation of systems engineering data for trial This snapshot