INFORMATION TO USERS
This manuscript has been reproduced from the microfilm master UMI films the text directly from the original or copy submitted Thus, some thesis and dissertation copies are in typewriter face, while others may be from any type of computer printer
The quality of this reproduction is dependent upon the quality of the copy submitted Broken or indistinct print, colored or poor quality illustrations and photographs, print bleedthrough, substandard margins, and improper alignment can adversely affect reproduction
in the unlikely event that the author did not send UMI a complete manuscript and tnere are missing pages, these will be noted Also, if unauthorized copyright material had to be removed, a note will indicate the deletion
Oversize materials (e.g., maps, drawings, charts) are reproduced by sectioning the original, beginning at the upper left-hand comer and continuing from left to right in equal sections with small overlaps
ProQuest Information and Learning
300 North Zeeb Road, Ann Arbor, MI 48106-1346 USA 800-521-0600
Trang 3MANAGING SIMULATION BASED TRAINING
A FRAMEWORK FOR OPTIMIZING LEARNING, COST, AND TIME
A DISSERTATION
SUBMITTED TO THE DEPARTMENT OF MANAGEMENT SCIENCE AND ENGINEERING AND THE COMMITTEE ON GRADUATE STUDIES
OF STANFORD UNIVERSITY
IN PARTIAL FULLFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF
DOCTOR OF PHILSOPHY
Trang 4UMI Number: 3085223
Copyright 2003 by Richmond, Noah Joseph
All rights reserved
® UMI
UMI Microform 3085223
Copyright 2003 by ProQuest Information and Learning Company All rights reserved This microform edition is protected against
unauthorized copying under Title 17, United States Code
ProQuest Information and Learning Company 300 North Zeeb Road
P.O Box 1346
Trang 5© Copyright Noah J Richmond 2003 All Rights Reserved
Trang 6I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy
William Perry
I certify that I have read this dissertation and that, in my opinion, it is fully adequate in
scope and quality as a dissertation for the degree of Doctor of Philosophy
Trang 7
MANAGING SIMULATION BASED TRAINING
A FRAMEWORK FOR OPTIMIZING LEARNING, COST, AND TIME
-Abstract-
This study provides a management framework for optimizing training programs for learning, cost, and time when using simulation based training (SBT) and reality based training (RBT) as resources Simulation is shown to be an effective means for
implementing activity substitution as a way to reduce risk The risk profile of 22 US Air
Force vehicles are calculated, and the potential risk reduction is calculated under the
assumption of perfect substitutability of RBT and SBT Methods are subsequently developed to relax the assumption of perfect substitutability The transfer effectiveness tatio (TER) concept is defined and modeled as a function of the quality of the simulator used, and the requirements of the activity trained The Navy F/A-18 is then analyzed in a case study illustrating how learning can be maximized subject to constraints in cost and time, and also subject to the decision maker’s preferences for the proportional and absolute use of simulation Solution methods for optimizing multiple activities across shared resources are next provided Finally, a simulation strategy including an operations planning program (OPP), an implementation program (IP), an acquisition program (AP), and a pedagogical research program (PRP) is detailed The study provides the theoretical tools to understand how to leverage SBT, a case study demonstrating these tools’
efficacy, and a set of policy recommendations to enable the US military to better utilize SBT in the future
Trang 8In Memory of Gian-Carlo Rota
Trang 9
Managing Simulation Based Training
A Framework for Optimizing Learning, Cost, and Time
Noah Joseph Richmond
Executive Summary: Simulation Strategy 1
Organization and Principle Findings 1
Chapter I: Risk Exposure and Reduction via Simulation 1 Chapter IA: Military Personnel Risk Exposure and Risk Reduction Strategies 2 Chapter IB: Risk Assessment and Impact of 1:1 SBT:RBT exchange 3 Chapter II: Transfer Effectiveness Ratio and Total Training Resources 4 Chapter III: Optimization Framework and Computational Form 5
Chapter IIIA: Optimization Framework 5
Chapter IIIB: Computational Form 6
Chapter IV: Case Study and Algorithmic Techniques 6
Chapter IVA: F/A-18 Training Program and Formulation 6 Chapter [VB and Chapter [VC: Formulation and Solution Methods of the LATT
Activity in the (L,C,T) Framework 7
Chapter IVD: Formulation of the Multi-Activity Strategic F/A-18 Optimization 8
Closing Remarks 9
Chapter [ 10
Risk Reduction of Military Aviation through Simulation 10
Flight Training: Key Element of Aviation Accidents II
Strategies for Risk Reduction 12
Reduced Risk Training 13
Estimating the Value of Simulation Based Training 13
If SBT is a perfect substitute for RBT, how should SBT be used? 14
Chapter II 33
The Link between simulation and reality based training: TER 33
What is TER? 33
Transfer Effectiveness Ratio: Qualitative Definition 33
TER: Mathematical Definition 34
What Factors Determine TER? 36
Physiology of TER 37
The Technology of TER 40
The Interaction of Physiology and Technology 44
Chapter IITA 50
Structure of the Tactical Model 50
The Total Training Resources (TTR) Function 51
The Optimization Framework 58
Chapter IIIB 62
Computational Form 62
Trang 10
Boundary Solutions
Optimization with Respect to Cost
Optimization with Respect to Time Optimization with Respect to Cost and Time Chapter IVA Introduction: The Navy F/A-18 fighter Overview Strategic Significance Risk Profile The Inter-deployment Training Cycle Readiness: Peaks and Troughs Stages of Training Logistical Constraints
Skill Sets and Hierarchies
The Primary Mission Area
PMA Types and Associated Skills
The PMA Points System
PMA Missions
Activities
Modeling Readiness
Chapter [VB
Formulation: Low-Altitude Tactical Training (LATT)
Construction of the Learning functions
Estimating T
Estimating L
Navy Flight Simulators: Assessing TER
Simulator Types and Measures of Quality
Simulator Types: PTT, OFT, TOFT, and WTT
Capital Assets: Type, Quantity, and Location
Measures of Quality: Evaluating Vision and Acceleration Systems TER Estimation Logistic TER estimation for the F/A-18 Cost of Training Optimization Summary: Low-Altitude Tactical Training (LATT) Chapter [VC
Solution Methods and Solution Properties
Present Solution for LATT
Boundary Solutions for LATT
Parametric Solutions: Simultaneous Cost and Time Constraints
Optimal Training Programs
Constraints in a and B
Optimal Training for different values of alpha
Optimal training for different values of B
Trang 11Chapter IV D
The Total Readiness Function
PMA Readiness Functions
Combining PMA Readiness Functions to form Total Readiness
Variations in Readiness Measures
Shared Resource Constraints: Cost and Time
Shared Resource Constraints: Simulator Capacity
Total and proportional use of SBT and RBT
A Two Activity Example
Solution Techniques and [nterpretations: Overview
Strategic-Tactical Optimality Condition (STOC)
Interpreting Results: The Implied Allocation Model (AM)
What If Analyses: Leveraging STOC and IAM
Chapter V: Simulation Strategy
Summary of Results
Developing and Implementing a Simulation Strategy
Components of a Simulation Strategy
Trang 12Executive Summary: Simulation Strategy
This study provides a management framework for understanding the impact of simulation based training Chapters [ through IV provide the analytic machinery necessary to
analyze, formulate, and optimize training programs This chapter previews and
summarizes the principle findings of the study Chapter V closes with recommendations for present policy planning, and future research
Organization and Principle Findings
The chart below illustrates the organization of the study along single activity (“micro”) and complete training program (“macro”) lines, and between theory and application RES x x X x X x xX X xX X XxX X xX X
Methods for understanding single activity analysis, as well as complete training programs are developed Real data is supplied for the analysis of the total risk exposure of the US Air Force, and activity level analysis of the Navy’s F/A-18 A short overview of the primary findings of each chapter is next provided
Chapter |: Risk Exposure and Reduction via Simulation
Trang 13question, “If SBT and RBT are perfect substitutes, how should SBT be used?” The chapter thus provides a descriptive analysis of current risk exposure, strategies for its reduction, and a first-cut analysis for the cost savings that can be achieved using simulation
Chapter IA: Military Personnel Risk Exposure and Risk Reduction Strategies
The military has seen a dramatic reduction is risk exposure in the last twenty years In
1980, 2400 servicemen died The Air Force alone lost 350 due to unintentional injury By
1993, 1300 lives were lost The Air Force cut losses due to accidents to 100 persons While total accidental deaths fell by 71%, the accidental death rate fell a less dramatic 58% because the military downsized
The loss reductions were driven by improved training doctrine and equipment reliability
Though accident rates have been halved throughout the force, risk exposure remains high, and more recent gains have leveled off Today, five military personnel per month die in aviation accidents Over ten years, these deaths exceed combined losses from the Gulf War and military operations in Kosovo Training is now riskier than war; accident rates are sufficiently high to threaten operational tempo While the rate averages five per month, there are periods when three or four times the average may occur nearly
simultaneously When multiple accidents occur, operations must be addressed politically in order to maintain confidence in the military
It is therefore imperative to find new ways to reduce risk exposure Chapter JA reviews
Trang 14Chapter IB: Risk Assessment and Impact of 1:1 SBT:RBT exchange If SBT and RBT are perfect substitutes, then one training hour of SBT can be used to
replace one training hour of RBT without impacting the quality of learning that takes place The risk associated with the live flight hour is eliminated, replaced by the (nearly) risk free simulation training hour—when a simulator is “crashed” there is no loss of life or equipment! While the assumption of perfect substitutability is naive, it is a good point of departure Chapter IB provides an analysis of risk reduction through the use of
simulation under this assumption The remainder of this study provides methods for analyzing the more complex problem where training methods have different pedagogical
values, costs, and exercise times
Chapter IB begins by showing that time, i.e a training hour, is the proper measure of the independent variable Loss rates, the dependent variables, are then defined as the average number of deaths per training hour The loss rates are calculated for each of 22 vehicles
in the US Air force by analyzing their complete historical risk profiles Because loss rates
change over time, it is necessary to model the trend in these rates The cumulative average loss rates are calculated, first for loss of life, and then for cost of capital These
rates are combined to forecast total losses for each vehicle type for a five year horizon
This analysis is referred to as the Single Vehicle Loss Model (SVLM), and measures how risky each vehicle platform is using a single summary statistic
These SVLM statistics are then combined in the Multi-Vehicle Loss Model (MVLM) to
show how losses can be reduced through the use of simulation The MVLM shows that total forecast losses for the five year period 2002-2006 is $3.3 Billion Savings can be made by substituting simulation based training for live flight Because simulation is assumed to be a perfect substitute, the optimal solution is degenerate—all live flight should be replaced with simulation because of its cost advantage As a result, two
constraints are added: (1) Proportional and (2) Total use of simulation Respectively
denoted by a and , the first limits the percentage of hours that may be substituted away for any single vehicle, while the latter limits the total number of hours substituted across
Trang 15savings as a function of a and B As expected, the larger the total number of hours permitted, the more savings that accrue However, marginal savings decline with increased volume, because the optimal approach is to substitute away the riskiest (most costly) hours first
Chapter Il: Transfer Effectiveness Ratio and Total Training Resources
Much of the debate surrounding the use of simulation today is based upon a lack of agreement on the impact of substituting one training activity for another Chapter I is performed under the restrictive analysis of perfect substitutability, despite the fact that different activities have different pedagogical value, and a method to account for these
differences is needed
Chapter II provides the theoretical foundation necessary to lift the assumption of perfect substitutability The impact on pedagogical value is separated into two parts: (1)
Calculating aggregate training, and (2) Modeling learning as a function of aggregate training The keystone is the concept of the Transfer Effectiveness Ratio (TER) The transfer effectiveness ratio is a method of converting the contribution of a single training method into a measurable, common currency of ‘training resources.’ The total of these
contributions is referred to as Total Training Resources (TTR), and learning is modeled
as a function of the single variable TTR
Chapter [I provides methods for modeling and assessing TER The analysis proceeds in four steps The first is the definition of TER, which is expressed as a simple
Trang 16of training method thereby leads to predictable outcomes—the amount of learning accomplished
The third and final step discusses current methods for assessing TER Advantages and disadvantages of three current methods are discussed: (1) Pilot and expert opinion, (2) In-
simulator learning, and (3) Direct transfer In addition, a new method, termed ‘logistic
analysis,’ is developed It models TER as a function of two variables, Frequency and Quality Frequency is the rate at which practice cycles can be completed, and Quality is a
composite measure of the capabilities of the technology to deliver realistic synthetic stimuli This method for TER estimation thus builds on the previous analysis of
physiology and technology, and is the only method that separately accounts for the quality and quantity of training delivered
Chapter Ill: Optimization Framework and Computational Form Chapter III describes the management framework in which complete training programs are modeled and optimized The chapter has two parts Part A formulates the objective
function and constraints in their most intuitive form Part B provides an equivalent
program more suitable for computation
Chapter IIIA: Optimization Framework
The optimization framework provides a method for the efficient trading off of learning, cost, and time for the training of a single activity Mathematical definitions of learning,
cost, and time are provided Each is described as a function of the types and quantities of
training resources consumed In addition, the constraints a and B on proportional and total
use of simulation are formalized The equations provide a framework in which one of
Trang 17Chapter I1IB: Computational Form
The optimization of principal interest to the military is maximizing learning subject to constraints in cost and time Learning, L(x), is designated as the objective function By construction, L(x) is a composite function consisting of a concave function of a linear function of x As a result, the embedded linear function can be maximized instead of L(x) Because the constraints in cost, C(x), and time, T(x), are linear, minor algebra results in a linear program of the form:
Maximize f (x)
sử
Ax<b x20
Referred to as the “computational form,” this program can be solved rapidly using simplex or any other basic solution methodology
Chapter IV: Case Study and Algorithmic Techniques
Chapter IV applies the theory of Chapters II and II in a case study of the Navy’s F/A-18 The principle strike fighter deployed on aircraft carriers, the F/A-18 forms the backbone of sea-based, power projection capabilities Because of it’s sheer numbers, and
comparatively high risk profile, the F/A-18 is an ideal subject for a case study The analysis proceeds in four steps First, a thorough review of the Navy’s current training program for the F/A-18 is provided The Low Altitude Tactical Training (LATT) activity
is then formulated in the optimization framework A numerical analysis for LATT
follows Chapter IV closes by showing how multiple activities can be simultaneously optimized over a set of common resources so as to maximize total readiness
Chapter IVA: F/A-18 Training Program and Formulation
Trang 18“activities,” of the F/A-18 By modeling activity level readiness as a learning function L(x) in the optimization framework, the tools of Chapters II and III can be applied
Chapter IVA also describes the Navy’s simulator assets, and shows how they are modeled as training resources in the optimization framework The TER for each
simulator when applied to Low Altitude Tactical Training (LATT) is determined using the logistic analysis method Provision of the cost of training for each method completes the necessary data sets Chapter IVA concludes by providing the general form for the LATT tactical optimization program
Chapter !VB and Chapter IVC: Formulation and Solution Methods of the LATT Activity in the (L,C,T) Framework
Chapter IVB takes up where Chapter [VA ends, translating the general form of LATT
into its computational form Learning is designated as the objective function, subject to constraints in cost and time Chapter [VC then demonstrates solution methods The numerical analysis begins by parametrically maximizing learning subject to simultaneous constraints in cost and time The program is first optimized with a and B unrestricted, and
subsequently repeated with a gradually restricted and B free, and then B restricted while a
is free The step by step analysis shows how the constraints interact to select the optimal training program Two principle concepts arise: (1) The efficient frontier, and (2) a
solution heuristic
The efficient frontier is found by plotting the training methods on two dimensions: TER and Cost The efficient frontier is the boundary formed by taking convex combinations of those points with the best combinations of TER and Cost When a and B are unrestricted, the training methods selected must lie on the efficient frontier The efficient frontier is a useful concept for understanding which technologies offer advantageous TER-Cost combinations in the framework of a system of training platforms
The solution heuristic provides a way for understanding what solutions on the efficient
Trang 19deviate from the efficient frontier When a and B are free, problems in which time is scarce will favor methods on the efficient frontier with a high TER ratio When budgets are tight, methods with a high TER/$ ratio will be preferred The heuristic states that as a
is constrained, the most efficient (i.e., closes to the efficient frontier) RBT methods will
enter the solution mix When B is constrained, RBT may or may not be used Together, these principles provide a way of looking at the training methods on a simple graph, and then understanding how the program’s constraints shape the optimal solution
Chapter IVD: Formulation of the Multi-Activity Strategic F/A-18 Optimization
The final piece of the methodology puzzle is to show how several different activities can
be optimized over a shared set of resources The total readiness (TR) function shows how
the objective functions L(x) of the individual activities can be joined into a single
strategic objective function It is simply a linear combination where the weights assigned
reflect the relative importance of the activities trained: TR(Z) =Ze L(x)
The constraints of the multi-activity (strategic) level problem are analogous to the single activity (tactical) problem Total cost and time constraints are enforced as limits on the sum of the individual costs and time limits Likewise, a and B limit the proportional and total use of simulation across all activities Capacity constraints for total available hours on each training method are added
Two properties follow from this structure The first is the Strategic-Tactical Optimality
Condition (STOC) This condition states that the solution to the strategic program consists of solutions to the tactical programs that are optimal when viewed alone The property holds when the constraints of the tactical problems are set to the values implied by the strategic solution For example, if the strategic solution requires that $4,000 and five hours be spent for Low Altitude Tactical Training, then the tactical solution is
Trang 20The STOC property is useful because it leads to the Implied Allocation Model (LAM) This model describes strategic optimization as occurring in two steps First, the strategic constraints are “allocated” across the individual sub-problems For example, the total cost budget is divided across the individual tactical models, each of which may use up to its
allocated portion of the total available budget Second, after allocation, each model is
optimized given its now fixed resource constraints Optimal solutions for the strategic model can be found informally by trying different allocations, or through a standard search algorithm
The STOC property parallels how actual budgeting is accomplished It provides a method for conducting the what if analyses and re-planning current training regimes For
example, the strategic program can be optimized given current budget and time
allocations It can then be re-optimized with these tactical allocations left free The two model runs indicate where budgets should be tightened or increased, where time should be reduced or increased, and in what training activities simulation should be emphasized or reduced Guidance on how to change the training program is thereby provided
Closing Remarks
Trang 21Chapter |
Risk Reduction of Military Aviation through Simulation Aviation accidents are a major source of risk for the US military Flight training in particular is a dangerous business In World War II, 120,000 American aviators were killed Of these, 15,000—one in eight—died in training.' Today, Accidents are the number one cause of death amongst US servicemen Accidents have superceded enemy action as the military’s number one source of fatalities, claiming half of all fatalities from the period spanning 1980 to 1993 The charts below show that for both men and women,
accidents amount to a clear majority of deaths, claiming respectively 58.4% and 50.7% of
persons This rate more than doubles that of death due to disease and illness.”
'đ8 Title
Fuzure Air Force ta Oismi Sut on co Maes = artes by Cause ofCeam 1930 1993 No5 292 Figure Ar Force - to Crsintabon cf Female F atanves oy Dause of Death, 1980-1999 N=337 Os ease ang liness Cusease and uress 8962 720% TS 216% } Homicide "42x i?e “39 1% tara Sot ce ald 140% Se nteraonal inate : 2048 98 KV ` ‘rentenbonal Injury 176 %0 7% “ a a ” Creer BE an 41 !0% Other ° 103% * hchudes deaths O.¢ 20 hustite action (n= ZH} and hose deaths where
Couse ss ther unclessfed_ cr uns mown [ns 18) * Includes ore death where Cause 1s csher
` ⁄/
Accidents during flight compose one of the greatest risks undertaken today: It is one of
Trang 22(disease, homicide, and suicide) are not associated with major capital losses, accidents are the primary peace time driver for the loss of capital equipment Peacetime losses from the last 10 years exceed the combined losses of the 1991 Persian Gulf and Kosovo air campaigns
The degree to which the problem of accidental death can interfere with military
operations cannot be overstated Beyond even the loss of life and capital equipment— accepted risks in military service—is the strategic issue of maintaining operational tempo Twice in the past five years the military has temporarily ceased non-essential operations precisely because these costs became too high The strategic military mission of maintaining readiness was compromised This limitation has the potential to
significantly undermine the capability of the US military; the most recent stand down
came during the war in Afghanistan, illustrating the seriousness of this problem
Flight Training: Key Element of Aviation Accidents
Training accidents are a key driver of aviation accidents In the period spanning 1984 to 1994, 1523 aircraft crashed and 1600 lives were lost in training exercises Pro-rated, lives lost during flight training amount to over half of those lost through unintentional injury, hence over 25% of all fatalities There are approximately 30,000 pilots in the armed
forces today From an operations perspective, there are typically 1.5 major accidents for
every 100,000 flight hours As a result of the force size, operational tempo, and
associated risk of flight training, accidents resulting in death during flight accrue at a rate of about five aircraft per month.* The costs of these accidents are enormous Consider the losses due to flight training alone: At an estimated loss rate of 60 planes and 100 lives per year, with an aircraft cost of $30 million, life value of $2 million, and loss of skills of $5 million per person, the total cost of these accidents amounts to $2.5 Billion per annum The strategic value of op tempo is perhaps an order of magnitude greater
3 http://www.cnn.com/2002/US/02/08/air.force.safety/index.html See also http://www.cnn.com/US/9709/17/military.crash/index.html
* This statistic does not take into account type of aircraft or missions flown, but conveys the general level of risk faced Note that a Class A mishap, in which the aircraft is essentially lost or destroyed, typically results in more than one death Hence the loss rate of five aircraft per month results in 60 aircraft per year, and 100 lives per year
Trang 23Strategies for Risk Reduction
The US military faces a clear problem that can be summarized in three key points: (1) Unintentional accidents are responsible for the majority of deaths of US
servicemen
(2) Unintentional accidents are responsible for a vast majority of peace time capital
equipment losses
(3) Unintentional accidents can raise risk levels to the point that it is preferable to reduce operational tempo and sacrifice readiness, the core metric of (latent) military efficacy
There are three basic strategies that the military can pursue to offset these losses: (1) Reduce the frequency of activities that lead to risk
(2) Make the activities that lead to risk safer
(3) Find alternatives to the activities that lead to risk
Option (1) is essentially the concession of op tempo: The reduction of lost life and equipment at the expense of reduced readiness.” Options (2) and (3) are preferable because they attempt to maintain readiness while simultaneously reducing risk The rigorous pursuit of option (2) explains the past reduction in the rate of unintentional accidents By all accounts, this strategy has been successful, accounting for a 25% reduction in the total rate of death amongst servicemen from the early eighties Previous gains have come through the improvement in the reliability of military equipment, and improved safety precautions in training doctrine However, the low-hanging fruit has been picked—there are fundamental risks that cannot be engineered away, either through improved reliability or operations management Unless the activities undertaken during mission rehearsal are significantly changed, it will be difficult to further improve safety Hence we turn to option (3): Reduced Risk Training
> This option can include the removal of certain training activities, and brief periods during which the military “stands down.” For example, in September of 1997, Secretary of Defense William Cohen ordered a 24 hours suspension of routine flights This period coincided with the patrol of no-fly zones in Bosnia and Iraq See http://www.cnn.com/US/9709/17/military.crash/index.html
Trang 24Reduced Risk Training
According to the military credo, “We train as we fight,” pilots learn by doing The consequence is that high quality training equates to risky, high cost training—a cost appraised in billions of dollars per year Pursing option (3) requires that the military find new ways to train its personnel that have the same pedagogical value as the current high- risk training methods but with a lower risk profile Simulation is one approach, but has met with resistance
Juxtaposed to the military credo, ““We train as we fight,” the concept of substituting low
risk activities for high risk activities—including the use of simulators—is anathema
Two assumptions typically bolster this viewpoint: (1) Reality based training is “more real” than Simulation based training, and (2) More realism is better However, both assumptions are patently false Empirical evidence shows that (1) Reality based training need not be “more real” than Simulation based training, and (2) More realism need not have any pedagogical value whatsoever—it can, in fact, be counter-producti ve." The Gordian knot that links high quality training to high risk activities can be cut Activity substitution, specifically simulation, offers a means to this end However, it must be used appropriately, or the premonition of inadequate training may be fulfilled This chapter
sets forth to answer the following critical question: How can simulation be used to reduce
the risk of training, and what is its potential value?
Estimating the Value of Simulation Based Training
Simulation Based Training reduces risk exposure by substituting an essentially risk free activity for a high risk activity By reducing the amount of real flight training, the military can reduce the number of accidents that occur, and the subsequent loss of life, expertise, and capital This study next calculates the exact savings gained by using SBT as a substitute for RBT Real data is used The number of lives and dollars saved are calculated as functions of both the proportional and absolute quantity of SBT used Perfect substitutability of RBT and SBT is assumed for this first cut analysis
® See Taylor, Lintern, Kunde, Tschopp, and Talleur, “Effects of Scene Content, Field of View and Amount of Simulator Training in First Officer Training.” Appearing in Flight Simulation Technology, Capabilities, and Benefits, [7-18 May 1995, pp 7.1-7.11
Trang 25If SBT is a perfect substitute for RBT, how should SBT be used? If SBT is a perfect substitute for RBT, then the calculation of how much SBT to use is direct If we replace one unit of RBT with one unit of SBT, then no change in readiness occurs However, the risk associated with the unit of RBT is removed, replaced by the risk associated with the unit of SBT added Because SBT leads to neither loss of life nor capital equipment loss, the risk associated with SBT is zero Thus every unit of SBT substituted in effectively removes the risk of one unit of RBT From this fundamental relationship we can build a function that shows the dollars and lives saved as a function of the amount of SBT used (in place of RBT) This analysis is carried out in 6 basic steps:
(1) Define the independent unit of measure
(2) Define a loss rate associated with the independent unit of measure (3) Calculate the single vehicle loss rate for human life
(4) Expand the single vehicle loss rate to include the cost of lost capital
(5) Combine data from the single vehicle loss models to generate summary statistics of losses across aircraft types
(6) Combine data from the single vehicle loss models and multi-vehicle loss models
to calculate risk reduction as a function of proportional and absolute use of SBT
This calculations tells the DM exactly what the impact of different SBT usage policies will be on cost through loss of life and equipment The policy maker can then make explicit tradeoffs between the use of SBT and RBT, and anticipate the results
The Independent Unit of Measure: Flight Hours
In order to determine how to reduce risk through activity substitution, it is necessary to have a, ‘common currency,’ for the exchange of units of one activity for those of another The essential criterion is that each activity unit should have associated with it a standard quantity of risk Then substituting units of one activity for units of the other will have a predictable effect on the amount of risk incurred Without having a standard unit of activity, and an associated standard quanta of risk, there can be no rational trade off of risks through changes in activity levels The selection of the unit of measure determines
Trang 26how this exchange will be made The independent unit of measure used in this study is flight hours
Estimable Loss Rates
Risk is a Statistical phenomena Showing that a flight hour has associated with it a ‘standard’ quanta of risk is complex because it requires the estimation of a probability density: We must calculate the probability of an accident per flight hour We must calculate a mean and confidence interval for the losses per flight hour In mathematical terms, the loss rate is considered ‘estimable’ if the mean can be calculated with low or ne
bias (accuracy), and the confidence interval is small (precision) In the real world,
‘enough’ accidents have to occur for us to be able to develop the first order mean and
standard deviation statistics
Unfortunately (from an analytic perspective), the loss rate changes over time As a result,
the mean—a function of time—cannot be computed by taking the simple average of
accidents over flight hours Likewise, the standard formulae for the standard deviation for stationary means cannot be applied Fortunately, the rate of loss for most aircraft
approaches an asymptotic limit As a result, the mean at any time T can be calculated as
the cumulative mean—the total number of accidents over the cumulative flight hours The benefit of this method over a stationary mean is that it illustrates the change in
accident rates over time, and that the emerging trends can be forecast The ability to forecast future loss rates is critical, because the policy decisions that this study informs are designed around the amount of simulation that should be used now and in the future While this method will slightly overestimate the accident rate at time T (because the function is approaching an asymptote from above), the use of an unbiased moving average is precluded by the relative rarity of accidents Such a measure would be too
volatile to be useful, or to forecast
Trang 27The Single Vehicle Loss Model We next describe how to estimate and forecast the loss rates for a single vehicle We first show through graphical inspection that risk is an estimable function of flight hours We then validate this explanation with computational results
Our objective is to forecast the total number of deaths that will be incurred by a particular aircraft type for the next five years The calculation is computed in two parts First, we forecast the number of flight hours for each year, and the rate of death for each year Second, these forecasts are used to compute the total number of forecast deaths as the vector product of the flight hours and death rates:
flight hours ¢ death rates
Forecast Deaths = -
discount factor
The discount factor is included in this model to reflect the change in real value of nominal losses over time
Flight Hours: Confirming the Risk Measure
Trang 28There is a small non-linearity in the relationship between these variables, which is significant Note that the line depicting cumulative fatalities ‘bows’ out and away from cumulative flight hours This curvature occurs because, with each passing year, fewer accidents accrue per flight hour: The rate of accidents is undergoing a very progressive decline
Estimating the Death Rates
This trend is confirmed with
A-10 Fatality Rate and Cumulative Average Fatality Rate :
a second plot, illustrating the Ị
death rate for cach year, and the cumulative death rate In the chart at right, the bars Deaths per 100,000 Hours depict the single period death rate, equal to the number of persons killed per 100,000 eer FFX Ke KF PELL ESS Year flight hours ÍmBFATAL RATE ===FATAL AVE RATE
There is a clear estimable mean that slowly declines over time This mean is depicted by the blue line, which shows the cumulative average fatality rate The downward trend is initially exponential in character, then progressively more linear An asymptote appears to exist at approximately I-1.5 deaths per 100,000 flight hours The equations for these statistics are given below: deaths, Fatality Rate d, = ————_ Oey BBE =F 700,000 > deaths, Cum Ave Fatality Rate d* = —* Yh, /;00,000 i=]
death rate, period t = d, flight hours, period t = h,
Trang 29The standard deviation of the cumulative average fatality rate is estimated as a generalized auto-regressive conditional heteroscedastic (GARCH) process The basic equation is 0° =a@eo,,+fe", where £ =d, —d,™” The variance at time t is calculated as a weighted average of the previous periods’ estimated variance, and the
current one-period observed variance This one period variance is ý”, the square of the
difference of the single period death rate from the cumulative average death rate This difference is interpreted as the one period deviation of the observed value from the estimated mean
The chart at right illustrates the time series d™*, ¢,, and a All three statistics are well
behaved As noted, d,”* follows a gradual exponential decline, then asymptotically
approaches a limit at 1-1.5 deaths per 100,000 flight
hours The series ¢, is
conditionally normal, and the resulting series o7 is stable, approaching a mean value of one (hence
G, approaches one also)
Trang 30In conclusion, the estimate of d®* is numerically r yp precise (i.e., confidence is high) The & chart below illustrates confidence intervals centered around a mean death rate of | person per 100,000 flight hours The results indicate that the confidence intervals are valid (i.e., contain only non-negative values) up to a confidence level of 90% These values are sufficient for a first cut analysis Though the estimated loss of life will vary by a factor of 2, the risk levels associated with different aircraft vary by as much as a factor of 100 As a result, these values allow a definitive prioritization between aircraft The policy maker can identify the riskier aircraft with considerable confidence, a useful tool when
determining how best to leverage SBT Confidence Intervals for Mu=1, Sigma=0.6" Mu 1.00 1.00 1.00 1.00 1.00 1.00 Sigma 0.60 0.60 0.60 0.60 0.60 0.60 Conf Level 50% 70% 90% 95% 98% 99% High Est 1.40 1.62 1.99 2.18 2.40 2.85 Low Est 0.60 0.38 0.01 -0.18 -0.40 -0.85
Forecasting Flight Hours and Death Rates
The final step is to model both the future flight hours and death rates, and then to combine them as a vector product:
flight hours ¢ death rates Forecast Deaths = -
discount factor
For these estimates, the ten year period spanning 1992 to 2001 is used for historical data, and the five year forecast spans the years 2002 to 2006.® Data is fit both using the linear model y =ax+b, and also the exponential model y=e**?
” Setting o = 0.6 when p = | is equivalent to setting o equal to 60% of pt, reflecting the observed relationship between o, and d,""*
Trang 31The maximum likelihood estimator can be solved for explicitly In the linear case, we have a= no 0Q) and b=Y —aX , where X and ¥ are respectively the
nà x” - (> xƑ
mean of the independent variable x (year), and dependent variable y (death rate, or flight
hours) For the exponential model the calculation ¡s the same, except that we replace the
observed dependent values y with the natural logarithm of these values, effectively estimating In y=ax+b The total error of both models is computed as the sum of squared errors over the estimated and observed values, and the model with the smaller total error is selected for forecasting
Future values are estimated with the selected model for the forecast years The chart at right depicts the ten year death rates and flight hours for the A-10 The shaded box
indicates forecast values
Forecast Cumulative Average Fatality Rate and Hours
Trang 32In the case of the A-10, the trend is clearly linear, and the results are intuitive (the standard deviation is similarly forecast, but not shown) The final calculation is:
flight hours ¢ death rates discount factor Forecast Deaths = 4 => (1+ hd, Ind, Ind, hại, (I+r} (L+r} (L+ r} (L+r} FD = hyd, +
death rate, period t = d, flight hours, period t =h, risk free rate of return=r
For the ALO, the estimated flight hours, death rates, and ‘real deaths’ for periods zero through four are shown in the following chart Forecast Flight Hours, Loss Rates, and Deaths by Period for A-10 Thunderbolt Period ” T0——T 2 3 4 Loss Rate 1.405 1377 - 1349 1.322 1296 ‘Total Nominal Deaths | 1.416 1.322 1.233 1.150 1.073 6.194 Real Deaths’ _| 1.416 1.282 1.161 1.050 0.950 5.861
The estimated 5.86 deaths is the mean forecast deaths By forecasting the standard
deviation of the loss rates, it is possible to estimate a low and high forecast The method
used for this forecast is the same as for the mean loss rate and the number of flight hours: Linear and exponential models are fit, and the series is forecast using the method with the smaller sum of square errors The high (low) forecasts for the loss rates is then calculated as the mean loss rates plus (minus) the forecast standard deviation.'® The results for the A-10 are illustrated below
A discount rate of 3% was used See Office of Management and Budget Mid Session Review: http://www whitehouse z0v/omb/budget/fy2002/msr04 html
' It is possible to have a negative loss rate for the low estimate using this method In this case, we replace the negative loss rate with a loss rate of zero
Trang 33High LossRate ` | 2.356 2325 - 2.296 2266 2.238 Low, Mean, and High Real Death Estimates for A-10 Thunderbolt _ Total Mean Real Deaths [1416 1282 L161 1.050 0.950 5861 ad
The Single Vehicle Loss Model: Summary of Results
The principal value of the single vehicle loss model is that it allows us to quantify, forecast, and compare the risk attributed to aviation accidents by aircraft type For the A- 10 Thunderbolt, the expected number of deaths over five years is 5.9 persons, with a low of 1.8 persons, and a high of 10.0 persons This range reflects the mean estimate plus or
minus one standard deviation, hence a confidence interval of 70% The values fall within
one order of magnitude
Because of the enormity of value attributed to a human life, this information is sufficient to enable effective decision making The questions of what percentage of reality based training should be replaced with simulation based training, and how much simulation should be used in total can be answered The multi-vehicle loss model, subsequently
described, is a method for leveraging this information to inform these decisions
However, we first summarize the findings of the single vehicle loss model across all
airplanes, and then extend the model to allows us to account for the loss of capital
equipment and skills in addition to the loss of life
Trang 34Losses by Aircraft Type
The Single Vehicle Loss Model (SVLM) is used to determine the amount of risk incurred
by each aircraft, and
Mean, Low, and High Death Estimates by Aircraft
the comparative risk 5 Year Projection
across air craft 100
types The chart at % 80 a 1
right illustrates the J 60 — — | : High |
ø : + Low |
low, mean, and high 2 40 ‹ Mean!
death estimates for 20
0 et etetetetete+ee+ hf tt ttt i
each of twenty two PRSSERSTSLYRERaREe Seas
aircraft types ; Aircraft
There are three critical points to observe First, the aircraft fall loosely into three
categories: Low, medium, and high risk The low risk group consists of the T41, B2, C10, C17, TI, T39, C12, F4, C9, F1 L7, and B52 All of these aircraft are predicted to incur losses of fewer than two persons in the next five years The medium risk group consists of the T43, C141, T37, B1, T38, Al0 and F15 aircraft Each of these aircraft are
predicted to have more than two, but fewer than 10 lives lost The high risk group
contains the C5, F16, C135, and C130 aircraft, all of which are predicted to lose 10 or more lives
The second key observation is that the low, medium, and high risk rankings are consistent
with respect to the vehicles’ low, mean, and high loss rate estimates With the exception of the high estimate of the T43, and aircraft with highly similar loss rates, ranking the aircraft by their low risk estimates produces nearly the same results as ranking the aircraft
by their mean risk rankings.'' Ranking by the high risk rankings produces similar results
As a result, the analysis results are robust from the standpoint of relative risk
Trang 35
The third point is that the amount of risk exposure between aircraft can vary by an order
of magnitude At the low end, some aircraft have negligible death rates because either
there are few flight hours flown, or because the risk per flight hour is very low (or both)
These vehicles are expected to have no or few losses—one a year would be a surprise In contrast, there would be no surprise if the three riskiest airplanes resulted in losses of 50 or more persons each As a result, the return to investment on simulation will vary dramatically based on which airplanes the military chooses to target for simulation
Targeting Simulation Hours
When applying simulation to reduce risk, the economical solution is to replace reality based training hours with simulation based training hours for those aircraft that pose the
greatest risk per hour In order to have the maximum possible effect, the military ought to
pursue the ‘low-hanging fruit’ first—that is, the deaths that are cheapest to prevent The strategy of replacing flight hours with the highest loss rates is the most economical because simulator hours are approximately fixed in price, irrespective of airplane type rẻ The value of substituting a simulator hour for a reality based training hour is thus
determined by the amount of risk removed We thus prioritize those aircraft with the
highest loss rates The chart illustrates the low, medium, and high estimates together with
the total flight hours for the five year forecast period
Mean, Low, and High Death Estimates, & Flight Hours, by Airplane, Projected Five Years @ Hours a High « Low su Mean Flight Hours =a On TF An THORN MDM XS — ow w oO tTtomoeenrr-eMre ornre Ow 3 =e OF a 8 e oO - oO ca - eee ö K < uw k= = oO Aircraft
2 The price driver of a simulator hour is the type of system used, and the extent of amortization of the fixed costs Simulators vary in price from tens of thousands of dollars for low fidelity, fixed based systems to tens of millions of dollars for realistic cockpit simulators with synergistic motion platforms The price of each type of system is, however, relatively independent of the actual aircraft simulated
Trang 36The loss rates govern the lives saved per training hour substituted, while the total forecast deaths for a given airplane places a ceiling on the maximum possible savings The chart
below illustrates the relationship between total flight hours and deaths accrued Aircraft with similar total deaths appear in the same ‘row.’ Those with similar flight hours
appears in the same column For two aircraft with the same number of total deaths, the aircraft to the left on the chart has a higher loss rate because it has fewer associated flight hours Estimated Deaths, Flight hours by Aircraft Type | 100.0 C145 C130 | Fié | C5 ị 10.0 - Fis | 0 1a BÍ Giái 1 137 | ề BS2 § 1.0 +—Ft17 | a ị c9 F4 | T39 | 0.0 0 200000 400000 600000 800000 1000000 1200000 1400000 1600000 Flight Hours Ị
Recall that the loss models’ confidence intervals precision is typically one order of magnitude, hence the logarithmic risk scale is an intuitive way to compare the risk of
Trang 37The Value of Human Losses
As described, the SVLM describes how human losses accrue with aircraft accidents The cost associated contains two major components: The value associated with the loss of life, and the cost of replacing the skill set of the aviator The first figure is typically found as the implied expected value of a human life by the manifest risk preferences of decision makers In other words, if a decision maker is willing to spend $20,000 to reduce the risk of loss of life by 1%, then the associated value of a human life is
100 e $20,000 = $2 million Estimating the intrinsic value of a life at $2.0M, and the value of skills lost at $5.0M, the cost attributed to loss of a skilled personnel is approximately
$7.0M Totaling the loss of life across all aircraft produces low, mean, and high estimates
of 21, 158, and 360 lives for respective dollar equivalents of $147M, $1.1B, and $2.5B
respectively
The Value of Capital Losses
The second major cost is the loss of the aircraft The value of lost aircraft is taken to be
the dollar cost of the vehicle Note that this definition also has complications Older aircraft, when lost, are not replaced by equivalent aircraft Some analysts would argue that the value of such losses should be lower, because some useful proportion of the vehicle’s life has already been consumed These analysts value aircraft in a way similar to how insurance companies treat automobiles—they depreciate with time Others point out that older vehicles will be replaced with more expensive, newer vehicles Because the unit cost of vehicles has steadily increased, the dollar cost of replacement is actually higher than the purchase price of the lost airplane A more thorough approach might
consider the value derived from the aircraft through actual use Aircraft central to the
military’s planning might be assessed at a higher value than those considered ancillary The options are endless, for which reason this study begins at the beginning—the dollar cost of the vehicle Methods for more accurately assessing a vehicles ‘worth’ presents one area of future research for subsequent studies
Trang 38
The chart below illustrates the loss of aircraft (in units) Low, medium, and high
estimates are provided with the total flight hours for the five year forecast period Losses
are expressed in numbers of aircraft, not the value of the aircraft lost Estimated Aircraft Losses, Flight hours by Aircraft Type 100.0 F1B " F15 10.0 AAD T38 Tô C130 C135 = FIy 8 10 ~ < B52 G541 F4 0.1 TT4ãCS - > | | C12 | 0.0 0 2000 400000 600000 800000 1000000 1200000 1400000 1600000 | Flight Hours ị
There are large differences in risk exposure per flight hour, and both the risk per hour and total flight hours are needed to accurately gage the losses that will be incurred by any given aircraft type Plotting losses against flight hours on a logarithmic plot illustrates this point Note that the vertical axis traverses 4 orders of magnitude
Mean, Low, and High Aircraft Losses, |
Trang 39The Value of Combined Losses
Ultimately, the loss of lives and aircraft must be combined into a single measure
The chart below illustrates the combined losses of life, skills, and capital equipment by aircraft type Three critical features emerge on inspection First, the scope of the problem
can now be deduced Over the five year forecast period, the total of mean losses is $3.3
Billion, while the high and low estimates are respectively $6.0 Billion and $1.1 Billion
Mean, Low, and High Value Losses, Flight Hours, by Airplane, Projected Five Years 1600000 2.00E+09 | 1400000 F 1.80E+09 | 120000 [ 180E+09 „— | £ L 1.406409 2 fo} S 5000000 | 1.40E+09 2 ia Hours | 3 1206+09 Tao = 900000 [ 100E+09 @ | g @ 60000 [ 800E+08 ® : ® LOW x sưng j{ 600E:0s we Ls Mean | ; RỊT 4.00E+08 > 200000 4 ; R Ề Bt 2.00E+08 04 Š HAS “LLỆ,,BLL o.ooE+oo Ị WÑ' >> = “= Ơ Q@ *# Ơ Œ >> Œ@ «w> €@ h CC 0 0 = © ư © i PSST RFOT OFF R SBE eOM Eee | Aircraft ị
Second, the cost by aircraft type varies Aircraft Value Lost: % of Total
considerably The F16 contributes fully 37.9% of - Other, 2.8% lost value The top five sources of loss, the F16, | - C141, 1.1%
| | 7 C135, C130, BI, and F15 together contribute approximately 80% of the cost In contrast, the | | |} B52 1.5% || (3521 F117 2.8%
bottom ten contribute less than 3% of the losses | | [ A10, 4.0% C5, 5.3% Finally, the hours flown—and hence the amount of F16, 37.8% F15, 8.6% simulation needed to replace flight hours—varies
widely by aircraft type Substituting 10% of the
flight hours of the F-16 equates to 140,000 B1, 10.7%
simulator hours, while the B1 yields 1,000
C135, 12.9% C130, 12.4%
Trang 40The Multi-Vehicle Loss Model
The Single Vehicle Loss Model (SVLM) calculates the risk associated with each aircraft
It is a normative model, describing the mean and standard deviation of loss rates—our
principle measure of risk In contrast, the Multi Vehicle Loss Model (MVLM) is a prescriptive model, and describes how to apportion simulator hours across multiple
vehicles based upon the data produced by the SVLM Given a fixed number of simulator
hours to substitute for reality based training, the MVLM specifies which RBT hours to replace, and computes the savings resulting from the removal of the risk associated with the eliminated flight hours The method combines a single criterion for optimality, and
two constraints:
(1) Optimality: The hours with the highest associated risk, measured through dollar
equivalent losses, are eliminated first
(2) Constraint 1: A maximum percentage of training hours may be replaced for each
aircraft type This percentage, a, is held fixed across all aircraft types
(3) Constraint 2: A maximum total number of simulation hours, f, is specified This parameter reflects the maximum capacity to deliver simulation hours
For example, suppose that there are only two types of aircraft, A and B Suppose further that A has 100 flight hours, and B has 200 flight hours The losses associated with A are
$1.0 per hour, and with B are $0.5 per hour Suppose that we wish to allocate B = 10
flight hours We should first substitute flight hours for A, because each hour eliminated results in a cost avoidance of $1.0 If a = 5%, then we can substitute away up to 5 type A flight hours, and 10 type B flight hours In this case, we use up the maximum 5 type A flight hours, and an additional 5 type B flight hours for a total cost avoidance of $7.5 If a is increased to 10%, then we substitute away up to 10 type A flight hours, and 20 type B flight hours In this case we will exclusively use type A flight hours for a cost
avoidance of $10 because B = 10 If we increase the maximum simulation hours by
raising B to 20, we will first use 10 type A flight hours, and then 10 type B hours for a
savings of $15