Calibration of stochastic computer models

CALIBRATION OF STOCHASTIC COMPUTER MODELS YUAN JUN (B.Eng., Shanghai Jiao Tong University) A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF INDUSTRIAL & SYSTEMS ENGINEERING NATIONAL UNIVERSITY OF SINGAPORE 2013 DECLARATION I hereby declare that the thesis is my original work and it has been written by me in its entirely. I have duly acknowledged all the sources of information which have been used in the thesis. This thesis has also not been submitted for any degree in any university previously. yuanjun YUAN JUN 25 July 2013 Acknowledgements First and foremost, I would like to express my deepest and sincerest gratitude to my supervisor, A/Prof. NG Szu Hui, for her inspiration, encouragement, and guidance throughout my Ph.D study in the Department of Industrial and Systems Engineering at National University of Singapore. I am extraordinarily grateful for her patience, suggestions and comments to all of my research work. All these works would not have been possible without her efforts. I would also like to express my great appreciation to all my friends in Singapore for their continued help and support. Finally, i would like to thank my family for their support. Their love and encouragement have given me the strength to face challenges. iii Contents Acknowledgements .iii Summary vii List of Tables ix List of Figures . xi 1. INTRODUCTION . 1.1 Computer Model 1.1.1 Deterministic Computer Model . 1.1.2 Stochastic Computer Model 1.2 Computer Model Calibration, Validation and Prediction 1.3 Objective and Scope 1.4 Organization . 11 2. LITERATURE REVIEW 14 2.1 Review of Automatic Calibration Approach . 15 2.2 Review of Surrogate Based Calibration Approach 17 2.3 Review of Sequential Calibration Approach . 20 2.4 Review of Integrated Approach . 22 3. CALIBRATION OF STOCHASTIC COMPUTER MODELS USING STOCHASTIC APPROXIMATION METHODS . 24 3.1 Introduction 24 3.2 Modeling Details 27 3.2.1 Stochastic Model Formulation 27 3.2.2 Data . 29 3.2.3 Objective Function 29 3.3 Applying SA to Stochastic Computer Model Calibration . 30 3.3.1 Stochastic Approximation Methods 30 3.3.2 Application of SA for Stochastic Computer Model Calibration . 33 3.3.3 Selection of the Calibration Parameter Values . 39 3.3.4 Implementation Details . 41 3.4 Parameter Uncertainty and Predictive Uncertainty 46 iv 3.5 Examples 50 3.5.1 Chemical Kinetic Model Example 51 3.5.2 Adenoma Prevalence Microsimulation Model 54 3.5.3 Stochastic Biological Model . 60 3.6 Discussion 65 4. BAYESIAN CALIBRATION, VALIDATION AND PREDICTION OF STOCHASTIC COMPUTER MODELS . 67 4.1 Introduction 67 4.2 Model Formulation 69 4.2.1 Stochastic Model . 69 4.2.2 Gaussian Process Model 70 4.2.3 Prior Distributions for Parameters . 71 4.3 Calibration, Prediction and Validation 73 4.3.1 Observed Data . 73 4.3.2 Calibration . 73 4.3.3 Prediction 81 4.3.4 Validation 85 4.4 Examples 86 4.4.1 The Simple Quadratic Example 87 4.4.2 Mitochondrial DNA Deletions Example . 90 4.5 Discussion 93 5. SEQUENTIAL CALIBRATION APPROACH 95 5.1 Introduction 95 5.2 Effects of Different Initial Data . 98 5.3 A General Two Stage Sequential Calibration Approach . 103 5.4 EIMSPE Based Sequential Approach 104 5.4.1 EIMSPE Based Sequential Design 104 5.4.2 Examples . 108 5.5 Entropy Based Sequential Approach . 112 5.5.1 Entropy Criterion . 112 v 5.5.2 Both Computer Experiments and Real Process Runs are Available for Sequential Design . 114 5.5.3 Computation 117 5.6 Improved EIMSPE Based Sequential Approach . 118 5.7 Examples for Different Sequential Approaches 120 5.7.1 Simple Quadratic Example 120 5.7.2 Kinetic Model Example 124 5.8 Insights of Follow up Design Points’ Location for Different Criteria 129 5.9 The Combined Sequential Approach . 132 5.9.1 Significance Test Based on Kullback-Leibler Divergence . 135 5.9.2 Significance Test Procedure 137 5.9.3 Example for the Combined Approach . 138 5.10 Discussion 141 6. AN INTEGRATED APPROACH TO STOCHASTIC COMPUTER MODEL CALIBRATION, VALIDATION AND PREDICTION 143 6.1 Introduction 143 6.2 Integrated Calibration, Validation and Prediction Approach 145 6.2.1 Calibration . 146 6.2.2 Prediction 148 6.2.3 Validation 148 6.3 Implementation Details of the Integrated Approach 149 6.4 Case Study . 150 6.5 Discussion 154 7. CONCLUSION 155 7.1 Main Findings 155 7.2 Future Works . 158 References 160 Appendix A 168 Appendix B 170 vi Summary This thesis studies the calibration of stochastic computer models. Computer models are widely used to simulate complex and costly real processes and systems. When a computer model is used to predict the behavior of the real system for decision making, it is often important to calibrate the computer model so as to improve the model’s predictive accuracy. Calibration is a process to adjust the unknown input parameters in the computer model by comparing the computer model output with the real observed data so as to ensure that the computer model fits well to the real process. With the complexity of both the real process and the computer model, the available real observations and simulation data may be limited. Therefore, an effective and efficient calibration method is usually required to improve the calibration accuracy and predictive performance with limited data resources. This thesis first proposes an automated calibration approach for stochastic computer models based on the stochastic approximation (SA) method that can search for the optimum calibration parameter values accurately and efficiently. Moreover, an approach to quantify and account for the calibration parameter uncertainty in the subsequent application of the calibration model for prediction is further provided using asymptotic approximations. The results show that the proposed SA approach performs well in terms of calibration accuracy and significantly better in terms of computational search time compared with another direct calibration search method, the genetic algorithm. In order to use the limited data resources more efficiently when computer models are extremely time consuming and computationally expensive, and better quantify the various uncertainties, this thesis proposes a surrogate based Bayesian approach for stochastic computer model calibration and prediction. The proposed Bayesian approach is much more efficient as it uses the surrogates, where simpler and faster statistical approximations are used instead of the original complex computer models. Moreover, the proposed approach accounts for various uncertainties including the calibration parameter vii uncertainty in the follow up prediction and computer model analysis. The numerical results show the accuracy and efficiency of the proposed Bayesian calibration approach. Furthermore, in order to effectively allocate the limited data resources, a general two-stage sequential approach is proposed for stochastic computer model calibration and prediction. Different criteria are provided and studied in the proposed sequential approach and several examples are used to illustrate and compare their calibration and prediction performance. Other than calibration, it is also important to validate the computer model so as to assess the model’s predictive capability. Validation is a process to confirm whether the computer model precisely represents the real system. This thesis further provides a general framework to combine calibration, validation and prediction. Based on the proposed framework, an integrated approach is proposed for stochastic computer model calibration, validation and prediction. viii List of Tables 3.1. Mean and standard deviation of Q(θ) at the 100th iteration, iteration number before termination and operation time (in seconds) before termination, for n runs of each algorithm with the same starting in the chemical kinetic model example. 53 3.2. Mean and standard deviation of Q(θ) at the 100th iteration, iteration number before termination and operation time (in seconds) before termination for SPSA and GA, for n runs of each algorithm with the same starting in the adenoma prevalence microsimulation model example 57 3.3. The final calibration parameter values, simulated prevalence rates (including discrepancy term) and mean observed loss value (MOLV) for the SPSA algorithm under the two scenarios. . 57 3.4. t test results between two algorithms under observed loss value, iteration number and operation time (in seconds) measurements in adenoma prevalence microsimulaiton model. 57 3.5. Mean and standard deviation of Q(θ) at the 100th iteration, iteration number before termination and operation time (in hours) before termination for FDSA, SPSA and GA, for n runs of each algorithm with the same starting in the stochastic biological model. . 63 3.6. t test results between each pair of algorithms under observed loss value, iteration number and operation time (in hours) measurements in stochastic biological model. . 64 4.1. Mean, minimum and maximum ratios of the underestimated predictive variance using the proposed approach to the overall predictive variance using the MCMC approach among 100 evenly selected input points for 10 sets of random observed data. . 90 4.2. Means and 95% equal-tailed CIs of calibration parameters . 93 5.1. Mean (variance) of the plug-in parameters estimated for different variance and the target values obtained with sufficient data. 100 5.2. Mean (variance) of the plug-in parameters estimated for different number of initial data and the target values obtained with sufficient data. 102 5.3. Predicted EIMSPE and observed EIMSPE for both sequential design scenarios. . 111 5.4. Average posterior mean, mode and variance for different allocation choices. 126 5.5. Mean (variance) of the plug-in parameters estimated for different sequential approaches and the target values obtained with sufficient data. . 127 ix 5.6. Approximate values of the Kullback-Leibler divergence and the values obtained from the MCMC method for different number of follow up computer model design points. . 141 5.7. Average entropy and RMSPE values for entropy based sequential approach (T1), improved EIMSPE based sequential approach (T2) and combined approach (T3) with different additional runs. 141 x for calibration when the cost of computation is high. Particularly, SPSA is one of the best choices when the dimension of the calibration parameter is high. Furthermore, an approach is provided to estimate the calibration parameter uncertainty based on asymptotic normality. Then their effects on the prediction uncertainty are analyzed using the delta method. The SA method discussed in Chapter is an efficient and effective calibration approach which can be directly applied with limited data resources. However, this approach may not be efficient when the computer models are extremely time consuming and computational expensive as it usually requires a relatively large number of simulation runs before convergence. In this situation, the surrogate based methods are preferred as they are much more efficient by using the simpler surrogates instead of directly using the original complex computer models. In Chapter 4, a surrogate based Bayesian approach is proposed for stochastic computer model calibration and prediction. The proposed approach can account for various uncertainties including the calibration parameter uncertainty and it provides the analytical results for calibration and prediction. In the proposed approach, the posterior distribution of the calibration parameter and the predictive distributions for both the computer model outputs and the real process outputs are derived by combing all the available computer model data and the real observations. The predictive distribution for the inadequacy term is also derived which measures the discrepancy between the computer model and the real process. Two numerical examples are provided to illustrate the accuracy of the proposed Bayesian calibration approach. The numerical results indicate that the proposed approach performs well in both calibration and prediction. The approach proposed in Chapter is a one-time calibration approach. To further efficiently use the limited data resources to obtain more accurate calibration parameter and improve the model predictive performance, the 156 sequential calibration approach is more favorable as it can allocate resources more efficiently. In Chapter 5, a general two-stage sequential approach is proposed which can improve the calibration accuracy and the predictive performance by effectively allocating the limited resources. In this sequential approach, different criteria are provided and studied, including the EIMSPE criterion and the entropy criterion. First, an EIMSPE based sequential calibration approach is proposed which is to reduce the overall prediction error. The results from the two numerical examples show that the EIMSPE based sequential approach performs better than the one-stage approach in both calibration and prediction. Then an entropy based sequential approach is proposed for stochastic computer model calibration which is directly to improve the calibration accuracy. Moreover, an improved EIMSPE based sequential approach is provided to overcome the shortcomings of the EIMSPE based sequential approach. Two examples are used to illustrate the performance of the entropy based sequential approach and the improved EIMSPE based sequential approach by comparing with the EIMSPE based sequential approach. The results show that the improved EIMSPE based sequential approach has better calibration and prediction performance than the EIMSPE based sequential approach. The entropy based sequential approach always has the best calibration performance and sometimes even has better predictive performance. This is because that the entropy based sequential approach with more accurate calibration parameter value may also result in more accurate surrogate model. Hence better predictive performance can be obtained. As the predictive performance is usually the final objective of using the computer model, it is required to calibrate the computer model sufficiently but not expected to waste resources to over calibrate the computer model. Therefore, a combined approach that combines entropy criterion and the improved EIMSPE criterion is provided to improve the performance of both 157 calibration and prediction by effectively allocating the resources. The results of an illustrative example indicate the advantage of using the combined approach. When the computer model is used to represent and predict the real process or system’s behavior for better decision making, it is important to improve the calibration accuracy and assess the model validity so as to improve the predictive accuracy and capability. In Chapter 6, a general framework is provided to combine calibration, validation and prediction as a whole procedure. An integrated approach is then proposed to stochastic computer model calibration, validation and prediction based on this framework. A practical example is used to illustrate the integrated approach. The results show that the integrated approach performs well in calibration, validation and prediction. 7.2 Future Works The current work can be extended in several directions. 1. For the SA methods in Chapter 3, future works include considering the calibration parameter non-uniqueness problem in further detail and incorporating uncertainty and sensitivity analysis in the application of the calibration model. 2. For the Bayesian calibration approach in Chapter 4, it is assumed that the stochastic error has a constant variance and is independent of the design variable and the calibration parameter. This may not be the case in some situations. Therefore, one possible important future work is to take into account the dependency of the input parameters. 3. For the Bayesian calibration approach, the mathematically convenient conjugate priors are assumed for the unknown parameters. A more in depth study on the effects of different priors is also an area for further study. 4. In the proposed calibration approaches, a computer model is used to represent or predict the real process, and a GP model is used as a 158 surrogate model to the computer model for its generality and flexibility. However, in many real applications, there may be several possible computer models that can be used and the GP model may not be the best surrogate model in all situations. Therefore, the computer model form uncertainty and surrogate model uncertainty are important issues to be addressed in the future work. 5. In the proposed Bayesian calibration approach discussed in Chapter 4, some parameters are estimated. However, the results from Section 5.2 show that their estimation may not be accurate when the calibration parameter is not accurate and this may further influence the predictive performance. Therefore, one important future work is to provide a criterion that design on all the estimated parameters so as to better improve the model accuracy and predictive performance. 159 REFERENCES Ahmed, M. A., & Alkhamis, T. M. (2002). Simulation-based optimization using simulated annealing with ranking and selection. Computers & Operations Research, 29(4), 387-402. Andrieu, C., Moulines, E., & Priouret, P. (2005). Stability of stochastic approximation under verifiable conditions. SIAM Journal on Control and Optimization, 44(1), 283-312. Ang, A. H.-S., & Tang, W. H. (2007). Probability concepts in engineering: emphasis on applications to civil and environmental engineering. 2nd edn. New York: John Wiley. Ankenman, B., Nelson, B. L., & Staum, J. (2010). Stochastic kriging for simulation metamodeling. Operations Research, 58(2), 371-382. Arendt, P. D., Apley, D. W., & Chen, W. (2012). Quantification of model uncertainty: calibration, model discrepancy, and identifiability. Journal of Mechanical Design, 134(10). Asmussen, S., & Glynn, P. W. (2007). Stochastic simulation: algorithms and analysis. New York: Springer. Balakrishna, R., Antoniou, C., Ben-Akiva, M., Koutsopoulos, H. N., & Wen, Y. (2007). Calibration of microscopic traffic simulation models - methods and application. Transportation Research Record (1999), 198-207. Bastos, L. S., & O’Hagan, A. (2009). Diagnostics for Gaussian process emulators. Technometrics, 51(4), 425-438. Chandra, W., & Lin, D. K. J. (2012). Parameter calibration and inadequacy modeling in a computer experiment. Quality and Reliability Engineering International, 28(1), 35-46. 160 Cheng, R. C. H., & Holland, W. (1997). Sensitivity of computer simulation experiments to errors in input data. Journal of Statistical Computation and Simulation, 57(1-4), 219-241. Cho, J. J., Ding, Y., Chen, Y., & Tang, J. (2010). Robust calibration for localization in clustered wireless sensor networks. IEEE Transactions on Automation Science and Engineering, 7(1), 81-95. Cox, D. D., Park, J. S., & Singer, C. E. (2001). A statistical method for tuning a computer code to a data base. Computational Statistics & Data Analysis, 37(1), 77-92. Craig, P. S., Goldstein, M., Rougier, J. C., & Seheult, A. H. (2001). Bayesian forecasting for complex systems using computer simulators. Journal of the American Statistical Association, 96(454), 717-729. Cressie, N. A. C. (1993). Statistics for spatial data. New York : John Wiley. Currin, C., Toby, M., Morris, M., & Don, Y. (1991). Bayesian prediction of deterministic functions, with applications to the design and analysis of computer experiments. Journal of the American Statistical Association, 86(416), 953-963. Drignei, D. (2011). A general statistical model for computer experiments with time series output. Reliability Engineering & System Safety, 96(4), 460-467. Frank, M. V. (1999). Treatment of uncertainties in space nuclear risk assessment with examples from Cassini mission applications. Reliability Engineering & System Safety, 66(3), 203-221. Frazier, P., Powell, W. B., & Simao, H. P. (2009). Simulation model calibration with correlated knowledge-gradients. In Proceedings of the 2009 Winter Simulation Conference, 364-376, Institute of Electrical and Electronics Engineers, Inc., Piscataway, New Jersey. 161 Goldsman, D., & Nelson, B. L. (1998). Comparing systems via simulation, Handbook of simulation: principles, methodology, advances, application, and practice. Banks, J. Ed. New York: Wiley, 273-306. Greenwood, A., Vanguri, S., Eksioglu, B., Jain, P., Hill, T., Miller, J., & Walden, C. (2005). Simulation optimization decision support system for ship panel shop operations. In Proceedings of the 2005 Winter Simulation Conference, 2078-2086, Institute of Electrical and Electronics Engineers, Inc., Piscataway, New Jersey. Greenwood, P. E., & Nikulin, M. S. (1996). A guide to chi-squared testing. New York: Wiley. Guzman-Cruz, R., Castaneda-Miranda, R., Garcia-Escalanta, J. J., Lopez-Cruz, I. L., Lara-Herrera, A., & de la Rosa, J. I. (2009). Calibration of a greenhouse climate model using evolutionary algorithms. Biosystems Engineering, 104(1), 135-142. Han, G., Santner, T. J., & Rawlinson, J. J. (2009). Simultaneous determination of tuning and calibration parameters for computer experiments. Technometrics, 51(4), 464-474. Helton, J. C. (1994). Treatment of uncertainty in performance assessments for complex-systems. Risk Analysis, 14(4), 483-511. Henderson, D. A., Boys, R. J., Krishnan, K. J., Lawless, C., & Wilkinson, D. J. (2009). Bayesian emulation and calibration of a stochastic computer model of mitochondrial DNA deletions in substantia nigra neurons. Journal of the American Statistical Association, 104(485), 76-87. Higdon, D., Kennedy, M., Cavendish, J. C., Cafeo, J. A., & Ryne, R. D. (2004). Combining field data and computer simulations for calibration and prediction. SIAM Journal on Scientific Computing, 26(2), 448-466. 162 Higdon, D., Gattiker, J., Williams, B., & Rightley, M. (2008). Computer model calibration using high-dimensional output. Journal of the American Statistical Association, 103(482), 570-583. Hills, R. G. & Trucana, T. G. (2002). Statistical validation of engineering and scientific models: a maximum likelihood based metric. SAND2001-1783. Albuquerque, NM: Sandia National Laboratories. Jones, D. R., Schonlau, M., & Welch, W. J. (1998). Efficient global optimization of expensive black-box functions. Journal of Global Optimization, 13(4), 455-492. Kanso, A., Chebbo, G., & Tassin, B. (2006). Application of MCMC-GSA model calibration method to urban runoff quality modeling. Reliability Engineering & System Safety, 91(10-11), 1398-1405. Kennedy, M. C., & O'Hagan, A. (2001). Bayesian calibration of computer models. Journal of the Royal Statistical Society Series B-Statistical Methodology, 63, 425-450. Kiefer, J., & Wolfowitz, J. (1952). Stochastic estimation of the maximum of a regression function. Annals of Mathematical Statistics, 23(3), 462-466. Kleijnen, J. (2008). Design and analysis of simulation experiments. New York: Springer. Kong, C. Y., McMahon, P. M., & Gazelle, G. S. (2009). Calibration of disease simulation model using an engineering approach. Value in Health, 12(4), 521-529. Kumar A. (2008). Sequential calibration of computer models [dissertation]. Ohio: Ohio State University. Kushner, H. J., & Yin, G. (2003). Stochastic approximation and recursive algorithms and applications. New York: Springer. 163 Law, A. M., & McComas, M. G. (2001). How to build valid and credible simulation models. In B. A. Peters, J. S. Smith, D. J. Medeiros, and M. W. Rohrer, eds., In Proceedings of the 2001 Winter Simulation Conference, 22-29, Institute of Electrical and Electronics Engineers, Inc., Piscataway, New Jersey. Lee, J. B., & Ozbay, K. (2009). New calibration methodology for microscopic traffic simulation using enhanced simultaneous perturbation stochastic approximation approach. Transportation Research Record (2124), 233-240. Lindley, D. V. (1956). On a measure of the information provided by an experiment. Annals of Mathematical Statistics, 27(4), 986-1005. Liu, S. M., Butler, D., Brazier, R., Heathwaite, L., & Khu, S. T. (2007). Using genetic algorithms to calibrate a water quality model. Science of the Total Environment, 374(2-3), 260-272. Loeppky, J., Bingham, D., & Welch, W. (2006). Computer model calibration or tuning in practice. Available from: http://www.stat.ubc.ca/Research/TechReports/techreports/221.pdf Ma, J. T., Dong, H., & Zhang, H. M. (2007). Calibration of microsimulation with heuristic optimization methods. Transportation Research Record (1999), 208-217. Marti, R. (2003). Multi-start methods, Handbook of metaheuristics. Glover, F., & Kochenberger, G. Eds. New York: Springer, 355-368. Maryak, J. L., & Chin, D. C. (2008). Global random optimization by simultaneous perturbation stochastic approximation. IEEE Transactions on Automatic Control, 53(3), 780-783. Massey, F. J. (1951). The Kolmogorov-Smirnov test for goodness of fit. Journal of the American Statistical Association, 46(253), 68-78. 164 Matott, L. S., & Rabideau, A. J. (2008). Calibration of complex subsurface reaction models using a surrogate-model approach. Advances in Water Resources, 31(12), 1697-1707. Nelson, B. L., Swann, J., Goldsman, D., & Song, W. M. (2001). Simple procedures for selecting the best simulated system when the number of alternatives is large. Operations Research, 49(6), 950-963. O'Hagan, A. (2006). Bayesian analysis of computer code outputs: a tutorial. Reliability Engineering & System Safety, 91(10-11), 1290-1300. Oberkampf, W. L., & Roy, C. J. (2010). Verification and validation in scientific computing. Cambridge University Press. Qian, P. Z. G., & Wu, C. F. J. (2008). Bayesian hierarchical modeling for integrating low-accuracy and high-accuracy experiments. Technometrics, 50(2), 192-204. Rao, S. & Balakrishnan, N. (1999). Computational electromagnetic – a review. Current Science, 77, 1343-1347. Robbins, H., & Monro, S. (1951). A stochastic approximation method. Annals of Mathematical Statistics, 22(3), 400-407. Rutter, C. M., Miglioretti, D. L., & Savarino, J. E. (2009). Bayesian calibration of microsimulation models. Journal of the American Statistical Association, 104(488), 1338-1350. Sacks, J., Welch, W. J., Toby, J. M., & Wynn, H. P. (1989). Design and analysis of computer experiments. Statistical Science, 4(4), 409-423. Saltelli, A., Chan, K., & Scott, E. M. (2000). Sensitivity analysis. New York: Wiley. Santner, T. J., Notz, W., & Williams, B. J. (2003). Design and analysis of computer experiments. New York: Springer. 165 Senarath, S. U. S., Ogden, F. L., Downer, C. W., & Sharif, H. O. (2000). On the calibration and verification of two-dimensional, distributed, Hortonian, continuous watershed models. Water Resources Research, 36(6), 14951510. Shewry, M. C., & Wynn, H. P. (1987). Maximum entropy sampling. Journal of Applied Statistics, 14(2), 165-170. Song, K. S. (2002). Goodness-of-fit tests based on Kullback-Leibler discrimination information. IEEE Transactions on Information Theory, 48(5), 1103-1117. Spall, J. C. (1992). Multivariate stochastic-approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, 37(3), 332-341. Spall, J. C. (2000). Adaptive stochastic approximation by the simultaneous perturbation method. IEEE Transactions on Automatic Control, 45(10), 1839-1853. Spall, J. C. (2003). Introduction to stochastic search and optimization: estimation, simulation, and control. Hoboken, N.J: Wiley-Interscience. Strul, H. et al. (2006). The prevalence rate and anatomic location of colorectal adenoma and cancer detected by colonoscopy in average-risk individuals aged 40-80 years. American Journal of Gastroenterology, 101(2), 255-262. Trucano, T. G., Swiler, L. P., Igsua, T., Oberkampf, W. L., & Pilch, M. (2006). Calibration, validation, and sensitivity analysis: what’s what. Reliability Engineering & System Safety, 91(10-11), 1331-1357. Vasicek, O. (1976). A test for normality based on sample entropy. Journal of the Royal Statistical Society. Series B (Methodological), 38(1), 54-59. Wang, S. C., Chen, W., & Tsui, K. L. (2009). Bayesian validation of computer models. Technometrics, 51(4), 439-451. 166 Watson, C., & Johnson, M. (2004). Hurricane loss estimation models: Opportunities for improving the state of the art. Bulletin of the American Meteorological Society, 85, 1713-1726. Wei, G. C. G., & Tanner, M. A. (1990). A monte-carlo implementation of the EM algorithm and the poor man’s data augmentation algorithms. Journal of the American Statistical Association, 85(411), 699-704. Williams, B. J., Loeppky, J. L., Moore, L. M., & Macklem, M. S. (2011). Batch sequential design to achieve predictive maturity with calibrated computer models. Reliability Engineering & System Safety, 96(9), 12081219. Yin, G. (1999). Rates of convergence for a class of global stochastic optimization algorithms. SIAM Journal on Optimization, 10(1), 99-120. Yin, J., Ng, S. H., & Ng, K. M. (2011). Kriging metamodel with modified nugget-effect: the heteroscedastic variance case. Computers & Industrial Engineering, 61(3), 760-777. Zhang, X. S., Srinivasan, R., Zhao, K. G., & Van Liew, M. (2009). Evaluation of global optimization algorithms for parameter calibration of a computationally intensive hydrologic model. Hydrological Processes, 23(3), 430-441. 167 APPENDIX A Proof of Proposition 4.1 First, integrate out β: p (θ , φ , σ | d ) = ∫ p (θ , β , σ , φ | d )d β ∝ p (θ ) p (φS ) p (φδ ) p (σ ε2 ) p (σ e2 ) p (σ S2 ) p (σ δ2 ) Vd (θ ) ∫ p(β | (σ S −1/2 , σ δ2 )) exp  − {(d − H (θ ) β )T Vd (θ ) −1 (d − H (θ ) β )} / d β where ∫ p(β | (σ S ∝ σ S2V , σ δ2 )) exp  − {(d − H (θ ) β )T Vd (θ ) −1 (d − H (θ ) β )} / d β ∫ exp − {(β − b) −1/2 T (σ S2V ) −1 ( β − b)} / − {(d − H (θ ) β )T Vd (θ ) −1 (d − H (θ ) β )} /  d β (Collect terms that dependent on β and group other constant terms) =σ S2V −1/2 ∫ exp −β ∝ σ S2V exp  −bT (σ S2V ) −1 b / − d T Vd (θ ) −1 d /  T −1/2 A−1β / + υ T β d β exp  −bT (σ S2V ) −1 b / − d T Vd (θ ) −1 d /  A 1/2 exp υ T Aυ /  with A-1=[H(θ)TVd(θ)-1H(θ)+(σS2V)-1] and υ=[H(θ)TVd(θ)-1d+(σS2V)-1b]. To prove ∫ exp  − β T A−1β / + υ T β d β ∝ A 1/2 exp υ T Aυ /  , we consider a multivariate normal distribution NN+n(Aυ,A). As 1/2 (2π )( N + n )/2 | A |= = ∫ exp −(β − Aυ ) T A−1 ( β − Aυ ) / d β T −1 T T ∫ exp −β A β / + υ β − υ Aυ / 2d β , rearranging the terms gives the results. Second, integrate out σS2: collecting all the terms that dependent on σS2 and integrating out it we obtain 168 p (θ , φ ,τ 12 ,τ 22 ,τ 32 | d ) = ∫ p (θ , φ , σ ε2 , σ δ2 , σ e2 , σ S2 | d )dσ S2 ∝ p (θ ) p (φS ) p (φδ ) ∫ p (σ S2 ) p (σ S2τ 12 ) p (σ S2τ 22 ) p (σ S2τ 32 ) Vd (θ ) −1/2 σ S2V −1/2 A 1/2 exp  −bT (σ S2V ) −1 b / − d T Vd (θ ) −1 d /  exp υ T Aυ /  dσ S2 ∝ p (θ ) p (φS ) p (φδ ) ∫ (σ S2 ) −α S −1 exp  −γ S / σ S2  (σ S2τ 12 ) −αε −1 exp  −γ ε / (σ S2τ 12 )  (σ S2τ 22 ) −αδ −1 exp  −γ δ / (σ S2τ 22 )  (σ S2τ 32 ) −αe −1 exp  −γ e / (σ S2τ 32 )  (σ S2 ) − ( N + n )/2 Vd (θ ) (σ S2 ) − ( kS + kδ )/2 V −1/2 (σ S2 )( kS + kδ )/2 A −1/2 1/2 exp  −(bT V −1b + d T Vd (θ ) −1 d − υ T Aυ ) / (2σ S2 )  dσ S2 = p (θ ) p (φS ) p (φδ )(τ 12 ) −αε −1 (τ 22 ) −αδ −1 (τ 32 ) −αe −1 Vd (θ ) −1/2 V −1/2 A 1/2 ∫ (σ −α S −αε −αδ −α e − − ( N + n )/2 S ) exp  −(γ S + γ ε / τ 12 + γ δ / τ 22 + γ e / τ 32 +(bT V −1b + d T Vd (θ ) −1 d − υ T Aυ ) / 2) / σ S2  dσ S2 ∝ p (θ ) p (φS ) p (φδ )(τ 12 ) −αε −1 (τ 22 ) −αδ −1 (τ 32 ) −αe −1 Vd (θ ) −1/2 VS +δ −1/2 A 1/2 (γ S + γ ε / τ 12 + γ δ / τ 22 + γ e / τ 32 +(bT V −1b + d T Vd (θ ) −1 d − υ T Aυ ) / 2) −α S −αε −αδ −αe −3−( N + n )/2 −1 where A  H (θ )T Vd (θ ) −1 H= = (θ ) + V −1  , υ  H (θ )T Vd (θ ) −1 d + V −1b  . 169 APPENDIX B Proof of Proposition 4.5 Given that ( S ( x0 ), y T )T is a multivariate normal distribution given the parameters βS, σS2, ϕS, τ12 and θ, we can obtain the conditional distribution of S(x0) based on y and these parameters: S ( x0 ) | y , β S , σ S2 , φS ,τ 12 , θ ~ N ( hS ( x0 , θ )T β S + RS ( ( x0 , θ ), DS ) ( R (D , D ) + τ S S S N I ) (y −H −1 S ( DS ) β S ) , ( σ S2 − RS ( ( x0 ,θ ), DS ) ( RS ( DS , DS ) + τ 12 I N ) RS ( DS , ( x0 ,θ ) ) −1 )) This gives us p ( S ( x0 ) | y , β S , σ S , φS ,τ 12 , θ ) . Assume that ϕS and τ12 are known, we can obtain the predictive distribution of S(x0) conditional on y , ϕS, τ12 and θ by integrating out βS and σS2: p ( S ( x0 ) | y , φS ,τ 12 , θ ) = ∫ p ( S ( x0 ) | y , β S , σ S2 , φS ,τ 12 , θ ) p ( β S , σ S | y , φS , τ , θ ) d β S d σ . (B.1) S As βS and σS2 are assumed to be independent of θ, p(βS,σS2| y ,ϕS,τ12,θ) in the right hand side of Equation (B.1) can p ( β S , σ S2 | y , φS ,τ 12 ) ∝ p ( β S , σ S2 ) p ( y | β S , σ S2 , φS ,τ 12 ) . be Since obtained p(βS,σS2) from and p( y |βS,σS2,ϕS,τ12) are known, the form of (B.1) can be obtained. To integrate out βS and σS2 is similar in spirit to that of Appendix A. After integrating out βS and σS2, we have Proposition 4.5. 170 LIST OF PUBLICATIONS Yuan, J., & Ng, S. H. (2013). An entropy based sequential calibration approach for stochastic computer models. To appear in the Proceedings of the Winter Simulation Conference 2013. Yuan, J., & Ng, S. H. (2013). A sequential approach for stochastic computer model calibration and prediction. Reliability Engineering & System Safety, 111(0), 273-286. Yuan, J., Ng, S. H., & Tsui, K. L. (2013). Calibration of stochastic computer models using stochastic approximation methods. IEEE Transactions on Automation Science and Engineering, 10(1), 171-186. Yuan, J., Han, C. W., & Ng, S. H. (2012). Applying the informational approach to global optimization to the homoscedastic stochastic simulation. In Proceedings of IEEE International Conference on Industrial Engineering and Engineering Management, 1676-1680, Institute of Electrical and Electronics Engineers, Inc., Piscataway, New Jersey. Yuan, J., & Ng, S. H. (2011). Bayesian calibration of stochastic computer models. In Proceedings of IEEE International Conference on Industrial Engineering and Engineering Management, 1695-1699, Institute of Electrical and Electronics Engineers, Inc., Piscataway, New Jersey. Yuan, J., Ng, S. H., & Tsui, K. L. (2010). Application of stochastic approximation methods for stochastic computer models calibration. In Proceedings of IEEE International Conference on Industrial Engineering and Engineering Management, 1606-1610, Institute of Electrical and Electronics Engineers, Inc., Piscataway, New Jersey. 171 [...]... general model calibration, validation and prediction framework In Chapter 6, we will discuss the integrated approach for stochastic computer model calibration, validation and prediction 23 Chapter 3 CALIBRATION OF STOCHASTIC COMPUTER MODELS USING STOCHASTIC APPROXIMATION METHODS 3.1 Introduction In this chapter, an automatic calibration approach is proposed for stochastic computer model Computer models are... by Yin et al (2011) and the stochastic kriging model proposed by Ankenman et al (2010) have been proposed to deal with stochastic computer models 1.2 Computer Model Calibration, Validation and Prediction With continually increasing computer power, analysis of real systems relies more and more heavily on computer models Computer models are often used to predict the behavior of the real system so as to... proposed for computer model calibration and prediction so as to better use the data resources Kumar (2008) proposed a sequential calibration approach based on the deterministic computer model As the stochastic computer model calibration is much harder to analyze, the sequential calibration approach should further be proposed to solve the stochastic computer model calibration problem • Computer models are... model analysis and consider the effects of the stochastic error that inherent the stochastic computer model, and better use the limit data to improve the calibration and prediction performance The results of this study may provide more insights into the development and statistical analysis of stochastic computer models, especially for stochastic computer model calibration It is understood that in some... outputs from a stochastic model may be different for the same input levels In most applications, the real systems of interest are often stochastic in nature The stochastic models are usually required to assess the probability distribution of the outcome of interest and the expected output is a typical measure of performance of such systems With the increasing application of the stochastic computer model,... measurement of deletion accumulation; the circles denote the real observations and dots denote the predictive means 153 xii Chapter 1 INTRODUCTION This thesis contributes to the calibration of stochastic computer models The background of computer model and the differences between deterministic and stochastic computer models are first introduced in Section 1.1 In Section 1.2, computer model calibration, ... distribution of the stochastic simulation output Therefore, the stochastic computer models can be much more difficult and time consuming to analyze than the deterministic computer models This 4 motivates the using of surrogate model to solve the stochastic computer model problems In recent years, the surrogate model designed for the deterministic situations has been extended to take accounts of the stochastic. .. sequential calibration approach for stochastic computer model Overall, this thesis propose different efficient calibration approaches for stochastic computer model calibration when the computer model is time consuming to run and limit data resources are available More specifically, this thesis extend the current approaches to further solve the stochastic computer model calibration problems, quantify the calibration. .. deterministic computer model problems 1.1.2 Stochastic Computer Model Different from the deterministic computer models, there is randomness in the stochastic computer models and simulation outputs from a stochastic model may be different for the same input level In most applications, the real systems of interest, such as complex engineering systems and expensive commercial systems, are often stochastic. .. proof of the feasibility and convergence of the SA methods for general stochastic simulation calibration under various conditions needs to be further discussed In addition, the effect of the calibration parameters' uncertainty on the calibrated computer model's outputs needs to be quantified In Chapter 3, we will discuss the details of applying the stochastic approximation methods to stochastic computer . SA to Stochastic Computer Model Calibration 30 3.3.1 Stochastic Approximation Methods 30 3.3.2 Application of SA for Stochastic Computer Model Calibration 33 3.3.3 Selection of the Calibration. calibration of stochastic computer models. Computer models are widely used to simulate complex and costly real processes and systems. When a computer model is used to predict the behavior of. stochastic computer models. The background of computer model and the differences between deterministic and stochastic computer models are first introduced in Section 1.1. In Section 1.2, computer