Optimal computing budget allocation for multi objective simulation optimization

OPTIMAL COMPUTING BUDGET ALLOCATION FOR MULTI-OBJECTIVE SIMULATION OPTIMIZATION LI JUXIN (B.Eng., Shanghai Jiao Tong University, China) A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF INDUSTRIAL & SYSTEMS ENGINEERING NATIONAL UNIVERSITY OF SINGAPORE 2012 Declaration I hereby declare that this thesis is my original work and it has been written by me in its entirety. I have duly acknowledged all the sources of information which have been used in the thesis. This thesis has also not been submitted for any degree in any university previously. Li Juxin Aug. 2012 i Acknowledgements First and foremost, I would like to express my sincere gratitude to my two supervisors, Associate Professor Lee Loo Hay and Associate Professor Chew Ek Peng. Their constructive advice, invaluable support and patient guidance throughout the whole course of my candidature have been of great value for me. The study reported in this thesis would not have been possible without their supervision. Deepest gratitude are also due to all the other faculty members of the Department of Industrial and Systems Engineering, from whom the knowledge and insights gained have helped in a number of ways in my research. Special thanks to all my graduate friends, especially Zhou Qi, Wang Qiang, Chen Liqin, Fu Yinghui, Bae Minju and Nugroho Pujowidianto, for sharing the literature and ideas, and rendering invaluable assistance. I am deeply indebted to my parents, for their understanding, unconditional love and support through the duration of my study. This dissertation is dedicated to them. ii Contents Declaration i Acknowledgements ii Summary vi List of Tables viii List of Figures ix Symbols and Nomenclature x Introduction 1.1 Objectives of the Study . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Significance of the Research . . . . . . . . . . . . . . . . . . . . . 1.3 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . Literature Review 2.1 Simulation Optimization . . . . . . . . . . . . . . . . . . . . . . . 2.1.1 Ranking and Selection and Computing Budget Allocation 2.2 Computing Budget Allocation Problems on Finite Sets . . . . . . 2.2.1 Classification of Problems . . . . . . . . . . . . . . . . . . 2.2.2 Solution Approaches . . . . . . . . . . . . . . . . . . . . . 2.3 Computing Budget Allocation Strategies . . . . . . . . . . . . . . 2.3.1 Problems with a Single Performance Measure . . . . . . 2.3.2 Problems with Multiple Performance Measures . . . . . . 2.3.3 Summary of the Works . . . . . . . . . . . . . . . . . . . . 2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 10 10 12 15 15 19 21 23 Finding a Subset of Good Systems for Multi-objective Simulation Optimization on Finite Sets 25 iii 3.1 3.2 3.3 3.4 3.5 3.6 Introduction . . . . . . . . . . . . . . . . . . . . . 3.1.1 Problem Statement . . . . . . . . . . . . 3.1.2 Organization . . . . . . . . . . . . . . . . Preliminaries . . . . . . . . . . . . . . . . . . . . Probability of Correct Selection . . . . . . . . . Computing Budget Allocation Strategy . . . . . 3.4.1 An Approximate Closed-form Solution 3.4.2 A Sequential Allocation Procedure . . . Numerical Experiments . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Optimal Computing Budget Allocation to Select the Non-dominated Systems: a Large Deviations Perspective 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . 4.1.2 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Notations and Assumptions . . . . . . . . . . . . . . . . . . . . . . 4.3 Rate Function of the Probability of False Selection . . . . . . . . 4.4 The Optimal Allocation Strategy . . . . . . . . . . . . . . . . . . . 4.4.1 Optimal Allocation Strategy Using a Solver . . . . . . . . 4.4.2 Optimality Conditions . . . . . . . . . . . . . . . . . . . . 4.5 The Multivariate Normal Case . . . . . . . . . . . . . . . . . . . . 4.5.1 Optimal Sampling Allocation Using a Solver . . . . . . . 4.5.2 An Approximate Closed-form Solution to Sampling Allocations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.5.3 Closed-form Solutions to the Nested Optimization Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Numerical Experiments . . . . . . . . . . . . . . . . . . . . . . . . 4.7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Combining Computing Budget Allocation with Multi-objective Optimization via Simulation 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.1.1 Objectives of This Study . . . . . . . . . . . . . . . . . . . 5.1.2 Organization . . . . . . . . . . . . . . . . . . . . . . . . . . 5.2 Multi-objective Evolutionary Algorithms . . . . . . . . . . . . . . 5.2.1 Challenges for Multi-objective Optimization via Simulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iv 25 26 27 27 29 33 33 37 38 52 54 54 56 57 57 58 62 63 64 66 67 68 70 73 80 82 82 84 85 85 87 5.3 5.4 5.5 Combination of MOEAs and Computing Budget Allocation . . 5.3.1 Multi-objective Genetic Algorithms with Optimal Computing Budget Allocation . . . . . . . . . . . . . . . . . 5.3.2 Multi-objective Estimation of Distribution Algorithms with Optimal Computing Budget Allocation . . . . . . 5.3.3 Discussions on the Convergence of the Combined Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . Numerical Experiments . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 The Experiments Scheme . . . . . . . . . . . . . . . . . 5.4.2 Performance Metrics . . . . . . . . . . . . . . . . . . . . 5.4.3 Test Problems . . . . . . . . . . . . . . . . . . . . . . . . 5.4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 . 90 . 92 . . . . . . . 93 94 94 95 96 98 105 Conclusions 107 6.1 Conclusions of the Study . . . . . . . . . . . . . . . . . . . . . . . 107 6.2 Discussions and Future Research . . . . . . . . . . . . . . . . . . . 109 Bibliography 112 v Summary Complex systems are very common in real world situations and multiple performance measures are usually of interest. Simulation has been widely employed in evaluating these systems and selecting the desired ones. Performances of these systems are frequently stochastic in nature and therefore selection based on simulation output bears uncertainty. Correct selection would require considerable sampling from simulation models. However, simulation runs of complex systems tend to be expensive and simulation budget is often limited. It is therefore vital to determine an optimal sampling allocation strategy such that the desired systems can be correctly selected with the highest evidence. This thesis describes how computing budget allocation concerns are addressed in the multi-objective simulation optimization context. The concept of Pareto optimality is incorporated to resolve the trade-offs among the multiple competing performance measures, where preferences of the decision maker are not required. Evidence of correct selection is maximized through mathematical programming models that are built from either a probability or a large deviations perspective. Finite time performance and asymptotic properties of the proposed strategies are both investigated. The problem of finding a subset of good systems from a finite set is first studied under a multi-objective simulation optimization context. The alternative systems are measured by their ranks to be Pareto-optimal, often referred to as the domination counts within the finite set. Probability of correct selection is used as the evidence of correct selection, and the objective is to determine an optimal computing budget allocation that maximizes this probability. Bonferroni bounds are employed to provide estimates for the probability, from which asymptotic allocation strategies are derived assuming multivariate normally distributed samples. The efficacy of the proposed allocation schemes in finite time are illustrated through numerical examples. vi To develop sampling laws in a general context and resolve the possible suboptimality brought by probability bounds into the sampling laws, the problem of selecting the non-dominated systems is revisited from a large deviations perspective. Focusing on the asymptotic rate of decay of the probability of incorrect selection rather than the probability itself, a mathematically robust formulation of the problem is established to determine the optimal computing budget allocation that maximizes the rate of decay. Sampling correlations are explicitly modelled into the related rate functions. The optimal sampling allocation is proposed to be computed using numerical solvers in a general context. The formulation and the solution approach are then applied to problems under multivariate normal assumptions, for which rate functions are well-defined. An approximate closed-form solution to sampling allocation which is computationally more efficient is also suggested as an alternative to the solution approach using a solver, while both approaches explicitly characterize sampling correlations. Numerical examples illustrate the benefit gained in terms of convergence rate by the proposed solution approaches. This study also deals with extending the optimal computing budget allocation strategies on finite sets to optimization via simulation problems with a relatively large solution space. Population-based search heuristics, for instance, evolutionary algorithms, are usually employed to drive the search for multi-objective optimization via simulation problems. Computing budget allocation techniques are embedded into iterations of the select population-based search algorithms, targeting for a higher confidence in selecting promising systems for reproduction. Efficacy and efficiency enhancement is demonstrated numerically in terms of convergence and coverage measures for these search heuristics. The findings may suggest the great potential in search quality and speed that can be gained from designing algorithm-specific sampling laws for population-based search heuristics. Overall, the study reported in this thesis provides effective and efficient allocation strategies for decision makers who are faced with limited budget to simulate complex and stochastic systems. While these allocation strategies asymptotic in nature, numerical experiments illustrate that the proposed methods also provide robust and reliable performances in finite time. vii List of Tables 2.1 A summary table of the literature on R&S . . . . . . . . . . . . . 22 3.1 3.2 3.3 Computing budget allocation for Experiment . . . . . . . . . . . 44 Computing budget allocation for Experiment . . . . . . . . . . . 45 Computing budget allocation for Experiment . . . . . . . . . . . 47 4.1 4.2 4.3 4.4 4.5 Means for Experiment . . . . . . . . . . . . . . . . . . Sampling allocations and rates for Experiment . . . . Means for Experiment . . . . . . . . . . . . . . . . . . Sampling allocations and rates for Experiment . . . . Relative differences in rate and time for Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 75 76 78 79 5.1 5.2 5.3 5.4 Running settings of MOGAs for Test Problem . Running settings of MOGAs for Test Problem . Running settings of MOEDAs for Test Problem Running settings of MOEDAs for Test Problem . . . . . . . . . . . . . . . . . . . . . . . . 99 100 102 103 viii . . . . . . . . . . . . List of Figures 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 Probability of correct selection for Experiment . . . . . . . . . . Probability of correct selection for Experiment with correlation Spread of systems for Experiment . . . . . . . . . . . . . . . . . Probability of correct selection for Experiment . . . . . . . . . . Spread of systems for Experiment . . . . . . . . . . . . . . . . . Probability of correct selection for Experiment . . . . . . . . . . Spread of systems for Experiment . . . . . . . . . . . . . . . . . Probability of correct selection for Experiment . . . . . . . . . . Probability of correct selection for Experiment 5: the neutral case Probability of correct selection for Experiment 5: the flat case . . Probability of correct selection for Experiment 5: the steep case . 4.1 Rate of decay for Experiment . . . . . . . . . . . . . . . . . . . . 76 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 5.11 5.12 The flow chart for the MOGA + MOCBA framework . . . . . The flow chart for the MOEDA + MOCBA-subset framework Objective space and Pareto front for Test Problem . . . . . . Objective space and Pareto front for Test Problem . . . . . . Convergence measures of MOGAs for Test Problem . . . . Coverage measures of MOGAs for Test Problem . . . . . . Convergence measures of MOGAs for Test Problem . . . . Coverage measures of MOGAs for Test Problem . . . . . . Convergence measures of MOEDAs for Test Problem . . . . Coverage measures of MOEDAs for Test Problem . . . . . . Convergence measures of MOEDAs for Experiment . . . . Coverage measures of MOEDAs for Test Problem . . . . . . ix . . . . . . . . . . . . . . . . . . . . . . . . 40 41 42 43 45 46 47 48 49 50 51 91 93 97 98 99 100 101 101 102 103 104 104 Chapter Conclusions 6.1 Conclusions of the Study This study explores the sampling allocation strategy for multi-objective simulation optimization problems. In view of the practical needs within a multiobjective simulation optimization context, the desired systems are defined either as a good subset containing promising systems for reproduction, or as the non-dominated systems. The goal of the study is to find an optimal computing budget allocation that can maximize the evidence of correct selection. This evidence can either be expressed in a probability term or indirectly in terms of the rate of decay from a large deviations perspective. Specifically for the subset selection problem, we employ the probability of correct (or false) selection as an evidence and aim to find a sampling allocation that can maximize a lower bound estimate of this probability. Asymptotic analyses are applied to establish approximate closed-form sampling allocation strategies. Numerical experiments illustrate that the proposed allocation schemes can generally obtain a significantly higher empirical probability of correct selection than the compared allocation does. In other words, to maintain the same pre-specified confidence level of correct selection, our rules can make substantial savings of computing budget than the compared alternative allocations. This observation provides clear evidence that the proposed allocation rules can utilize the available samples more efficiently, without significant compromise in computing the allocation. A plausible explanation of such findings is that the proposed OCBA rule can learn the sample information during the sequential allocation procedure and benefit from intelligently adjusting the allocation scheme with updated in107 formation. This study has taken a major step towards allocating limited computing resource in an optimal manner for multi-objective subset selection problems and has provided implementation guidelines for practitioners. Another contribution of this study is that it can enhance the search efficiency when integrated with multi-objective search algorithms, for which an elite set is needed in the intermediate iterations to reproduce promising solutions. The proposed methodology provides a powerful technique for generating a set of seeding solutions for genetic algorithms or evolutionary algorithms for problems with multiple performance measures. This study also examines the Pareto set selection problem from a large deviations perspective. A robust mathematical framework is employed and the problem is formulated into an non-linear programming problem with nested optimization problems, targeting a sampling allocation that can maximize the rate of decay of the probability of false selection. The optimal solution to the formulated NLP problem can be secured from using solvers in a general context. Optimality conditions to the NLP formulation are also presented by exploiting the problem structure. For problems with a multivariate normal assumption in particular, an approximate closed-form solution is suggested by elaborating distribution-specific characteristics in the derivation. This approximate solution can serve as an alternative to the solver approach while providing noncompromising solutions efficiently and reliably. Moreover, closed-form solutions are provided to the nested optimization problems for the NLP formulation, which can further lessen the computing time when embedded into the solver approach or the approximate heuristic approach. Numerical results also justify the advantage of the proposed allocation schemes. The solutions from using a solver (MOCBA*) and the approximate MOCBA+ approach can significantly outperform other allocation schemes for a set of problems with multiple competing objectives. This is probably attributed to the fact that all the objectives are taken into consideration for comparison while using large deviations theory, whereas the MOCBA approach in Lee et al. (2010c) considers only the most dominating objective and thus loses certain information essential for identifying systems. Another contribution is that we explicitly characterize the sampling correlations between performance measure by using large deviations. This correlation is usually dissolved by the Bonferroni bounds of probabilities and is therefore implicitly considered. By exploiting the rates of decay of the probability of false 108 selection from a large deviations perspective, sampling correlations appear in related rate terms and are therefore explicitly characterized in the problem formation and consequently, in the solution. By investigating sampling allocations using a mathematically more robust framework, this study is significant in that the commonly accepted assumption of independent sampling between objectives can be ruled out in theory and application. Numerical tests indicate that allocation strategies incorporating sampling allocations manage to guarantee a faster convergence rate, compared with those sampling rules that not consider the effect of correlation on the sampling allocation. The findings have also provided valuable insight into the interpretation of the allocation rules in terms of the rate of decay of the probability of false selection that is targeted to minimize. The underlying intelligence adherent to the allocation strategy can therefore be better explained. Moreover, the proposed approach may have great potential in application since it does not require any interaction from the decision maker to find the desired systems. 6.2 Discussions and Future Research Being an exploratory study, this work makes several assumptions, which may not always hold in practice. For example, constraint measures are not considered for the targeted problem settings, which may restrict the application of the allocation scheme to some extent. Although the duality between constrained optimization and multi-objective optimization is well known, research work should take into consideration the deficiency of such dualities in term of problem size scaling and non-equivalence (Deb, 2001). In the aspect of generality of problem settings, further study is needed to incorporate constraints on system performances into modelling. To achieve this, the definition of evidence of correct selection should be systematically explored, due to the additional complexity introduced by the constraint measures in detecting feasibility. The problem settings assume that all performance measures are following a certain sampling distribution. There are often cases that the performances of interest include both discrete and continuous ones and they are not necessarily following a joint distribution. To extend this study to more general cases, estimation of the sampling distribution is critical. Machine learning techniques may be employed to explore the problem structure. For example, feature selec109 tion can be applied to find those relevant measures and cluster them into groups. Pattern recognition techniques can then be used to identify the pattern or distributions within group and the correlations among groups. As a promising venue for further research, it would be interesting to extend the proposed approaches in this study to a more general context. This study assumes that no systems are having objectives equal to that of other systems, since noises make it impossible to distinguish these systems from simulation output. Future studies may incorporate the indifference-zone (IZ) concept into a new problem formation to resolve this issue. The indifference zone parameter may be determined by the minimum detectable difference for the finite set of systems. Another fundamental assumption is that the systems are ranked by means only. Variance may also be of interest for decision making under uncertainty. While our analysis being asymptotic in nature partially address this concern, there are also instances where a decision maker will prefer a balance between the projected mean and the variability from a finite time perspective. Ranking of systems considering both mean and variance information are reported for the indifference-zone ranking and selection problems (Batur and Choobineh, 2010a,b), and it would be interesting to incorporate this concept into multiobjective simulation optimization. From a methodological perspective, it is noted that the suggested sampling allocation schemes are derived based on asymptotic analyses, or the a large number of computing budget, while the motivation of the sampling allocation comes from a limited computing budget context. For finite time analyses, the probability of correct selection is often guaranteed by very conservative bounds. Gradient-based heuristics are usually then developed to estimate a sampling allocation. For example, the indifference-zone approach requires a problemspecific parameter delta, or the minimum detectable difference, to derive the allocation procedures. Such procedures tend not to be robust and cannot provide insights or interpretation of the sampling scheme. On the other hand, the asymptotic analysis allows for a large deviations perspective to look into the decaying property of the probability of false selection. The optimal rate of decay can then be achieved. The asymptotic optimality separates the impact of finitetime factors and gives the trend of the sampling allocation when the computing budget is increased. This optimality also provides insights and interpretations of the sampling scheme. It is an intuitive philosophical testing to compare the 110 rules from asymptotic analyses with those heuristics and the suggested procedures prove to be both robust and efficient in finite-time scenarios. Empirical results of finite-time simulation optimization experiments justify this testing and verify the trend to the optimal rate. It is noted that for the suggested procedures to achieve the expected performance in carrying out practical experiments, the general guideline for choosing the proper parameters applies. If the budget is even not sufficient enough to initialize the samples with equal replicates, the suggested procedures would terminate prematurely and degrade to equal allocation in such cases. Although we provide a general solution framework for sampling allocation, we not give explicit sampling laws for general cases. The explicit sampling laws presented in this study apply to problems under a multi-variate assumption, which is a common practice in the literature. For problems with samples following other distributions, the proposed sampling rules for the MVN case need to be carefully adapted to the case by using batch means or other methodologies (Hart et al., 2005). It would also be interesting to examine explicit sampling laws for problems with other particular distributions or general distributions. The main challenge is to estimate the distribution pattern or the rate functions. It is acknowledged that sampling correlations between systems are not taken into account due to the complexity of evaluating probabilities with highly correlated terms. Based on the simulation results for multi-objective objective ranking and selection problems with correlated objectives, we speculate that sampling correlations between individual systems would also have significant impact on the determination of the computing budget allocation. Thus future studies should incorporate this type of sampling correlation into modelling. The computing budget allocation rules applicable to more general contexts can then be provided. 111 Bibliography Almomani, M. H., Rahman, R. A., 2012. Selecting a good stochastic system for the large number of alternatives. Communications in Statistics - Simulation and Computation 41 (2), 222–237. Andrad´ottir, S., 2006. An overview of simulation optimization via random search. In: Henderson, S. G., Nelson, B. L. (Eds.), Simulation. Vol. 13 of Handbooks in Operations Research and Management Science. Elsevier, Ch. 20, pp. 617 – 631. Andrad´ottir, S., Goldsman, D., Kim, S.-H., 2005. Finding the best in the presence of a stochastic constraint. In: Winter Simulation Conference. pp. 732– 738. Andrad´ottir, S., Kim, S.-H., 2010. Fully sequential procedures for comparing constrained systems via simulation. Naval Research Logistics (NRL) 57 (5), 403–421. Batur, D., Choobineh, F., 2010a. A quantile-based approach to system selection. European Journal of Operational Research 202 (3), 764–772. Batur, D., Choobineh, F. F., 2010b. Mean-variance based ranking and selection. In: Winter Simulation Conference. pp. 1160–1166. Batur, D., Kim, S.-H., 2005. Procedures for feasibility detection in the presence of multiple constraints. In: Winter Simulation Conference. pp. 692–698. Batur, D., Kim, S.-H., 2006. Fully sequential selection procedures with parabolic boundary. IIE Transactions 38, 749–764. Batur, D., Kim, S.-H., 2010. Finding feasible systems in the presence of constraints on multiple performance measures. ACM Transactions on Modeling and Computer Simulation 20, 1–26. Bechhofer, R. E., 1954. A single-sample multiple decision procedure for ranking means of normal populations with known variances. The Annals of Mathematical Statistics 25, 16–39. Bechhofer, R. E., Santner, T. J., Goldsman, D. M., 1995. Design and analysis of experiments for statistical selection, screening, and multiple comparisons. Wiley Series in Probability and Statistics. Wiley (New York). Blanchet, J., Liu, J., Zwart, B., 2008. Large deviations perspective on ordinal optimization of heavy-tailed systems. In: WSC ’08: Proceedings of the 40th Conference on Winter Simulation. Winter Simulation Conference, pp. 489– 494. Boesel, J., Nelson, B. L., Kim, S.-H., 2003. Using ranking and selection to clean up after simulation optimization. Operations Research 51 (5), 814–825. 112 Bosman, P., Thierens, D., April 2003a. The balance between proximity and diversity in multiobjective evolutionary algorithms. IEEE Transactions on Evolutionary Computation (2), 174–188. Bosman, P., Thierens, D., 2005. The naive MIDEA: A baseline multiobjective ea. In: Coello Coello, C., Hernndez Aguirre, A., Zitzler, E. (Eds.), Evolutionary Multi-Criterion Optimization. Vol. 3410 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg, pp. 428–442. Bosman, P. A., 2009. On empirical memory design, faster selection of bayesian factorizations and parameter-free gaussian edas. In: Proceedings of the 11th Annual conference on Genetic and evolutionary computation. GECCO ’09. ACM, New York, NY, USA, pp. 389–396. Bosman, P. A., Thierens, D., 2002. Multi-objective optimization with diversity preserving mixture-based iterated density estimation evolutionary algorithms. International Journal of Approximate Reasoning 31 (3), 259 – 289. Bosman, P. A. N., Thierens, D., 2003b. The balance between proximity and diversity in multiobjective evolutionary algorithms. IEEE Transactions on Evolutionary Computation 7, 174–188. Bosman, P. A. N., Thierens, D., 2006. Multi-objective optimization with the naive midea. In: Towards a New Evolutionary Computation. Springer, pp. 123–157. Branke, J., Chick, S. E., Schmidt, C., 2005. New developments in ranking and selection: an empirical comparison of the three main approaches. In: Winter Simulation Conference. pp. 708–717. Branke, J., Chick, S. E., Schmidt, C., 2007. Selecting a selection procedure. Management Science 53 (12), 1916–1932. Branke, J., Schmidt, C., 2004. Sequential sampling in noisy environments. In: Yao, X., Burke, E., Lozano, J., Smith, J., Merelo-Guervs, J., Bullinaria, J., Rowe, J., Tino, P., Kabn, A., Schwefel, H.-P. (Eds.), Parallel Problem Solving from Nature - PPSN VIII. Vol. 3242 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg, pp. 202–211. Broadie, M., Han, M., Zeevi, A., 2007. Implications of heavy tails on simulation-based ordinal optimization. In: WSC ’07: Proceedings of the 39th conference on Winter simulation. IEEE Press, Piscataway, NJ, USA, pp. 439– 447. Bui, L. T., Abbass, H. A., Essam, D., 2005. Fitness inheritance for noisy evolutionary multi-objective optimization. In: Proceedings of the 2005 conference on Genetic and evolutionary computation. GECCO ’05. ACM, New York, NY, USA, pp. 779–785. Butler, J., Morrice, D. J., Mullarkey, P. W., 2001. A multiple attribute utility theory approach to ranking and selection. Management Science 47, 800–816. 113 Chen, C.-H., He, D., Fu, M., Lee, L. H., 2008a. Efficient simulation budget allocation for selecting an optimal subset. INFORMS Journal on Computing 20 (4), 579–595. Chen, C. H., Lin, J., Y¨ucesan, E., Chick, S. E., 2000a. Simulation budget allocation for further enhancing the efficiency of ordinal optimization. Discrete Event Dynamic Systems 10 (3), 251–270. Chen, C. H., Y¨ucesan, E., 2005. An alternative simulation budget allocation scheme for efficient simulation. International Journal of Simulation and Process Modelling 1, 49–57. Chen, E. J., Kelton, W. D., 2005. Sequential selection procedures: Using sample means to improve efficiency. European Journal of Operational Research 166, 133–153. Chen, E. J., Lee, L. H., 2009. A multi-objective selection procedure of determining a pareto set. Computers & Operations Research 36 (6), 1872–1879. Chen, H. C., Chen, C. H., Y¨ucesan, E., May 2000b. Computing efforts allocation for ordinal optimization and discrete event simulation. Automatic Control, IEEE Transactions on 45 (5), 960–964. Chen, H. C., Dai, L., Chen, C. H., Y¨ucesan, E., 1997. New development of optimal computing budget allocation for discrete event simulation. In: WSC ’97: Proceedings of the 29th conference on Winter simulation. pp. 334–341. Chen, M.-R., Lu, Y.-Z., Yang, G., 2008b. Multiobjective optimization using population-based extremal optimization. Neural Computing & Applications 17, 101–109, 10.1007/s00521-007-0118-6. Chick, S. E., Inoue, K., 2001. New two-stage and sequential procedures for selecting the best simulated system. Operations Research 49 (5), 732–743. Chick, S. E., Inoue, K., 2002. Corrigendum: New selection procedures. Operations Research 50 (3), 566. Chick, S. E., Wu, Y., 2005. Selection procedures with frequentist expected opportunity cost bounds. Operations Research 53 (5), 867–878. Coello, C. A. C., 1999. A comprehensive survey of evolutionary-based multiobjective optimization techniques. Knowledge and Information Systems 1, 129–156. Coello, C. A. C., Lamont, G. B., Veldhuizen, D. A. V., 2007. Evolutionary Algorithms for Solving Multi-Objective Problems, 2nd Edition. Genetic and Evolutionary Computation. Springer. Dai, L., 1996. Convergence properties of ordinal comparison in the simulation of discrete event dynamic systems. Journal of Optimization Theory and Applications 91 (2), 363–388. 114 Damerdji, H., Nakayama, M. K., 1999. Two-stage multiple-comparison procedures for steady-state simulations. ACM Transactions on Modeling and Computer Simulation 9, 1–30. Deb, K., 2001. Multi-Objective Optimization using Evolutionary Algorithms. John Wiley & Sons, Inc. Deb, K., Jain, S., 2002. Running performance metrics for evolutionary multiobjective optimization. Tech. rep., Kanpur Genetic Algorithms Laboratory, Indian Institue of Technology Kanpur. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T., 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation (2), 182–197. Deb, K., Sinha, A., Kukkonen, S., 2006. Multi-objective test problems, linkages, and evolutionary methodologies. In: Proceedings of the 8th annual conference on Genetic and evolutionary computation. GECCO ’06. ACM, New York, NY, USA, pp. 1141–1148. Dembo, A., Zeitouni, O., 1998. Large deviations techniques and applications. 2nd Edition. Vol. 38 of Stochastic Modelling and Applied Probability. Springer. Dudewicz, E. J., Dalal, S. R., 1975. Allocation of observations in ranking and selection with unequal variances. Sankhya-The Indian Journal of Statistics Series B 37 (FEB), 28–78. Eskandari, H., Geiger, C., 2008. A fast pareto genetic algorithm approach for solving expensive multiobjective optimization problems. Journal of Heuristics 14, 203–241, 10.1007/s10732-007-9037-z. Eskandari, H., Geiger, C., 2009. Evolutionary multiobjective optimization in noisy problem environments. Journal of Heuristics 15, 559–595, 10.1007/s10732-008-9077-z. Fogel, D. B., 1994. Asymptotic convergence properties of genetic algorithms and evolutionary programming: Analysis and experiments. Cybernetics and Systems 25 (3), 389–407. Frazier, P., Powell, W., Dayanik, S., 2008. A knowledge-gradient policy for sequential information collection. SIAM Journal on Control and Optimization 47 (5), 2410–2439. Frazier, P., Powell, W., Dayanik, S., 2009. The knowledge-gradient policy for correlated normal beliefs. INFORMS Journal on Computing. Fu, M., Chen, C. H., Hu, J., Xiong, X., 2007. Simulation allocation for determining the best design in the presence of correlated sampling. Informs Journal on Computing 19 (1), 101–111. 115 Fu, M., Chen, C.-H., Shi, L., 2008. Some topics for simulation optimization. In: Simulation Conference, 2008. WSC 2008. Winter. pp. 27–38. Glynn, P., Juneja, S., 2004. A large deviations perspective on ordinal optimization. In: WSC ’04: Proceedings of the 36th conference on Winter simulation. pp. 577–585. Glynn, P. W., Juneja, S., 2008. A large deviations view of asymptotic efficiency for simulation estimators. In: WSC ’08: Proceedings of the 40th Conference on Winter Simulation. pp. 396–406. Glynn, P. W., Juneja, S., 2011. Ordinal optimization: A nonparametric framework. In: Simulation Conference (WSC), Proceedings of the 2011 Winter. pp. 4057–4064. Goh, C., Chiam, S., Tan, K., june 2006. An investigation on noisy environments in evolutionary multi-objective optimization. In: Cybernetics and Intelligent Systems, 2006 IEEE Conference on. pp. –7. Goh, C., Tan, K., june 2007. An investigation on noisy environments in evolutionary multiobjective optimization. Evolutionary Computation, IEEE Transactions on 11 (3), 354 –381. Goldsman, D., hee Kim, S., Marshall, W. S., Nelson, B. L., 2002. Ranking and selection for steady-state simulation: Procedures and perspectives. Informs Journal on Computing 14, 2–19. Goldsman, D., Nelson, B. L., 1994. Ranking, selection and multiple comparisons in computer simulations. In: Proceedings of the 1994 Winter simulation conference. pp. 192–199. Goldsman, D., Nelson, B. L., Schmeiser, B. W., 1991. Methods for selecting the best system. In: Winter Simulation Conference. pp. 177–186. Gregorio, T.-P., Carlos, C. C., Luis, S.-Q., 2007. Emopso: A multi-objective particle swarm optimizer with emphasis on efficiency. In: Obayashi, S., Deb, K., Poloni, C., Hiroyasu, T., Murata, T. (Eds.), Evolutionary Multi-Criterion Optimization. Vol. 4403 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg, pp. 272–285. Gupta, S. S., 1965. On some multiple decision (selection and ranking) rules. Technometrics 7, 225–245. Hart, W. E., Krasnogor, N., Smith, J. E., 2005. Recent Advances in Memetic Algorithms. Vol. 166 of Studies in Fuzziness and Soft Computing. Springer. He, D., Chick, S. E., Chen, C., 2007. Opportunity cost and ocba selection procedures in ordinal optimization for a fixed number of alternative systems. IEEE Transactions on Systems, Man, and Cybernetics 37, 951–961. He, D., Lee, L. H., Chen, C.-H., Fu, M. C., Wasserkrug, S., February 2010. Simulation optimization using the cross-entropy method with optimal computing budget allocation. ACM Trans. Model. Comput. Simul. 20, 4:1–4:22. 116 Healey, C. G., Goldsman, D., hee Kim, S., 2007. Ranking and selection techniques with overlapping variance estimators. In: Winter Simulation Conference. pp. 522–529. Healey, C. M., Goldsman, D., Kim, S.-H., 2009. Ranking and selection techniques with overlapping variance estimators for simulations. Sequential Analysis 28 (4), 459–474. Ho, Y. C., Sreenivas, R. S., Vakili, P., 1992. Ordinal optimization of deds. Discrete Event Dynamic Systems (1), 61–88. Ho, Y.-C., Zhao, Q.-C., Jia, Q.-S., 2007. Ordinal Optimization: Soft Optimization for Hard Problems. Springer. Hong, L., Nelson, B., dec. 2009. A brief introduction to optimization via simulation. In: Proceedings of the 2009 Winter Simulation Conference (WSC). pp. 75 –85. Hong, L. J., 2006. Fully sequential indifference-zone selection procedures with variance-dependent sampling. Naval Research Logistics 53, 464–476. Hong, L. J., Nelson, B. L., 2005. The tradeoff between sampling and switching: New sequential procedures for indifference-zone selection. IIE Transactions 37 (7), 623–634. Hong, L. J., Nelson, B. L., 2006. Discrete optimization via simulation using COMPASS. Operations Research 54 (1), 115–129. Hong, L. J., Nelson, B. L., 2007. Selecting the best system when systems are revealed sequentially. Iie Transactions 39, 723–734. Huband, S., Barone, L., While, R. L., Hingston, P., 2005. A Scalable Multiobjective Test Problem Toolkit. Hunter, S., Pasupathy, R., dec. 2010. Large-deviation sampling laws for constrained simulation optimization on finite sets. In: Proceedings of the 2010 Winter Simulation Conference (WSC). pp. 995 –1002. Hunter, S., Pujowidianto, N., Chen, C.-H., Lee, L. H., Pasupathy, R., Yap, C. M., dec. 2011. Optimal sampling laws for constrained simulation optimization on finite sets: The bivariate normal case. In: Proceedings of the 2011 Winter Simulation Conference (WSC). pp. 4289 –4297. Inoue, K., Chick, S. E., Chen, C. H., 1999. An empirical evaluation of several methods to select the best system. ACM Transactions on Modeling and Computer Simulation (4), 381–407. Jin, Y., Zhou, A., Zhang, Q., Sendhoff, B., Tsang, E., 2008. Modeling regularity to improve scalability of model-based multiobjective optimization algorithms. In: Knowles, J., Corne, D., Deb, K., Chair, D. R. (Eds.), Multiobjective Problem Solving from Nature. Natural Computing Series. Springer Berlin Heidelberg, pp. 331–355. 117 ´ Kabirian, A., Olafsson, S., 2009. Selection of the best with stochastic constraints. In: Winter Simulation Conference. pp. 574–583. Kim, I., de Weck, O., 2006. Adaptive weighted sum method for multiobjective optimization: a new method for pareto front generation. Structural and Multidisciplinary Optimization 31, 105–116, 10.1007/s00158-005-0557-6. Kim, S.-H., 2005. Comparison with a standard via fully sequential procedures. ACM Transactions on Modeling and Computer Simulation 15, 155–174. Kim, S.-H., Nelson, B. L., 2001. A fully sequential procedure for indifferencezone selection in simulation. ACM Transactions on Modeling and Computer Simulation 11, 251–273. Kim, S. H., Nelson, B. L., 2003. Selecting the best system: selecting the best system: theory and methods. In: WSC ’03: Proceedings of the 35th conference on Winter simulation. pp. 101–112. Kim, S.-H., Nelson, B. L., 2006a. On the asymptotic validity of fully sequential selection procedures for steady-state simulation. Operations Research 54, 475–488. Kim, S.-H., Nelson, B. L., 2006b. Selecting the best system. In: Henderson, S. G., Nelson, B. L. (Eds.), Simulation. Vol. 13 of Handbooks in Operations Research and Management Science. Elsevier, Ch. 17, pp. 501 – 534. Kim, S.-H., Nelson, B. L., 2007. Recent advances in ranking and selection. In: Winter Simulation Conference. pp. 162–172. Koenig, L. W., Law, A. M., 1985. A procedure for selecting a subset of size m containing the l best of k independent normal populations, with applications to simulation. Communications in Statistics - Simulation and Computation 14 (3), 719–734. Law, A. M., McComas, M. G., 2000. Simulation-based optimization. In: Winter Simulation Conference. pp. 46–49. Lee, L. H., Chen, C. H., Chew, E. P., Li, J., Pujowidianto, N. A., Zhang, S., 2010a. A review of optimal computing budget allocation algorithms for simulation optimization problem. International Journal of Operations Research 7(2), 19–31. Lee, L. H., Chew, E., Teng, S., Li, J., 2009. Application of evolutionary algorithms for solving multi-objective simulation optimization problems. In: Goh, C.-K., Ong, Y.-S., Tan, K. C. (Eds.), Multi-Objective Memetic Algorithms. Vol. 171 of Studies in Computational Intelligence. Springer Berlin / Heidelberg, pp. 91–110. Lee, L. H., Chew, E. P., Teng, S., 2007. Finding the pareto set for multi-objective simulation models by minimization of expected opportunity cost. In: Proceedings of the 39th conference on Winter simulation: 40 years! The best is yet to come. WSC ’07. IEEE Press, Piscataway, NJ, USA, pp. 513–521. 118 Lee, L. H., Chew, E. P., Teng, S., 2010b. Computing budget allocation rules for multi-objective simulation models based on different measures of selection quality. Automatica 46 (12), 1935 – 1950. Lee, L. H., Chew, E. P., Teng, S., Goldsman, D., 2004. Optimal computing budget allocation for multi-objective simulation models. In: WSC ’04: Proceedings of the 36th conference on Winter simulation. pp. 586–594. Lee, L. H., Chew, E. P., Teng, S., Goldsman, D., 2010c. Finding the nondominated pareto set for multi-objective simulation models. IIE Transactions 42 (9), 656–674. Lee, L. H., Lau, T. W. E., Ho, Y. C., Jan 1999. Explanation of goal softening in ordinal optimization. Automatic Control, IEEE Transactions on 44 (1), 94– 99. Lozano, J., Larraaga, P., Graa, M., Albizuri, F., 1999. Genetic algorithms: bridging the convergence gap. Theoretical Computer Science 229 (12), 11 – 22. Malone, G. J., Kim, S.-H., Goldsman, D., Batur, D., 2005. Performance of variance updating ranking and selection procedures. In: Winter Simulation Conference. pp. 825–832. Marler, R., Arora, J., 2004. Survey of multi-objective optimization methods for engineering. Structural and Multidisciplinary Optimization 26, 369–395. Mo, L., Dai, G., Zhu, J., 2010. The RM-MEDA based on elitist strategy. In: Cai, Z., Hu, C., Kang, Z., Liu, Y. (Eds.), Advances in Computation and Intelligence. Vol. 6382 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg, pp. 229–239. Morrice, D. J., Butler, J. C., 2006. Ranking and selection with multiple ”targets”. In: Winter Simulation Conference. pp. 222–230. Nakayama, M. K., 1997. Multiple-comparison procedures for steady-state simulations. Annals of Statistics 25, 2433–2450. Nebro, A., Durillo, J., Coello Coello, C., Luna, F., Alba, E., 2008. A study of convergence speed in multi-objective metaheuristics. In: Rudolph, G., Jansen, T., Lucas, S., Poloni, C., Beume, N. (Eds.), Parallel Problem Solving from Nature PPSN X. Vol. 5199 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg, pp. 763–772. Nelson, B., Banerjee, S., 2001. Selecting a good system: procedures and inference. IIE Transactions 33, 149–166, 10.1023/A:1007639112626. Nelson, B. L., Goldsman, D., 2001. Comparisons with a standard in simulation experiments. Management Science 47 (3), 449–463. Nelson, B. L., Swann, J., Goldsman, D., Song, W., 2001. Simple procedures for selecting the best simulated system when the number of alternatives is large. Operations Research 49 (6), 950–963. 119 Okabe, T., Jin, Y., Olhofer, M., Sendhoff, B., 2004. On Test Functions for Evolutionary Multi-objective Optimization. Osogami, T., August 2009. Finding probably best systems quickly via simulations. ACM Transactions on Modeling and Computer Simulation 19, 12:1– 12:19. Pelikan, M., Sastry, K., Goldberg, D., 2006. Multiobjective estimation of distribution algorithms. In: Scalable Optimization via Probabilistic Modeling. Springer, pp. 223–248. Pichitlamken, J., Nelson, B. L., 2002. Optimization via simulation: a combined procedure for optimization via simulation. In: Winter Simulation Conference. pp. 292–300. Pichitlamken, J., Nelson, B. L., 2003. A combined procedure for optimization via simulation. ACM Transactions on Modeling and Computer Simulation 13, 155–179. Pichitlamken, J., Nelson, B. L., Hong, L. J., 2006. A sequential procedure for neighborhood selection-of-the-best in optimization via simulation. European Journal of Operational Research 173, 283–298. Pujowidianto, N., Lee, L. H., Chen, C.-H., Yap, C. M., dec. 2009. Optimal computing budget allocation for constrained optimization. In: Proceedings of the 2009 Winter Simulation Conference (WSC). pp. 584 –589. Rinott, Y., 1978. On two-stage selection procedures and related probabilityinequalities. Communications in Statistics - Theory and Methods (8), 799– 811. Siwik, L., Natanek, S., 2008. Elitist evolutionary multi-agent system in solving noisy multi-objective optimization problems. In: Evolutionary Computation, 2008. CEC 2008. (IEEE World Congress on Computational Intelligence). IEEE Congress on. pp. 3319 –3326. Sullivan, D. W., Wilson, J. R., 1989. Restricted subset selection procedures for simulation. Operations Research 37 (1), 52–71. Swisher, J. R., Jacobson, S. H., Y¨ucesan, E., 2003. Discrete-event simulation optimization using ranking, selection, and multiple comparison procedures: A survey. ACM Transactions on Modeling and Computer Simulation 13 (2), 134–154. Szechtman, R., Y¨ucesan, E., 2008. A new perspective on feasibility determination. In: Proceedings of the 40th Conference on Winter Simulation. WSC ’08. Winter Simulation Conference, pp. 273–280. Tan, K., Goh, C., 2008. Handling uncertainties in evolutionary multi-objective optimization. In: Zurada, J., Yen, G., Wang, J. (Eds.), Computational Intelligence: Research Frontiers. Vol. 5050 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg, pp. 262–292. 120 Tan, K. C., Khor, E. F., Lee, T. H., 2005. Multiobjective Evolutionary Algorithms and Applications, 1st Edition. Advanced Information and Knowledge Processing. Springer. Tan, K. C., Lee, T. H., Khor, E. F., 2001. Evolutionary algorithms with dynamic population size and local exploration for multiobjective optimization. IEEE Transactions on Evolutionary Computation 5, 565–588. Tekin, E., Sabuncuoglu, I., 2004. Simulation optimization: A comprehensive review on theory and applications. IIE Transactions 36 (11), 1067–1081. Teng, S., Lee, L. H., Chew, E. P., 2007. Multi-objective ordinal optimization for simulation optimization problems. Automatica 43 (11), 1884–1895. Teng, S., Lee, L. H., Chew, E. P., 2010. Integration of indifference-zone with multi-objective computing budget allocation. European Journal of Operational Research 203 (2), 419 – 429. Tsai, S. C., Nelson, B. L., 2009. Fully sequential selection procedures with control variates. IIE Transactions 42 (1), 71–82. Tsai, S. C., Wei, J., 2011. Multinomial selection for comparison with a standard. Communications in Statistics - Simulation and Computation 40 (6), 926–947. Varadhan, S. R. S., 1984. Multidimensional version of cramer’s theorem. In: Large Deviations and Applications. Society for Industrial and applied mathematics. Wang, Y., Luangkesorn, L., Shuman, L., 2011. Best-subset selection procedure. In: Proceedings of the 2011 Winter Simulation Conference (WSC). pp. 4310 –4318. Wilson, J. R., 2001. A multiplicative decomposition property of the screeningand-selection procedures of Nelson et al. Operations Research 49 (6), 964– 966. Zhang, Q., Muhlenbein, H., 2004. On the convergence of a class of estimation of distribution algorithms. Evolutionary Computation, IEEE Transactions on (2), 127–136. Zhang, Q., Zhou, A., Jin, Y., feb. 2008. RM-MEDA: A regularity model-based multiobjective estimation of distribution algorithm. Evolutionary Computation, IEEE Transactions on 12 (1), 41 –63. Zhao, Q. C., Ho, Y. C., Jia, Q. S., 2005. Vector ordinal optimization. Journal of Optimization Theory and Applications 125 (2), 259–274. Zhou, A., Zhang, Q., Jin, Y., Tsang, E. P. K., Okabe, T., 2005. A model-based evolutionary algorithm for bi-objective optimization. In: IEEE Congress on Evolutionary Computation. pp. 2568–2575. Zitzler, E., Deb, K., Thiele, L., 2000. Comparison of multiobjective evolutionary algorithms: Empirical results. Evolutionary Computation (2), 173–196. 121 Zitzler, E., Laumanns, M., Thiele, L., 2001. SPEA2: Improving the strength pareto evolutionary algorithm. Tech. Rep. 103, Computer Engineering and Networks Laboratory, ETH Zurich. Zitzler, E., Thiele, L., 1999. Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Transactions on Evolutionary Computation (4), 257–271. 122 [...]... efficient computing budget allocation rules for multi- objective simulation optimization problems on finite sets More specifically, the aims of this research are to 1 study the problem of finding a subset of good systems in a multi- objective simulation optimization context, provide a computing budget allocation 4 that can maximize the probability of correct selection, subject to a limited computing budget. .. in the objective space Dconvergence : convergence metric of a searching algorithm Dcoverage : coverage metric of a searching algorithm SO : Simulation Optimization OvS : Optimization via Simulation R & S : Ranking and Selection IZ : Indifference-zone OCBA : Optimal Computing Budget Allocation MOEA : Multi- objective Evolutionary Algorithm MOGA : Multi- objective Genetic Algorithm MOEDA : Multi- objective. .. usually to find a computing budget allocation that maximizes the selection quality or maintain the selection quality above a certain confidence level Therefore a ranking and selection problem of interest is also a computing budget allocation problem In the following text, we use ranking and selection and optimal computing budget allocation interchangeably where necessary 2.2 Computing Budget Allocation Problems... o When all the performance measures act as objectives, the problems become multi- objective simulation optimization ones A number of studies have been carried out to tackle such problems Butler et al (2001) transform the multi- objective problem into a single objective one using multiple attribute utility theory and incorporating the R procedure for the ranking and selection Prior information on decision-maker’s... Simulation Optimization Simulation optimization combines two well-established paradigms, namely, simulation and optimization (Fu et al., 2008; Tekin and Sabuncuoglu, 2004) Simulation intends to be a modelling tool for evaluating complex systems in practice, whereas optimization aims to find the system with the best decision variables (Fu et al., 2008) Simulation optimization involves optimization on a simulation. .. to simulation optimization have been developed, examples include sample path optimization, response surface methods and searching heuristics Among these approaches, simulation budget allocation or sampling allocation becomes vital in conducting efficient simulation experiments for a finite and small enough set of alternatives and is the research area of interest in this study The computing budget allocation. .. single objective simulation optimization problems only, and there is the unmet practical need to develop efficient computing budget allocation rules for multiobjective simulation optimization problems Moreover, investigation into generalization and optimality of the derived allocation rules is still lacking and a study in a rigorous mathematical framework is necessary The main objective of this study is... initial studies on multi- objective ranking and selection with Pareto-optimality have been reported, examples include the work done by Lee et al (2004), Lee et al (2010b) and Lee et al (2010c) that deal with selection of non-dominated systems under a multi- objective simulation optimization context The practical need of selecting a subset containing good designs for multiobjective simulation optimization is... explore the computing budget allocation problem for multi- objective simulation optimization using expected 20 opportunity cost as an evidence of correct selection Allocation schemes with regard to different evidence of correct selection are discussed and compared in Lee et al (2010b) Teng et al (2010) deal with a multi- objective ranking and selection problem where alternative have close performances... measure for system x, which can be a scalar or vector for the single objective or multi- objective case respectively It is noted that the solution space may also be explicitly specified by constraints like g(x) ≤ c, where g(x) are constraint measures and c stands for constants The objective of simulation optimization problems is to find the feasible x’s with the minimum true objectives, where performance . 22 3.1 Computing budget allocation for Experiment 2 . . . . . . . . . . . 44 3.2 Computing budget allocation for Experiment 3 . . . . . . . . . . . 45 3.3 Computing budget allocation for Experiment. with single objective simulation optimization problems only, and there is the unmet practical need to develop efficient computing budget allocation rules for multi- objective simulation optimization. OPTIMAL COMPUTING BUDGET ALLOCATION FOR MULTI- OBJECTIVE SIMULATION OPTIMIZATION LI JUXIN (B.Eng., Shanghai Jiao Tong University, China) A THESIS SUBMITTED FOR THE DEGREE OF