Gregory Levitin (Ed.) Computational Intelligence in Reliability Engineering Evolutionary Techniques in Reliability Analysis and Optimization With 99 Figures and 64 Tables 123 Dr Gregory Levitin Research && Development Division The Israel Electric Corporation Ltd PO Box 10 31000 Haifa Israel E-mail: levitin@iec.co.il Library of Congress Control Number: 2006931548 ISSN print edition: 1860-949X ISSN electronic edition: 1860-9503 ISBN-10 3-540-37367-5 Springer Berlin Heidelberg New York ISBN-13 978-3-540-37367-4 Springer Berlin Heidelberg New York This work is subject j to copyright py g All rights g are reserved, whether the whole or p part of the material is concerned, specifically p y the rights g of translation, reprinting, p g reuse of illustrations, recitation, broadcasting, g reproduction p on microfilm or in anyy other way, y and storage g in data banks Duplication p of this p publication or p parts thereof is p permitted onlyy under the p provisions of the German Copyright py g Law of Septem p ber 9, 1965, in its current version, and permission p for use must always y be obtained from Springer-Verlag Violations are liable to prosecution under the German Copyright Law Springer is a part of Springer Science+Business Media springer.com © Springer-Verlag Berlin Heidelberg 2007 The use of ggeneral descriptive p names, registered g names, trademarks, etc in this p publication does not imply, p y even in the absence of a specific p statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use Cover design: g deblik, Berlin Typesetting by the editor and SPi Printed on acid-ffree paper SPIN: 11418368 89/SPi 543210 Preface This two-volume book covers the recent applications of computational intelligence techniques in reliability engineering Research in the area of computational intelligence is growing rapidly due to the many successful applications of these new techniques in very diverse problems “Computational Intelligence” covers many fields such as neural networks, fuzzy logic, evolutionary computing, and their hybrids and derivatives Many industries have benefited from adopting this technology The increased number of patents and diverse range of products developed using computational intelligence methods is evidence of this fact These techniques have attracted increasing attention in recent years for solving many complex problems They are inspired by nature, biology, statistical techniques, physics and neuroscience They have been successfully applied in solving many complex problems where traditional problem-solving methods have failed The book aims to be a repository for the current and cutting-edge applications of computational intelligent techniques in reliability analysis and optimization In recent years, many studies on reliability optimization use a universal optimization approach based on metaheuristics These metaheuristics hardly depend on the specific nature of the problem that is solved and, therefore, can be easily applied to solve a wide range of optimization problems The metaheuristics are based on artificial reasoning rather than on classical mathematical programming Their important advantage is that they not require any information about the objective function besides its values corresponding to the points visited in the solution space All metaheuristics use the idea of randomness when performing a search, but they also use past knowledge in order to direct the search Such algorithms are known as randomized search techniques Genetic algorithms are one of the most widely used metaheuristics They were inspired by the optimization procedure that exists in nature, the biological phenomenon of evolution The first volume of this book starts with a survey of the contributions made to the optimal reliability design literature in the resent years The next chapter is devoted to using the metaheuristics in multiobjective reliability optimization The volume also contains chapters devoted to different applications of the genetic algorithms in reliability engineering and to combinations of this algorithm with other computational intelligence techniques VI Preface The second volume contains chapters presenting applications of other metaheuristics such as ant colony optimization, great deluge algorithm, cross-entropy method and particle swarm optimization It also includes chapters devoted to such novel methods as cellular automata and support vector machines Several chapters present different applications of artificial neural networks, a powerful adaptive technique that can be used for learning, prediction and optimization The volume also contains several chapters describing different aspects of imprecise reliability and applications of fuzzy and vague set theory All of the chapters are written by leading researchers applying the computational intelligence methods in reliability engineering This two-volume book will be useful to postgraduate students, researchers, doctoral students, instructors, reliability practitioners and engineers, computer scientists and mathematicians with interest in reliability I would like to express my sincere appreciation to Professor Janusz Kacprzyk from the Systems Research Institute, Polish Academy of Sciences, Editor-in-Chief of the Springer series “Studies in Computational Intelligence”, for providing me with the chance to include this book in the series I wish to thank all the authors for their insights and excellent contributions to this book I would like to acknowledge the assistance of all involved in the review process of the book, without whose support this book could not have been successfully completed I want to thank the authors of the book who participated in the reviewing process and also Prof F Belli, University of Paderborn, Germany, Prof Kai-Yuan Cai, Beijing University of Aeronautics and Astronautics, Dr M Cepin, Jozef Stefan Institute, Ljubljana , Slovenia, Prof M Finkelstein, University of the Free State, South Africa, Prof A M Leite da Silva, Federal University of Itajuba, Brazil, Prof Baoding Liu, Tsinghua University, Beijing, China, Dr M Muselli, Institute of Electronics, Computer and Telecommunication Engineering, Genoa, Italy, Prof M Nourelfath, Université Laval, Quebec, Canada, Prof W Pedrycz, University of Alberta, Edmonton, Canada, Dr S Porotsky, FavoWeb, Israel, Prof D Torres, Universidad Central de Venezuela, Dr Xuemei Zhang, Lucent Technologies, USA for their insightful comments on the book chapters I would like to thank the Springer editor Dr Thomas Ditzinger for his professional and technical assistance during the preparation of this book Haifa, Israel, 2006 Gregory Levitin Contents Recent Advances in Optimal Reliability Allocation Way Kuo, Rui Wan 1.1 Introduction 1.2 Problem Formulations 1.3 Brief Review of Advances In P1-P4 1.3.1 Traditional Reliability-Redundancy Allocation Problem (P1) 1.3.1.1 Active and Cold-Standby Redundancy 1.3.1.2 Fault-Tolerance Mechanism 10 1.3.2 Percentile Life Optimization Problem (P2) 11 1.3.3 MSS Optimization (P3) 12 1.3.4 Multi-Objective Optimization (P4) 16 1.4 Developments in Optimization Techniques 18 1.4.1 Meta-Heuristic Methods 19 1.4.1.1 Ant Colony Optimization Method 19 1.4.1.2 Hybrid Genetic Algorithm 20 1.4.1.3 Tabu Search 21 1.4.1.4 Other Meta-Heuristic Methods 22 1.4.2 Exact Methods 22 1.4.3 Other Optimization Techniques 23 1.5 Comparisons and Discussions of Algorithms Reported in Literature 25 1.6 Conclusions and Discussions 27 References 29 Multiobjective Metaheuristic Approaches to Reliability Optimization Sadan Kulturel-Konak, Abdullah Konak, David W Coit 37 2.1 Introduction 37 2.2 Metaheuristics and Multiobjective Optimization 39 2.2.1 Metaheuristics 39 2.2.2 Metaheuristic Approaches to Multiobjective Optimization 41 2.3 Multiobjective Tabu Search and Reliability Optimization 42 2.3.1 The Multinomial Tabu Search Algorithm to Solve Redundancy Allocation Problem 42 2.3.1.1 Multiobjective System Redundancy Allocation Problem 42 2.3.1.2 Multinomial Tabu Search Algorithm 43 2.3.1.3 Computational Experiments 45 2.4 Multiobjective Genetic Algorithms 47 2.4.1 Multiobjective GA Approaches to Reliability Optimization 48 VIII Contents 2.5 Multiobjective Optimization of Telecommunication Networks Considering Reliability 50 2.5.1 Network Survivability and Reliability 50 2.5.2 Multiobjective Elitist GA with Restricted Mating 52 2.5.2.1 Problem Encoding 52 2.5.2.2 Crossover Operator 52 2.5.2.3 Mutation 53 2.5.2.4 Overall Algorithm 53 2.5.3 Computational Experiments 55 2.6 Other Metaheuristic Techniques to Multiobjective Reliability Optimization 56 2.6.1 Ant Colony Optimization 56 2.6.2 Simulated Annealing 57 2.6.3 Conclusions 57 References 58 Genetic Algorithm Applications in Surveillance and Maintenance Optimization Sebastián Martorell, Sofía Carlos, José F Villanueva, Ana Sánchez 63 3.1 Introduction 63 3.2 Analysis of Published Research 67 3.3 Overview of Testing and Maintenance 71 3.3.1 RAMS and the role of T&M at NPP 71 3.3.2 Failure types and T&M activities 77 3.4 Decision-making Based on RAMS+C 78 3.4.1 Basis for the RAMS+C informed decision-making 78 3.4.2 Quantification models of RAMS+C 80 3.5 Optimization Problem 84 3.5.1 Problem Formulation 84 3.5.2 Solution Approaches 84 3.6 Genetic Algorithms 86 3.6.1 Origin, fundamentals and first applications 86 3.6.2 Pioneering GA 87 3.6.3 Development of multi-objective GA 89 3.7 Research Areas 91 References 94 Genetic Algorithms and Monte Carlo Simulation for the Optimization of System Design and Operation Marzio Marseguerra, Enrico Zio, Luca Podofillini 101 4.1 Introduction 101 4.1.1 Motivations for the GA-MC Approach 102 4.1.1.1 Use of Monte Carlo Simulation for the System Modeling 102 4.1.1.2 Use of Multi-Objective Genetic Algorithms for the System Optimization 103 4.1.1.3 Combination of GA-MC 104 Contents IX 4.1.2 Application to System Optimization Under Uncertainty 105 4.1.3 Structure of the Chapter 106 4.2 Fundamentals of Monte Carlo Simulation 106 4.2.1 The System Transport Model 107 4.2.2 Monte Carlo Simulation for Reliability Modeling 107 4.3 Genetic Algorithms 112 4.3.1 Introduction 112 4.3.2 The Standard GA Procedure 115 4.4 Embedding the Monte Carlo Simulation in the Genetic Algorithm Search Engine 117 4.5 Optimization of the Design of a Risky Plant 118 4.5.1 Problem Statement 118 4.5.2 Applying the GA-MC Approach 121 4.6 Multi-objective Genetic Algorithms 127 4.7 Optimizing the Technical Specifications of the Reactor Protection Instrumentation System of a PWR 129 4.7.1 System Description 130 4.7.2 Testing Procedures and TSs of the RPIS 131 4.7.3 Modeling Assumptions 132 4.7.4 Multiobjective Optimization 140 4.8 Conclusions 145 References 147 New Evolutionary Methodologies for Integrated Safety System Design and Maintenance Optimization B Galván, G Winter, D Greiner, D Salazar, M Méndez 151 5.1 Introduction 151 5.2 Evolutionary Algorithms (EA) 155 5.2.1 Multiple Objective Evolutionary Algorithms 155 5.2.2 Evolutionary Algorithms with Mixed Coding Schemes: A Double Loop Approach 157 5.2.3 Flexible Evolutionary Algorithms 163 5.3 Integrated Safety Systems Optimization 167 5.3.1 The System Model 167 5.3.2 Components Unavailability and Cost Models 169 5.3.3 Computing the System Model Using Restricted Sampling 171 5.3.4 Coding Schemes 172 5.4 Application example: The Containment Spray System of a Nuclear Power Plant 173 5.4.1 System Description 173 5.4.2 Models and Parameters Selected 175 5.4.3 System Model 176 5.5 Results 177 5.6 Conclusions 184 References 186 X Contents Optimal Redundancy Allocation of Multi-State Systems with Genetic Algorithms Zhigang Tian, Ming J Zuo, Hong-Zhong Huang 191 6.1 Introduction 191 6.1.1 Optimal Redundancy Allocation of Multi-state Systems 191 6.1.2 Optimal Redundancy Allocation of Multi-state Systems with Genetic Algorithms 193 6.1.3 Content of this Chapter 195 6.2 System Utility Evaluation 195 6.2.1 Multi-state Series-parallel Systems by Barlow and Wu (1978) 196 6.2.2 The Multi-state Load Sharing Series-parallel Systems by Levitin et al (1998) 198 6.2.3 Multi-state k-out-of-n Systems 200 6.2.4 General Comments 201 6.3 Optimization Models for Multi-state Systems 201 6.3.1 Single-objective Optimization Models 202 6.3.2 The Multi-objective Optimization Model 203 6.4 Implementation of Genetic Algorithms 206 6.4.1 General Framework of GA 206 6.4.2 Encoding and Decoding 207 6.4.3 Fitness Function Value 208 6.4.4 Fitness Function Value 208 6.5 An Application Example 208 6.6 Concluding Remarks 211 References 212 Intelligent Interactive Multiobjective Optimization of System Reliability Hong-Zhong Huang, Zhigang Tian, Ming J Zuo 215 7.1 Introduction 215 7.2 Multiobjective Optimization Problem 217 7.2.1 Problem Formulation 217 7.2.2 Pareto Solution 217 7.2.3 Weighted-sum Method 218 7.2.4 Augment Weighted Tchebycheff Programs 218 7.3 Designer’s Preference Structure Model 219 7.3.1 Model Parameter Vector 219 7.3.2 General Multiobjective Optimization Procedure 219 7.3.3 Designer’s Preference Structure Model 221 7.3.4 Preference Information Elicitation 222 7.4 IIMOM Procedure 222 7.5 Application of IIMOM to Reliability Optimization Problem 225 7.5.1 Problem Definition 225 7.5.2 The Mapping from Weight Vector to Preference Value 227 7.5.3 Results and Discussions 228 7.5.4 Discussion on the Performances of IIMOM 234 Contents XI 7.6 Conclusions 235 References 236 Reliability Assessment of Composite Power Systems Using Genetic Algorithms Nader Samaan, Chanan Singh 237 8.1 Introduction 237 8.2 Reliability Evaluation of Composite Generation-Transmission Systems 239 8.3 Genetic Algorithms Approach for the Assessment of Composite Systems Annualized Indices 241 8.3.1 Construction of System State Array 244 8.3.2 Evolution of a New Generation 246 8.3.3 Stopping Criterion 247 8.3.4 State Evaluation Model 249 8.3.5 Assessment of Composite System Adequacy Indices 251 8.3.6 Case Studies for the Assessment of Annualized Indices 252 8.4 Reliability Indices Considering Chronological Load Curves 254 8.4.1 Modeling of Chronological Load Curve 255 8.4.2 Genetic Algorithms Sampling with m Cluster Load Vectors 257 8.4.2.1 Genetic Algorithms Parallel Sampling 257 8.4.2.2 Genetic Algorithm Sampling for Maximum Cluster Load Vector with Series State Revaluation 258 8.4.3 State Evaluation Model 260 8.4.4 Case Studies for the Assessment of Annual Indices 262 8.4.4.1 Fully Correlated Load Buses 262 8.4.4.2 Partially Correlated Load Buses 264 8.5 Calculation of Frequency and Duration Indices 266 8.5.1 Modeling of the Chronological Load 267 8.5.1.1 Calculating Transition Rates between Load Clusters 268 8.5.2 Calculating Failure State Contribution to System Failure Frequency 269 8.5.3 Non-Sequential Monte Carlo Sampling 271 8.5.4 Genetic Algorithm Sampling for Maximum Load State with Series State Reevaluation 272 8.5.5 Case Studies for the Assessment of Frequency and Duration Indices 273 8.6 Consideration of Multi-State Components 276 8.6.1 Representation of Generating Unit Derated States 276 8.6.2 Consideration of Common Mode Failures in Transmission Lines 278 8.6.3 Case Studies with Multi-State Components 279 8.6.3.1 Generating Unit Derated States 279 8.6.3.2 Common Mode Outage 281 8.7 Summary and Conclusions 282 References 283 Appendix: The RBTS Test System Data 285 384 Liang Tian and Afzel Noore Fig Static feed-forward neural network in (a) and recurrent neural network in (b) with feedback connections Each processing element of a recurrent neural network is denoted by the following generalized equations [5]: N [l ] N[l −1] m=1 m=1 p [l,n](ti )=∑w[l,m][l,n]q[l,m] (ti −1)+ ∑w[l−1,m][l,n]q[l−1,m] (ti ) +b[l,n] q[l , n] (ti ) = Φ [l , n] ( p[l , n] (ti )) where, (7) (8) Computational Intelligence Methods in Software Reliability Prediction 385 p[l , n ] (t i ) is the internal state variable of the n th neuron in the l th layer at time ti q[l , n] (t i ) is the output of the n th neuron in the l th layer at time ti b[l ,n ] is the bias of the n th neuron in the l th layer w[l ,m ][l ',n ] is the weight associated with the link between the m th neuron of the l th layer to the n th neuron of the l 'th layer Φ (⋅) is the activation function 12.3.2 Bayesian Regularization A desirable neural network model should have small errors not only in the training data set, but also in the validation or testing data set The ability to adapt to previously known data as well as unknown data requires improving generalization Regularization constrains the size of the network parameters When the parameters in a network are kept small, the response of the network will be smooth With regularization, the performance function is modified by adding a term that consists of the mean of the sum of squares of the neural network weights and biases The mean squared error with regularization performance, msereg , is given by: msereg = β × mse + (1 − β ) × msw (9) where, β is the performance ratio and represents the relative importance of errors vs weight and bias values, mse is the mean squared error during training, and msw is the mean squared weights and biases By using this modified performance function, msereg , the neural network is forced to have smaller weights and biases, which causes the network to respond smoother, represent the true function rather than capture the noise, and is less likely to overfit The major problem with regularization is to choose the performance ratio coefficient β MacKay [22] has done extensive work on the application of Bayes rule for optimizing regularization Hessian matrix computation is required for regularization optimization In order to minimize the computational overhead, Foresee and Hagan [12] proposed using a GaussNewton approximation to the Hessian matrix, which is readily available while Levenberg-Marquardt algorithm is used as neural network training scheme 386 Liang Tian and Afzel Noore The Bayesian optimization of the regularization coefficient with a Gauss-Newton approximation to the Hessian matrix is described in the following procedure [12]: Initialize β ( β = 1) and the weights Minimize the performance function msereg by using LevenbergMarquardt algorithm Compute the effective number of parameters using Gauss-Newton approximation to the Hessian matrix available in the Levenberg-Marquardt training algorithm Derive the new estimate of β Repeat from Step 2-4 until the convergence is obtained Thus, the optimized value for β is chosen We show how Bayesian regularization can be combined with recurrent training scheme (RNN-BR) for improving the generalization capability 12.3.3 Modeling Rationale More specifically, in this failure time modeling, the input-output pattern fed into the network is the software failure temporal sequence Thus, the recurrent network can learn and recognize the inherent temporal patterns of input-output pair For one-step-ahead prediction, the input sequence and the desired output sequence should have one step delay during the training process The desired objective is to force the network to recognize the onestep-ahead temporal pattern A sample input sequence and the corresponding one-step-ahead desired output sequence is defined as: Input Sequence: x(t ), x(t1 ),⋅ ⋅ ⋅, x(ti −1 ), x(ti ), x(ti +1 ),⋅ ⋅ ⋅ Output Sequence: x(t1 ), x(t ),⋅ ⋅ ⋅, x(ti ), x(ti +1 ), x(ti + ),⋅ ⋅ ⋅ where x(ti ) is the software failure time in the training data sequence, and ti is the software failure time sequence index The activation function in our modeling approach is linear for the output layer, and it is hyperbolic tangent sigmoidal for hidden layer neurons Once the network is trained based on sufficient training data sequence, the unknown data sequence will be presented to the network to validate the performance Computational Intelligence Methods in Software Reliability Prediction 387 12.4 Adaptive Support Vector Machine (A–SVM) Learning 12.4.1 SVM Learning in Function Approximation As a novel type of machine learning algorithm, support vector machine (SVM) has gained increasing attention from its original application in pattern recognition to the extended application in function approximation and regression estimation Based on the structural risk minimization (SRM) principle, the learning scheme of SVM is focused on minimizing an upper bound of the generalization error that includes the sum of the empirical training error and a regularized confidence interval, which will eventually result in better generalization performance Moreover, unlike other gradient descent based learning scheme that requires nonlinear optimization with the danger of getting trapped into local minima, the regularized risk function of SVM can be minimized by solving a linearly constrained quadratic programming problem, which can always obtain a unique and global optimal solution Thus, the possibility of being trapped at local minima can be effectively avoided [33,41,42] The basic idea of SVM for function approximation is mapping the data x into a high-dimensional feature space by a nonlinear mapping and then performing a linear regression in this feature space Assume that a total of l pairs of training patterns are given during SVM learning process, ( x1 , y1 ), ( x2 , y ),⋅ ⋅ ⋅, ( xi , yi ),⋅ ⋅ ⋅, ( xl , yl ) where the inputs are n-dimensional vectors xi ∈ ℜ n , and the target outputs are continuous values yi ∈ ℜ The SVM model used for function approximation is: f ( x) = w ⋅ φ ( x) + b (10) where φ (x) is the high-dimensional feature space that is nonlinearly mapped from the input space x Thus, a nonlinear regression in the lowdimensional input space is transferred to a linear regression in a highdimensional feature space [41] The coefficients w and b can be estimated by minimizing the following regularized risk function R: 1 l R = || w ||2 +C ∑ | yi − f ( xi ) |ε l i =1 (11) 388 Liang Tian and Afzel Noore where if | yi − f ( xi ) |≤ε , ⎧0 | yi − f ( xi ) |ε = ⎨ ⎩| yi − f ( xi ) |−ε otherwise (12) || w ||2 is the weights vector norm, which is used to constrain the model structure capacity in order to obtain better generalization performance The second term is the Vapnik’s linear loss function with ε -insensitivity zone as a measure for empirical error The loss is zero if the difference between the predicted and observed value is less than or equal to ε For all other cases, the loss is equal to the magnitude of the difference between the predicted value and the radius ε of ε -insensitivity zone C is the regularization constant, representing the trade-off between the approximation error and the model structure ε is equivalent to the approximation accuracy requirement for the training data points Further, two positive slack variables ξi and ξ i* are introduced We have ⎧ξ i for data " above" an ε tube, | yi − f ( xi ) | −ε = ⎨ * ⎩ξ i for data " below" an ε tube (13) Thus, minimizing the risk function R in Eq (11) is equivalent to minimizing the objective function Rw,ξ ,ξ * l Rw,ξ ,ξ * = || w ||2 + C (ξ i + ξ i* ) i =1 ∑ (14) subject to constraints ⎧ yi − w ⋅ φ ( xi ) − b ≤ ε +ξ i i =1, , l , ⎪ * ⎨w ⋅ φ ( xi ) + b − yi ≤ε +ξ i i =1, , l , * ⎪ξ , ξ ≥0 i =1, , l ⎩ i i (15) This constrained optimization problem is typically solved by transforming into the dual problem, and its solution is given by the following explicit form: l f ( x) = ∑ (α i −α i* ) K ( xi , x) + b i =1 (16) Computational Intelligence Methods in Software Reliability Prediction 389 12.4.2 Lagrange Multipliers In Eq (16), α i and α i* are the Lagrange multipliers with α i × α i* =0 and α i , α i* ≥0 for any i = 1, , l They can be obtained by maximizing the following form: l l l l − ε ∑ (α i + α i* ) + ∑ yi (α i −α i* ) − ∑ ∑ (α i − α i* )(α j −α *j ) K ( xi , x j i =1 j =1 i =1 i =1 (17) subject to constraints ⎧ l * l ⎪ ∑α i = ∑α i ⎨i =1 i =1 ⎪ * ≤C i =1, , l α , α ≤ i i ⎩ (18) After learning, only some of coefficients (α i −α i* ) in Eq (16) differ from zero, and the corresponding training data points are referred to as support vectors It is obvious that only the support vectors can fully decide the decision function in Eq (16) 12.4.3 Kernel Function In Eq (16), K ( xi , x) is defined as the kernel function, which is the inner product of two vectors in feature space φ ( xi ) and φ (x) By introducing the kernel function, we can deal with the feature spaces of arbitrary dimensionality without computing the mapping relationship φ (x) explicitly Some commonly used kernel functions are polynomial kernel function and Gaussian kernel function Fig Dynamic software reliability prediction framework 390 Liang Tian and Afzel Noore 12.4.4 Formulation of the SVM-Predictor The software reliability prediction system shown in Fig consists of a failure history database and an iteratively and dynamically updated SVM learning-predicting process When a software failure, xi , occurs, the failure history database is updated and the accumulated failure data ( x1 , x2 , , xi ) is made available to the SVM learning process The number of failure data increases over time during a dynamic system Accordingly, the SVM learning process is iteratively and dynamically updated after every occurrence of new failure time data in order to capture the most current feature hidden inside the software failure sequence After the SVM learning process is complete based on the currently available history failure data, next-step failure information, xˆ i + , will be predicted Unlike the existing mapping characteristics, the inter-relationship among software failure time data is investigated More specifically, the input-output pattern fed into the network is the failure temporal sequence The SVM learning scheme is applied to the failure time data, forcing the network to learn and recognize the inherent internal temporal property of software failure sequence For one-step-ahead prediction, the input sequence and the desired output sequence should have one step delay during the learning process The desired objective is to force the network to recognize the one-step-ahead temporal pattern Once the network is trained based on all the currently available history failure data using the SVM learning procedure, the one-step-ahead failure time will be predicted Accordingly, the SVM learning process is iteratively and dynamically updated after every occurrence of new failure time data in order to capture the most current feature hidden in the software failure sequence 12.5 Validation of New Approaches Based on the modeling procedure described in D–ENN, RNN–BR, and A– SVM, three sets of experiments with four data sets are conducted By using the common data sets and performance evaluation metrics, it is possible to quantitatively validate the effectiveness of the new approaches The data set descriptions and experimental results are summarized in the following sections Computational Intelligence Methods in Software Reliability Prediction 391 12.5.1 Data Sets Description and Pre-processing The performance of these models have been tested using the same realtime control application and flight dynamic application data sets as cited in Park et al [26] and Karunanithi et al [18] We choose a common baseline to compare the results with related work cited in the literature All four data sets used in the experiments are summarized as follows: Dataset-1: Real-time command and control application consisting of 21,700 assembly instructions and 136 failures Dataset-2: Flight dynamic application consisting of 10,000 lines of code and 118 failures Dataset-3: Flight dynamic application consisting of 22,500 lines of code and 180 failures Dataset-4: Flight dynamic application consisting of 38,500 lines of code and 213 failures Dataset-1 is obtained from Musa et al [25] Dataset-2, Dataset-3, and Dataset-4 are equivalent to DATA-11, DATA-12, and DATA-13 as cited in Park et al [26] and Karunanithi et al [18] All the inputs and outputs of the network are scaled and normalized within the range of [0.1,0.9] to minimize the impact of absolute values For this purpose, the actual values are scaled using the following relationship: y= x 0.8 x + (0.9 − 0.8 × max ) ∆ ∆ (19) where, y is the scaled value we feed into our network, x is the actual value before scaling, x max is the maximum value in the samples xmin is the minimum value among all the samples, and ∆ is defined as ( x max − x ) After the training process, we test the prediction performance by scaling back all the network outputs to their actual values using the following equation: x= y − 0.9 × ∆ + xmax 0.8 (20) 12.5.2 Experimental Results Our choice for using specific performance measures for assessing the predictive accuracy was based on similar measures used by other researchers By using common data sets and performance evaluation metrics, it is 392 Liang Tian and Afzel Noore possible to quantitatively compare the efficacies of various approaches In addition, the relative error (RE) and/or average relative error (AE) are widely used in [9,17-19,24,26,30,32] for assessment of predictive accuracy of cumulative patterns Let xˆi be the predicted value of failure time and xi be the actual value of failure time n is the number of data points in the test data set Relative Error (RE) is given by: xˆ − x RE = | i i | xi Average Relative Prediction Error (AE) is given by: ) n xi − xi |×100 AE = ∑ | n i =1 xi (21) (22) The larger the value of predictability, or smaller the value of AE, the closer are the predicted values to the actual values Table summarizes the results of the three approaches when applied to software reliability prediction modeling Park et al [26] applied failure sequence number as input and failure time as desired output in feed-forward neural network (FF-NN) Based on the learning pair of execution time and the corresponding accumulated number of defects disclosed, Karunanithi et al [18] employed both feed-forward neural network (FF-NN) and recurrent neural network (R-NN) structures to model the failure process These results are also summarized in Table Table Comparison of Average Relative Prediction Error Comparison of Test Data Sets FF-NN R-NN FF-NN D-ENN RNN-BR A-SVM (Ref [26]) (Ref [18]) (Ref [18]) Dataset-1 2.72 1.83 2.44 2.58 2.05 2.50 Dataset-2 2.65 2.06 1.52 3.32 2.97 5.23 Dataset-3 1.16 0.97 1.24 2.38 3.64 6.26 Dataset-4 1.19 0.98 1.20 1.51 2.28 4.76 Data Set For example, using D-ENN approach with Dataset-3, the average relative prediction error (AE) is 1.16%; using RNN-BR approach, the average relative prediction error (AE) is 0.97%; and using A-SVM approach, the average relative prediction error (AE) is 1.24% These errors are lower than the results obtained by Park et al [26] (2.38%) using feed-forward neural network, Karunanithi et al [18] (3.64%) using recurrent neural network, and Computational Intelligence Methods in Software Reliability Prediction 393 Karunanithi et al [18] (6.26%) using feed-forward neural network In all four data sets, numerical results show that these modeling approaches have significantly improved prediction performance compared to other existing approaches 12.6 Discussions and Future Work In this chapter, we survey the literature on the most commonly used computational intelligence methods, including fuzzy logic, neural networks, genetic algorithms and their applications in software reliability prediction Secondly, we generalized some of the limitations of the existing modeling approaches Three emerging computational intelligence modeling approaches are then introduced facing those challenges Specifically, the new modeling approaches that focus on improving the dynamic learning and generalization capabilities are: In the dynamic learning model using evolutionary connectionist approach, as the number of available software failure data increases over time, the optimization process adaptively determines the number of input neurons and the number of neurons in the hidden layer The corresponding globally optimized neural network structure is iteratively and dynamically reconfigured and updated as new data arrive to improve the prediction accuracy Regularized recurrent neural network model is used for improving generalization capability Recurrent neural network has the inherent capability of developing an internal memory, which may naturally extend beyond the externally provided lag spaces Moreover, by adding a penalty term of sum of connection weights, Bayesian regularization approach is applied to the network training scheme to improve the generalization performance and lower the susceptibility of overfitting In the adaptive prediction model using support vector machines, the learning process is focused on minimizing an upper bound of the generalization error that includes the sum of the empirical training error and a regularized confidence interval This eventually results in better generalization performance Further, this learning process is iteratively and dynamically updated after every occurrence of new software failure data in order to capture the most current feature hidden inside the data sequence All the approaches have been successfully validated on applications related to software reliability prediction Numerical results show that these modeling approaches have improved prediction performance Some related 394 Liang Tian and Afzel Noore discussions and future research directions are summarized in the following sections 12.6.1 Data Type Transformation There are two common types of software failure data: time-betweenfailures data (time-domain data) and failure-count data (interval-domain data) The individual times at which failure occurs are recorded for timedomain data collection The time can either be actual failure time or time between successive failures The interval-domain approach is represented by counting the number of failures occurring during a fixed interval period, such as the number of failures per hour The D-ENN, RNN-BR, and A-SVM software reliability growth modeling approaches are flexible and can take different types of data as input Our approaches were originally intended for using time-domain data (actual failure time) as input to make predictions If it is assumed that the data collected are interval-domain data, it is possible to develop new models by changing the input-output pair of the network One type of software failure data can be transformed into another type in order to meet the input data requirement for a specific model Interval-domain data can be obtained by counting the number of failures occurring within a specified time period in time-domain data However, if it is needed to transform interval-domain data to time-domain data, this conversion can be achieved by randomly or uniformly allocating the failures for the specified time intervals, and then recording the individual times at which failure occurred Some software reliability tools integrate the capability of data transformation between two data types, such as CASRE (Computer-Aided Software Reliability Estimation) [21] 12.6.2 Modeling Long-Term Behavior The reason we focused on short-term prediction (one-step-ahead) in this research was to establish a baseline for comparison purposes with other known approaches We also believe it is more meaningful to make onestep-ahead prediction in certain types of applications in order to make early stage preventive action and avoid catastrophic events Meanwhile, it is of great interest for modeling and predicting long-term behavior of software failure process as well For example, suppose xi is the number of failures in the first i specified time intervals, and x0 , x1 , , xi −1 are used for predicting xi Once the predicted value of xi , Computational Intelligence Methods in Software Reliability Prediction 395 denoted by xˆi , is obtained, it is then used as input to the network to generate the predicted value of xi +1, denoted by xˆ i +1 Further, xˆi and xˆ i +1 are used as input to obtain xˆ i + , and so forth These new modeling approaches are also flexible and can deal with this situation by changing the inputoutput sequences 12.6.3 Assessment of Predictive Accuracy Our choice for using specific performance measures for assessing the predictive accuracy was largely predicated on similar measures used by other researchers We believe it is reasonable to compare our results with existing work using the same data sets and same performance evaluation metrics This provides us the opportunity to quantitatively gauge the efficacy of these new approaches We also acknowledge that there are other metrics for validating prediction accuracy Some examples are: sum of squared error (SSE) as used in [3,29,45]; mean of the sum of squared error (MSSE) as used in [4,13,43]; mean absolute deviation (MAD) as used in [16]; u-plot and prequential likelihood function as used in [6,7] 12.6.4 Incorporating Environmental Factors Recent studies show that using software testing time as the only influencing factor may not be appropriate for predicting software reliability [27,28] Some environmental factors should be integrated Examples of related environmental factors are program complexity, programmer skills, testing coverage, level of test-team members, and reuse of existing code [27,44] This is an area of research that can provide comprehensive insight to the problem of software reliability prediction Acknowledgements The authors would like to thank the reviewers for their helpful comments, and Drs J Y Park, N Karunanithi, Y K Malaiya, and D Whitley for providing data sets DATA-11, DATA-12, DATA-13 396 Liang Tian and Afzel Noore References [1] Adnan WA, Yaacob MH (1994) An integrated neural-fuzzy system of software reliability prediction In: Proceedings of the 1st International Conference on Software Testing, Reliability and Quality Assurance New Delhi, India, pp 154–158 [2] Adnan WA, Yaakob M, Anas R, Tamjis MR (2000) Artificial neural network for software reliability assessment In: 2000 TENCON Proceedings of Intelligent Systems and Technologies for the New Millennium Kuala Lumpur, Malaysia, pp 446–451 [3] Aljahdali SH, Sheta A, Rine D (2001) Prediction of software reliability: A comparison between regression and neural network non-parametric models In: Proceedings of ACS/IEEE International Conference on Computer Systems and Applications Beirut, Lebanon, pp 470–473 [4] Aljahdali SH, Sheta A, Rine D (2002) Predicting accumulated faults in software testing process using radial basis function network models In: Proceedings of the ISCA 17th International Conference on Computers and their Applications San Francisco, CA, pp 26–29 [5] Bhattacharya A, Parlos AG, Atiya AF (2003) Prediction of MPEG-coded video source traffic using recurrent neural networks IEEE Trans Signal Processing 51:2177–2190 [6] Brocklehurst S, Chan PY, Littlewood B, Snell J (1990) Recalibrating software reliability models IEEE Trans Software Eng 16:458–470 [7] Brocklehurst S, Littlewood B (1992) New ways to get accurate reliability measures IEEE Software 9:34–42 [8] Cai KY (1996) Introduction to Fuzzy Reliability, chapter Fuzzy Methods in Software Reliability Modeling Kluwer International Series in Engineering and Computer Science Kluwer Academic Publishers [9] Cai KY, Cai L, Wang WD, Yu ZY, Zhang D (2001) On the neural network approach in software reliability modeling The Journal of Systems and Software 58:47–62 [10] Cai KY, Wen CY, Zhang ML (1991) A critical review on software reliability modeling Reliability Engineering and System Safety 32:357–371 [11] Cai KY, Wen CY, Zhang ML (1993) A novel approach to software reliability modeling Microelectronics and Reliability 33:2265–2267 [12] Foresee FD, Hagan MT (1997) Gauss-Newton approximation to Bayesian learning In: Proceedings of the 1997 IEEE International Conference on Neural Networks Houston, TX, pp 1930–1935 [13] Fujiwara T, Yamada S (2002) C0 coverage-measure and testing-domain metrics based on a software reliability growth model International Journal of Reliability, Quality and Safety Engineering 9:329–340 [14] Guo P, Lyu MR (2004) A pseudoinverse learning algorithm for feedforward neural networks with stacked generalization applications to software reliability growth data Neurocomputing 56:101–121 Computational Intelligence Methods in Software Reliability Prediction 397 [15] Gupta N, Singh MP (2005) Estimation of software reliability with execution time model using the pattern mapping technique of artificial neural network Computers and Operations Research 32:187–199 [16] Ho SL, Xie M, Goh TN (2003) A study of the connectionist models for software reliability prediction Computers and Mathematics with Applications 46:1037–1045 [17] Karunanithi N (1993) A neural network approach for software reliability growth modeling in the presence of code churn In: Proceedings of the 4th International Symposium on Software Reliability Engineering Denver, CO, pp 310–317 [18] Karunanithi N, Whitley D, Malaiya YK (1992a) Prediction of software reliability using connectionist models IEEE Trans Software Eng 18:563–574 [19] Karunanithi N, Whitley D, Malaiya YK (1992b) Using neural networks in reliability prediction IEEE Software 9:53–59 [20] Leung FHF, Lam HK, Ling SH, Tam PKS (2003) Tuning of the structure and parameters of a neural network using an improved genetic algorithm IEEE Trans Neural Networks 14:79–88 [21] Lyu MR (1996) Handbook of Software Reliability Engineering McGrawHill, New York [22] MacKay DJC (1992) Bayesian interpolation Neural Computation 4:415–447 [23] Mihalache A (1992) Software reliability assessment by fuzzy sets IPB Buletin Stiintific - Electrical Engineering 54:91–95 [24] Musa JD (1998) Software Reliability Engineering, chapter Software Reliability Models McGraw-Hill Osborne Media, New York [25] Musa JD, Iannino A, Okumoto K (1987) Software Reliability: Measurement, Prediction, Application McGraw-Hill Series in Software Engineering and Technology McGraw-Hill Book Company [26] Park JY, Lee SU, Park JH (1999) Neural network modeling for software reliability prediction from failure time data Journal of Electrical Engineering and Information Science 4:533–538 [27] Pham H (2000) Software Reliability, chapter Software Reliability Models with Environmental Factors Springer [28] Pham H (2003) Software reliability and cost models: Perspectives, comparison, and practice European Journal of Operational Research 149:475–489 [29] Pham H, Zhang X (2003) NHPP software reliability and cost models with testing coverage European Journal of Operational Research 145:443–454 [30] Sitte R (1999) Comparison of software-reliability-growth predictions: Neural networks vs parametric-recalibration IEEE Trans Reliability 48:285–291 [31] So SS, Cha SD, Kwon YR (2002) Empirical evaluation of a fuzzy logicbased software quality prediction model Fuzzy Sets and Systems 127:199– 208 [32] Stringfellow C, Andrews AA (2002) An empirical method for selecting software reliability growth models Empirical Software Engineering 7:319–343 [33] Tian L, Noore A (2004a) A novel approach for short-term load forecasting using support vector machines International Journal of Neural Systems 14:329–335 398 Liang Tian and Afzel Noore [34] Tian L, Noore A (2004b) Software reliability prediction using recurrent neural network with Bayesian regularization International Journal of Neural Systems 14:165–174 [35] Tian L, Noore A (2005a) Dynamic software reliability prediction: An approach based on support vector machines International Journal of Reliability, Quality and Safety Engineering 12:309–321 [36] Tian L, Noore A (2005b) Evolutionary neural network modeling for software cumulative failure time prediction Reliability Engineering and System Safety 87:45–51 [37] Tian L, Noore A (2005c) Modeling distributed software defect removal effectiveness in the presence of code churn Mathematical and Computer Modelling 41:379–389 [38] Tian L, Noore A (2005d) On-line prediction of software reliability using an evolutionary connectionist model Journal of Systems and Software 77:173– 180 [39] Tsoukalas LH, Uhrig RE (1996) Fuzzy and Neural Approaches in Engineering John Wiley & Sons, New York [40] Utkin LV, Gurov SV, Shubinsky MI (2002) A fuzzy software reliability model with multiple-error introduction and removal International Journal of Reliability, Quality and Safety Engineering 9:215–227 [41] Vapnik VN (1999) An overview of statistical learning theory IEEE Trans Neural Networks 10:988–999 [42] Xing F, Guo P (2005) Support vector regression for software reliability growth modeling and prediction Lecture Notes in Computer Science 3496:925–930 [43] Yamada S, Fujiwara T (2001) Testing-domain dependent software reliability growth models and their comparisons of goodness-of-fit International Journal of Reliability, Quality and Safety Engineering 8:205–218 [44] Zhang X, Shin MY, Pham H (2001) Exploratory analysis of environmental factors for enhancing the software reliability assessment Journal of Systems and Software 57:73–78 [45] Zhang X, Teng X, Pham H (2003) Considering fault removal efficiency in software reliability assessment IEEE Trans Systems, Man, and CyberneticsPart A: Systems and Humans 33:114–120 колхоз 1/8/07 ... Type I MSS, ω s1 ( g1 , g ) = min( g1 , g ) (4 ) ω p1 ( g1 , g ) = g1 + g (5 ) g1 g g1 + g (6 ) ω p ( g1 , g ) = g1 + g (7 ) For Type II MSS, ω s ( g1 , g ) = Later, Ref [55] combines importance and... Recent Advances in Optimal Reliability Allocation, Computational Intelligence in Reliability Engineering (SCI) 39, 1–36 (2 007) www.springerlink.com © Springer-Verlag Berlin Heidelberg 2007 Way Kuo... Op ( g1 , g ) = max( g1 , g ) (1 3) ω sC ( g1 , g ) = max( g1 , g ) (1 4) ω Cp ( g1 , g ) = min( g1 , g ) (1 5) FC (GC , WC ) = WC − GC ≥ (1 6) FO (GO , WO ) = WO − GO ≥ (1 7) Thus, the system availability