Applying the meta heuristic algorithms for mutation based test data generation for simulink models

Applying the meta-heuristic algorithms for mutation-based Test data generation for Simulink models Le Thi My Hanh, Nguyen Thanh Binh, Khuat Thanh Tung DATIC Laboratory University of Science and Technology Danang December 2014 Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Contents Introduction  Simulink  Mutation testing Related work Mutation testing for Simulink  Mutation operators  Mutant generation and execution - MuSimulink Test data generation  Random test data generation  Mutation-based test data generation Experimentation Conclusion Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Simulink Mutation testing Simulink  Block diagram environment  multi-domain simulation, model-based design  Being widely used for developing embedded systems, control systems…  aerospace, automobile and electronics systems  Provides a graphical editor, customizable block libraries  A Simulink model consists of two main elements: blocks and lines  blocks are the functional units, used to generate, manipulate and output signals  blocks are connected by lines that provide the mechanism to transfer signals  It is integrated with Matlab, enabling to incorporate Matlab algorithms into models and export simulation results to Matlab for further analysis Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Simulink Mutation testing Software testing  Important activity in software process in order to assure the quality  Discovering faults  Detecting faults as soon as possible in software process to reduce development costs  Detecting faults in design phase  Several testing techniques: black-box, white-box  Mutation testing: popular and easy to automate  Mutation testing for Simulink models Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Simulink Mutation testing Mutation testing  Fault-based testing technique  Measuring the quality of a test set according to its ability to detect specific faults  Faults are inserted into the original program  Mutation operators: representing common faults usually found in program  Faulty program is called mutant  Example Original Mutant if (a > 5000) if (a < 5000) Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Simulink Mutation testing Mutation testing  Process  Generating mutants  Executing the original program and the mutant on the same test data  Comparing the output of the original program and the output of the mutant  If they are different, the mutant is killed  If they are not different, the mutant is alive Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Simulink Mutation testing Mutation testing  Concepts  Killed mutant  There exists a test case distinguishing the original program and the mutant  Alive mutant  There is no test case distinguishing the original program and the mutant  Equivalent mutant  The outputs of the original program and the mutant are always the same  Mutation score (MS)  The score of the number of killed mutants over the total number of non equivalent mutants  Measuring the quality of the set of test cases Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Simulink Mutation testing Mutation testing  Disadvantages  determining equivalent mutants  generating mutants  cost of executing mutant  test data generation Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Related work Mutation testing • For some programming languages: C, Fortran, Ada, Java, SQL, • Three mutation operators for Simulink: add, multiply, assign for signals carried on inputs (Zhan Clark) Mutation testing tools • Mothra (Fortran), MuJava (Java), Proteum (C), SQLMutation (SQL), Cost reduction techniques Do fewer • Selective mutation • Mutant sampling Do faster • Schema-based Mutation Analysis • Separate compilation • Compilation bottleneck Do smarter • Weak mutation • Distributed Architectures: SIMD (Krauser - 1991), MIMD (Offut - 1992) Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Objectives Mutation operators MuSimulink Tool Objectives  Defining a set of mutation operators for Simulink  Basing on common faults committed by designers  Developing a mutation testing tool for Simulink - MuSimulink  Automatic generation of mutants  Executing mutants  Improving the cost by applying parallelism  Generating test data  Improving test data by meta-heuristic algorithms 10 Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Random test data generation Mutation-based test data generation Mutation-based test data generation  Using meta-heuristic algorithms  Genetic Algorithm (GA)  Simulated Annealing (SA)  Artificial Immune System (AIS) 15 Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Random test data generation Mutation-based test data generation Using Genetic Algorithm Individual representation Set of test cases Genetic Algorithm Roulette Wheel Selection Technique 16 Individual selection Retaining the best individual in each generation Father Crossover operator Two points crossover Mother Child Child Mutation operator Fitness Function Single point mutation: replacing the test case at randomly chosen locus by another (random generated test case) The total number of killed mutants by all the tests combined into the individual Introduction Related work Mutation operators for Simulink Mutant generation and execution Test data generation Experimentation Conclusion Random test data generation Mutation-based test data generation Using Simulated Annealing Simulated Annealing Solution representation Search process Being controlled by gradually changing the value of a parameter called temperature, from high to low Depending on the current temperature Mutating the current solution to get a new solution Selecting the best solution New solutions with lower cost are always accepted The probability of accepting a worse state is given by the equation: 𝑒 −𝛿/𝑇 𝛿 = 𝑓 𝑛𝑒𝑤𝑆𝑜𝑙𝑢𝑡𝑖𝑜𝑛 − 𝑓 𝑐𝑢𝑟𝑆𝑜𝑙𝑢𝑡𝑖𝑜𝑛 T : current temperature Cost function 17 Set of test cases f = – Mutation Score Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Random test data generation Mutation-based test data generation Artificial Immune System Using Artificial Immune System 18 Inspired by human immune system Four main immunological theories B-cells produce antibodies Interaction between antibody and antigen affinity negative selection theory immune network theory danger theory Clonal selection theory Selecting n of the highest affinity antibodies The number of clones for an antibody is proportional to its affinity The mutation is inversely proportional to the affinity Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Random test data generation Mutation-based test data generation Using Artificial Immune System  Mapping between AIS and mutation testing AIS 19 Mutation testing B-cell Model under test Antigen Mutant Antibody Test data Affinity Mutation score Clonal Selection Evolving test data Memory cells The memory set to save test data which is able to kill mutants Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Comparing approaches The improvement of genetic algorithm Experimentation  Configuration  CPU: Intel Core Quad CPU Q6600 , 2.4 GHz  Ram: 2GB  OS: Windows  Using Simulink models 20 Model SmplSw Quadratic_v1 RandMdl Tiny Quadratic_v2 No of Input Var No of Mutants 92 161 188 144 140 Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Comparing approaches The improvement of genetic algorithm The effectiveness of the algorithms The best mutation scores of experimentations Model No of mutants The average execution time Mutation Score (%) Time (s) AIS GA SA AIS GA SA SmplSw 92 96.74 95.65 94.57 2772.85 14348.81 10474.15 Quadratic_v1 161 88.82 86.33 87.58 8619.93 21445.35 19943.68 RandMdl 188 94.68 86.17 88.30 6629.87 23370.20 21821.83 Tiny 144 93.06 86.81 88.19 4451.38 12474.87 10360.35 Quadratic_v2 140 81.43 79.29 75.71 12670.05 34858.63 32120.53 21 Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Comparing approaches The improvement of genetic algorithm The effectiveness of the algorithms % 100.00 90.00 80.00 70.00 60.00 50.00 40.00 30.00 20.00 10.00 0.00 22 Comparing the execution time of approaches GA SmplSw Quadratic_v1 RandMdl SA Tiny AIS Quadratic_v2 Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Comparing approaches The improvement of genetic algorithm Hybrid Artificial Immune Genetic Algorithm  Individual representation  Each individual contains only a test data: v = {a1, a2,…, an},  n is the number of variable and/or inport of Simulink model and is the value of ith input data  Genetic operators  Selection operator and mutation operator are the same as the GA  Crossover operator  Single point crossover between two parent individuals to create two offspring  Only one child individual is created and its value will be the arithmetic mean of values in two parent individuals  If the number of inputs of Simulink model is one  This crossover always occur in the each generation so we not use crossover probability  The fitness for each individual  the fitness computed for each individual by basing on alive mutants in that generation  The fitness function for each individual is the total number of killed mutants by that individual per the total number of alive mutants in current genetic generation  test data that are able to kill at least one mutant which is not killed by any 23 individual else will be added to memory set Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Comparing approaches The improvement of genetic algorithm The effectiveness of the proposed hybrid algorithm Parameter values of the algorithms Test Set HAIGA nG pSz GA mP AIS nTCpi pSz nG cR mR nIs pSz nISP nC rPS 20 500 0.5 25 20 20 0.9 0.5 20 500 20 20 20 1000 0.5 40 25 20 0.9 0.5 20 1000 20 20 10 1000 0.5 40 25 10 0.9 0.5 10 1000 20 20 The empirical results of Tiny model Mutation Score (%) Test Set HAIGA GA AIS 24 Time (s) HAIGA GA AIS 90.97 86.81 88.89 7168.9 10009.9 2937.8 93.06 88.89 93.06 12802.1 17632.1 4020.5 93.06 86.81 93.06 7629.6 3130.1 9839.5 Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Comparing approaches The improvement of genetic algorithm The effectiveness of the proposed hybrid algorithm Parameter values of the algorithms Test Set HAIGA nG pSz GA mP AIS nTCpi pSz nG cR mR nIs pSz nISP nC rPS 20 500 0.5 25 20 20 0.9 0.5 20 500 20 20 20 1000 0.5 40 25 20 0.9 0.5 20 1000 20 20 10 1000 0.5 40 25 10 0.9 0.5 10 1000 20 20 The empirical results of Tiny model Mutation Score (%) Time (s) Test Set HAIGA GA AIS HAIGA GA AIS 160000 The number of interpreted mutants 140000 GA 120000 HAIGA AIS 100000 80000 90.97 86.81 88.89 7168.9 10009.9 2937.8 93.06 88.89 93.06 12802.1 17632.1 4020.5 40000 93.06 86.81 93.06 7629.6 9839.5 3130.1 20000 60000 25 10 11 Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Conclusion  Mutation testing  An effective technique for evaluating the quality of test cases  Costly  big number of mutants  Set of mutation operators  Automatic framework for Simulink  Generating mutants, Generating test data, Executing mutants, Analysing results 26 Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Conclusion  Test data generation  Using meta-heuristic algorithms  Genetic Algorithm (GA)  Simulated Annealing (SA)  Artificial Immune System (AIS)  the time of test data generation of AIS is the least, and GA is the most  the AIS is stable and it has also significantly improved the number of killed mutants compared to the GA and SA methods  Improving Genetic Algorithm  Hybrid Artificial Immune Genetic Algorithm  The number of killed mutants is greater than that of conventional genetic algorithm and even AIS  Has less execution time than conventional genetic algorithm, but more than AIS 27 Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Conclusion  Ongoing and future work  Test data generation  Improving fitness function to determine test data which kills a specific mutant  Applying more algorithms 28 View publication stats ... work Mutation testing for Simulink Test data generation Experimentation Conclusion Random test data generation Mutation- based test data generation Test data generation  Automatic test data generation. ..  Applying the meta- heuristic algorithms 13 Introduction Related work Mutation testing for Simulink Test data generation Experimentation Conclusion Random test data generation Mutation- based test. .. Executing the mutant models on test data Verifying if the test data kills mutants? Marking the test data 14 Introduction Related work Mutation testing for Simulink Test data generation Experimentation