A Method and Tool Support for Automated Data Flow Testing of Java Programs : M.A Thesis Information Technology : 60 48 01

VIETNAMNATIONALUNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY PHAM VAN CUONG A METHOD AND TOOL SUPPORT FOR AUTOMATED DATA FLOW TESTING OF JAVA PROGRAMS Major: Computer Science Supervisor: Dr Pham Ngoc Hung Hanoi 2014 ACKNOWLEDGEMENT First of all, I would like to express my sincere gratitude to my supervisor Dr Pham Ngoc Hung, University of Engineering and Technology, Viet Nam National University (VNU), Ha Noi for his enthusiastic guidance, warm encouragement and helpful research experiences I am grateful to thank all the teachers in University of Engineering and Technology, VNU who provide invaluable knowledge and life skills for me during the four academic years I would like to also thank my friends in K18-CS class who helped me during the four academic years Last, but not least, my family is really the biggest motivation for me My parents always encourage me when I have stress and difficulty I would like to send them great love and gratefulness AUTHORSHIP I hereby declare that this submission is my own work and to the best of my knowledgeit contains no materials previously published or written by another person, or substantial proportions of material which have been accepted for the award of any other degreeor diploma at University of Engineering and Technology (UET/Coltech) or any othereducational institution, except where due acknowledgement is made in the thesis Anycontribution made to the research by others, with whom I have worked at UET/Coltechor elsewhere, is explicitly acknowledged in the thesis I also declare that the intellectualcontent of this thesis is the product of my own work, except to the extent that assistance from others in the project's design and conception or in style, presentation and linguisticexpression is acknowledged Signature: ………………… Table of content Abstract Chapter 1: Introduction 1.1.Introduce to data flow testing 1.2 Applications of data flow testing 1.3 Related work 1.4 The goal of research 1.5 The organization of this thesis Chapter 2: Theory of Data Flow Testing 2.1 Basic idea 2.2 Static data flow testing 10 2.3 Dynamic data flow testing 12 2.3.1 Overview of dynamic data flow testing 12 2.3.2 Data flow graph 13 2.3.3 Data Flow Terms 13 2.3.3 Data Flow Testing Criteria 14 2.4 Summary 16 Chapter 3: A Method for Data Flow Testing 16 3.1 Data Flow Graph 17 3.2 Test path generation 21 3.3 Test case generation 24 3.3.1 Test data 24 3.3.2 Expected output 28 Chapter 4: Experiment and Discussion 31 4.1 Experiment 31 4.2 Discussion 37 Chapter 5: Conclusion 39 References 41 LIST FIGURES AND TABLES Fig.1.1 Limitation of different fault detection techniques[10] Fig.2.1 State transition diagram of a program variable[2] 10 Fig.3.1 Basic components of control flow 17 Fig.3.2 Structure of control statements 18 Fig.3.3 A source code and its data flow graph 19 Fig.3.4 Relative Strength of Testing Strategies 23 Fig.3.5 Process of generating test input data for data flow testing 25 Fig.3.6 Model of test oracle use an alternative program 29 Fig 3.7 Methods written by two different ways return the same results 29 Fig.4.1 Open a file java 31 Fig.4.2 Display data flow of a Java program 32 Fig.4.3 Generating test data 32 Fig.4.4 The test results of a Java program 33 Fig.4.5 The source code of the programmer 34 Fig.4.6 The source code of the programmer 34 Fig.4.7 Data flow of the application to calculate bill 36 Table 3.1 Def and c-use Sets of Nodes in Figure 3.3 20 Table 3.2 Predicates and P-use Set of Edges in Figure 3.3 20 Table 3.3 A test case of U is created from the path p 28 Table 3.4 The results of the two methods 30 Table 4.1 Billing rules 33 Table 4.2 The results of testing bill application 35 Abstract This thesis proposes a method and a tool support for automated data flow testing of Java programs The key purpose of this method is to detect improper uses of data values due to coding errors Given source code of a Java program, the proposed method analyzes and visualizes the program as a data flow graph All test paths for covering all definition-use pairs of all variables are then generated A test case corresponding to each generated test path is produced by identifying values to the input parameters so that the test path is executable The expected outputs of these test cases are identified automatically An implemented tool supporting the improved method and experimental results are also presented This tool is promising to be applied in practice Key words: software testing, data flow testing, white-box testing, dataflow anomaly, data flow coverage Chapter 1: Introduction 1.1 Introduce to data flow testing The main goals of software testing are to reveal bugs and to ensure that the system being developed complies with the customer‟s requirements To make testing effective, it is recommended that test planning/development begin at the onset of the project Software testing techniques can be divided into kinds: black box and white box techniques Black box testing is mainly a validation technique that checks to see if the product meets the customer‟s requirements On the other hand, white box testing is a verification technique which uses the source code to guide the selection of test data.Furthermore, software testing has been considered as the major solution in improving quality of software systems Currently, software companies focus only on the black box testing techniques in order to validate whether software products meet the customer‟s requirements By this approach, they only detect the errors/mistakes which can be observed by users As a result, all potential errors of program code can be not detected Moreover, detecting such errors has been recognized as a key difficult and expensive task in practice In addition, the testers in charge this task are required high level knowledge and skills for analyzing source code These issues are still open problems in software companies, especially in Vietnam Data flow testing has been known as a key white box testing technique that can be used to detect improper uses of data values due to coding errors [4] These errors are inadvertently introduced in a program by programmers Forinstance, a software programmer might use a variable without defining it, or he/she may define a variable, but not initialize it and then uses that variable in a predicate (e.g int x; if(x==100);) [4] The problem of errors in variables is common problems of programmers Each variable is classified as either a definition occurrence or a use occurrence A definition occurrence of a variable is where a value isassociated with the variable A use occurrence of a variable is where the value of the variable is referred Each use occurrence is further classified as a computational use (c-use) or a predicate use (p-use) If the value of the variable is used to decide whether a predicate is true for selecting execution paths, the occurrence is called a predicate use Otherwise, the occurrence is called a computational use Their criteria require that test data to be included which cause the traversal of sub-paths from a variable definition to either some or all of the p-uses, c-uses, or their combination We may note that there is much similarity between control flow testing and data flow testing Moreover, there is a key difference between the twoapproaches The similarities stem from the fact that both approaches identify program paths and emphasize on generating test cases from those program paths Thedifference between the two lies in the fact that control flow test selection criteriaare used in the former, whereas data flow test selection criteria are used in thelatter approach 1.2 Applications of data flow testing The primary purpose of dynamic data-flow testing is to uncover possible bugs in data usage during the execution of the code To achieve this, test cases are created which trace every definition to each of its use and every use is traced to each of its definition Ntafos [10] has reported on the results of an experiment comparing with the effectiveness of three test selection techniques The data flow testing, control flow testing, and random testing detected 90%, 85.5%, and 79.5% respectively, of the known defects Furthermore, Fig.1.1 shows the limitation of different fault detection techniques [10] These facts imply that data flow testing is one of the most effective methods for examining structure of programs Fig.1.1 Limitation of different fault detection techniques[10] 1.3 Related work Much of the formalization of define/use testing was done in the early 1980s [1,2]; the definitions in this chapter are compatible with those in [1,2], an article which summarizes most of define/use testing theory Firstly, with regards to test data, Tonella [21] performed the unit testing of classes using genetic algorithm In this approach test cases are generated for unit testing of classes using algorithm McMinn and Holcombe [20] proposed a solution for the state problem in evolutionary testing using ant colony model.Cheon et al [18] proposed automation of Java program testing at unit level using evolutionary approaches However, the types of data of these methods are limited Secondly, with regards to generate expected output is also known as test oracle, the oracles so far exactly compared the expected outputs with the actual However, this is not always feasible.Statistical Methods [13] and Artificial neural networks (ANNs)[14] have been used to identify the expected output These methods however, are not always feasible and require the implementation under test (IUT) Applying metamorphic testing to situations in which there is no test oracle has previously been studied by Chen et al [22] In some cases, these works have looked at situations in which there cannot be an oracle for a particular application; in others, the work has considered the case in which the oracle is simply absent or difficult to implement Although some tools supporting data flow testing such as BPAS - ATCGS (Basic Program Analyzer System Automatic Test Case Generation System) [8], JaBUTi [9], DFC (Data Flow Coverage) [3],etc, these tools only generate all paths for covering given source code In fact, we need a tool that assists the tester in creating test data [5] that include expected output Some free versions only allow testing the programs that are fixed in these tools and they are difficult to be extended in order to satisfy the specific data flow testing purposes of a certain software company 1.4 The goal of research One of the major difficulties in software testing is the automatic generation of test data and expected outputs In this thesis, wewill present a method to create test data and expected outputs bases on data flow testing of Java programs Given source code of a program, this method analyzes and visualizes the program as a data flow graph All test paths corresponding to all paths of the data flow graph for covering all definition-use pairs of all variables in the program are then generated All test cases of generated test paths are produced by giving values to the input parameters The set of the values to the input parameters and expected outputs of the produced test cases are also generated automatically In order to show the practical usefulness of the proposed method, a tool supports the method is implemented The obtained experimental results by applying this tool for some typical programs are completely reliable in detecting Final, let x = {2} and 1100){ if(Usage=100){ Bill=Bill+0.9; } } } return Bill; } Fig.4.6 The source code of the programmer The application to calculate the bill is tested by the JDFT tool and the outputs are given in Figure 4.5 and Table 4.2 34 Table 4.2 The results of testing bill application Stt Path Input(Usage) All dupath of variable Usage (1,2,3,4,5,6,7) 199 (1,2,3,5,6,7) Error (1,2,3,4,5,6,8) 295 (1,2,3,5,6,8) Error (1,2,3,4) 151 (1,2,3,4,5,6) 169 (1,2,3,5,6) Error (1,2,3,5,11) (1,2,3,4,5,11) 94 All dupath of variable Bill (2,3,5,6,7) Error (2,3,5,6,8) Error (2,3,5,6,8,9,11) 351 (2,3,5,11) -1 (2,3,5,6,8,9,11) Error (4,5,6,7) 157 (4,5,6,8) 269 (4,5,6,8,9,10) 354 (4,5,6,8,9,10,11) 388 10 (4,5,6,8,9,11) 250 11 (7,11) 157 35 Expected outputs Actual results Pass 89.5 Error 99.5 Error 65.5 74.5 Error 40.0 89.5 Error 99.5 Error 65.5 74.5 Error 40.0 True Error True Error True True Error True True Error Error 94.59 Error 68.5 96.9 94.86 97.92 95.0 68.5 Error Error 106.0 Error 68.5 96.9 106.3 109.7 95.0 68.5 Error Error False True Error True True False False True True Fig.4.7 Data flow of the application to calculate bill It is clear from the table 4.1 that there are some different results when the values of variable Usage is greater than 300 Therefore we have to check source codes of two programmers by using the paths which are false For example, the path (4,5,6,8,9,10,11) with the input parameter is 388 and the result is false Base on the path, we see that the difference between the source codes is the statements (Bill = Bill * 0.9;) and (Bill = Bill + 0.9;) When we fix this error and test the application again and the outputs between expected outputs and actual results are completely equal In addition, we also have some paths which are not executed because we not find the values to the input parameter (Usage) In this case, we need check logic errors of the program or unit program The above results are clear to show the practical usefulness of the implemented tool In addition, this tool also can detect the unnecessary statements of the program under testing 36 4.2 Discussion The proposed method is a complete solution for automated data flow testing It not only solves the key issues of white-box testing that generates test data, but also solves the problem about generating expected outputs Firstly, with regard to generate test data, by dividing boolean functions into three main groups helps us to minimize the disadvantages of exhaustive method [7] that is running time Generally, these functions will tend to fall into group and group Therefore, finding solution for these group is easy, whereas the boolean functions of group is fewer and research space of this group is also solution of group and group hence calculation time is reduced significantly While a brute-force search is simple to implement, and will always find a solution if it exists, its cost is proportional to the number of candidate solutions which in many practical problems tends to grow very quickly as the size of the problem increases Therefore, brute-force search is typically used when the problem size is limited, or when there are problem-specific heuristics that can be used to reduce the set of candidate solutions to a manageable size The method is also used when the simplicity of implementation is more important than speed.In addition, although there are some methods to generate test data such as Genetic Algorithm [12], Ant Colony Algorithms [11], these methods are restricted about basic data types, while our method has solved all basic data types However, it is still the restrictions as in some cases where the program includes complex boolean expressions, the calculation time is still high Secondly, for generating expected output, there are some automatic methods for generating expected output such as Statistical Oracle [13], Neural Network [14], and MT [15], but these methods are normally unavailable or too expensive to apply By using an alternate program, this method is not only simple, but it is also easy to apply for software companies Moreover, it also helps programmers have multiple perspectives on one problem, but this can also 37 make workload of programmers as well as the volume of program code increase significantly Finally, comparing with other data flow testing tools has not been done, but the obtained experimental results are clear to show that the method is promising to be applied in industry 38 Chapter 5: Conclusion We have presented a method for data flow testing of Java programs The key idea of this method is to generate all test cases such that they cover all definition-use pairs of all variables used in the Java program under testing The expected outputs of the generated test cases are also computed automatically by using an alternate program A tool supports the proposed method is implemented in Java Some typical examples are tested in order to show practical usefulness of the tool The obtained experimental results are clear to show that the implemented tool candetect several common errors in coding Theoretical analysis and practical testing results show that the approach is simpler, easier, and more effective in generating test data automatically.Experiments have been carried out to evaluate the effectiveness of the proposed The results of these experiments showed the practical usefulness of the JDFT tool and candetect several common errors in coding One of the major difficulties in software testing is the automatic generation of test data that satisfy a given adequacy criterion However, test data can be generated manually, but this may become difficult In addition, generating the expected outputsis still challenging to test applications and functions forwhich the correct output for arbitrary input cannot beknown in advance, e.g some computational science or machine learning applications Future work, we focus on solving the problem about test data, especially find out test paths which are not executable to remove unnecessary statements due to these statements is never executed We are also investigating to apply some practical programs with their sizes are larger than the sizes of the applied systems in order to show the practical usefulness of the implemented tool In addition, more experiments are needed in order to evaluate and compare the 39 proposed method and the existing methods for data flow testing In addition, we also are going to apply this tool in some Vietnamese software companies 40 References S Rapps and E.J.Weyuker, Selecting software test data using data flow information, IEEE Transactions on Software Engineering, vol.11, no 4, pp 367-375 (1985) Parrish, A.S.; Zweben, S.H, On the relationships among the all-uses, all-DU-paths, and all-edges testing criteria, Software Engineering, IEEE Transactions, vol.21, issue 12, pp 1006-1009 (1995) Ilona Bluemke, A coverage analysis tool for java Programs, In the 4th IFIP TC Central and East European conference on Advances in Software Engineering Techniques, pp 215-228 (2009) Lee Copeland, A Practitioner’s Guide to Software Test Design, STQE Publishing (2004) N.Narmada and D P Mohapatra, Automatic test data generation for data flow testing using particle swarm optimization Communications in Computer and Information Science, Vol.95, No.1, pp 1-12 (2010) PDM Homepage, http://pmd.sourceforge.net/ Exhaustive research, http://en.wikipedia.org/wiki/Brute-force search BPAS-ATCGS Homepage, http://www.cs.ucy.ac.cy/ cs04pp2/WebHelp/index.htm JaBUTi Homepage, http://jabuti.incubadora.fapesp.br (access, December 2007) 10 S.C Ntafos, On required element testing IEEE Transactions on Software Engineering, vol.10, issue.6, pp 795803 (1984) 11 Ahmed S Ghiduk, A new software data-flow testing approach via ant colony algorithms, 2010 UniCSE, ISSN: 2219-2158 12 Moheb R Girgis, Automatic test data generation for data flow testing using a genetic algorithm, J UCS, vol 11, no 6, pp 898-915 (2005) 13 Johannes Mayer, Ralph Guderlei, Test oracles using statistical methods, Proceedings of the First International Workshop on Software Quality, Lecture Notes in Informatics P-58, pp 179-189 (2004) 14 Meenakshi Vanmali, Mark Last, Abraham Kandel, Using a neural network in the software testing process, International Journal of Intelligent Systems, vol 17, Issue 1, pp 4562, January 2002 15 Peifeng Hu, Zhenyu Zhang, W K Chan, T H Tse, An empirical comparison between direct and indirect test result checking approaches, Proceedings of the 3rd International Workshop on Soft Quality Assurance (SOQUA06), pp - 13 (2006) 16 J C Huang Detection of Data Flow Anomaly through Program Instrumentation IEEE Transactions on Software Engineering, May 1979, pp 226–236 41 17 L D Fosdick and L J Osterweil Data Flow Analysis in Software Reliability Computing Surveys, September 1976, pp 305–330 18 Cheon, Y., Kim, M.Y., Perumandla, A.: A complete Automation of Unit Testing for Java Programs In: The 2005 International Conference on Software Engineering Research and Practice (SERP), Las Vegas, Nevada, USA (June 2005) 19 Dharsana, C.S.S., Askarunish, A.: Java based Test case Generation and Optimization Using Evolutionary Testing In: International Conference on Computational Intelligence and Multimedia Applications, Sivakasi, India (December 2007) 20 McMinn, P., Holcombe, M.: The State Problem for Evolutionary Testing, vol 2724 In: LNCS, pp.2488-2497 Springer, Heidelberg (2003) 21 Tonella, P.: Evolutionary Testing of Classes In: Proceedings of the ACM SIGSOFT International Symposium of Sofware Testing and Analysis, Boston, MA, pp 119-128 (July 2004) 22 T.Y Chen, J Feng, and T.H Tse Metamorphic testing of programs on partial differential equations: a case study In Proceedings of the 26th Annual International Computer Software and Applications Conference (COMPSAC 2002), pages 327–333 IEEE Computer Society Press, Los Alamitos, California, 2002 42