Software bug management from bug reports to bug signatures

SOFTWARE BUG MANAGEMENT FROM BUG REPORTS TO BUG SIGNATURES CHENGNIAN SUN NATIONAL UNIVERSITY OF SINGAPORE 2013 SOFTWARE BUG MANAGEMENT FROM BUG REPORTS TO BUG SIGNATURES CHENGNIAN SUN (BEng., Northeastern Univeristy (China), 2007) A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF COMPUTER SCIENCE SCHOOL OF COMPUTING NATIONAL UNIVERSITY OF SINGAPORE 2013 Declaration I hereby declare that the thesis is my original work and it has been written by me in its entirety. I have duly acknowledged all the sources of information which have been used in the thesis. This thesis has also not been submitted for any degree in any university previously. Chengnian Sun 28 March 2013 Acknowledgements I would like to express my gratitude to all those who advised, helped and collaborated with me to complete this thesis. First and foremost, my deep and sincere appreciation goes to my supervisor Dr. Siau-Cheng Khoo for his encouragement, continuous support and patient guidance, which give me critical thinking and fundamental skills to survive in research. I also appreciate the freedom from him which allows me to what I am interested in. I would like to thank my thesis advisory committee: Dr. Jin Song Dong and Dr. Wei Ngan Chin for their constructive suggestions on my research. Without them, I cannot conclude my Ph.D. study so soon. I also thank my mentors Dr. Haidong Zhang and Dr. Jian-Guang Lou at Microsoft Research Asia for offering me the internship, thus expanding my research interest, and the insightful discussion with me. Much gratitude is owed to Dr. David Lo and Dr. Jing Jiang of Singapore Management University for their great help on my research topic selection at the early stage of my Ph.D. study. Without them, the publication of my first paper would be much postponed. My special gratitude goes to Dr. Stan Jarzabek for his recommending me to the NUS Ph.D. program and the warm hospitality when I first came to Singapore in 2008. He helped ease the relocation trouble and relieve my anxiety. I am also grateful to my collaborators, who have helped diversify my research: Hong Cheng from Chinese University of Hong Kong, Yuan Tian from Singapore Management University, Xiaoyin Wang from Peking University, Jing Du from Chinese Academy of Science, and Ning Chen from Nanyang Technological University. My appreciation goes to my senior Dr. Sandeep Kumar, for the generous help and information. I gratefully acknowledge the support, friendship and help from my lab mates in Programming Language and Software Engineer Lab and friends from Programming Language and Software Engineer Lab 1, which leaves a wonderful memory of my Ph.D. life. Last but not the least, I would like to thank my family. First, I am grateful to my parents Chuanli Sun and Shuyan Wang for their unconditional love, and the enduring great effort to raise me to who I am today. Second, I rent a flat in Singapore, but my wife Dr. Shaojie Zhang gives me a home here. Without her, my life these years would not be as bright and happy as it is now. Moreover, she agrees to use family fund to buy lenses and accessories for my future “photography career”, which makes me very happy:-) Contents List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ii List of Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i Introduction and Overview 1.1 Duplicate Bug Report Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Bug Signature Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Discriminative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Thesis Outline and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Acknowledgement of Published Work . . . . . . . . . . . . . . . . . . . . . . . A Discriminative Model Approach for Duplicate Bug Report Retrieval 2.1 2.2 13 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.1.1 Duplicate Bug Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.1.2 Information Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.1.3 Building Discriminative Models via SVM . . . . . . . . . . . . . . . . 18 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2.1 Overall Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.2.2 Data Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2.3 Training a Discriminative Model . . . . . . . . . . . . . . . . . . . . . 21 i 2.3 2.4 2.5 2.2.4 Applying Models for Duplicate Detection . . . . . . . . . . . . . . . . 26 2.2.5 Model Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.3.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 2.3.2 Experimental Details and Result . . . . . . . . . . . . . . . . . . . . . 31 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.4.1 Runtime Overhead . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 2.4.2 Feature Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Chapter Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Towards More Accurate Retrieval of Duplicate Bug Reports 3.1 3.2 3.3 3.4 37 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.1.1 Duplicate Bug Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.1.2 BM25F . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.1.3 Optimizing Similarity Functions by Gradient Descent . . . . . . . . . . 42 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.2.1 Extending BM25F for Structured Long Queries . . . . . . . . . . . . . 44 3.2.2 Retrieval Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.2.3 Optimizing REP with Gradient Descent . . . . . . . . . . . . . . . . . 47 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.3.1 Experimental Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.3.2 Effectiveness of BM25Fext . . . . . . . . . . . . . . . . . . . . . . . . . 56 3.3.3 Effectiveness of Retrieval Function . . . . . . . . . . . . . . . . . . . . 56 Chapter Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 ii An Information Theoretic Approach to Bug Signature Identification 61 4.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 4.2 Bug Signature Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.3 4.4 4.5 4.6 4.2.1 Discriminative Significance . . . . . . . . . . . . . . . . . . . . . . . . 70 4.2.2 Equivalence Classes of Bug Signatures . . . . . . . . . . . . . . . . . . 71 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 4.3.1 Reducing Subgraph Isomorphism . . . . . . . . . . . . . . . . . . . . . 75 4.3.2 Reducing Graph Mining . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4.3.3 Upper Bound of DS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4.3.4 Intra-Procedural Signature Mining . . . . . . . . . . . . . . . . . . . . 81 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 4.4.1 Proximity to Actual Bug . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.4.2 Experiment Set-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.4.3 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.5.1 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.5.2 Graph Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.5.3 Inter- vs Intra-Procedural Mining . . . . . . . . . . . . . . . . . . . . . 89 4.5.4 Threats of Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Chapter Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Mining Succinct Predicated Bug Signatures 5.1 5.2 93 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.1.1 Overall Workflow and Instrumentation Scheme . . . . . . . . . . . . . 96 5.1.2 Itemset Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 iii 5.3 5.4 5.5 5.2.1 Bug Signature . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 5.2.2 Discriminative Significance . . . . . . . . . . . . . . . . . . . . . . . . 103 5.2.3 Top-k Bug Signatures . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 5.3.1 Gr-tree to Mine Itemset Generators . . . . . . . . . . . . . . . . . . . . 105 5.3.2 Algorithm Skeleton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 5.3.3 Mining Discriminative Generators Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 5.4.1 Objective Comparison with LEAP . . . . . . . . . . . . . . . . . . . . 114 5.4.2 Proximity to Actual Bug . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.4.3 Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 5.4.4 A Debugging Session for sed . . . . . . . . . . . . . . . . . . . . . . . . 119 5.4.5 A Debugging Session for tcas . . . . . . . . . . . . . . . . . . . . . . . 120 5.4.6 Threats to Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5.4.7 Limitation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Chapter Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Related Work 6.1 125 Duplicate Bug Report Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . 125 6.1.1 6.2 . . . . . . . . . . . . . . . . . . . . 109 Other Bug Report Related Studies . . . . . . . . . . . . . . . . . . . . 127 Bug Signature Identification . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Conclusion 131 Future Work 135 Bibliography 137 iv Summary Bugs are prevalent in software systems, and how to effectively manage bugs has been an important activity in the software development life cycle. For example, in software testing and maintenance phases, bug report repositories are used for filing bugs and tracking their status changes; when a bug occurs, automatic techniques can be employed to extract useful information from failure runs for developers to diagnose the bug. In this thesis, we study two specific problems in bug management: duplicate bug report retrieval and bug signature identification. In particular, we focus on how discriminative techniques can be applied to these two problems. In a bug tracking system, different testers or users may submit multiple reports on the same bugs, referred to as duplicates, which may cost extra maintenance efforts in triaging and fixing bugs. In order to identify such duplicates accurately, we propose two approaches. First, we leverage recent advances on using discriminative models for information retrieval to detect duplicate bug reports, by considering words in summary and description fields with different degrees of importance and using learning technique to automatically infer word importance. Second, we propose an enhanced retrieval function (REP) to measure the similarity between two bug reports, which fully utilizes the information available in a bug report including not only the similarity of textual content in summary and description fields, but also similarity of non-textual fields such as product, component, version, etc The case studies show that our technique can identify 71–78% duplicate reports for real-world software projects by retrieving top-20 similar reports. The other problem we investigate is identifying bug signatures from a buggy program for debugging purpose. A bug signature is a set of program elements highlighting the cause or effect of a bug, and provides contextual information for debugging. In order to mine a signature for a buggy program, two sets of execution profiles of the program, one capturing the correct execution and the other capturing the faulty, are examined to identify the program elements contrasting faulty from correct. In this thesis, we adopt the viewpoint that a bug signature should be (1) inclusive of the cause or the effect of a bug manifestation, and (2) succinct in its representation. We first investigate mining signatures consisting of control flow information from profiles recording executed basic blocks and branches. Several mining techniques – including graph mining techniques – have been employed to discover such signatures, with some success. However, there has not been a clear characterization of good bug signatures. In this thesis, we take an information theoretic approach to bug signature identification. We classify signatures into various equivalence classes, and devise efficient algorithms for discovering signatures representative of these equivalence classes. v Sandusky et al. study the statistics of duplicate bug reports [69]. Hooimeijer and Weimer develop a model to predict bug report quality [31]. Bettenburg et al. survey developers of Eclipse, Mozilla, and Apache to study what developers care most in a bug report [10]. Bettenburg et al. later show that duplicate bug reports might be useful [11]. Duplicate bug reports could potentially provide different perspectives to the same bug potentially enabling developers to better fix the bug in a faster amount of time. Still there is a need to detect bug reports that are duplicate of one another. 6.2 Bug Signature Identification This section surveys four lines of research closely related to our work, i.e. bug signature mining, fault localization, graph mining and discriminative pattern mining. Bug Signature Mining. RAPID by Hsu et al. [32] utilizes BIDE [83] to discover bug signatures as longest common subsequences from a multi-set of sequences of suspicious program elements in the faulty execution traces. These longest common subsequences are sorted by their lengths and output as bug signature. Cheng et al. [18] extend RAPID by replacing sequences of program elements by corresponding behavior graphs. They employ a top-k discriminative graph mining algorithm, based on LEAP, to mine bug signatures. They have shown that their approach outperforms RAPID. In Chapter 4, we emphasize the succinctness and accuracy of bug signature and efficiency of signature detection by leveraging the unique characteristics of software behavior graphs and reducing graph mining problem to customized item-set generator mining, we successfully show that our method attains scalability while ensuring succinctness and accuracy. In Chapter 5, we introduce predicated bug signatures by incorporating data predicates. This further extends the ability of bug signatures to capture more bug scenarios than solely control-flow-based signatures, e.g. those by MBS and LEAP. 128 Fault Localization. A closely related and active research direction is that of pinpointing the location of bugs given failure. The approach can be categorized into spectrum-based and model-based fault localization. In the spectrum-based approaches program profiles or spectra obtained from failed and correct executions are analyzed [89, 39, 1, 65, 46, 22, 48, 7, 8, 91, 35, 21]. Model based approaches build formal models of a program and perform logical reasoning on the models [30, 54, 25]. Some of the spectrum-based fault localization techniques include comparing a failed execution with the nearest correct execution [65], finding minimum state differences [89] and predicates that are correlated with failures [46, 91, 48], ranking program elements based on their “suspiciousness", such as Tarantula [39] and Ochiai [3], and reducing confounding bias in statistical debugging metrics [29, 9]. Our approach is more closely related to Spectrum-based approach. We however derive minimal bug signatures to capture the locations of the bug or its manifestation. We adapt the metric used in [65, 22] to determine the proximity of a signature to a bug from the program’s PDG. In addition, we devise new metric to measure the relevancy of such a signature to a bug. Moreover, our technique is orthogonal to those in [29, 9], and the performance can be further improved by adopting these techniques to reduce the control and data flow dependence confounding bias. Discriminative Pattern Mining. Hong et al. mine discriminative itemset patterns based on information gain in [19]. They later scale-up the approach further by introducing a mining procedure that fuses the mining and filtering process [20]. Nijssen et al. employ a constraint solver to scale up the mining of discriminative itemset further [60]. Lee et al., Lo et al., and Kim et al. mine discriminative patterns from sequences in various formats in [43, 49, 40, 50]. Yan et al. mine discriminative subgraph patterns [86] which is later extended by Cheng et al. [18] to mine bug signatures. Jin et al. employ an evolutionary algorithm to mine discriminative subgraph patterns [37]. Different from the above studies, in this thesis we apply information theory to discovery of 129 discriminative software behavior in terms of control flow elements or predicates. that could discriminate correct and faulty software executions. We apply the mined patterns to extract bug signatures. 130 Chapter Conclusion The prevalence of bugs in software systems has made software bug management an important aspect in software life cycle. Effective and efficient management can not only alleviate developers’ burden in developing and maintaining software, but also save time and financial cost for organizations. In this thesis, we present several solutions to problems in bug management, i.e., duplicate bug report retrieval and bug signature identification. Moreover, all our solutions are based on discriminative analysis. In general, for each problem, we analyze the historical data, and partition them into two classes positive and negative according to the characteristics of the problem. After contrasting the two classes, we get the discriminative difference, which is used as the solution or for building the solution. The two problems we address here correspond to two serial activities in software life cycle: reporting bugs and fixing bugs. Bug tracking systems enable end users to notify developers of the bugs they encountered, and eventually leading improvement in software quality. However, such activities, if not properly coordinated, can lead to bug report management challenge. Specifically, triagers are faced with managing a mountain of reports submitted by users, and have difficulty identifying 131 duplicate reports. One of the solutions to this problem is thus to retrieve similar reports from the bug repository for each new report, enabling a triager to determine if the new report is indeed not a duplicate. When developers start diagnosing reported bugs, it will be a blessing to have some tools to help them identify bugs in the programs. There have been considerable automatic debugging approaches proposed in recent years, and one of them is bug signature identification capturing contextual information for a bug under which the bug manifests. For the first problem of duplicate bug report retrieval, we propose two approaches to improving the retrieval accuracy of duplicate reports. The two techniques share the same characteristics, that is, we use machine learning techniques to automatically learn a retrieval function measuring the similarity of two bug reports, rather than handcraft one. The learning is done by contrasting pairs of duplicate reports from pairs of non-duplicate reports, namely, in a discriminative model manner. In Chapter 2, we argue that the degree of the importance of terms in summary and description fields should not be manually specified, but be inferred from the existing reports in bug repository via learning techniques. In Chapter 3, we propose an enhanced retrieval function REP , consisting of a better textual similarity component based on BM25F retrieval measure and several other components characterizing the similarity of non-textual fields, such as project, component and report type. The case studies show that our techniques outperform simple similarity measures, i.e., vector space model. By retrieving top-20 similar reports, 71–78% of the duplicate reports in real-world software projects can be correctly identified. For the second problem, we propose two solutions to bug signature identification. In Chapter 4, we mine control-flow-based signatures, which are sub-graphs of the static control-flow graphs, capturing the cause or effect of bugs. We prove that the NP-complete sub-graph isomorphism checking in this problem can be reduced to subset inclusion checking 132 which can be implemented in linear time. Hence the complex graph mining problem can be reduce to a simpler itemset mining. Also we propose two desired properties inclusiveness and succinctness to require signatures cover the bug without irrelevant or redundant information in an information-theoretic way. The empirical result shows significant improvement – in comparison with related work – in both the accuracy of the bug signatures produced (with relative improvement up to 1578%), and the time efficiency of the mining task 225–2939 times faster. In Chapter 5, we propose the notion “predicated bug signatures”, which does not only contain control-flow information, but also data predicates capturing the program states at runtime. From the case studies, with the presence of data predicates, the functionality of our signatures is extended to enable developers to diagnose a class of bugs, the manifestation of which does not trigger any deviation in control-flow transitions from correct executions, and thus cannot be detected by control-flow-based signatures. 133 134 Chapter Future Work For duplicate bug report retrieval problem, we are investigating the indexing technique to speed up the retrieval process. Currently, the retrieval is performed by scanning each report in the repository, which will slow down if the repository contains a large number of reports. Different from existing search engine systems where a query is un-structured and usually short with several terms, the query in duplicate report retrieval is structured with multiple fields and can be lengthy. These differences bring challenges and also optimization opportunities for instant duplicate report retrieval. For bug signature identification, we have two future directions. First, in Chapter 4, the signatures mined can be disjoint, and it will be interesting to see whether connected signatures can further improve the predictive power of bug signatures. Imposing connectivity constraint on signatures requires a new mining algorithm. Since all the elements in a signature should be connected, the search space of this new problem is much smaller than that of control-flow signatures, namely in the former space, all disjoint patterns are filtered away. Second, We are in the process of developing a debugging environment that assists programmers in inferring the cause of bugs from the data predicates present in signatures. As we 135 define a signature as the cause or effect of a bug, the signature may not pinpoint the exact location of its bug but the manifestation. Thus we plan to incorporate the information of programs and employ program analysis techniques (i.e. slicing [82]) to automatically infer the cause. There are two research questions we plan to address. 1. How can profiles help improving the precision of slicing? As we mine bug signatures from profiles, and profiles record the runtime behaviors of programs (executed statements/basic blocks, taken branches, evaluated predicates, etc.,), we can perform the slicing on the profiles, different from classical static slicing on program code. This difference can boost the precision of slicing as the part of program which is not executed at runtime is pruned away. However, this slicing is also different from dynamic slicing. In dynamic slicing, the execution trace with necessary data information is recorded from the beginning of the execution to the end, which is usually very long due to looping. In contrast, a profile aggregates the whole trace into a finite-sized graph for each execution. This abstraction significantly lowers the profiling and signature-mining overhead, but may have an impact on the precision of slicing. 2. How can multiple bug signatures be used to locate the cause? We usually get top-k bug signatures for a buggy program, can we utilize these signatures to better infer the cause of the bug? Do they share common characteristics yet with differences, and these differences describe the bug from different perspectives? This is the other question we will explore in the future. 136 Bibliography [1] R. Abreu. Spectrum-Based Fault Localization in Embedded Software. PhD thesis, Delft University of Technology, 2009. 1.2, 6.2 [2] R. Abreu, P. Zoeteweij, and A. J. van Gemund. Spectrum-Based Multiple Fault Localization. In ASE, pages 88–99, 2009. 1.2 [3] R. Abreu, P. Zoeteweij, and A. J. C. van Gemund. On the Accuracy of Spectrum-based Fault Localization. In Mutation Testing: Academic and Industrial Conference Practice and Research Techniques (TAICPART-MUTATION), pages 89–98, 2007. 6.2 [4] R. Agrawal and R. Srikant. Fast Algorithms for Mining Association Rules in Large Databases. In VLDB, pages 487–499, 1994. 4.3.3, 5.1.2 [5] J. Anvik, L. Hiew, and G. Murphy. Who Should Fix This Bug? 361–370, 2006. 6.1.1 In ICSE, pages [6] J. Anvik, L. Hiew, and G. C. Murphy. Coping With an Open Bug Repository. In eclipse ’05: Proceedings of the 2005 OOPSLA workshop on Eclipse technology eXchange, pages 35–39, 2005. 1.1, 6.1.1 [7] S. Artzi, J. Dolby, F. Tip, and M. Pistoia. Directed Test Generation for Effective Fault Localization. In ISSTA, pages 49–60, 2010. 6.2 [8] S. Artzi, J. Dolby, F. Tip, and M. Pistoia. Practical Fault Localization for Dynamic Web Applications. In ICSE, pages 265–274, 2010. 1.2, 6.2 [9] G. K. Baah, A. Podgurski, and M. J. Harrold. Causal Inference for Statistical Fault Localization. In ISSTA, pages 73–84, 2010. 1.2, 4.5.4, 6.2 [10] N. Bettenburg, S. Just, A. Schröter, C. Weiss, R. Premraj, and T. Zimmermann. What Makes a Good Bug Report? In ESEC/FSE, pages 308–318, 2008. 1.1, 6.1.1 137 [11] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim. Duplicate Bug Reports Considered Harmful . Really? In ICSM, pages 337–345, 2008. 6.1.1 [12] N. Bettenburg, R. Premraj, T. Zimmermann, and S. Kim. Extracting Structural Information from Bug Reports. In MSR, pages 27–30, 2008. 6.1.1 [13] D. Beyer, T. A. Henzinger, R. Jhala, and R. Majumdar. The Software Model Checker Blast: Applications to Software Engineering. International Journal on Software Tools for Technology Transfer, 9(5):505–525, Oct. 2007. [14] Bugzilla. Bugzilla, 2013. [Online; accessed 21-March-2013]. [15] C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to Rank Using Gradient Descent. In ICML, pages 89–96, 2005. 3.1.3 [16] C. Cadar, D. Dunbar, and D. Engler. KLEE: Unassisted and Automatic Generation of High-coverage Tests for Complex Systems Programs. In OSDI, pages 209–224, 2008. [17] C.-C. Chang and C.-J. Lin. LIBSVM: A Library For Support Vector Machines, 2001. Software available at http://www.csie.ntu.edu.tw/∼cjlin/libsvm. 2.1.3, 2.2.3.3 [18] H. Cheng, D. Lo, Y. Zhou, X. Wang, and X. Yan. Identifying Bug Signatures Using Discriminative Graph Mining. In ISSTA, pages 141–152, 2009. 4, 4.3, 4.3.1, 4.4, 4.4.3, 5, 5.4, 6.2 [19] H. Cheng, X. Yan, J. Han, and C.-W. Hsu. Discriminative Frequent Pattern Analysis for Effective Classification. In ICDE, pages 716–725, 2007. 4.3.3, 5.3.3.1, 6.2 [20] H. Cheng, X. Yan, J. Han, and P. S. Yu. Direct Discriminative Pattern Mining for Effective Classification. In ICDE, pages 169–178, 2008. 6.2 [21] T. M. Chilimbi, B. Liblit, K. Mehra, A. V. Nori, and K. Vaswani. HOLMES: Effective Statistical Debugging via Efficient Path Profiling. In ICSE, pages 34–44, 2009. 1.2, 6.2 [22] H. Cleve and A. Zeller. Locating Causes of Program Failures. In ICSE, pages 342–351, 2005. 4.4.1, 5.4.2, 6.2 [23] D. Cubranic and G. C. Murphy. Automatic Bug Triage Using Text Categorization. In SEKE, pages 92–97, 2004. 6.1.1 [24] H. Do, S. Elbaum, and G. Rothermel. Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact. Empirical Software Engineering, 10(4):405–435, Oct. 2005. 4.4.2 138 [25] A. Feldman and A. van Gemund. A Two-step Hierarchical Algorithm for Model-based Diagnosis. In National Conference on On Artificial Intelligence, pages 827–833, 2006. 6.2 [26] C. Flanagan, K. R. M. Leino, M. Lillibridge, G. Nelson, J. B. Saxe, and R. Stata. Extended Static Checking for Java. In PLDI, pages 234–245, 2002. [27] P. Francis, D. Leon, and M. Minch. Tree-based methods for classifying software failures. In ISSRE, pages 451–462, 2004. 6.1.1 [28] P. Godefroid, N. Klarlund, and K. Sen. DART: Directed Automated Random Testing. In PLDI, pages 213–223, 2005. [29] R. Gore and P. F. Reynolds, Jr. Reducing Confounding Bias in Predicate-level Statistical Debugging Metrics. In ICSE, pages 463–473, 2012. 1.2, 4.5.4, 6.2 [30] N. Gupta, H. He, X. Zhang, and R. Gupta. Locating Faulty Code Using Failure-inducing Chops. In ASE, pages 263–272, 2005. 4.5.4, 6.2 [31] P. Hooimeijer and W. Weimer. Modeling Bug Report Quality. In ASE, pages 34–43, 2007. 6.1.1 [32] H. Hsu, J. A. Jones, and A. Orso. RAPID: Identifying Bug Signatures to Support Debugging Activities. In ASE, pages 439–442, 2008. 1.2, 4, 4, 5, 6.2 [33] M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments on the Effectiveness of Dataflow- and Controlflow-based Test Adequacy Criteria. In ICSE, pages 191–200, 1994. 5.4 [34] N. Jalbert and W. Weimer. Automated Duplicate Detection for Bug Tracking Systems. In DSN, pages 52–61, 2008. 1.1, 2, 2.3, 2.3.2, 2.4.1, 3.3.1, 6.1 [35] D. Jeffrey, N. Gupta, and R. Gupta. Fault Localization Using Value Replacement. In ISSTA, pages 167–178, 2008. 6.2 [36] R. Jhala and R. Majumdar. Software Model Checking. ACM Computing Survey, 41(4):21:1–21:54, Oct. 2009. [37] N. Jin, C. Young, and W. Wang. GAIA: Graph Classification Using Evolutionary Computation. In SIGMOD, pages 879–890, 2010. 6.2 [38] JIRA. Jira, 2013. [Online; accessed 21-March-2013]. 139 [39] J. Jones and M. Harrold. Empirical Evaluation of the Tarantula Automatic FaultLocalization Technique. In ASE, pages 273–282, 2005. 1.2, 6.2 [40] H. Kim, S. Kim, T. Weninger, J. Han, and T. F. Abdelzaher. NDPMine: Efficiently Mining Discriminative Numerical Features for Pattern-Based Classification. In ECML/PKDD, pages 35–50, 2010. 6.2 [41] A. Ko and B. Myers. A Linguistic Analysis of How People Describe Software Problems. In IEEE Symposium on Visual Languages and Human-Centric Computing, pages 127– 134, 2006. 6.1.1 [42] M. Kuramochi and G. Karypis. Frequent Subgraph Discovery. In ICDM, pages 313–320, 2001. 4.3 [43] J.-G. Lee, J. Han, X. Li, and H. Cheng. Mining Discriminative Patterns for Classifying Trajectories on Road Networks. In TKDE, pages 713–726, 2010. 6.2 [44] F. Lerda and W. Visser. Addressing Dynamic Issues of Program Model Checking. In SPIN, pages 80–102, 2001. [45] J. Li, H. Li, L. Wong, J. Pei, and G. Dong. Minimum Description Length Principle: Generators Are Preferable to Closed Patterns. In AAAI, 2006. 4, 4.3, 4.3.2, 4.5.2, 5.1.2, 5.3.1, 2, 5.3.3 [46] B. Liblit, A. Aiken, A. X. Zheng, and M. I. Jordan. Bug Isolation via Remote Program Sampling. In PLDI, pages 141–154, 2003. 6.2 [47] B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan. Scalable Statistical Bug Isolation. In PLDI, pages 15–26, 2005. 1.2, 5.1.1 [48] C. Liu, X. Yan, L. Fei, J. Han, and S. Midkiff. SOBER: Statistical Model-based Bug Localization. In FSE, pages 286–295, 2005. 1.2, 6.2 [49] D. Lo, H. Cheng, H. Jiawei, S.-C. Khoo, and C. Sun. Classification of Software Behaviors for Failure Detection: a Discriminative Pattern Mining Approach. In KDD, pages 557– 566, 2009. 1.5, 2.2.3.1, 6.2 [50] D. Lo, H. Cheng, and Lucia. Mining Closed Discriminative Dyadic Sequential Patterns. In EDBT, pages 21–32, 2011. 6.2 [51] Lucia, D. Lo, L. Jiang, and A. Budi. Comprehensive Evaluation of Association Measures for Fault Localization. In ICSM, pages 1–10, 2010. 5.4.1, 5.4.6 140 [52] C. D. Manning, P. Raghavan, and H. Schtze. Introduction to Information Retrieval, pages 232–233. Cambridge University Press, New York, NY, USA, 2008. 3.2.1 [53] C. D. Manning, P. Raghavan, and H. Schtze. Introduction to Information Retrieval, pages 158–163. Cambridge University Press, New York, NY, USA, 2008. 3.3 [54] W. Mayer and M. Stumptner. Model-Based Debugging – State of the Art and Future Challenges. Electronic Notes in Theoretical Computer Science, 174(4):61–82, 2007. 6.2 [55] T. Menzies and A. Marcus. Automated Severity Assessment of Software Defect Reports. In ICSM08, pages 346–355, 2008. 6.1.1 [56] T. M. Mitchell. Machine Learning, pages 88–95. McGraw-Hill, New York, 1997. 3.1.3 [57] E. G. N. Vanetik and S. Shimony. Computing Frequent Graph Patterns from Semistructured Data. In ICDM, pages 458–465, 2002. 4.3 [58] R. Nallapati. Discriminative Models for Information Retrieval. In SIGIR, pages 64–71, 2004. 2, 2.2, 2.2.3.3 [59] A. T. Nguyen, T. T. Nguyen, T. N. Nguyen, D. Lo, and C. Sun. Duplicate Bug Report Detection with a Combination of Information Retrieval and Topic Modeling. In ASE, pages 70–79, 2012. 1.5 [60] S. Nijssen, T. Guns, and L. Raedt. Correlated Itemset Mining in ROC Space: a Constraint Programming Approach. In KDD, pages 647–656, 2009. 4.3.3, 5.3.3.1, 6.2 [61] C. Parnin and A. Orso. Are Automated Debugging Techniques Actually Helping Programmers? In ISSTA, pages 199–209, 2011. 1.2, [62] A. Podgurski, D. Leon, P. Francis, W. Masri, M. Minch, J. Sun, and B. Wang. Automated Support for Classifying Software Failure Reports. In ICSE, pages 465–475, 2003. 6.1.1 [63] J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 1993. 4.2.1, 5.4.1 [64] P. R.Duda and D.Stork. Pattern Classification. Wiley Interscience, 2nd edition, 2000. 2.4.2 [65] M. Renieris and S. Reiss. Fault Localization With Nearest Neighbor Queries. In ASE, 2003. 1.2, 4.4.1, 5.4.2, 6.2 141 [66] S. Robertson, H. Zaragoza, and M. Taylor. Simple BM25 Extension to Multiple Weighted Fields. In ACM International Conference on Information and Knowledge Management, pages 42–49, 2004. 3, 3.1.2, 3.2.3.2 [67] J. Rossler, G. Fraser, A. Zeller, and A. Orso. Isolating Failure Causes through Test Case Generation. In ISSTA, pages 309–319, 2012. 1.2 [68] P. Runeson, M. Alexandersson, and O. Nyholm. Detection of Duplicate Defect Reports Using Natural Language Processing. In ICSE, pages 499–510, 2007. 1.1, 2, 2.1.1, 2.3, 2.3.2, 2.4.1, 3, 3.3.1, 6.1 [69] R. J. Sandusky, L. Gasser, R. J. S, U. L. Gasser, and G. Ripoche. Bug Report Networks: Varieties, Strategies, and Impacts in a F/OSS Development Community. In MSR, pages 80–84, 2004. 6.1.1 [70] R. Santelices, J. A. Jones, Y. Yu, and M. J. Harrold. Lightweight Fault-localization Using Multiple Coverage Types. In ICSE, pages 56–66, 2009. 1.2 [71] C. Sun, J. Du, N. Chen, S.-C. Khoo, and Y. Ye. Mining Explicit Rules for Software Process Evaluation. In ICSSP, pages 118–125, 2013. 1.5 [72] C. Sun, S.-C. Khoo, and S. J. Zhang. Graph-based Detection of Library API Imitations. In ICSM, pages 183–192, 2011. 1.5 [73] C. Sun, D. Lo, S.-C. Khoo, and J. Jiang. Towards More Accurate Retrieval of Duplicate Bug Reports. In ASE, pages 253–262, 2011. 1.5 [74] C. Sun, D. Lo, X. Wang, J. Jiang, and S.-C. Khoo. A discriminative model approach for accurate duplicate bug report retrieval. In ICSE, pages 45–56, 2010. 1.5, 3, 3.2.3.2, 3.3, 3.3.1, 3.3.3, 3.3.3, 6.1 [75] A. Sureka and P. Jalote. Detecting Duplicate Bug Report Using Character N-GramBased Features. In APSEC, pages 366–374, 2010. 3, 3.3, 3.3.1, 3.3.3, 3.3.3, 3.4, 6.1 [76] J. Sutherland. Business Objects in Corporate Information Systems. In ACM Computing Surveys, volume 27, pages 274–276. ACM, June 1995. [77] G. Tassey. The Economic Impacts of Inadequate Infrastructure for Software Testing. National Institute of Standards and Technology. Planning Report 02-3.2002, 2002. [78] M. Taylor, H. Zaragoza, N. Craswell, S. Robertson, and C. Burges. Optimisation Methods for Ranking Functions with Multiple Parameters. In ACM international Conference on Information and Knowledge Management, pages 585–593, 2006. 3.1.3, 3.2.3.2 142 [79] Y. Tian, D. Lo, and C. Sun. Information Retrieval Based Nearest Neighbor Classification for Fine-Grained Bug Severity Prediction. In WCRE, pages 215–224, 2012. 1.5 [80] Y. Tian, D. Lo, and C. Sun. DRONE: Predicting Priority of Reported Bugs by MultiFactor Analysis. In ICSM, 2013. 1.5 [81] Y. Tian, C. Sun, and D. Lo. Improved Duplicate Bug Report Identification. In CSMR, pages 385–390, 2012. 1.5 [82] F. Tip. A Survey of Program Slicing Techniques. Journal of Programming Languages, 3:121–189, 1995. [83] J. Wang and J. Han. BIDE: Efficient Mining of Frequent Closed Sequences. In ICDE, pages 79–90, 2004. 4, 5, 6.2 [84] X. Wang, L. Zhang, T. Xie, J. Anvik, and J. Sun. An Approach to Detecting Duplicate Bug Reports Using Natural Language and Execution Information. In ICSE, pages 461–470, 2008. 1.1, 2, 2.3, 2.3.2, 2.3.2, 2.4.1, 3, 3.3.1, 6.1 [85] H. Xiao, J. Sun, Y. Liu, S.-W. Lin, and C. Sun. TzuYu: Learning Stateful Typestates. In ASE, 2013. 1.5 [86] X. Yan, H. Cheng, J. Han, and P. Yu. Mining Significant Graph Patterns by Leap Search. In SIGMOD, pages 433–444, 2008. 4, 4.3.1, 5, 6.2 [87] X. Yan and J. Han. gSpan: Graph-based Substructure Pattern Mining. In ICDM, pages 721–724, 2002. 4.3 [88] H. Zaragoza, N. Craswell, M. J. Taylor, S. Saria, and S. E. Robertson. Microsoft Cambridge at TREC 13: Web and Hard Tracks. In TREC, 2004. 3, 3.1.2 [89] A. Zeller. Isolating Cause-effect Chains from Computer Programs. In FSE, pages 1–10, 2002. 6.2 [90] S. J. Zhang, J. Sun, C. Sun, Y. Liu, J. Ma, and J. S. Dong. Constraint-based Automatic Symmetry Detection. In ASE, 2013. 1.5 [91] X. Zhang, N. Gupta, and R. Gupta. Locating Faults through Automated Predicate Switching. In ICSE, pages 272–281, 2006. 6.2 143 [...]... of vector space representation, we develop a discriminative model to retrieve similar bug reports from a bug repository We make use of the recent advances in information retrieval community that uses a classifier to retrieve similar documents from a collection[58] We build a model that contrasts duplicate bug reports from non-duplicate bug reports and utilize this model to extract similar bug reports, ... track software bugs and build more reliable systems, bug tracking systems have been introduced Bug tracking systems enable many users to report their findings in a unified environment These bug reports are then used to guide software corrective maintenance activities and result in more reliable software systems Via the bug tracking systems, users are able to report new bugs, track status of bug reports, ... 1 We employ in total 54 features to comprehensively evaluate the similarity between two reports 2 We propose a discriminative model based solution to retrieve similar bug reports from a bug tracking system Our model can automatically assign optimum weight to each feature and evolve along with the changes of bug repositories 3 We are the first to analyze the applicability of duplicate bug report detection... importance, to deteriorate in their performance More accurate results would mean more automation and less effort by triagers to find duplicate bug reports To address this need, we propose a discriminative model based approach that further improves accuracy in retrieving duplicate bug reports by up to 43% on real bug report datasets Different from the previous approaches that rank similar bug reports based... the notion of predicated bug signatures for debugging as a complementary to control-flow-based bug signatures described in Chapter 4 By utilizing both data predicates and control-flow information, predicated bug signatures can enhance the predictive power of bug signatures We also introduce and detail a novel “discriminative itemset generator” mining technique to generate succinct signatures which do not... function, to automatically predict the severity of bug reports Bugs are prevalent in software systems Some bugs are critical and need to be fixed right away, whereas others are minor and their fixes could be postponed until resources are available Our approach automatically analyzes bug reports reported in the past along with their assigned severity labels, and recommends severity labels to newly reported bug. .. of duplicate bug report retrieval, and propose a discriminative model approach to automatically learning a textual similarity function from history bug reports We show that the retrieval accuracy is improved over classical information retrieval measures, i.e., vector space model Chapter 3 introduces an enhanced metric REP to characterize the similarity of two bug reports for duplicate bug report retrieval... 1.2 Bug Signature Identification In order to eliminate a bug, a developer utilizes all means to identify the location of the bug, and figure out its cause This process is referred to as debugging Debugging has long been regarded as a painstaking task, especially when the symptom (or the manifestation) of a bug does not follow immediately the place where the bug is triggered For example, a noncrashing bug. .. language processing style – tokenization, stemming and stop words removal – described in Sub-section 2.1.2 The second step, training a discriminative model , trains a classifier to answer the question “How likely are two bug reports duplicates of each other?” The third step, retrieving duplicate bug reports, makes use of this classifier to retrieve relevant bug reports from the repository 20 ... is also employed to verify bug- freeness of a program, e.g., the Blast model checker [13] for C programs and JPF [44] for Java 2 Report Bugs After bugs are experienced by tools, developers or encountered by end users, they need to be effectively managed, yet another important issue to be addressed in software life cycle In most software projects, bug tracking systems are used for filing bugs and tracking . SOFTWARE BUG MANAGEMENT FROM BUG REPORTS TO BUG SIGNATURES CHENGNIAN SUN NATIONAL UNIVERSITY OF SINGAPORE 2013 SOFTWARE BUG MANAGEMENT FROM BUG REPORTS TO BUG SIGNATURES CHENGNIAN. duplicate reports for real-world software projects by retrieving top-20 similar reports. The other problem we investigate is identifying bug signatures from a buggy program for debugging purpose. A bug. These bug reports are then used to guide software corrective maintenance activities and result in more reliable software systems. Via the bug tracking systems, users are able to report new bugs,