New approaches to automated annotation of pathology level findings in medical images

New approaches to automated annotation of pathology-level findings in brain images DINH THIEN ANH Bachelor of Computing National University of Singapore A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY SCHOOL OF COMPUTING NATIONAL UNIVERSITY OF SINGAPORE 2013 Acknowledgements First and foremost, I would like to express my deepest gratitude to my thesis advisor, Dr. Tze-Yun Leong, for her incisive guidance, encouragement, patience and immense support through out my Ph.D career. And I have learned a lot from her. She has also provided me with an excellent research environment that is full of freedom. Without her help and belief, I would not have finished my dissertation. I am very grateful to have Dr. Choie Cheio Tchoyoson Lim from National Neuroscience Institute as my medical advisor. Despite his extremely busy schedule, he is always available to share with me his valuable medical knowledge and insightful feedbacks for my research. In addition, thanks to his generous reference, I have the honor to receive the Singapore Millennium Scholarship for my graduate study. I am also much indebted to Dr. Tomi Silander for being such an excellent mentor and for his inputs in my research. He has helped me to overcome so many obstacles in my research. Together with Dr. Tze Yun Leong, he has reviewed my thesis and provided many thoughtful suggestions, which help me to improve this thesis tremendously. I cannot thank him enough for his devotion. And I have also benefited so much from his wide knowledge and constructive advices. I am very fortunate to have several other mentors and collaborators along the way. I am thankful to Dr. Chew Lim Tan for his financial support which funded me as Research Assistant through the last year of my study. I would like to thank Dr. Boon Chuan Pang and Dr. Cheng Kiang Lee from National Neuroscience Institute for providing me the traumatic brain injury dataset. My sincere thank goes to Dr. Tianxia i Gong for providing me the labelled training dataset and her valuable experience in the medical imaging field. I wish to extend my thanks to Dr. Dinh Truong Huy Nguyen, Dr. Duc Hiep Chu, Thang Truong Duc, Dr. Bolan Su, Quang Loc Le, Thuy Ngoc Le, Thanh Trung Nguyen, Zhuoru Li, and many more great friends and colleagues through out the years for their friendship, ideas, encouragement and support. Without their accompanies, I would not have had that much fun in my life. My heartfelt gratitude goes to my fiance Ngoc Yen for her unconditional love, encouragement, patience, loyalty and for standing by me in both good and bad times. She has been virtually working as hard as me on this thesis. I am completely amazed at her willingness to proof read my writing countlessly. She is truly a gift that I am so blessed to have. Thank you dear from the bottom of my heart and I am looking forward to starting a family with you. Last but not least, I am extremely grateful to my parents for their unbounded love and sacrifice; and my elder brother and sister-in-law for their encouragement and understanding. My parents have been giving me many wonderful opportunities in life. I am forever thankful to have such an amazing family, and there is no word that can describe how much I love them. The past six years have been a bumpy ride for me. And their love, care, sacrifice, support and encouragement have made it become much easier. Thus, I owe this to my family. ii Table of Contents Introduction 1.1 Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Approaches and contributions . . . . . . . . . . . . . . . . . . . . . 1.3 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Road map of this thesis . . . . . . . . . . . . . . . . . . . . . . . . . The medical domains 11 2.1 Ischemic stroke . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.1.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Traumatic brain injury . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.2.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.2 2.3 An overview of the pathology-level medical image annotation system 19 3.1 Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.2 Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.3 Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.4 Evaluation metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Related work 27 4.1 27 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iii 4.2 4.3 4.4 4.5 4.1.1 Generative models vs. Discriminative models . . . . . . . . . 27 4.1.2 Ensemble learning . . . . . . . . . . . . . . . . . . . . . . . 29 Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.2.1 Global features . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.2.2 Local features . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Annotating natural images . . . . . . . . . . . . . . . . . . . . . . . 33 4.3.1 Translation paradigm . . . . . . . . . . . . . . . . . . . . . . 34 4.3.2 Relevance Models . . . . . . . . . . . . . . . . . . . . . . . 35 4.3.3 Other approaches . . . . . . . . . . . . . . . . . . . . . . . . 35 4.3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Annotating medical images . . . . . . . . . . . . . . . . . . . . . . . 38 4.4.1 Organ-level annotation . . . . . . . . . . . . . . . . . . . . . 39 4.4.2 Pathology-level annotation . . . . . . . . . . . . . . . . . . . 40 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 A generative model based approach 43 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 5.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 5.3 Image Processing Component . . . . . . . . . . . . . . . . . . . . . 46 5.3.1 Automated lesion segmentation . . . . . . . . . . . . . . . . 47 5.3.2 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . 48 5.4 Generative model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 5.5 Content-based retrieval . . . . . . . . . . . . . . . . . . . . . . . . . 56 5.6 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 5.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 A discriminative-model based approach 61 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 6.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 6.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 6.3.1 65 Feature extraction component . . . . . . . . . . . . . . . . . iv 6.3.2 Classification system . . . . . . . . . . . . . . . . . . . . . . 66 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6.4.1 Without global features . . . . . . . . . . . . . . . . . . . . . 74 6.4.2 With global features . . . . . . . . . . . . . . . . . . . . . . 76 6.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 6.4 Unsupervised classification by combining case-based classifiers 81 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 7.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 7.2.1 System architecture . . . . . . . . . . . . . . . . . . . . . . . 84 7.2.2 Gabor feature extraction . . . . . . . . . . . . . . . . . . . . 84 7.2.3 Sparse representation-based classifier . . . . . . . . . . . . . 87 7.2.4 Ensemble of weak classifiers . . . . . . . . . . . . . . . . . . 89 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 7.3.1 Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 7.3.2 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . 92 7.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 7.3 7.4 Automatic Traumatic Brain Injury prognosis 99 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 8.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 8.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 8.3.1 Preprocessing and feature extraction . . . . . . . . . . . . . . 103 8.3.2 Classification of CT image slices . . . . . . . . . . . . . . . . 104 8.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 8.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 8.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Prototype implementation and informal evaluation 9.1 113 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 v 9.1.1 GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 9.1.2 Annotator . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 9.2 Informal evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 9.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 10 Conclusion 121 10.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 10.2 Proposed approaches and contributions . . . . . . . . . . . . . . . . . 122 10.2.1 The generative model based approach . . . . . . . . . . . . . 122 10.2.2 The discriminative model based approach . . . . . . . . . . . 123 10.2.3 The unsupervised classification by combining case-based classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 10.3 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 vi Summary Medical image annotation aims to improve the e↵ectiveness and efficiency of keywordbased image retrieval. In this work, we focus on automated pathology annotation that tries to identify potential pathologies, abnormalities and diseases from brain images. This is a challenging task because pathology annotation demands a deep understanding of the structural and functional changes induced by diseases. Existing works in pathological annotation often require large and fully annotated training data, reliable segmentation, and domain knowledge for hand-crafted feature extraction and selection. Since these prerequisites are not always feasible, they reduce the level of automation, desirability, and practicality of the annotation systems. To mitigate the requirements of annotated training data and reliable segmentation, we propose to use probabilistic generative models, since they support the integration of expert knowledge and e↵ectively handle the uncertainties inherent in the images and segmentation. However, when a priori knowledge is not available, these generative models are not able to achieve their best performance. In this case, we suggest using a discriminative model which incorporates an automated feature selection method to tackle the problem. Specifically, sparse group lasso provides a flexible selection mechanism that helps to handle annotation problems without relying on the domain knowledge. The performance of existing annotation methods heavily depends on the quality of hand-crafted features extracted from an automatic image segmentation. To achieve good performance, constructing the system requires a considerable amount of manual work. We propose to combine an unsupervised feature extraction technique with a case-based classification in an ensemble learning framework to improve the adaptability and automation of the annotation systems. The unsupervised nature of this non-parametric technique can significantly reduce the time and e↵ort for system calibration. To evaluate these approaches, we select two important neurological disorders - ischemic stroke and traumatic brain injury, as illustrative domains because imaging findings of these diseases play significant roles in their diagnosis. Despite the additional challenges due to the relaxation of the common prerequisites in existing systems, our vii proposed frameworks still show reasonable performance. An informal evaluation with expert users has also demonstrated the practical promise of the proposed system. viii Chapter 10 Conclusion We conclude by reflecting on the lessons learned from this research. We summarize the contributions of this work and compare our approaches with related work. Finally, we discuss some directions for future research. 10.1 Summary We began with the comparison of existing methodologies for addressing a subclass of medical image annotation problems. We identified that generative models, discriminative models and ensemble learning are three common bases of the current techniques. Building on these common bases and integrating di↵erent features of these techniques, we proposed three novel approaches for annotating pathology-level information in medical images. To evaluate the feasibility and e↵ectiveness of these approaches, we implemented them in two important medical imaging domains: ischemic stroke and traumatic brain injury. In addition, we found that pathology-level annotation techniques can be used for a clinical outcome prediction. We also conducted an informal evaluation of the prototype system of one of our proposed methods. This exercise demonstrated the practical promise of the overall framework. 121 10.2 Proposed approaches and contributions In pathology-level image annotation, many existing systems operate under several prerequisite conditions which limit their automation, robustness and practicality. The first prerequisite is to have annotated training data. Although medical images are abundant, annotated training data is rare because it is labour-intensive and time-consuming to annotate them. Secondly, existing frameworks often require accurate segmentation results. The inability of the systems to deal with imperfect segmentation results restricts their usability in practice. Moreover, automated segmentation techniques still require a considerable amount of manual calibration work. Third, domain knowledge is required for hand-crafted feature extraction and selection. However, such prior knowledge is not always available. Lastly, the availability of large training dataset is often assumed. The main contributions of our proposed approaches is to improve the automation and practicality of existing pathology image annotation systems by relaxing some of the assumptions or prerequisites mentioned above. We proposed three novel approaches to pathology annotation of brain images. 10.2.1 The generative model based approach To address the issues with the lack of annotated training data and reliable segmentation, we propose using the probabilistic generative model which naturally supports the incorporation of an expert knowledge and e↵ectively handles the uncertainties inherent in the images and image segmentation. Unlike existing generative image annotation methods based on the translation paradigm [87, 29, 7], the relevance models [51, 33] and the closely related method by Carneiro et al. [13], the proposed model can capture spatial constraints among abnormal findings in an image, which are essential in identifying diseases and disorders and ruling out artifacts. Furthermore, since these techniques usually require a large training dataset for model construction, they 122 are not suitable for dealing with small training medical image dataset. Our empirical study demonstrates the feasibility of building a pathology-level annotation system with a limited training dataset. This proposed solution is also the first attempt to perform subtype annotation in the ischemic stroke domain. 10.2.2 The discriminative model based approach Di↵erent from the generative model approach, the discriminative model with an automated feature selection can avoid the dependency on prior knowledge. When the domain knowledge is not available, our framework that adopts the sparse group lasso feature selection technique and the SVM can achieve better performance. Similar to the generative model, fully annotated training data and perfect segmentation results for the relevant brain regions are optional. In addition, the sparse group lasso provides a flexible selection mechanism to capture the structure of the dataset. We have also combined both region-based features and global features in our model. Since existing works [21, 70, 115] only handle annotating regions of interests (ROI) from CT brain images, the main highlight of the proposed method is to work directly with a CT brain scan. Comparing our method with the state-of-the-art approach in classifying ROIs [37], our classifier has demonstrated very encouraging results. 10.2.3 The unsupervised classification by combining case-based classifiers The performance of existing methods often depends on the quality of hand-crafted features extracted from automatic image segmentation. Therefore, extracting useful features and calibrating automatic segmentation heavily depend on manual work. We have presented an ensemble classification framework with sparse Gabor-feature based classifiers to tackle these limitations. By doing so, we can achieve an automated and robust image annotation system. In comparing with the most recent methods in 123 annotating TBI images [38], we have obtained reasonable results without relying on a time-consuming process of manually selecting handcrafted features. We further extend the unsupervised method to deal with 3-D brain images. Unlike many previous methods [20, 99] which require expensive and time-consuming manual interpretation of the original brain images, the proposed method automatically predicts an outcome of a CT scan using only weakly labelled training data. As a result, it is suitable for classifying large brain image datasets. In an initial human evaluation, senior radiologists found the accuracy of the automated prognosis to be potentially useful in practice. Although the above approaches are demonstrated in ischemic stroke and TBI images, they are also applicable to other brain imaging domains because no modalitydependent assumption is made in our work. However, there are open questions and technical limitations that have not been addressed in this dissertation. In the next section, we will discuss some potential directions that could further improve the proposed approaches. 10.3 Future work We identify five areas for improvement below. We often encounter the imbalance problem in medical domain, which means the critically ill patients normally constitute a small portion of the whole patient population. Imbalanced data often causes negative e↵ects on the performance of machine learning algorithms. In the scope of our work, this problem has not been addressed adequately. Existing techniques in tackling these problems such as SMOTE and modelbased sampling [15, 50] should be studied and incorporated into our proposed system. Second, one of the major problems in dealing with medical images is the lack of sufficient training data. Training an accurate model is a very challenging task. Therefore, incorporating textual information and metadata of medical images can help 124 to improve annotation accuracy. However, metadata is often captured in free text and subjective. How to integrate both visual information and textual information into a coherent annotation model could be a promising future direction. Third, in this work, the annotation process is done separately from the retrieval process. Hence, it could be inefficient for image retrieval. Ranking images within each category or concept has not been fully taken into account yet. While the previously proposed generative model can be conveniently converted into the retrieval system, it is more challenging for our methods based on discriminative approaches (Chapter 6) to achieve the same purpose [113]. Additional e↵orts should be put into studying this problem. The fourth issue is the lack of standard vocabulary and taxonomy for annotation. Currently, arbitrary vocabularies are being used. Hierarchical taxonomy and ontologies not only would standardize the annotation vocabulary but also make the annotation system more comprehensive for the image retrieval process. Last but not least, the predictive power of our annotation model could be enhanced by utilizing real-time feedback from the users in practice. Relevance feedback method [95] could be a good starting point in this direction. 125 126 Bibliography [1] Harold P Adams, Birgitte H Bendixen, L Jaap Kappelle, Jose Biller, Besty B Love, David Lee Gordon, Eugene E Marsh, and the TOAST Investigators. Classification of subtype of acute ischemic stroke. definitions for use in a multicenter clinical trial. toast. trial of org 10172 in acute stroke treatment. Stroke, 24(1):35–41, 1993. [2] Brian T Andrews, Bennie W Chiles III, Walter L Olsen, and Lawrence H Pitts. The e↵ect of intracerebral hematoma location on the risk of brain-stem compression and on clinical outcome. Journal of neurosurgery, 69(4):518–522, 1988. [3] Uri Avni, Hayit Greenspan, Eli Konen, Michal Sharon, and Jacob Goldberger. X-ray categorization and retrieval on the organ and pathology level, using patch-based visual words. IEEE Transactions on Medical Imaging, 30(3):733– 746, March 2011. [4] John Bamford, P Sandercock, Martin Dennis, C Warlow, and J Burn. Classification and natural history of clinically identifiable subtypes of cerebral infarction. Lancet, 337(8756):1521–1526, 1991. [5] Christopher M Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006. [6] David M Blei and Michael I Jordan. Modeling annotated data. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 127–134. ACM, 2003. [7] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. The Journal of Machine Learning Research, 3:993–1022, 2003. [8] John M Boone. Radiological interpretation 2020: Toward quantitative image assessment. Medical physics, 34:4173, 2007. [9] Guillaume Bouchard and Bill Triggs. The tradeo↵ between generative and discriminative classifiers. In 16th IASC International Symposium on Computational Statistics (COMPSTAT’04), pages 721–728, 2004. [10] Alan C. Bovik, Marianna Clark, and Wilson S. Geisler. Multichannel texture analysis using localized spatial filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(1):55–73, 1990. 127 [11] Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge University Press, 2004. [12] Leo Breiman. Bagging predictors. Machine Learning, 24(2):123–140, August 1996. [13] Gustavo Carneiro, Antoni B Chan, Pedro J Moreno, and Nuno Vasconcelos. Supervised learning of semantic classes for image annotation and retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(3):394–410, 2007. [14] BB Chaudhuri and Nirupam Sarkar. Texture segmentation using fractal dimension. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(1):72–77, 1995. [15] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16:321–357, 2002. [16] Heng-Da Cheng, Xiaopeng Cai, Xiaowei Chen, Liming Hu, and Xueling Lou. Computer-aided detection and classification of microcalcifications in mammograms: a survey. Pattern recognition, 36(12):2967–2991, 2003. [17] David A Clausi and MEME Jernigan. Designing gabor filters for optimal texture separability. Pattern Recognition, 33(11):1835–1849, 2000. [18] Paul Clough, Michael Grubinger, Thomas Deselaers, Allan Hanbury, and Henning Muller. Overview of the imageclef 2006 photographic retrieval and object annotation tasks. Evaluation of Multilingual and Multi-modal Information Retrieval, pages 579–594, 2007. [19] Paul Clough, Henning Muller, Thomas Deselaers, Michael Grubinger, Thomas M Lehmann, Je↵ery Jensen, and William Hersh. The clef 2005 cross– language image retrieval track. Accessing Multilingual Information Repositories, pages 535–557, 2006. [20] MRC CRASH Trial Collaborators, P Perel, M Arango, T Clayton, P Edwards, E Komolafe, S Poccock, I Roberts, H Shakur, E Steyerberg, et al. Predicting outcome after traumatic brain injury: practical prognostic models based on large cohort of international patients. BMJ, 336(7641):425–9, 2008. ´ c and Sven Lonˇcari´c. Rule-based labeling of CT head image. [21] Dubravko Cosi´ In Artificial Intelligence in Medicine, volume 1211 of Lecture Notes in Computer Science, pages Rule–based labeling of CT head image. Springer Berlin / Heidelberg, 1997. [22] Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In International Conference on Computer Vision Pattern Recognition, volume 2, pages 886–893, INRIA Rhone-Alpes, ZIRST-655, av. de l’Europe, Montbonnot-38334, June 2005. 128 [23] Thomas Deselaers and Thomas M Deserno. Automatic medical image annotation in imageclef 2007: Overview, results, and discussion. Pattern Recognition Letters, 29(15):1988–1995, 2008. [24] Thomas Deselaers and Thomas M Deserno. Medical image annotation in imageclef 2008. Evaluating Systems for Multilingual and Multimodal Information Access, pages 523–530, 2009. [25] Thomas G. Dietterich. Ensemble methods in machine learning. Multiple classifier systems, pages 1–15, 2000. [26] Thien Anh Dinh, Tomi Silander, C C Tchoyoson Lim, and Tze-Yun Leong. A generative model based approach to retrieving ischemic stroke images. AMIA Annual Symposium Proceeding, 2011:312–321, 2011. [27] Thien Anh Dinh, Tomi Silander, C. C. Tchoyoson Lim, and Tze-Yun Leong. An automated pathological class level annotation system for volumetric brain images. AMIA Annual Symposium Proceedings, 2012:1201–1210, 2012. [28] Thien Anh Dinh, Tomi Silander, Bolan Su, Tianxia Gong, Boon Chuan Pang, C C Tchoyoson Lim, Chiang Kiang Lee, Chew Lim Tan, and Tze-Yun Leong. Unsupervised medical image classification by combining case-based classifiers. 14th World Congress on Health and Medical Informatics (MEDINFO 2013), 2013. [29] Pinar Duygulu, Kobus Barnard, Joao FG de Freitas, and David A Forsyth. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In Proceedings of the 7th European Conference on Computer Vision-Part IV, number 16 in ECCV ’02, pages 97–112, London, UK, UK, 2002. Springer-Verlag. [30] EssamA. El-Kwae, Haifeng Xu, and MansurR. Kabuka. Content-based retrieval in picture archiving and communication systems. Journal of Digital Imaging, 13(2):70–81, 2000. [31] Issam El-Naqa, Yongyi Yang, Miles N. Wernick, Nikolas P. Galatsanos, and Robert M. Nishikawa. A support vector machine approach for detection of microcalcifications. IEEE Transactions on Medical Imaging, 21(12):1552–1563, 2002. [32] Jianping Fan, Yuli Gao, Hangzai Luo, and Guangyou Xu. Automatic image annotation by using concept-sensitive salient objects for image content representation. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 361–368. ACM, 2004. [33] SL Feng, Raghavan Manmatha, and Victor Lavrenko. Multiple bernoulli relevance models for image and video annotation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., volume 2, pages II–1002. IEEE, 2004. 129 [34] Yoav Freund and Robert E Schapire. A desicion-theoretic generalization of online learning and an application to boosting. In Computational learning theory, pages 23–37. Springer, 1995. [35] Yul Gao and Jianping Fan. Incorporating concept ontology to enable probabilistic concept reasoning for multi-level image annotation. In Proceedings of the 8th ACM international workshop on Multimedia information retrieval, pages 79–88. ACM, 2006. [36] P. Ghosh, S. Antani, L.R. Long, and G.R. Thoma. Review of medical image retrieval systems and future directions. In Computer-Based Medical Systems (CBMS), 2011 24th International Symposium on, pages 1–6, 2011. [37] Tianxia Gong, Shimiao Li, Chew Lim Tan, Boon Chuan Pang, CC Tchoyoson Lim, Cheng Kiang Lee, Qi Tian, and Zhuo Zhang. Automatic pathology annotation on medical images: A statistical machine translation framework. In 20th International Conference on Pattern Recognition (ICPR), pages 2504–2507. IEEE, 2010. [38] Tianxia Gong, Shimiao Li, Jie Wang, Chew Lim Tan, Boon Chuan Pang, CC Tchoyoson Lim, Cheng Kiang Lee, Qi Tian, and Zhuo Zhang. Automatic labeling and classification of brain ct images. In 18th IEEE International Conference on Image Processing (ICIP), pages 1581–1584. IEEE, 2011. [39] Rafael C Gonzalez and Richard E Woods. Digital Image Processing. Prentice Hall, 2008. [40] Jonathon S Hare, Paul H Lewis, Peter GB Enser, and Christine J Sandom. Mind the gap: Another look at the problem of the semantic gap in image retrieval. In Proceedings of SPIE, volume 6073, page 607309. SPIE and IS&T, 2006. [41] Trevor Hastie, Robert Tibshirani, and Jerome H Friedman. The elements of statistical learning. Springer, 2009. [42] Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on Knowledge and Data Engineering, 21(9):1263–1284, 2009. [43] Ralf Herbrich and Thore Graepel. A pac-bayesian margin bound for linear classifiers. Information Theory, IEEE Transactions on, 48(12):3140–3150, 2002. [44] William Hersh, Mark Mailhot, Catherine Arnott-Smith, and Henry Lowe. Selective automated indexing of findings and diagnoses in radiology reports. Journal of Biomedical Informatics, 34(4):262 – 273, 2001. [45] William Hersh, Henning Muller, and Jayashree Kalpathy-Cramer. The imageclefmed medical image retrieval task test collection. Journal of Digital Imaging, 22:648–655, 2009. [46] Jarmo Ilonen, Joni-Kristian, K¨am¨ar¨ainen, and Heikki K¨alvi¨ainen. Efficient computation of gabor features. Research report, Lappeenranta University of Technology, Department of Information Technology, 2005. 130 [47] Amit Jain and Glenn Healey. A multiscale representation including opponent color features for texture recognition. IEEE Transactions on Image Processing, 7(1):124–128, 1998. [48] Anil K Jain and Farshid Farrokhnia. Unsupervised texture segmentation using gabor filters. Pattern Recognition, 24(12):1167–1186, 1991. [49] Anil K Jain and Aditya Vailaya. Image retrieval using color and shape. Pattern recognition, 29(8):1233–1244, 1996. [50] Nathalie Japkowicz. Learning from imbalanced data sets: a comparison of various strategies. In AAAI workshop on learning from imbalanced data sets, volume 68, 2000. [51] Jiwoon Jeon, Victor Lavrenko, and Raghavan Manmatha. Automatic image annotation and retrieval using cross-media relevance models. In Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages 119–126. ACM, 2003. [52] Anna K Jerebko, James D Malley, Marek Franaszek, and Ronald M Summers. Support vector machines committee classification method for computer-aided polyp detection in ct colonography. Academic Radiology, 12(4):479–486, 2005. [53] Rong Jin, Joyce Y Chai, and Luo Si. E↵ective automatic image annotation via a coherent language model and active learning. In Proceedings of the 12th annual ACM international conference on Multimedia, pages 892–899. ACM, 2004. [54] Thorsten Joachims. Training linear SVMs in linear time. In 12th ACM SIGKDD international Conference on Knowledge Discovery and Data Mining, pages 217–226, Philadelphia, PA, USA, 2006. ACM Press. [55] Yacine Kabir, Michel Dojat, Benoit Scherrer, C Garbay, and F Forbes. Multimodal mri segmentation of ischemic stroke lesions. In 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2007. EMBS 2007, pages 1595–1598. IEEE, 2007. [56] Jayashree Kalpathy-Cramer and William Hersh. Multimodal medical image retrieval: Image categorization to improve search precision. In Proceedings of the International Conference on Multimedia Information Retrieval, MIR ’10, pages 165–174, New York, NY, USA, 2010. ACM. [57] Dong-Wha Kang, Julio A Chalela, Mustapha A Ezzeddine, and Steven Warach. Association of ischemic lesion patterns on early di↵usion-weighted imaging with toast stroke subtypes. Archives of neurology, 60(12):1730, 2003. [58] Seung-Jean Kim, Kwangmoo Koh, Michael Lustig, Stephen Boyd, and Dimitry Gorinevsky. An interior-point method for large-scale l1-regularized least squares. IEEE Journal of Selected Topics in Signal Processing, 1(4):606–617, 2007. 131 [59] William A Knaus, Elizabeth A Draper, Douglas P Wagner, and Jack E Zimmerman. Apache ii: a severity of disease classification system. Critical care medicine, 13(10):818, 1985. [60] William A Knaus, Douglas P Wagner, Elizabeth A Draper, Jack E Zimmerman, Marilyn Bergner, Paulo G Bastos, Carl A Sirio, Donald J Murphy, Ted Lotring, Anne Damiano, and Frank E Harrel. The apache iii prognostic system. risk prediction of hospital mortality for critically ill hospitalized adults. Chest, 100(6):1619–1636, 1991. [61] Ludmila I. Kuncheva and Christopher J. Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach. Learn., 51(2):181–207, May 2003. [62] Victor Lavrenko, R Manmatha, and Jiwoon Jeon. A model for learning the semantics of pictures. In Proceedings of the 17th International Conference on Neural Information Processings Systems (NIPS) 2003. NIPS, 2003. [63] Kuen-Long Lee and Ling-Hwei Chen. An efficient computation method for the texture browsing descriptor of mpeg-7. Image and Vision Computing, 23(5):479–489, 2005. [64] Lance J Lee, Chelsea S Kidwell, Je↵ry Alger, Sidney Starkman, and Je↵rey L Saver. Impact on stroke subtype diagnosis of early di↵usion-weighted magnetic resonance imaging and magnetic resonance angiography. Stroke, 31(5):1081– 1089, 2000. [65] Thomas M Lehmann, Henning Schubert, Daniel Keysers, Michael Kohnen, and Berthold B Wein. The irma code for unique classification of medical images. In Proceedings SPIE, volume 5033, pages 109–117. SPIE, 2003. [66] Michael S. Lew, Nicu Sebe, Chabane Djeraba, and Ramesh Jain. Contentbased multimedia information retrieval: State of the art and challenges. ACM Transaction in Multimedia Comput. Commun. Appl., 2(1):1–19, February 2006. [67] Chun-Guang Li, Jun Guo, and Hong-Gang Zhang. Local sparse representation based classification. In 20th International Conference on Pattern Recognition (ICPR), volume 2010, pages 649–652. IEEE, 2010. [68] Shimiao Li, Tianxia Gong, Jie Wang, Ruizhe Liu, Chew Lim Tan, Tze-Yun Leong, Boon Chuan Pang, C C Tchoyoson Lim, Cheng Kiang Lee, Qi Tian, and Zhuo Zhang. TBIdoc: 3D content-based CT image retrieval system for traumatic brain injury. In In Proceedings SPIE Medical Imaging 2010, volume 7624, pages 762427–10, 2010. [69] Xi Li, Weiming Hu, Hanzi Wang, and Zhongfei Zhang. Linear discriminant analysis using rotational invariant {L1} norm. Neurocomputing, 73(1315):2571 – 2579, 2010. Pattern Recognition in Bioinformatics Advances in Neural Control. 132 [70] Chun-Chih Liao, Furen Xiao, Jau-Min Wong, and I Jen Chiang. A Knowledge Discovery Approach to Diagnosing Intracranial Hematomas on Brain CT: Recognition, Measurement and Classification. In Medical Biometrics, volume 4901 of Lecture Notes in Computer Science, pages 73–82. Springer Berlin / Heidelberg, 2007. [71] CC Tchoyoson Lim, Gouliang Yang, W.L. Nowinski, and Francis Hui. Medical image resource center–making electronic teaching files from pacs. Journal of Digital Imaging, 16(4):331–336, 2003. [72] Yuanqing Lin and D D Lee. Bayesian L1-Norm Sparse Learning. In ICASSP 2006 Proceedings. IEEE International Conference on Acoustics, Speech and Signal Processing( ICASSP), volume 5, pages 605–608, 2006. [73] Boqiang Liu, Qingwei Yuan, Zhongguo Liu, Xiaomei Li, and Xiaohong Yin. Automatic segmentation of intracranial hematoma and volume measurement. In 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 2008. EMBS 2008., pages 1214–1217. IEEE, 2008. [74] Manhua Liu, Daoqiang Zhang, and Dinggang Shen. Ensemble sparse classification of alzheimer’s disease. Neuroimage, 60(2):1106–1116, 2012. [75] Ruizhe Liu, Chew Lim Tan, Tze-Yun Leong, Cheng Kiang Lee, Boon Chuan Pang, CC Tchoyoson Lim, Qi Tian, Suisheng Tang, and Zhuo Zhang. Hemorrhage slices detection in brain ct images. In 19th International Conference on Pattern Recognition (ICPR), pages 1–4. IEEE, 2008. [76] Fuhui Long, Hongjiang Zhang, and David Dagan Feng. Fundamentals of content-based image retrieval. Multimedia Information Retrieval and Management, 17:1–26, 2003. [77] David G Lowe. Object recognition from local scale-invariant features. In The Proceedings of the Seventh IEEE International Conference on Computer Vision, volume 2, pages 1150–1157. IEEE, 1999. [78] David G Lowe. Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91–110, 2004. [79] Ameesh Makadia, Vladimir Pavlovic, and Sanjiv Kumar. Baselines for image annotation. International Journal of Computer Vision, 90(1):88–105, 2010. [80] Lawrence F Marshall, Sharon Bowers Marshall, Melville R Klauber, Marjan van Berkum Clark, Howard M Eisenberg, John A Jane, Thomas G Luerssen, Anthony Marmarou, and Mary A Foulkes. A new classification of head injury based on computerized tomography. Special Supplements, 75(1S):14–20, 1991. [81] Anne L Martel, Steven J Allder, Gota S Delay, Paul S Morgan, and Alan R Moody. Measurement of infarct volume in stroke patients using adaptive segmentation of di↵usion weighted mr images. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 1999, pages 22–31. Springer, 1999. 133 [82] Milan Matesin, Sven Loncaric, and Damir Petravic. A rule-based approach to stroke lesion analysis from ct brain images. In Proceedings of the 2nd International Symposium on Image and Signal Processing and Analysis, 2001. ISPA 2001., pages 219–223. IEEE, 2001. [83] Lukas Meier, Sara Van De Geer, and Peter B¨uhlmann. The group lasso for logistic regression. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 70:53–71 ST – The group lasso for logistic regression, 2008. [84] Donald Metzler and R Manmatha. An inference network approach to image retrieval. Image and video retrieval, pages 2130–2131, 2004. [85] Krystian Mikolajczyk and Cordelia Schmid. Scale and affine invariant interest point detectors. International Journal of Computer Vision, 60(1):63–86, 2004. [86] Florent Monay and Daniel Gatica-Perez. On image auto-annotation with latent space models. In Proceedings of the eleventh ACM international conference on Multimedia, pages 275–278. ACM, 2003. [87] Yasuhide Mori, Hironobu Takahashi, and Ryuichi Oka. Image-to-word transformation based on dividing and vector quantizing images with words. In First International Workshop on Multimedia Intelligent Storage and Retrieval Management, 1999. [88] Henning Muller, Nicolas Michoux, David Bandon, and Antoine Geissbuhler. A review of content-based image retrieval systems in medical applicationsclinical benefits and future directions. International Journal of Medical Informatics, 73(1):1–24, 2004. [89] Kevin P Murphy. Machine learning: a probabilistic perspective. Cambridge, MA, 2012. [90] AbdallahBashir Musa. A comparison of 1-regularizion, pca, kpca and ica for dimensionality reduction in logistic regression. International Journal of Machine Learning and Cybernetics, pages 1–13, 2013. [91] Henning Mller, Jayashree Kalpathy-Cramer, Jr. Kahn, CharlesE., William Hatt, Steven Bedrick, and William Hersh. Overview of the imageclefmed 2008 medical image retrieval task. In Carol Peters, Thomas Deselaers, Nicola Ferro, Julio Gonzalo, GarethJ.F. Jones, Mikko Kurimo, Thomas Mandl, Anselmo Peas, and Vivien Petras, editors, Evaluating Systems for Multilingual and Multimodal Information Access, volume 5706 of Lecture Notes in Computer Science, pages 512–522. Springer Berlin Heidelberg, 2009. [92] Andrew Y Ng. Feature selection, L1 vs. L2 regularization, and rotational invariance. In Russ Greiner and Dale Schuurmans, editors, The 21st international Conference on Machine learning, ICML ’04, pages 78–, Ban↵, Alberta, Canada, 2004. ACM Press. 134 [93] Andrew Y Ng and Michael I Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. In Advances in neural information processing systems, volume 2, pages 841–848, 2002. [94] Donald A Ross, Walter L Olsen, Amy M Ross, Brian T Andrews, and Lawrence H Pitts. Brain shift, level of consciousness, and restoration of consciousness in patients with acute intracranial hematoma. Journal of neurosurgery, 71(4):498–502, 1989. [95] Gerard Salton and Chris Buckley. Improving retrieval performance by relevance feedback. Readings in information retrieval, 24:355–364, 1997. [96] Linda Shapiro and George C Stockman. Computer Vision. Prentice Hall. [97] Stephen M Smith. Fast robust automated brain extraction. Human brain mapping, 17(3):143–155, 2002. [98] D. David Stark and Walter G. Bradley. Magnetic resonance imaging. Number in Magnetic Resonance Imaging. Mosby, 1999. [99] Ewout W Steyerberg, Nino Mushkudiani, Pablo Perel, Isabella Butcher, Juan Lu, Gillian S McHugh, Gordon D Murray, Anthony Marmarou, Ian Roberts, J Dik F Habbema, et al. Predicting outcome after traumatic brain injury: development and international validation of prognostic scores based on admission characteristics. PLoS medicine, 5(8):e165, 2008. ¨ [100] Hasan Kamil Sucu, Fazıl Gelal, Metin G¨okmen, F¨usun Demirc¸ivi Ozer, and S¸evket Tektas¸. Can midline brain shift be used as a prognostic factor to predict postoperative restoration of consciousness in patients with chronic subdural hematoma? Surgical neurology, 66(2):178–182, 2006. [101] Daniel C Sullivan. Imaging as a quantitative. Radiology, 248(2):328–332, 2008. [102] Ricky K. Taira, Stephen G. Soderland, and Rex M. Jakobovits. Automatic structuring of radiology free-text reports. RadioGraphics, 21(1):237–245, 2001. PMID: 11158658. [103] Robert Tibshirani. Regression shrinkage and selection via the lasso: a retrospective. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 73:273–282, 2011. [104] Tatiana Tommasi, Barbara Caputo, Petra Welter, Mark Oliver Guld, and Thomas M Deserno. Overview of the clef 2009 medical image annotation track. Multilingual Information Access Evaluation II. Multimedia Experiments, pages 85–93, 2011. [105] Andrius Uˇsinskas. Ischemic stroke segmentation on ct images using joint features. Informatica, 15(2):283–290, 2004. 135 [106] Vincent F van Ravesteijn, Cees van Wijk, Frans M Vos, Roel Truyen, Joost F Peters, Jaap Stoker, and Lucas J van Vliet. Computer-aided detection of polyps in ct colonography using logistic regression. IEEE Transactions on Medical Imaging, 29(1):120 –131, 2010. [107] Liyang Wei, Yongyi Yang, Robert M Nishikawa, and Yulei Jiang. A study on several machine-learning methods for classification of malignant and benign clustered microcalcifications. IEEE Transactions on Medical Imaging, 24(3):371–380, 2005. [108] John Wright, Allen Y Yang, Arvind Ganesh, S Shankar Sastry, and Yi Ma. Robust face recognition via sparse representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(2):210–227, 2009. [109] Furen Xiao, Chun-Chih Liao, Ke-Chun Huang, I Chiang, Jau-Min Wong, et al. Automated assessment of midline shift in head injury patients. Clinical neurology and neurosurgery, 112(9):785–790, 2010. [110] Gouliang Yang, YF Tan, SC Loh, and CC Tchoyoson Lim. Neuroradiology imaging database: using picture archive and communication systems for brain tumour research. Singapore medical journal, 48(4):342, 2007. [111] Jian Yao, Zhongfei (Mark) Zhang, Sameer Antani, Rodney Long, and George Thoma. Automatic medical image annotation and retrieval. Neurocomputing, 71(1012):2012 – 2022, 2008. Neurocomputing for Vision Research Advances in Blind Signal Processing. [112] Jian Yao, Zhongfei Mark Zhang, Sameer Antani, Rodney Long, and George Thoma. Automatic medical image annotation and retrieval. Neurocomputing, 71(10):2012–2022, 2008. [113] Dengsheng Zhang, Md. Monirul Islam, and Guojun Lu. A review on automatic image annotation techniques. Pattern Recognition, 45(1):346–362, January 2012. [114] Dengsheng Zhang, Aylwin Wong, Maria Indrawan, and Guojun Lu. Contentbased image retrieval using gabor texture features. In IEEE Pacific-Rim Conference on Multimedia, 2000. [115] Wei-Li Zhang and Xi-Zhao Wang. Feature Extraction and Classification for Human Brain CT Images. In 2007 International Conference on Machine Learning and Cybernetics, pages 1155–1159. IEEE, 2007. 136 [...]... overview of the pathology- level medical image annotation system In this chapter, we describe a general framework for the pathology- level annotation of medical images The framework supports automated or semi -automated annotation of pathology- level information in input images The framework is able to handle images in 2-D or 3-D format and from di↵erent modalities (such as CT, MRI, X-ray) Annotated images. .. content, existing techniques for annotating natural images are not always suitable for annotating medical images As a result, automated annotation techniques for medical images need to be designed di↵erently There are two types of medical image annotation: organ -level annotation and pathology- level annotation Organ level annotation is the process of annotating general aspects of the images such as... training data size 1.4 Road map of this thesis The following is a road map of the remaining chapters of this thesis In Chapter 2, we discuss the domain knowledge and related work in ischemic stroke imaging and traumatic brain injury imaging In Chapter 3, we present the general framework of a pathology- level image annotation system In Chapter 4, we survey existing techniques in annotating natural and medical. .. (Figure 11) In stroke imaging, clinical imaging features are the abnormalities which often are the hyper intensive regions in the MRI images The objective of pathology annotation in ischemic stroke is to automatically annotate a 3-D MRI brain scan with its corresponding pathology class For instance, when small scatter lesions are observed in one vascular territory of a patient’s brain, the scan images can... have carefully examined the challenges in pathology- level annotation • We have proposed three novel approaches to existing unaddressed challenges of pathology- level annotation problem 7 • Our work provides a solid step toward improving the automation and practicality of existing pathology- level annotation systems • The proposed methods are evaluated into two important neurological domains, ischemic stroke... medical images Before going into the details of these techniques, we will briefly present the 9 strengths and weaknesses of three general machine learning approaches commonly used for these tasks, including discriminative models, generative models and ensemble learning We will then illustrate the common methods in annotating natural images and the challenges in applying to the medical imaging domain We... stroke subtype [57] The following are the main challenges in pathology annotation in this domain: 1 Semantic gap: The mapping between low -level image features and high -level image semantics is challenging in pathology annotation due to the complexity of the pathology A subtle change in the image could indicate a di↵erent pathological class Uncertainty and noise in the medical image feature extraction... framework consists of three main components: feature extraction, modelling and annotation (Figure 3-1) The life cycle of the annotation system can be divided into two di↵erent phases: 1) Training phase where training samples are used to construct modelling and annotation components and 2) Annotating phase where the system is actually used for annotating new images 19 Training' (volumetric)' images' Tes5ng'... attempt in an automated segmentation of stroke lesions from DWI images is introduced by Martel et al [81] They introduce a method of using the adaptive thresholding algorithm with spatial constraints for segmentation Matesin et al [82] apply seeded region growing algorithm and rule-based labeling to recognize brain lesions from CT images Usinskas et al [105] introduce an unsupervised classifier to identify... not only to ischemic stroke but also to other brain diseases or disorders (such as Traumatic brain injury) and in di↵erent image modalities (such as CT, MRI or X-ray) 1.2 Approaches and contributions The aim of this thesis is to propose solutions to the challenges of annotating medical images, especially of the brain, at the pathology level The annotation is guided by abnormalities found in the image . New approaches to automated annotation of pathology- level findings in brain images DINH THIEN ANH Bachelor of Computing National University of Singapore A THESIS SUBMITTED FOR THE DEGREE OF. designed di↵erently. There are two types of medical image annotation: organ -level annotation and pathology- level annotation. Organ level annotation is the process of annotating gen- eral aspects of the images such as. following are the main challenges in pathology annotation in this domain: 1. Semantic gap: The mapping between low -level image features and high -level image semantics is challenging in pathology annotation