Applying deep learning techniques for the localization and classification of digestive tract lesions = ứng dụng kỹ thuật học sâu trong khoanh vùng và phân loại tổn thương đường tiêu hóa

HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Master’s Thesis in Data Science and Artificial Intelligence Applying Deep Learning Techniques for the Localization and Classification of Digestive Tract Lesions PHAN NGOC LAN Lan.PN202634M@sis.hust.edu.vn Supervisor: Dr Dinh Viet Sang Department: Computer Science Ha Noi, 10/2021 Declaration of Authorship and Topic Sentences Personal information Full name: Phan Ngoc Lan Phone number: 094 979 1149 Email: Lan.PN202634M@sis.hust.edu.vn Major: Data Science and Artificial Intelligence Topic Applying Deep Learning techniques for the localization and classification of digestive tract lesions Contributions • Propose an extension of the polyp segmentation problem that accounts for neoplasm detection; • Propose a novel neural network architecture to address the problem; • Introduce a new annotated image dataset for the proposed problem; • Perform evaluations of the model on the new dataset, with comparisons to existing segmentation models Declaration of Authorship I hereby declare that my thesis, titled "Applying Deep Learning techniques for the localization and classification of digestive tract lesions", is the work of myself and my supervisor Dr Dinh Viet Sang All papers, sources, tables, used in this thesis have been thoroughly cited Supervisor confirmation Ha Noi, October 2021 Supervisor Dr Dinh Viet Sang Acknowledgments I would like to thank my supervisor, Dr Dinh Viet Sang, for his continued support and guidance throughout the course of my Masters’ studies He has been a great teacher and mentor for me since my undergraduate years, and I am proud to have completed this thesis under his supervision I would also like to thank Dr Dao Viet Hang and the team of doctors and physicians at the Institute of Gastroenterology and Hepatology Their tireless efforts have resulted in the NeoPolyp dataset presented in this thesis, and this work would not have been possible without their contributions The work in this thesis is also supported by the VINIF research project “Development of a Real-time AI-assisted System to Detect Colon Polyps and Identify Lesions at High Risk of Malignancy During Endoscopy”, code VINIF.2020.DA17 I would like to thank Vingroup and the Vingroup Innovation Foundation who have funded the project, along with the many students, faculty members and research staff who have helped me in my research I want to thank my family, my fiancee, and my friends, who have given me their unconditional love and support to finish my Masters’ studies Finally, I would like to again thank Vingroup and the Vingroup Innovation Foundation, who have supported my studies through their Domestic Master/Ph.D Scholarship program Parts of this work were published in the paper “NeoUNet: Towards accurate polyp segmentation and neoplasm detection” by Phan Ngoc Lan, Nguyen Sy An, Dao Viet Hang, Dao Van Long, Tran Quang Trung, Nguyen Thi Thuy and Dinh Viet Sang in the Proceedings of the 16th International Symposium on Visual Computing, 2021 Phan Ngoc Lan was funded by Vingroup Joint Stock Company and supii ported by the Domestic Master/Ph.D Scholarship Programme of Vingroup Innovation Foundation (VINIF), Vingroup Big Data Institute (VINBIGDATA), code VINIF.2020.ThS.BK.02 Abstract Medical image segmentation is a highly challenging task in computer vision with many important applications While the advent of deep learning techniques has created important breakthroughs in this field, there is still much room for improvement In this thesis, we focus on segmentation for digestive tract lesions, particularly colon polyps and esophageal lesions We identify a shortcoming in previous formulations of polyp segmentation, in which neoplasm classification is often ignored To address this issue, we propose a new problem formulation called Polyp Segmentation and Neoplasm Detection (PSND) In addition, this thesis proposes a deep neural network called NeoUNet to solve lesion segmentation and the PSND problem The proposed model is built upon U-Net, with a novel hybrid loss function that takes advantage of incomplete labels To validate NeoUNet, two medical image datasets are collected with the help of experts Our experiments show the effectiveness of NeoUnet over existing state-of-the-art models for image segmentation Keywords: Convolutional Neural Network, Medical Image Processing, Image Segmentation, U-Net, Colonoscopy Author Phan Ngoc Lan iv Contents List of Figures List of Tables Introduction 1.1 Problem overview 1.2 Thesis contributions 1.3 Thesis structure Theoretical Basis 2.1 Machine learning 2.2 Artificial neural networks 2.3 Convolutional neural networks 11 2.4 Attention mechanisms 16 2.5 Convolutional neural networks for semantic segmentation 18 2.6 Polyp segmentation and neoplasm classification 23 2.7 Problem formulation 23 Proposed Methods 3.1 NeoUNet 26 26 3.2 3.1.1 Motivation 26 3.1.2 Architecture overview 26 3.1.3 Encoder backbone 27 3.1.4 Attention mechanism 29 3.1.5 Decoder module 30 3.1.6 Loss function 31 Implementation details 33 Experiments 4.1 41 Dataset 41 4.1.1 NeoPolyp 41 4.1.2 Esophageal lesions 42 4.2 Experiment settings 43 4.3 Evaluation metrics 45 4.4 Results and discussion 46 4.4.1 Evaluating the HarDNet68 backbone 46 4.4.2 Comparison with baseline models 47 4.4.3 Evaluating the effect of undefined polyps 50 Conclusion 54 Bibliography 56 List of Figures 1.1 Example images of colon polyps and esophageal lesions Images on the right denote pixels with lesions in white 2.1 A 4-layer neural network1 2.2 Simple visualization of gradient descent2 2.3 Example of a computational graph Computation nodes store their derived gradients w.r.t their inputs 2.4 Speed comparison on several deep learning tasks between Xeon CPUs and NVIDIA Tesla GPUs 10 2.5 An example convolution layer5 12 2.6 An example of max-pooling6 12 2.7 LeNet-5 architecture [29] 13 2.8 Architecture of VGG-167 14 2.9 Example of a skip connection [16] 14 2.10 Architecture of Inception V1 (GoogLeNet) [49] 15 2.11 Example of dropout [14] 16 2.12 Architecture of EfficientNet-B0 [3] 17 2.13 Attention mechanism proposed in [5] 18 2.14 Transformer architecture [55] The network processes items in the sequence one-by-one, passing the output to the decoder for the next item 19 2.15 Architecture of the Fully Convolutional Network [34] 20 2.16 Overall U-Net architecture [43] 21 2.17 Overall PraNet architecture [13] 22 2.18 Overall HarDNet-MSEG architecture [19] 22 2.19 Classification targets for the polyp segmentation problem and the polyp segmentation and neoplasm detection problem 24 2.20 Expected outputs for polyp segmentation and PSND Black regions denote background pixels White regions denote polyp regions Green and red regions denote non-neoplastic and neoplastic polyp regions, respectively 24 2.21 Example of an image with an undefined polyp Pixels annotated in yellow denote the undefined polyp area 25 3.1 Overview of NeoUNet’s architecture 27 3.2 Structure of an example Harmonic Dense Block The value on each layer denotes the number of output channels 3.3 28 HarDNet68 architecture HDB layers may not be to scale with actual depths 29 3.4 Diagram of the additive attention gate module [38] 30 4.1 Pixel-wise distribution of polyp class labels in the NeoPolyp dataset Percentages are calculated on polyp pixels only (not including background pixels) 4.2 Learning rate over each step for the cosine annealing with warmup schedule 4.3 42 44 Examples of how Dice score and IoU scores are calculated Blue areas denote sets of pixels that are used for calculation Orangelined rectangles denote prediction mask pixels, and green-lined 4.4 rectangles denote ground-truth mask pixels 46 Qualitative results on the NeoPolyp test set 48 4.5 NeoUNet outputs for test images with undefined labels 4.6 Sample images and ground-truth labels from the NeoPolyp dataset Yellow pixels denote the undefined labels 4.7 49 52 Sample images and ground-truth labels from the esophageal lesion dataset 53 Figure 4.6: Sample images and ground-truth labels from the NeoPolyp dataset Yellow pixels denote the undefined labels Figure 4.7: Sample images and ground-truth labels from the esophageal lesion dataset Chapter Conclusion This thesis has presented the Polyp Segmentation and Neoplasm Detection problem, a challenging extension to the polyp segmentation and semantic segmentation problem in general PSND addresses a shortcoming in common formulations of polyp segmentation, which ignores neoplasm classes for polyps The proposed problem includes difficult challenges: high classification granularity, incomplete datasets and high requirements for annotators In order to address these challenges, the thesis proposes NeoUNet, a UNet-inspired neural network with components designed for PSND and also works well for other types of lesion segmentation NeoUNet features a robust and lightweight HarDNet68 backbone, coupled with a powerful attention gate mechanism and a novel loss formulation to take advantage of semi-labeled data NeoUNet’s design aims to provide a balanced trade-off between accuracy (with large attention gates and decoder layers) and latency (with the lightweight HarDNet backbone) To validate the effectiveness of NeoUNet, we present NeoPolyp, a curated dataset of colonoscopy images labeled with the help of medical professionals Our experiments show that NeoUNet outperforms several existing methods for polyp segmentation, proving that further research into specialized networks for PSND should yield even more improvements We hope that our work can be a solid baseline for further research into the PSND problem and digestive tract lesion segmentation in general For future works, we plan to continue improving the NeoUNet architecture to fit specific medical applications, as well as look into other promising architectures such 54 as Transformers for further improvement Bibliography [1] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al Tensorflow: A system for large-scale machine learning In 12th U SEN IX Symposium on Operating Systems Design and Implementation (OSDI 16), pages 265–283, 2016 [2] Nabila Abraham and Naimul Mefraz Khan A novel focal tversky loss function with improved attention u-net for lesion segmentation In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pages 683–687 IEEE, 2019 [3] Tashin Ahmed and Noor Hossain Nuri Sabab Classification and understanding of cloud structures via satellite images with efficientunet arXiv preprint arXiv:2009.12931, 2020 [4] Mohammad Ali Armin, Hans De Visser, Girija Chetty, Cedric Dumas, David Conlan, Florian Grimpen, and Olivier Salvado Visibility map: a new method in evaluation quality of optical colonoscopy In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 396–404 Springer, 2015 [5] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio Neural machine translation by jointly learning to align and translate arXiv preprint arXiv:1409.0473, 2014 [6] Yoshua Bengio and Daniel Faggella The rise of neural networks and deep learning in our everyday lives – a conversation with yoshua bengio, 2019 [7] James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio Theano: a cpu and gpu math expression compiler 56 In Proceedings of the Python for scientific computing conference (SciPy), volume Austin, TX, 2010 [8] Jorge Bernal, F Javier Sánchez, Gloria Fernández-Esparrach, Debora Gil, Cristina Rodríguez, and Fernando Vilari˜ no Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs saliency maps from physicians Computerized Medical Imaging and Graphics, 43:99– 111, 2015 [9] Giuseppe GA Celano A resnet-50-based convolutional neural network model for language id identification from speech recordings SIGTYP 2021, page 136, 2021 [10] Ping Chao, Chao-Yang Kao, Yu-Shan Ruan, Chien-Hsiang Huang, and Youn-Long Lin Hardnet: A low memory traffic network In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3552–3561, 2019 [11] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs IEEE transactions on pattern analysis and machine intelligence, 40(4):834–848, 2017 [12] William Falcon et al Pytorch lightning GitHub Note: https://github.com/PyTorchLightning/pytorch-lightning, 3, 2019 [13] Deng-Ping Fan, Ge-Peng Ji, Tao Zhou, Geng Chen, Huazhu Fu, Jianbing Shen, and Ling Shao Pranet: Parallel reverse attention network for polyp segmentation In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 263–273 Springer, 2020 [14] Marek Florkowski Classification of partial discharge images using deep convolutional neural networks Energies, 13:5496, 10 2020 [15] Ian Goodfellow, Yoshua Bengio, and Aaron Courville Deep Learning MIT Press, 2016 http://www.deeplearningbook.org [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun Deep residual learning for image recognition In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016 [17] Hanzhang Hu, Debadeepta Dey, Allison Del Giorno, Martial Hebert, and J Andrew Bagnell Log-densenet: how to sparsify a densenet arXiv preprint arXiv:1711.00002, 2017 [18] Jie Hu, Li Shen, and Gang Sun Squeeze-and-excitation networks In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018 [19] Chien-Hsiang Huang, Hung-Yu Wu, and Youn-Long Lin Hardnet- mseg: A simple encoder-decoder polyp segmentation neural network that achieves over 0.9 mean dice and 86 fps arXiv preprint arXiv:2101.07172, 2021 [20] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger Densely connected convolutional networks In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017 [21] Sergey Ioffe and Christian Szegedy Batch normalization: Accelerating deep network training by reducing internal covariate shift In International conference on machine learning, pages 448–456 PMLR, 2015 [22] Yuji Iwahori, Takayuki Shinohara, Akira Hattori, Robert J Woodham, Shinji Fukui, Manas Kamal Bhuyan, and Kunio Kasugai Automatic polyp detection in endoscope images using a hessian filter In MVA, pages 21–24, 2013 [23] Debesh Jha, Michael A Riegler, Dag Johansen, P˚ al Halvorsen, and H˚ avard D Johansen Doubleu-net: A deep convolutional neural network for medical image segmentation In Alba García Seco de Herrera, Alejandro Rodríguez González, K C Santosh, Zelalem Temesgen, Bridget Kane, and Paolo Soda, editors, 33rd IEEE International Symposium on Computer-Based Medical Systems, CBMS 2020, Rochester, MN, USA, July 28-30, 2020, pages 558–564 IEEE, 2020 [24] Debesh Jha, Pia H Smedsrud, Michael A Riegler, P˚ al Halvorsen, Thomas de Lange, Dag Johansen, and H˚ avard D Johansen Kvasir-seg: A segmented polyp dataset In International Conference on Multimedia Modeling, pages 451–462 Springer, 2020 [25] Debesh Jha, Pia H Smedsrud, Michael A Riegler, Dag Johansen, Thomas De Lange, P˚ al Halvorsen, and H˚ avard D Johansen Resunet++: An advanced architecture for medical image segmentation In 2019 IEEE International Symposium on Multimedia (ISM), pages 225–2255 IEEE, 2019 [26] Diederik P Kingma and Jimmy Ba Adam: A method for stochastic optimization arXiv preprint arXiv:1412.6980, 2014 [27] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton Imagenet classification with deep convolutional neural networks In Advances in neural information processing systems, pages 1097–1105, 2012 [28] R Lambert The paris endoscopic classification of superficial neoplastic lesions: esophagus, stomach, and colon: November 30 to december 1, 2002 Gastrointest Endosc, 58:S3–S43, 2003 [29] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al Gradient-based learning applied to document recognition Proceedings of the IEEE, 86(11):2278–2324, 1998 [30] Suck-Ho Lee, Il-Kwun Chung, Sun-Joo Kim, Jin-Oh Kim, Bong-Min Ko, Young Hwangbo, Won Ho Kim, Dong Hun Park, Sang Kil Lee, Cheol Hee Park, et al An adequate level of training for technical competence in screening and diagnostic colonoscopy: a prospective multicenter evaluation of the learning curve Gastrointestinal endoscopy, 67(4):683–689, 2008 [31] AM Leufkens, MGH Van Oijen, FP Vleggaar, and PD Siersema Factors influencing the miss rate of polyps in a back-to-back colonoscopy study Endoscopy, 44(05):470–475, 2012 [32] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár Focal loss for dense object detection In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017 [33] Wenqi Liu and Kun Zeng Sparsenet: A sparse densenet for image classification arXiv preprint arXiv:1804.05340, 2018 [34] Jonathan Long, Evan Shelhamer, and Trevor Darrell Fully convolutional networks for semantic segmentation In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015 [35] Benny Sukma Negara, Eki Satria, Suwanto Sanjaya, and Dimas Reynaldi Dwi Santoso Resnet-50 for classifying indonesian batik with data augmentation In 2021 International Congress of Advanced Technology and Engineering (ICOTEN), pages 1–4 IEEE, 2021 [36] U.S Department of Health and Human Services Colonoscopy Jul 2017 https://www.niddk.nih.gov/health-information/diagnostic-te sts/colonoscopy [37] U.S Department of Health and Human Services Upper gi endoscopy Jul 2017 https://www.niddk.nih.gov/health-information/diagn ostic-tests/upper-gi-endoscopy [38] Ozan Oktay, Jo Schlemper, Loăc Le Folgoc, Matthew C H Lee, Mattias P Heinrich, Kazunari Misawa, Kensaku Mori, Steven G McDonagh, Nils Y Hammerla, Bernhard Kainz, Ben Glocker, and Daniel Rueckert Attention u-net: Learning where to look for the pancreas CoRR, 2018 [39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala Pytorch: An imperative style, high-performance deep learning library In H Wallach, H Larochelle, A Beygelzimer, F d'Alché-Buc, E Fox, and R Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035 Curran Associates, Inc., 2019 [40] Rajat Raina, Anand Madhavan, and Andrew Y Ng Large-scale deep unsupervised learning using graphics processors In Proceedings of the 26th annual international conference on machine learning, pages 873– 880 ACM, 2009 [41] Edmar Rezende, Guilherme Ruppert, Tiago Carvalho, Fabio Ramos, and Paulo De Geus Malicious software classification using transfer learning of resnet-50 deep neural network In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 1011–1014 IEEE, 2017 [42] Eduardo Ribeiro, Andreas Uhl, Georg Wimmer, and Michael Hăafner Exploring deep learning and transfer learning for colonic polyp classification Computational and mathematical methods in medicine, 2016, 2016 [43] Olaf Ronneberger, Philipp Fischer, and Thomas Brox U-net: Convolutional networks for biomedical image segmentation In International Conference on Medical image computing and computer-assisted intervention, pages 234–241 Springer, 2015 [44] Frank Rosenblatt The perceptron: a probabilistic model for information storage and organization in the brain Psychological review, 65(6):386, 1958 [45] Seyed Sadegh Mohseni Salehi, Deniz Erdogmus, and Ali Gholipour Tversky loss function for image segmentation using 3d fully convolutional deep networks In International workshop on machine learning in medical imaging, pages 379–387 Springer, 2017 [46] Juan Silva, Aymeric Histace, Olivier Romain, Xavier Dray, and Bertrand Granado Toward embedded detection of polyps in wce images for early diagnosis of colorectal cancer International Journal of Computer Assisted Radiology and Surgery, 9(2):283–293, 2014 [47] Karen Simonyan and Andrew Zisserman Very deep convolutional networks for large-scale image recognition In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015 [48] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov Dropout: a simple way to prevent neural networks from overfitting The journal of machine learning research, 15(1):1929– 1958, 2014 [49] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich Going deeper with convolutions In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015 [50] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna Rethinking the inception architecture for computer vision In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818–2826, 2016 [51] Nima Tajbakhsh, Suryakanth R Gurudu, and Jianming Liang Automated polyp detection in colonoscopy videos using shape and context information IEEE transactions on medical imaging, 35(2):630–644, 2015 [52] Mingxing Tan and Quoc Le Efficientnet: Rethinking model scaling for convolutional neural networks In International Conference on Machine Learning, pages 6105–6114, 2019 [53] Mingxing Tan and Quoc V Le Efficientnetv2: Smaller models and faster training arXiv preprint arXiv:2104.00298, 2021 [54] Zhiqiang Tang, Xi Peng, Shijie Geng, Yizhe Zhu, and Dimitris N Metaxas Cu-net: Coupled u-nets In 29th British Machine Vision Conference, 2019 [55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin Attention is all you need In Advances in neural information processing systems, pages 5998–6008, 2017 [56] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang Residual attention network for image classification In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164, 2017 [57] Zhe Wu, Li Su, and Qingming Huang Cascaded partial decoder for fast and accurate salient object detection In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3907– 3916, 2019 [58] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He Aggregated residual transformations for deep neural networks In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492–1500, 2017 [59] Matthew D Zeiler Adadelta: an adaptive learning rate method arXiv preprint arXiv:1212.5701, 2012 [60] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang Unet++: Redesigning skip connections to exploit multiscale features in image segmentation IEEE transactions on medical imaging, 39(6):1856–1867, 2019 Thesis summary Topic: Applying Deep Learning techniques for the localization and classification of digestive tract lesions Author: Phan Ngoc Lan Supervisor: Dr Dinh Viet Sang Keywords: Convolutional Neural Network, Medical Image Processing, Image Segmentation, U-Net, Colonoscopy Reason for choosing the topic: Organs in the digestive tract are highly susceptible to various types of lesions (esophageal lesions, stomach lesions, colon polyps) While not immediately dangerous, some lesions can develop into cancer and cause fatalities if not removed in time These lesions can be detected and removed early via endoscopy However, endoscopies are still challenging to perform and may not be 100% accurate, as doctors can miss hard-to-detect lesions during the process Therefore, a pressing need in treating digestive tract lesions is improving endoscopic accuracy Automatic lesion segmentation is a potential solution to this problem A system that is capable of detecting and assessing lesions in real time can help doctors minimize errors during endoscopy, as well as train new doctors effectively Goals of the thesis • Summarize and assess image processing techniques for the lesion segmentation classification problems • Contribute datasets and tools for further research into the problem in the future • Propose new improvements and models for the problem • Apply proposed models to the problem and into real applications Main content and contribution This thesis formalizes an extension of polyp segmentation called Polyp Segmentation and Neoplasm Detection (PSND) We show that the problem is a hybrid of fine-grained classification and semantic segmentation We then propose NeoUNet, a novel neural network architecture based on U-Net to solve PSND and general segmentation NeoUNet features a hybrid loss function that allows it to take advantage of undefined labels in PSND datasets We introduce a dataset called NeoPolyp for the PSND problem and an esophageal lesion dataset, annotated with the help of trained medical professionals Experiments are performed to validate the performance of NeoUNet compared to state-of-the-art polyp segmentation models on the two datasets The authors published one paper for the proposal of NeoUNet and PSND (ISVC-2021) • We propose Neo-UNet, a novel neural network architecture designed for the Polyp Segmentation and Neoplasm Detection problem; • We introduce an esophageal lesion dataset and the NeoPolyp dataset, an annotated polyp image dataset with neoplasm information; • We perform experiments comparing NeoUNet with state-of-the-art polyp segmentation models to evaluate the model Methodology NeoUNet is a neural network architecture based on U-Net, consisting of an encoder branch and decoder branch, connected by a series of skip connections between corresponding blocks We use HarDNet68 as the network backbone, which provides solid trade-offs between accuracy and speed The network’s skip connections use the attention gate mechanism to adaptively filter the encoder’s outputs A hybrid loss function is proposed, comprised of a primary loss and a secondary loss to improve segmentation accuracy and take advantage of undefined labels in the training data Conclusion This thesis has presented the Polyp Segmentation and Neoplasm Detection problem, a challenging extension to the polyp segmentation and semantic segmentation problem in general In order to address these challenges, the thesis proposes NeoUNet, a U-Net-inspired neural network with components designed for PSND NeoUNet features a robust and lightweight HarDNet68 backbone, coupled with a powerful attention gate mechanism and a novel loss formulation to take advantage of semi-labeled data To validate the effectiveness of NeoUNet, we present an esophageal lesion dataset and NeoPolyp, a curated dataset of colonoscopy images labeled with the help of medical professionals Our experiments show that NeoUNet outperforms several existing methods for polyp segmentation, proving that further research into specialized networks for PSND should yield even more improvements ... Declaration of Authorship I hereby declare that my thesis, titled "Applying Deep Learning techniques for the localization and classification of digestive tract lesions" , is the work of myself and my... Topic Applying Deep Learning techniques for the localization and classification of digestive tract lesions Contributions • Propose an extension of the polyp segmentation problem that accounts for. .. While the advent of deep learning techniques has created important breakthroughs in this field, there is still much room for improvement In this thesis, we focus on segmentation for digestive tract