Nghiên cứu và phát triển các phương pháp nhận dạng cây dựa trên nhiều ảnh bộ phận của cây, có tương tác với người sử dụng

HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY NGUYEN THI THANH NHAN INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION Major: Computer Science Code: 9480101 INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION SUPERVISORS: Assoc Prof Dr Le Thi Lan Assoc Prof Dr Hoang Van Sam Hanoi − 2020 HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Nguyen Thi Thanh Nhan INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION Major: Computer Science Code: 9480101 DOCTORAL DISSERTATION OF COMPUTER SCIENCE SUPERVISORS: Assoc Prof Dr Le Thi Lan Assoc Prof Dr Hoang Van Sam Hanoi − 2020 DECLARATION OF AUTHORSHIP I, Nguyen Thi Thanh Nhan, declare that this dissertation entitled, ”Interactive and multi-organ based plant species identification”, and the work presented in it is my own I confirm that: This work was done wholly or mainly while in candidature for a Ph.D research degree at Hanoi University of Science and Technology Where any part of this dissertation has previously been submitted for a degree or any other qualification at Hanoi University of Science and Technology or any other institution, this has been clearly stated Where I have consulted the published work of others, this is always clearly attributed Where I have quoted from the work of others, the source is always given With the exception of such quotations, this dissertation is entirely my own work I have acknowledged all main sources of help Where the dissertation is based on work done by myself jointly with others, I have made exactly what was done by others and what I have contributed myself Hanoi, January, 2020 PhD Student Nguyen Thi Thanh Nhan SUPERVISORS i ACKNOWLEDGEMENT First of all, I would like to thank my supervisors Assoc Prof Dr Le Thi Lan at The International Research Institute MICA - Hanoi University of Science and Technology, Assoc Prof Dr Hoang Van Sam at Vietnam National University of Forestry for their inspiration, guidance, and advice Their guidance helped me all the time of research and writing this dissertation Besides my advisors, I would like to thank Dr Vu Hai, Assoc Prof Dr Tran Thi Thanh Hai for their great discussion Special thanks to my friends/colleagues in MICA, Hanoi University of Science and Technology: Hoang Van Nam, Nguyen Hong Quan, Nguyen Van Toi, Duong Nam Duong, Le Van Tuan, Nguyen Huy Hoang, Do Thanh Binh for their technical supports They have assisted me a lot in my research process as well as they are co-authored in the published papers Moreover, I would like to thank reviewers of scientific conferences, journals and protection council, reviewers, they help me with many useful comments I would like to express a since gratitude to the Management Board of MICA Institute I would like to thank the Thai Nguyen University of Information and Communication Technology, Thai Nguyen over the years both at my career work and outside of the work As a Ph.D student of the 911 program, I would like to thank this program for financial support I also gratefully acknowledge the financial support for attending the conferences from the Collaborative Research Program for Common Regional Issue (CRC) funded by ASEAN University Network (Aun-Seed/Net), under the grant reference HUST/CRC/1501 and NAFOSTED (grant number 106.06-2018.23) Special thanks to my family, to my parents-in-law who took care of my family and created favorable conditions for me to study I also would like to thank my beloved husband and children for everything they supported and encouraged me for a long time to study Hanoi, January, 2020 Ph.D Student Nguyen Thi Thanh Nhan ii CONTENTS DECLARATION OF AUTHORSHIP i ACKNOWLEDGEMENT ii CONTENTS v SYMBOLS vi SYMBOLS viii LIST OF TABLES x LIST OF FIGURES xiv INTRODUCTION 1 LITERATURE REVIEW 1.1 Plant identification 1.1.1 Manual plant identification 1.1.2 Plant identification based on semi-automatic graphic tool 1.1.3 Automated plant identification 1.2 Automatic plant identification from images of single organ 1.2.1 Introducing the plant organs 1.2.2 General model of image-based plant identification 1.2.3 Preprocessing techniques for images of plant 1.2.4 Feature extraction 1.2.4.1 Hand-designed features 1.2.4.2 Deeply-learned features 1.2.5 Classification methods 1.3 Plant identification from images of multiple organs 1.3.1 Early fusion techniques for plant identification from images multiple organs 1.3.2 Late fusion techniques for plant identification from images multiple organs 1.4 Plant identification studies in Vietnam 1.5 Plant data collection and identification systems 1.6 Conclusions iii of of 10 10 10 12 12 13 13 16 18 20 20 22 26 28 30 31 33 35 44 LEAF-BASED PLANT IDENTIFICATION METHOD BASED ON KERNEL DESCRIPTOR 2.1 The framework of leaf-based plant identification method 2.2 Interactive segmentation 2.3 Feature extraction 2.3.1 Pixel-level features extraction 2.3.2 Patch-level features extraction 2.3.2.1 Generate a set of patches from an image with adaptive size 2.3.2.2 Compute patch-level feature 2.3.3 Image-level features extraction 2.4 Experimental results 2.4.1 Datasets 2.4.1.1 ImageCLEF 2013 dataset 2.4.1.2 Flavia dataset 2.4.1.3 LifeCLEF 2015 dataset 2.4.2 Experimental results 2.4.2.1 Results on ImageCLEF 2013 dataset 2.4.2.2 Results on Flavia dataset 2.4.2.3 Results on LifeCLEF 2015 dataset 2.5 Conclusions FUSION SCHEMES FOR MULTI-ORGAN BASED PLANT IDENTIFICATION 3.1 Introduction 3.2 The proposed fusion scheme RHF 3.3 The choice of classification model for single organ plant identification 3.4 Experimental results 3.4.1 Dataset 3.4.2 Single organ plant identification results 3.4.3 Evaluation of the proposed fusion scheme in multi-organ plant identification 3.5 Conclusion 45 45 46 50 50 51 51 52 55 56 56 56 57 57 58 58 60 61 66 67 67 69 75 77 78 79 79 87 A FRAMEWORK FOR AUTOMATIC PLANT IDENTIFICATION WITHOUT DEDICATED DATASET AND A CASE STUDY FOR BUILDING IMAGE-BASED PLANT RETRIEVAL 88 4.1 Introduction 88 4.2 Challenges of building automatic plant identification systems 88 iv 4.3 4.4 4.5 4.6 The framework for building automatic plant identification system without dedicated dataset 92 Plant organ detection 93 Case study: Development of image-based plant retrieval in VnMed application 99 Conclusions 104 CONCLUSIONS AND FUTURE WORKS 105 4.6.1 Short term 106 4.6.2 Long term 106 Bibliography 108 PUBLICATIONS 121 APPENDIX 122 v ABBREVIATIONS No Abbreviation Meaning AB Ada Boost ANN Artificial Neural Network Br Branch CBF Classification Base Fusion CNN Convolution Neural Network CNNs Convolution Neural Networks CPU Central Processing Unit CMC Cumulative Match Characteristic Curve DT Decision Tree 10 En Entire 11 FC Fully Connected 12 Fl Flower 13 FN False Negative 14 FP False Positive 15 GPU Graphics Processing Unit 16 GUI Graphic-User Interface 17 HOG Histogram of Oriented Gradients 18 ILSVRC ImageNet Large Scale Visual Recognition Competition 19 KDES Kernel DEScriptors 20 KNN K Nearest Neighbors 21 Le Leaf 22 L-SVM Linear Support Vector Machine 23 MCDCNN Multi Column Deep Convolutional Neural Networks 24 NB Naive Bayes 25 NNB Nearest NeighBor 26 OPENCV OPEN source Computer Vision Library 27 PC Persional Computer 28 PCA Principal Component Analysis 29 PNN Probabilistic Neural Network 30 QDA Quadratic Discriminant Analysis vi 31 RAM Random Acess Memory 32 ReLU Rectified Linear Unit 33 RHF Robust Hybrid Fusion 34 RF Random Forest 35 ROI Region Of Interest 36 SIFT Scale-Invariant Feature Transform 37 SM SoftMax 38 SURF Speeded Up Robust Features 39 SVM Support Vector Machine 40 SVM-RBF Support Vector Machine-Radial Basic Function kernel 41 TP True Positive 42 TN True Negative vii MATH SYMBOLS No Symbol Meaning Summation - sum of all values in range of series R R π Set of real number d Set of real number has d dimensions π = 3.141592654 w L2 normalize of vector w xi The i-th element of vector x sign(x) The sign function that determines the sign Equals if x ≥ 0, −1 if x < ∈ Is member of max The function takes the largest number from a list 10 arctan(x) It returns the angle whose tangent is a given number 11 cos(θ) Function of calculating cosine value of angle θ 12 sin(θ) Function of calculating sine value of angle θ 13 m(z) The magnitude of the gradient vector at pixel z 14 The orientation of gradient vector at pixel z 15 θ(z) ˜ θ(z) The normalized gradient vector 16 exp(x) ex 17 argmax(x) It indicates the element that reaches its maximum value 18 ⊗ The Kronecker product 19 xT Transposition of vector x 20 Product of all values in range of series 21 q The query-image set 22 si (Ik ) The confidence score of the plant species i−th when using image Ik as a query from a single organ plant 23 c The predicted class of the species for the query q 24 C The number of species in dataset 25 km˜ The gradient magnitude kernel 26 ko The orientation kernel 27 kp The position kernel 28 m(z) ˜ The normalized gradient magnitude viii