Interactive and multiorgan based plant species identification =176

HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY NGUYEN THI THANH NHAN INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION Major: Computer Science Code: 9480101 INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION SUPERVISORS: Assoc Prof Dr Le Thi Lan Prof Dr Hoang Van Sam Hanoi − 2020 HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Nguyen Thi Thanh Nhan INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION Major: Computer Science Code: 9480101 DOCTORAL DISSERTATION OF COMPUTER SCIENCE SUPERVISORS: Assoc Prof Dr Le Thi Lan Prof Dr Hoang Van Sam Hanoi − 2020 DECLARATION OF AUTHORSHIP I, Nguyen Thi Thanh Nhan, declare that this dissertation entitled, ”Interactive and multi-organ based plant species identification”, and the work presented in it is my own I confirm that: This work was done wholly or mainly while in candidature for a Ph.D research degree at Hanoi University of Science and Technology Where any part of this dissertation has previously been submitted for a degree or any other qualification at Hanoi University of Science and Technology or any other institution, this has been clearly stated Where I have consulted the published work of others, this is always clearly attributed Where I have quoted from the work of others, the source is always given With the exception of such quotations, this dissertation is entirely my own work I have acknowledged all main sources of help Where the dissertation is based on work done by myself jointly with others, I have made exactly what was done by others and what I have contributed myself Hanoi, May, 2020 PhD Student Nguyen Thi Thanh Nhan SUPERVISORS i ACKNOWLEDGEMENT First of all, I would like to thank my supervisors Assoc Prof Dr Le Thi Lan at The International Research Institute MICA - Hanoi University of Science and Technology, Assoc Prof Dr Hoang Van Sam at Vietnam National University of Forestry for their inspiration, guidance, and advice Their guidance helped me all the time of research and writing this dissertation Besides my advisors, I would like to thank Assoc Prof Dr Vu Hai, Assoc Prof Dr Tran Thi Thanh Hai for their great discussion Special thanks to my friends/colleagues in MICA, Hanoi University of Science and Technology: Hoang Van Nam, Nguyen Hong Quan, Nguyen Van Toi, Duong Nam Duong, Le Van Tuan, Nguyen Huy Hoang, Do Thanh Binh for their technical supports They have assisted me a lot in my research process as well as they are co-authored in the published papers Moreover, I would like to thank reviewers of scientific conferences, journals and protection council, reviewers, they help me with many useful comments I would like to express a since gratitude to the Management Board of MICA Institute I would like to thank the Thai Nguyen University of Information and Communication Technology, Thai Nguyen over the years both at my career work and outside of the work As a Ph.D student of the 911 program, I would like to thank this program for financial support I also gratefully acknowledge the financial support for attending the conferences from the Collaborative Research Program for Common Regional Issue (CRC) funded by ASEAN University Network (Aun-Seed/Net), under the grant reference HUST/CRC/1501 and NAFOSTED (grant number 106.06-2018.23) Special thanks to my family, to my parents-in-law who took care of my family and created favorable conditions for me to study I also would like to thank my beloved husband and children for everything they supported and encouraged me for a long time to study Hanoi, May, 2020 Ph.D Student Nguyen Thi Thanh Nhan ii CONTENTS DECLARATION OF AUTHORSHIP i ACKNOWLEDGEMENT ii CONTENTS v SYMBOLS vi SYMBOLS viii LIST OF TABLES xi LIST OF FIGURES xvi INTRODUCTION 1 LITERATURE REVIEW 1.1 1.2 Plant identification 1.1.1 Manual plant identification 10 10 1.1.2 Plant identification based on semi-automatic graphic tool 11 1.1.3 Automated plant identification 12 Automatic plant identification from images of single organ 13 1.2.1 Introducing the plant organs 13 1.2.2 General model of image-based plant identification 16 1.2.3 Preprocessing techniques for images of plant 17 1.2.4 Feature extraction 19 1.2.4.1 Hand-designed features 20 1.2.4.2 Deeply-learned features 22 Training methods 25 Plant identification from images of multiple organs 28 1.2.5 1.3 10 1.3.1 1.3.2 Early fusion techniques for plant identification from images of multiple organs 30 Late fusion techniques for plant identification from images of multiple organs 31 1.4 Plant identification studies in Vietnam 33 1.5 Plant data collection and identification systems 35 1.6 Conclusions iii 43 LEAF-BASED PLANT IDENTIFICATION METHOD BASED ON KERNEL DESCRIPTOR 45 2.1 The framework of leaf-based plant identification method 45 2.2 Interactive segmentation 46 2.3 Feature extraction 50 Pixel-level features extraction Patch-level features extraction 50 51 2.3.1 2.3.2 2.3.2.1 Generate a set of patches from an image with adaptive size 2.3.2.2 51 Compute patch-level feature 52 2.3.3 Image-level features extraction 55 2.3.4 Time complexity analysis 56 2.4 Classification 57 2.5 Experimental results 2.5.1 Datasets 57 57 2.5.1.1 ImageCLEF 2013 dataset 57 2.5.1.2 Flavia dataset 57 2.5.1.3 LifeCLEF 2015 dataset 58 Experimental results 58 2.5.2.1 Results on ImageCLEF 2013 dataset 58 2.5.2.2 2.5.2.3 Results on Flavia dataset Results on LifeCLEF 2015 dataset 61 61 Conclusions 68 2.5.2 2.6 FUSION SCHEMES FOR MULTI-ORGAN BASED PLANT IDENTIFICATION 69 3.1 Introduction 69 3.2 The proposed fusion scheme RHF 71 3.3 3.4 The choice of classification model for single organ plant identification Experimental results 77 79 3.4.1 Dataset 80 3.4.2 Single organ plant identification results 81 3.4.3 Evaluation of the proposed fusion scheme in multi-organ plant 3.5 identification 81 Conclusion 89 TOWARDS BUILDING AN AUTOMATIC PLANT RETRIEVAL BASED ON PLANT IDENTIFICATION 90 4.1 Introduction 90 4.2 Challenges of building automatic plant identification systems 90 iv 4.3 The framework for building automatic plant identification system 94 4.4 Plant organ detection 96 4.5 Case study: Development of image-based plant retrieval in VnMed application 101 4.6 Conclusions 106 CONCLUSIONS AND FUTURE WORKS 107 4.6.1 Short term 108 4.6.2 Long term 108 Bibliography 110 PUBLICATIONS 124 APPENDIX 125 v ABBREVIATIONS No Abbreviation Meaning AB Ada Boost ANN Artificial Neural Network Br Branch CBF Classification Base Fusion CNN Convolution Neural Network CNNs Convolution Neural Networks CPU Central Processing Unit CMC Cumulative Match Characteristic Curve DT Decision Tree 10 En Entire 11 FC Fully Connected 12 Fl Flower 13 FN False Negative 14 FP False Positive 15 GPU Graphics Processing Unit 16 GUI Graphic-User Interface 17 HOG Histogram of Oriented Gradients 18 ILSVRC ImageNet Large Scale Visual Recognition Competition 19 KDES Kernel DEScriptors 20 KNN K Nearest Neighbors 21 Le Leaf 22 L-SVM Linear Support Vector Machine 23 MCDCNN Multi Column Deep Convolutional Neural Networks 24 NB Naive Bayes 25 NNB Nearest NeighBor 26 OPENCV OPEN source Computer V ision Library 27 PC Persional Computer 28 PCA Principal Component Analysis 29 PNN Probabilistic Neural Network 30 QDA Quadratic Discriminant Analysis vi 31 RAM Random Acess Memory 32 ReLU Rectified Linear Unit 33 RHF Robust Hybrid Fusion 34 RF Random Forest 35 ROI Region Of Interest 36 SIFT Scale-Invariant Feature Transform 37 SM SoftMax 38 SURF Speeded Up Robust Features 39 SVM Support Vector Machine 40 SVM-RBF Support Vector Machine-Radial Basic Function kernel 41 TP True Positive 42 TN True Negative vii MATH SYMBOLS No Symbol Meaning % 1% = 1/100 × Multiplication + Addition Summation - sum of all values in range of series P Multiplication R Set of real number Rd Set of real number has d dimensions / Division = Equality 10 ≥ Greater than or equal to 11 ≤ Less than or equal to 12 π π = 3.141592654 13 kwk L2 normalize of vector w 14 xi The i-th element of vector x 15 sign(x) The sign function that determines the sign Equals if x ≥ 0, −1 if x < 16 ∈ Is member of 17 max The function takes the largest number from a list 18 ∀ For all 19 m Spatial moment of an image 20 I (x, y ) The intensity value at (x, y) of an image 21 − Subtraction 22 O Complexity of an algorithm 23 arctan(x) 24 cos(θ ) Function of calculating cosine value of angle θ 25 sin(θ ) Function of calculating sine value of angle θ 26 m(z ) The magnitude of the gradient vector at pixel z 27 The orientation of gradient vector at pixel z 28 θ (z ) θ˜ (z ) 29 exp(x) ex It returns the angle whose tangent is a given number The normalized gradient vector viii [26] Arora A., Gupta A., Bagmar N., Mishra S., and Bhattacharya A (2012) A plant identification system using shape and morphological features on segmented leaflets: Team iitk, clef 2012 In CLEF (Online Working Notes/Labs/Workshop) [27] Bakic V., Mouine S., Ouertani-Litayem S., Verroust-Blondet A., Yahiaoui I., Goăeau H., and Joly A (2013) Inria’s participation at imageclef 2013 plant identification task In CLEF (Online Working Notes/Labs/Workshop) 2013 [28] Cerutti G., Antoine V., Tougne L., Mille J., Valet L., Coquin D., and Vacavant A (2012) Reves participation-tree species classification using random forests and botanical features In Conference and Labs of the Evaluation Forum (CLEF) [29] Cerutti G., Tougne L., Mille J., Vacavant A., and Coquin D (2013) Understanding leaves in natural images–a model-based approach for tree species identification Computer Vision and Image Understanding , 117(10):pp 1482–1501 [30] Cerutti G., Tougne L., Mille J., Vacavant A., and Coquin D (2013) A modelbased approach for compound leaves understanding and identification In 2013 IEEE International Conference on Image Processing , pp 1471–1475 [31] Rzanny M., Seeland M., Wăaldchen J., and Măader P (2017) Acquiring and preprocessing leaf images for automated plant identification: understanding the tradeoff between effort and information gain Plant Methods , 13(1):pp 1–11 [32] Nilsback M.E and Zisserman A (2009) An automatic visual flora-segmentation and classification of flower images Ph.D thesis, Oxford University [33] Wang X.F., Du J.X., and Zhang G.J (2005) Recognition of leaf images based on shape features using a hypersphere classifier In International Conference on Intelligent Computing , pp 87–96 [34] Chen Q., Abedini M., Garnavi R., and Liang X (2014) Ibm research australia at lifeclef2014: Plant identification task In CLEF (Working Notes) , pp 693–704 [35] Nakayama H (2013) Nlab-utokyo at imageclef 2013 plant identification task In CLEF (Working Notes) [36] Wang Z., Sun X., Zhang Y., Ying Z., and Ma Y (2016) Leaf recognition based on pcnn Neural Computing and Applications , 27(4):pp 899–908 [37] Jamil N., Hussin N.A.C., Nordin S., and Awang K (2015) Automatic plant identification: Is shape the key feature? Procedia Computer Science , 76:pp 436–442 112 [38] Mouine S., Yahiaoui I., and Verroust-Blondet A (2013) Combining leaf salient points and leaf contour descriptions for plant species recognition In International Conference Image Analysis and Recognition, pp 205–214 [39] Rodrigo R., Samarawickrame K., and Mindya S (2013) An intelligent flower analyzing system for medicinal plants In WSCG 2013 Conference on Computer Graphics, Visualization and Computer Vision, pp 41–44 V´aclav Skala-UNION Agency [40] Angelova A., Zhu S., Lin Y., Wong J., and Shpecht C (2012) Development and deployment of a large-scale flower recognition mobile app NEC Labs America Technical Report [41] Aakif A and Khan M.F (2015) Automatic classification of plants based on their leaves Biosystems Engineering , 139:pp 66–75 [42] Du J.X., Wang X.F., and Zhang G.J (2007) Leaf shape based plant species recognition Applied mathematics and computation , 185(2):pp 883–893 [43] Du J.X., Shao M.W., Zhai C.M., Wang J., Tang Y., and Chen C.L.P (2016) Recognition of leaf image set based on manifold–manifold distance Neurocomputing , 188:pp 131–138 [44] Hu R., Jia W., Ling H., and Huang D (2012) Multiscale distance matrix for fast plant leaf recognition IEEE transactions on image processing , 21(11):pp 4667–4672 [45] Lee K.B and Hong K.S (2013) An implementation of leaf recognition system using leaf vein and shape International Journal of Bio-Science and Bio-Technology , 5(2):pp 57–66 [46] Bo L., Ren X., and Fox D (2010) Kernel descriptors for visual recognition In Advances in neural information processing systems, pp 244–252 [47] Le T.L., Tran D.T., and Pham N.H (2014) Kernel descriptor based plant leaf identification In Image Processing Theory, Tools and Applications (IPTA), 2014 4th International Conference on, pp 1–5 [48] Yoo H.J (2015) Deep convolution neural networks in computer vision IEIE Transactions on Smart Processing & Computing , 4(1):pp 35–43 [49] Krizhevsky A., Sutskever I., and Hinton G.E (2012) Imagenet classification with deep convolutional neural networks In Advances in neural information processing systems, pp 1097–1105 113 [50] Mythili C and Kavitha V (2014) Recognition of plant leaf in medicine Journal of Convergence Information Technology , 9(2):pp 61–69 [51] Ghasab M.A.J., Khamis S., Mohammad F., and Fariman H.J (2015) Feature decision-making ant colony optimization system for an automated recognition of plant species Expert Systems with Applications , 42(5):pp 2361–2370 [52] Hall D., McCool C., Dayoub F., Sunderhauf N., and Upcroft B (2015) Evaluation of features for leaf classification in challenging conditions In Applications of Computer Vision (WACV), 2015 IEEE Winter Conference on , pp 797–804 [53] Wang B., Brown D., Gao Y., and La Salle J (2015) March: Multiscale-archheight description for mobile retrieval of leaf images Information Sciences , 302:pp 132–148 [54] Zhao C., Chan S.S., Cham W.K., and Chu L (2015) Plant identification using leaf shapes—a pattern counting approach Pattern Recognition , 48(10):pp 3203– 3215 [55] Prasad S., Peddoju S.K., and Ghosh D (2016) An adaptive plant leaf mobile informatics using rssc Multimedia Tools and Applications , pp 1–25 ISSN 15737721 doi:10.1007/s11042-016-4040-8 [56] VijayaLakshmi B and Mohan V (2016) Kernel-based pso and frvm: An automatic plant leaf type detection using texture, shape, and color features Computers and Electronics in Agriculture, 125:pp 99–112 [57] Nilsback M.E and Zisserman A (2006) A visual vocabulary for flower classification In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pp 1447–1454 [58] Nilsback M.E and Zisserman A (2008) Automated flower classification over a large number of classes In Computer Vision, Graphics & Image Processing, 2008 ICVGIP’08 Sixth Indian Conference on, pp 722–729 [59] Arivazhagan S., Shebiah R.N., Nidhyanandhan S.S., and Ganesan L (2010) Fruit recognition using color and texture features Journal of Emerging Trends in Computing and Information Sciences , 1(2):pp 90–94 [60] Huang Z.K (2006) Bark classification using rbpnn based on both color and texture feature International Journal of Computer Science and Network Security , 6(10):pp 100–103 [61] Zeiler M.D and Fergus R (2014) Visualizing and understanding convolutional networks In European conference on computer vision , pp 818–833 114 [62] Simonyan K and Zisserman A (2014) Very deep convolutional networks for large-scale image recognition CoRR , abs/1409.1556 [63] Szegedy C., Liu W., Jia Y., Sermanet P., Reed S., Anguelov D., Erhan D., Vanhoucke V., and Rabinovich A (2015) Going deeper with convolutions In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1–9 [64] He K., Zhang X., Ren S., and Sun J (2015) Deep residual learning for image recognition CoRR , abs/1512.03385 [65] http://cs231n.github.io/convolutional-networks/(retrievel 17/january/2018) [66] Yosinski J., Clune J., Bengio Y., and Lipson H (2014) How transferable are features in deep neural networks? In Advances in neural information processing systems, pp 3320–3328 [67] Russakovsky O., Deng J., Su H., Krause J., Satheesh S., Ma S., Huang Z., Karpathy A., Khosla A., Bernstein M., et al (2015) Imagenet large scale visual recognition challenge International Journal of Computer Vision , 115(3):pp 211 252 [68] Wăaldchen J., Rzanny M., Seeland M., and Măader P (2018) Automated plant species identicationtrends and future directions PLoS computational biology , 14(4):p e1005993 [69] Thyagharajan K and Raji I.K (2018) A review of visual descriptors and classification techniques used in leaf species identification Archives of Computational Methods in Engineering , pp 1–28 [70] Cortes C and Vapnik V (1995) Support-vector networks Machine learning , 20(3):pp 273–297 [71] Weston J and Watkins C (1998) Multi-class support vector machines Technical report, Citeseer [72] Chang C.C and Lin C.J (2011) Libsvm: A library for support vector machines ACM transactions on intelligent systems and technology (TIST), 2(3):pp 27:1– 27:27 [73] Burges C.J (1998) A tutorial on support vector machines for pattern recognition Data mining and knowledge discovery , 2(2):pp 121–167 [74] Cerutti G., Tougne L., Sacca C., Joliveau T., Mazagol P.O., Coquin D., and Vacavant A (2013) Late information fusion for multi-modality plant species 115 identification In Working notes for Conference and Labs of the Evaluation Forum [75] Choi S (2015) Plant identification with deep convolutional neural network: Snumedinfo at lifeclef plant identification task 2015 In CLEF (Working Notes) [76] Ghazi M.M., Yanikoglu B., and Aptoula E (2017) Plant identification using deep neural networks via optimization of transfer learning parameters Neurocomputing , 235:pp 228–235 [77] Ge Z., McCool C., Sanderson C., and Corke P (2015) Content specific feature learning for fine-grained plant classification In CLEF (Working Notes) [78] Lee S.H., Chan C.S., Mayo S.J., and Remagnino P (2017) How deep learning extracts and learns leaf features for plant classification Pattern Recognition , 71:pp 1–13 [79] He A and Tian X (2016) Multi-organ plant identification with multi-column deep convolutional neural networks In 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC)2016 , pp 002020–002025 [80] Mehdipour Ghazi M., Yankoglu B., Aptoula E., Muslu ă and Ozdemir ă O., M.C (2015) Sabanci-okan system in lifeclef 2015 plant identification competition CLEF (Conference and Labs of the Evaluation Forum) [81] Ahmed A., Atito S., Yanıko˘glu B., and Aptoula E (2017) Plant identification with large number of classes: Sabanciu-gebzetu system in plantclef 2017 CLEF [82] Jovi´c M., Hatakeyama Y., Dong F., and Hirota K (2006) Image retrieval based on similarity score fusion from feature similarity ranking lists In International conference on Fuzzy Systems and Knowledge Discovery , pp 461–470 [83] Kittler J., Hatef M., Duin R.P., and Matas J (1998) On combining classifiers IEEE transactions on pattern analysis and machine intelligence , 20(3):pp 226– 239 [84] Makihara Y., Muramatsu D., Iwama H., and Yagi Y (2013) On combining gait features In 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp 1–8 IEEE [85] He M., Horng S.J., Fan P., Run R.S., Chen R.J., Lai J.L., Khan M.K., and Sentosa K.O (2010) Performance evaluation of score level fusion in multimodal biometric systems Pattern Recognition , 43(5):pp 1789–1800 116 [86] Mohamed I., Diane L., and Fr´ed´eric P (2014) Plant species recognition using bag-of-word with svm classifier in the context of the lifeclef challenge Working Notes of CLEF [87] Champ J., Lorieul T., Servajean M., and Joly A (2015) A comparative study of fine-grained classification methods in the context of the lifeclef plant identication challenge 2015 [88] Săunderhauf N., McCool C., Upcroft B., and Perez T (2014) Fine-grained plant classification using convolutional neural networks for feature extraction In CLEF (Working Notes), pp 756–762 [89] Reyes A.K., Caicedo J.C., and Camargo J.E (2015) Fine-tuning deep convolutional networks for plant recognition In CLEF (Working Notes) [90] Dimitrovski I., Madjarov G., Kocev D., and Lameski P (2014) Maestra at lifeclef 2014 plant task: Plant identification using visual data In CLEF (Working Notes), pp 705–714 ˇ [91] Sulc M and Matas J (2017) Learning with noisy and trusted labels for finegrained plant recognition Working notes of CLEF [92] T´oth B.P., T´oth M.J., Papp D., and Szăucs G (2016) Deep learning and svm classification for plant recognition in content-based large scale image retrieval In CLEF (Working Notes) , pp 569–578 [93] Sz˝ucs G., Papp D., and Lovas D (2014) Viewpoints combined classification, method in image-based plant identification task In Working Notes for CLEF 2014 Conference, volume 1180, pp 763–770 [94] Affouard A., Goăeau H., Bonnet P., Lombardo J.C., and Joly A (2017) Pl@ntnet app in the era of deep learning In ICLR 2017 Workshop Track-5th International Conference on Learning Representations , pp 1–6 [95] Zhu H., Huang X., Zhang S., and Yuen P.C (2017) Plant identification via multipath sparse coding Multimedia Tools and Applications , 76(3):pp 4599– 4615 [96] Lee S.H., Chang Y.L., and Chan C.S (2017) Lifeclef 2017 plant identification challenge: Classifying plants using generic-organ correlation features Working Notes of CLEF , 2017 [97] Le T.L and Pham N.H (2013) Mica at imageclef 2013 plant identification task In CLEF (Working Notes) 117 [98] Pham N.H., Le T.L., Grard P., and Nguyen V.N (2013) Computer aided plant identification system In Computing, Management and Telecommunications (ComManTel), 2013 International Conference on, pp 134–139 [99] Nguyen Q.K., Le T.L., and Pham N.H (2013) Leaf based plant identification system for android using surf features in combination with bag of words model and supervised learning In 2013 International Conference on Advanced Technologies for Communications (ATC 2013), pp 404–407 [100] Hong P.T.T., Ha D.T.T., and Thuy N.T (2013) Leaf image classification using support vector machine Journal of Science and Development (in Vietnamese) , 11:pp 1045–1052 [101] Huan N.V and Tao N.V (2015) Technique of searching medicinal plants based on content for detection, management and exploitation In Proceedings of The 2015 National Conference on Electronics, Communications and Information Technology (in Vietnamese) (ECIT 2015), pp 353–357 [102] Anh H V., Hoa.T D., Bao.T N., and Van-Huy P (2019) Vietnamese herbal plant recognition using deep convolutional features International Journal of Machine Learning and Computing , 9:pp 363–367 [103] Duong-Trung N., Quach L.D., and Nguyen C.N (2019) Learning deep transferability for several agricultural classification problems International Journal of Advanced Computer Science and Applications, 10(1):pp 58–67 [104] Thanh T.K.N., Truong Q.B., Truong Q.D., and Xuan H.H (2018) Depth learning with convolutional neural network for leaves classifier based on shape of leaf vein In Asian Conference on Intelligent Information and Database Systems , pp 565– 575 [105] Thuc H.L.U., Duc N.V., Thien H.T., Tri L.V., and Hanh L.T.M (2017) Automatically identify some popular ornamental flowers in vietnam based on computer vision technique In Proceedings of The XX National Conference on Selected issues of information and communication technology(in Vietnamese), pp 366–371 [106] Thanh N.D (2017) Identification and classification fruits in color images: Thesis of software engineering master: 60480103 Master’s thesis, Vietnam National University, Hanoi - University of Engineering and Technology (in Vietnamese) [107] Joly A., Goăeau H., Bonnet P., Spampinato C., Glotin H., Rauber A., Vellinga W.P., Fisher R., and Măuller H (2014) Are species identification tools biodiversity-friendly? In Proceedings of the 3rd ACM International Workshop on Multimedia Analysis for Ecological Data, pp 3136 118 [108] Goăeau H., Bonnet P., Joly A., Baki´c V., Barbe J., Yahiaoui I., Selmi S., Carr´e J., Barth´el´emy D., Boujemaa N., et al (2013) Pl@ntnet mobile app In Proceedings of the 21st Association for Computing Machinery(ACM) international conference on Multimedia, pp 423424 [109] Goăeau H., Bonnet P., Joly A., Affouard A., Bakic V., Barbe J., Dufour S., Selmi S., Yahiaoui I., Vignau C., et al (2014) Pl@ntnet mobile 2014: Android port and new features In Proceedings of international conference on multimedia retrieval , p 527 [110] http://www.inaturalist.org/(retrieved 15/january/2017) [111] Scanlon E., Woods W., and Clow D (2014) Informal participation in science in the uk: Identification, location and mobility with ispot Educational Technology & Society , 17(2):pp 58–71 [112] Silvertown J., Harvey M., Greenwood R., Dodd M., Rosewell J., Rebelo T., Ansine J., and McConway K (2015) Crowdsourcing the identification of organisms: A case-study of ispot ZooKeys , 480:p 125 [113] https://itunes.apple.com/nz/app/flora-finder/id688613607?mt=8(retrieved 15/may/2017) [114] http://www.plantifier.com(retrieved 15/may/2017) [115] Nilsback M.E and Zisserman A (Dec 2008) Automated flower classification over a large number of classes In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing , pp 722729 doi:10.1109/ICVGIP.2008.47 [116] Joly A., Goăeau H., Glotin H., Spampinato C., Bonnet P., Vellinga W.P., Planqu´e R., Rauber A., Palazzo S., Fisher B., et al (2015) Lifeclef 2015: multimedia life species identification challenges In International Conference of the CrossLanguage Evaluation Forum for European Languages , pp 462–483 [117] Prasad S., Peddoju S.K., and Ghosh D (2017) Efficient plant leaf representations: A comparative study In Region 10 Conference, TENCON 2017-2017 IEEE , pp 1175–1180 [118] Wu S.G., Bao F.S., Xu E.Y., Wang Y.X., Chang Y.F., and Xiang Q.L (2007) A leaf recognition algorithm for plant classification using probabilistic neural network In 2007 IEEE international symposium on signal processing and information technology , pp 11–16 119 [119] Gu X., Du J.X., and Wang X.F (2005) Leaf recognition based on the combination of wavelet transform and gaussian interpolation In International Conference on Intelligent Computing , pp 253–262 [120] Cerutti G., Tougne L., Mille J., Vacavant A., and Coquin D (2011) Guiding active contours for tree leaf segmentation and identification In CLEF 2011, Conference on Multilingual and Multimodal Information Access Evaluation [121] Bo L and Sminchisescu C (2009) Efficient match kernel between sets of features for visual recognition In Advances in neural information processing systems , pp 135–143 [122] NGUYEN V.T (2015) Visual interpretation of hand postures for humanmachine interaction Ph.D thesis, Universit´e de La Rochelle [123] Achanta R., Hemami S., Estrada F., and Susstrunk S (2009) Frequency-tuned salient region detection In Computer vision and pattern recognition, 2009 cvpr 2009 ieee conference on, pp 1597–1604 [124] Meyer F and Beucher S (1990) Morphological segmentation Journal of visual communication and image representation, 1(1):pp 21–46 [125] Beucher S and Meyer F (1993) The morphological approach to segmentation: the watershed transformation Mathematical morphology in image processing , 34:pp 433–481 [126] Kornilov A and Safonov I (2018) An overview of watershed algorithm implementations in open source libraries Journal of Imaging , 4(10):pp 1–15 [127] Le T.L., Tran D.T., and Hoang V.N (2014) Fully automatic leaf-based plant identification, application for vietnamese medicinal plant search In Proceedings of the fifth symposium on information and communication technology , pp 146– 154 [128] Dey S., Bhattacharya B.B., Kundu M.K., and Acharya T.A (2002) A simple architecture for computing moments and orientation of an image Fundamenta Informaticae, 52(4):pp 285–295 [129] Priyabrata K (2018) Effective and efficient kernel-based image representations for classification and retrieval Ph.D thesis, Federation University Australia [130] Lowe D.G (1999) Object recognition from local scale-invariant features In Computer vision, 1999 The proceedings of the seventh IEEE international conference on, volume 2, pp 1150–1157 120 [131] Maji S., Berg A.C., and Malik J (2013) Efficient classification for additive kernel svms IEEE transactions on pattern analysis and machine intelligence , 35(1):pp 66–77 [132] Vedaldi A and Zisserman A (2012) Efficient additive kernels via explicit feature maps IEEE transactions on pattern analysis and machine intelligence , 34(3):pp 480–492 [133] Chaki J., Parekh R., and Bhattacharya S (2015) Recognition of whole and deformed plant leaves using statistical shape features and neuro-fuzzy classifier In 2015 IEEE 2nd international conference on recent trends in information systems (ReTIS), pp 189–194 [134] Chaki J., Parekh R., and Bhattacharya S (2015) Plant leaf recognition using texture and shape features with neural classifiers Pattern Recognition Letters , 58:pp 61–68 [135] Wang Z., Sun X., Ma Y., Zhang H., Ma Y., Xie W., and Zhang Y (2014) Plant recognition based on intersecting cortical model In 2014 International joint conference on neural networks (IJCNN), pp 975–980 [136] Kheirkhah F.M and Asghari H (2018) Plant leaf classification using gist texture features IET Computer Vision , 13(4):pp 369–375 [137] Tsolakidis D.G., Kosmopoulos D.I., and Papadourakis G (2014) Plant leaf recognition using zernike moments and histogram of oriented gradients In Hellenic Conference on Artificial Intelligence , pp 406–417 [138] Du J.x., Zhai C.M., and Wang Q.P (2013) Recognition of plant leaf image based on fractal dimension features Neurocomputing , 116:pp 150–156 [139] Priya C.A., Balasaravanan T., and Thanamani A.S (2012) An efficient leaf recognition algorithm for plant classification using support vector machine In International conference on pattern recognition, informatics and medical engineering (PRIME-2012), pp 428–432 [140] Jain A., Nandakumar K., and Ross A (2005) Score normalization in multimodal biometric systems Pattern recognition , 38(12):pp 2270–2285 [141] Wang F and Han J (2009) Multimodal biometric authentication based on score level fusion using support vector machine Opto-electronics review , 17(1):pp 59–64 121 [142] Abdiansah A and Wardoyo R (2015) Time complexity analysis of support vector machines (svm) in libsvm International journal computer and application , 128(3):pp 28–34 [143] Singh A and Kisku D.R (2019) Detection of rare genetic diseases using facial 2d images with transfer learning In 2018 8th International Symposium on Embedded Computing and System Design (ISED), pp 26–30 [144] Van Horn G., Mac Aodha O., Song Y., Cui Y., Sun C., Shepard A., Adam H., Perona P., and Belongie S (2018) The inaturalist species classification and detection dataset In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8769–8778 [145] Gaston K.J and O’Neill M.A (2004) Automated species identification: why not? Philosophical transactions of the Royal society B: Biological sciences , 359(1444):pp 655–667 [146] Săoderkvist O (2001) Computer vision classication of leaves from swedish trees Masters thesis, Linkăoping University [147] Mouine S., Yahiaoui I., and Verroust-Blondet A (2012) Advanced shape context for plant species identification using leaf image retrieval In Proceedings of the 2nd (Association for Computing Machinery)ACM international conference on multimedia retrieval , p 49 [148] Novotn`y P and Suk T (2013) Leaf recognition of woody species in central europe Biosystems engineering , 115(4):pp 444–452 [149] Mzoughi O., Yahiaoui I., and Boujemaa N (2012) Petiole shape detection for advanced leaf identification In 2012 19th IEEE International Conference on Image Processing , pp 1033–1036 [150] Lee S.H., Chan C.S., Wilkin P., and Remagnino P (2015) Deep-plant: Plant identification with convolutional neural networks In Image Processing (ICIP), 2015 IEEE International Conference on, pp 452–456 [151] Seeland M., Rzanny M., Alaqraa N., Wăaldchen J., and Măader P (2017) Plant species classication using flower images—a comparative study of local feature representations PloS one , 12(2):p e0170629 [152] Deng J., Dong W., Socher R., Li L.J., Li K., and Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database In Computer Vision and Pattern Recognition, 2009 CVPR 2009 IEEE Conference on , pp 248255 122 [153] Goăeau H., Joly A., Yahiaoui I., Baki´c V., Verroust-Blondet A., Bonnet P., Barth´el´emy D., Boujemaa N., and Molino J.F (2014) Plantnet participation at lifeclef2014 plant identification task In CLEF2014 Working Notes , pp 724– 737 [154] Jia Y., Shelhamer E., Donahue J., Karayev S., Long J., Girshick R., Guadarrama S., and Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding In Proceedings of the 22nd ACM international conference on Multimedia, pp 675–678 [155] Zintgraf L.M., Cohen T.S., Adel T., and Welling M (2017) Visualizing deep neural network decisions: Prediction difference analysis CoRR , abs/1702.04595 [156] Lan L.T., Nam H.V., Hai V., and Hai T.T.T (2014) Vietnamese medicinal plant retrieval system for android In The 17th nationalconference: selected problems about IT and Telecommunication, Taynguyen, Vietnam ISBN 978-604-67-04263 [157] Loi D.T (2014) The medicinal plants and herbal formulation in Vietnam Medical Publishing House (in Vietnamese) 123 PUBLICATIONS [1] Thi-Lan Le, Duong-Nam Duong, Van-Toi Nguyen, Hai Vu, Van-Nam Hoang and Thi Thanh-Nhan Nguyen, (2015) Complex Background Leaf-based Plant Identification Method Based on Interactive Segmentation and Kernel Descriptor, Proceedings of the 2nd International Workshop on Environmental Multimedia Retrieval, ISBN: 978-1-4503-3274-3, pp.3-8 [2] Thi-Lan Le, Duong-Nam Duong, Hai Vu and Thanh-Nhan Nguyen (2015) Mica at lifeclef 2015: Multi-organ plant identification, In CEUR-WS.org/Vol-1391CLEF2015 Working note proceedings, ISSN: 1613-0073, vol 1391 [3] Thi Thanh Nhan Nguyen, Van Tuan Le, Thi Lan Le, Hai Vu, Natapon Pantuwong and Yasushi Yagi (2016), Flower species identification using deep convolutional neural networks, AUN/SEED-Net Regional Conference on Computer and Information Engineering 2016, Yangon, Myanmar, ISBN: 978-99971-0-231-7, pp.51-56 [4] Thi Thanh-Nhan Nguyen, Thi-Lan Le, Hai Vu, Huy-hoang Nguyen and VanSam Hoang (2017), A combination of Deep Learning and Hand-Designed Feature for Plant Identification Based on Leaf and Flower, In Asian Conference on Intelligent Information and Database Systems, Studies in Computational Intelligence, volume 710, Springer, ISBN: 978-3-319-56659-7, pp 223-233 [5] Nguyen Thi Thanh Nhan, Do Thanh Binh, Nguyen Huy Hoang, Vu Hai, Tran Thi Thanh Hai, Thi-Lan Le (2018), Score-based Fusion Schemes for Plant Identification from Multi-organ Images, VNU Journal of Science: Computer Science and Communication Engineering, Vol 34, No.2, ISSN 2588-1086, pp.1-15 [6] Thi Thanh Nhan Nguyen, Thi-Lan Le, Hai Vu, Van-Sam Hoang, Thanh-Hai Tran (2018), Crowdsourcing for botanical data collection towards to automatic plant identification: A review, Journal of Computers and Electronics in Agriculture (SCIE), vol 155, ISSN:0168-1699, pp.412-425 [7] Nguyen Thi Thanh Nhan, Le Thi Lan, Vu Hai, Hoang Van Sam (2018), Automatic Plant Organ Detection from Images using Convolutional Neural Networks, Journal of Research and Development on Information and Communication Technology (in Vietnamese), vol V-1, No 39, ISSN: 1859-3526, pp 17-25 [8] Thi Thanh-Nhan Nguyen, Thi-Lan Le, Hai Vu, Van-Sam Hoang (2019), Towards an automatic plant identification system without dedicated dataset International Journal of Machie Learning and Computing (Scopus), vol 9, No.1, ISSN: 2010-3700, pp.26-34 APPENDIX A Table A.1 Details of 50 species used in chapter ID Species name #Leaf 54 Cotinus coggygria Scop 91 55 Pistacia lentiscus L 119 33 51 12 151 Daucus carota L 22 157 23 64 326 Ilex aquifolium L 103 55 49 23 329 Hedera helix L 244 46 58 44 333 Aristolochia clematitis L 40 31 46 26 505 Bidens pilosa L 39 43 35 24 702 Erigeron sumatrensis Retz 56 36 37 42 1212 Silybum marianum (L.) Gaertn 46 54 48 44 10 1321 Alnus glutinosa (L.) Gaertn 60 74 91 14 11 1325 Betula pendula Roth 88 34 64 38 12 1328 Carpinus betulus L 67 47 66 14 13 1329 Corylus avellana L 158 65 61 21 14 1837 Buddleja davidii Franch 39 77 47 24 15 1845 Opuntia ficus-indica (L.) Mill 64 55 23 29 16 1966 Lonicera xylosteum L 48 50 75 14 17 1968 Sambucus nigra L 74 78 37 20 18 1972 Viburnum opulus L 40 39 46 13 19 1973 Viburnum tinus L 179 109 64 16 20 2631 Diospyros kaki L.f 73 49 30 19 21 2648 Arbutus unedo L 58 84 72 24 22 2776 Amorpha fruticosa L 56 57 54 20 23 2841 Ceratonia siliqua L 49 29 50 31 24 2918 Gleditsia triacanthos L 40 30 87 16 25 3124 Robinia pseudoacacia L 89 90 56 20 26 3279 Castanea sativa Mill 52 55 66 17 27 3283 Quercus coccifera L 59 40 71 22 28 3288 Quercus ilex L 162 106 80 49 29 3435 Geranium robertianum L 33 125 #Flower 38 80 #Branch 43 44 #Entire 18 55 30 3465 Aesculus hippocastanum L 81 105 51 26 31 3506 Juglans regia L 53 36 62 27 32 3750 Laurus nobilis L 156 84 78 17 33 3797 Liriodendron tulipifera L 82 40 48 42 34 3798 Magnolia grandiflora L 71 85 60 14 35 3801 Alcea rosea L 43 77 48 23 36 3831 Malva sylvestris L 63 200 80 32 37 3846 Broussonetia papyrifera (L.) Vent 38 3849 Morus alba L 104 19 52 39 3942 Fraxinus angustifolia Vahl 47 45 100 20 40 3956 Olea europaea L 92 32 47 41 41 4026 Chelidonium majus L 58 83 42 63 42 4074 Pittosporum tobira (Thunb.) W.T.Aiton 43 4094 Plantago lanceolata L 39 64 10 82 44 4109 Platanus x hispanica Mill ex Mă unchh 99 42 47 33 45 4111 Platanus orientalis L 57 27 35 21 46 4721 Crataegus monogyna Jacq 135 151 79 21 47 5522 Veronica persica Poir 14 114 56 60 48 14869 Ulex europaeus L 66 93 44 50 49 14887 Verbascum thapsus L 94 110 50 80 50 30131 Euphorbia amygdaloides L 65 47 55 65 Total 3870 3309 2774 1661 126 76 127 39 75 104 52 17 113 41 ...HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Nguyen Thi Thanh Nhan INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION Major: Computer Science Code: 9480101... dissertation entitled, ? ?Interactive and multi-organ based plant species identification”, and the work presented in it is my own I confirm that: This work was done wholly or mainly while in candidature for... iii 43 LEAF -BASED PLANT IDENTIFICATION METHOD BASED ON KERNEL DESCRIPTOR 45 2.1 The framework of leaf -based plant identification method 45 2.2 Interactive segmentation