Interactive and multi organ based plant species identification

HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY NGUYEN THI THANH NHAN INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION Major: Computer Science Code: 9480101 INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION SUPERVISORS: Assoc Prof Dr Le Thi Lan Prof Dr Hoang Van Sam Hanoi − 2020 HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Nguyen Thi Thanh Nhan INTERACTIVE AND MULTI-ORGAN BASED PLANT SPECIES IDENTIFICATION Major: Computer Science Code: 9480101 DOCTORAL DISSERTATION OF COMPUTER SCIENCE SUPERVISORS: Assoc Prof Dr Le Thi Lan Prof Dr Hoang Van Sam Hanoi − 2020 DECLARATION OF AUTHORSHIP I, Nguyen Thi Thanh Nhan, declare that this dissertation entitled, ”Interactive and multi-organ based plant species identification”, and the work presented in it is my own I confirm that:  This work was done wholly or mainly while in candidature for a Ph.D research degree at Hanoi University of Science and Technology  Where any part of this dissertation has previously been submitted for a degree or any other qualification at Hanoi University of Science and Technology or any other institution, this has been clearly stated  Where I have consulted the published work of others, this is always clearly attributed  Where I have quoted from the work of others, the source is always given With the exception of such quotations, this dissertation is entirely my own work  I have acknowledged all main sources of help  Where the dissertation is based on work done by myself jointly with others, I have made exactly what was done by others and what I have contributed myself Hanoi, May, 2020 PhD Student Nguyen Thi Thanh Nhan SUPERVISORS i ACKNOWLEDGEMENT First of all, I would like to thank my supervisors Assoc Prof Dr Le Thi Lan at The International Research Institute MICA - Hanoi University of Science and Technology, Assoc Prof Dr Hoang Van Sam at Vietnam National University of Forestry for their inspiration, guidance, and advice Their guidance helped me all the time of research and writing this dissertation Besides my advisors, I would like to thank Assoc Prof Dr Vu Hai, Assoc Prof Dr Tran Thi Thanh Hai for their great discussion Special thanks to my friends/colleagues in MICA, Hanoi University of Science and Technology: Hoang Van Nam, Nguyen Hong Quan, Nguyen Van Toi, Duong Nam Duong, Le Van Tuan, Nguyen Huy Hoang, Do Thanh Binh for their technical supports They have assisted me a lot in my research process as well as they are co-authored in the published papers Moreover, I would like to thank reviewers of scientific conferences, journals and protection council, reviewers, they help me with many useful comments I would like to express a since gratitude to the Management Board of MICA Institute I would like to thank the Thai Nguyen University of Information and Communication Technology, Thai Nguyen over the years both at my career work and outside of the work As a Ph.D student of the 911 program, I would like to thank this program for financial support I also gratefully acknowledge the financial support for attending the conferences from the Collaborative Research Program for Common Regional Issue (CRC) funded by ASEAN University Network (Aun-Seed/Net), under the grant reference HUST/CRC/1501 and NAFOSTED (grant number 106.06-2018.23) Special thanks to my family, to my parents-in-law who took care of my family and created favorable conditions for me to study I also would like to thank my beloved husband and children for everything they supported and encouraged me for a long time to study Hanoi, May, 2020 Ph.D Student Nguyen Thi Thanh Nhan ii CONTENTS DECLARATION OF AUTHORSHIP i ACKNOWLEDGEMENT ii CONTENTS v SYMBOLS vi SYMBOLS viii LIST OF TABLES xi LIST OF FIGURES xvi INTRODUCTION 1 LITERATURE REVIEW 1.1 1.2 Plant identification 1.1.1 Manual plant identification 10 10 1.1.2 Plant identification based on semi-automatic graphic tool 11 1.1.3 Automated plant identification 12 Automatic plant identification from images of single organ 13 1.2.1 Introducing the plant organs 13 1.2.2 General model of image-based plant identification 16 1.2.3 Preprocessing techniques for images of plant 17 1.2.4 Feature extraction 1.2.4.1 Hand-designed features 19 20 1.2.4.2 Deeply-learned features 22 Training methods 25 Plant identification from images of multiple organs 28 1.2.5 1.3 10 1.3.1 Early fusion techniques for plant identification from images of multiple organs 30 Late fusion techniques for plant identification from images of multiple organs 31 1.4 Plant identification studies in Vietnam 33 1.5 Plant data collection and identification systems 35 1.6 Conclusions 43 1.3.2 iii 119 [119] Gu X., Du J.X., and Wang X.F (2005) Leaf recognition based on the combination of wavelet transform and gaussian interpolation In International Conference on Intelligent Computing , pp 253–262 [120] Cerutti G., Tougne L., Mille J., Vacavant A., and Coquin D (2011) Guiding active contours for tree leaf segmentation and identification In CLEF 2011, Conference on Multilingual and Multimodal Information Access Evaluation [121] Bo L and Sminchisescu C (2009) Efficient match kernel between sets of features for visual recognition In Advances in neural information processing systems , pp 135–143 [122] NGUYEN V.T (2015) Visual interpretation of hand postures for humanmachine interaction Ph.D thesis, Universit´e de La Rochelle [123] Achanta R., Hemami S., Estrada F., and Susstrunk S (2009) Frequency-tuned salient region detection In Computer vision and pattern recognition, 2009 cvpr 2009 ieee conference on, pp 1597–1604 [124] Meyer F and Beucher S (1990) Morphological segmentation Journal of visual communication and image representation, 1(1):pp 21–46 [125] Beucher S and Meyer F (1993) The morphological approach to segmentation: the watershed transformation Mathematical morphology in image processing , 34:pp 433–481 [126] Kornilov A and Safonov I (2018) An overview of watershed algorithm implementations in open source libraries Journal of Imaging , 4(10):pp 1–15 [127] Le T.L., Tran D.T., and Hoang V.N (2014) Fully automatic leaf-based plant identification, application for vietnamese medicinal plant search In Proceedings of the fifth symposium on information and communication technology , pp 146– 154 [128] Dey S., Bhattacharya B.B., Kundu M.K., and Acharya T.A (2002) A simple architecture for computing moments and orientation of an image Fundamenta Informaticae, 52(4):pp 285–295 [129] Priyabrata K (2018) Effective and efficient kernel-based image representations for classification and retrieval Ph.D thesis, Federation University Australia [130] Lowe D.G (1999) Object recognition from local scale-invariant features In Computer vision, 1999 The proceedings of the seventh IEEE international conference on, volume 2, pp 1150–1157 120 [131] Maji S., Berg A.C., and Malik J (2013) Efficient classification for additive kernel svms IEEE transactions on pattern analysis and machine intelligence , 35(1):pp 66–77 [132] Vedaldi A and Zisserman A (2012) Efficient additive kernels via explicit feature maps IEEE transactions on pattern analysis and machine intelligence , 34(3):pp 480–492 [133] Chaki J., Parekh R., and Bhattacharya S (2015) Recognition of whole and deformed plant leaves using statistical shape features and neuro-fuzzy classifier In 2015 IEEE 2nd international conference on recent trends in information systems (ReTIS), pp 189–194 [134] Chaki J., Parekh R., and Bhattacharya S (2015) Plant leaf recognition using texture and shape features with neural classifiers Pattern Recognition Letters , 58:pp 61–68 [135] Wang Z., Sun X., Ma Y., Zhang H., Ma Y., Xie W., and Zhang Y (2014) Plant recognition based on intersecting cortical model In 2014 International joint conference on neural networks (IJCNN), pp 975–980 [136] Kheirkhah F.M and Asghari H (2018) Plant leaf classification using gist texture features IET Computer Vision , 13(4):pp 369–375 [137] Tsolakidis D.G., Kosmopoulos D.I., and Papadourakis G (2014) Plant leaf recognition using zernike moments and histogram of oriented gradients In Hellenic Conference on Artificial Intelligence , pp 406–417 [138] Du J.x., Zhai C.M., and Wang Q.P (2013) Recognition of plant leaf image based on fractal dimension features Neurocomputing , 116:pp 150–156 [139] Priya C.A., Balasaravanan T., and Thanamani A.S (2012) An efficient leaf recognition algorithm for plant classification using support vector machine In International conference on pattern recognition, informatics and medical engineering (PRIME-2012), pp 428–432 [140] Jain A., Nandakumar K., and Ross A (2005) Score normalization in multimodal biometric systems Pattern recognition , 38(12):pp 2270–2285 [141] Wang F and Han J (2009) Multimodal biometric authentication based on score level fusion using support vector machine Opto-electronics review , 17(1):pp 59–64 121 [142] Abdiansah A and Wardoyo R (2015) Time complexity analysis of support vector machines (svm) in libsvm International journal computer and application , 128(3):pp 28–34 [143] Singh A and Kisku D.R (2019) Detection of rare genetic diseases using facial 2d images with transfer learning In 2018 8th International Symposium on Embedded Computing and System Design (ISED), pp 26–30 [144] Van Horn G., Mac Aodha O., Song Y., Cui Y., Sun C., Shepard A., Adam H., Perona P., and Belongie S (2018) The inaturalist species classification and detection dataset In Proceedings of the IEEE conference on computer vision and pattern recognition, pp 8769–8778 [145] Gaston K.J and O’Neill M.A (2004) Automated species identification: why not? Philosophical transactions of the Royal society B: Biological sciences , 359(1444):pp 655–667 [146] Săoderkvist O (2001) Computer vision classication of leaves from swedish trees Masters thesis, Linkăoping University [147] Mouine S., Yahiaoui I., and Verroust-Blondet A (2012) Advanced shape context for plant species identification using leaf image retrieval In Proceedings of the 2nd (Association for Computing Machinery)ACM international conference on multimedia retrieval , p 49 [148] Novotn`y P and Suk T (2013) Leaf recognition of woody species in central europe Biosystems engineering , 115(4):pp 444–452 [149] Mzoughi O., Yahiaoui I., and Boujemaa N (2012) Petiole shape detection for advanced leaf identification In 2012 19th IEEE International Conference on Image Processing , pp 1033–1036 [150] Lee S.H., Chan C.S., Wilkin P., and Remagnino P (2015) Deep-plant: Plant identification with convolutional neural networks In Image Processing (ICIP), 2015 IEEE International Conference on, pp 452–456 [151] Seeland M., Rzanny M., Alaqraa N., Wăaldchen J., and Măader P (2017) Plant species classication using flower images—a comparative study of local feature representations PloS one , 12(2):p e0170629 [152] Deng J., Dong W., Socher R., Li L.J., Li K., and Fei-Fei L (2009) Imagenet: A large-scale hierarchical image database In Computer Vision and Pattern Recognition, 2009 CVPR 2009 IEEE Conference on , pp 248255 122 [153] Goăeau H., Joly A., Yahiaoui I., Baki´c V., Verroust-Blondet A., Bonnet P., Barth´el´emy D., Boujemaa N., and Molino J.F (2014) Plantnet participation at lifeclef2014 plant identification task In CLEF2014 Working Notes , pp 724– 737 [154] Jia Y., Shelhamer E., Donahue J., Karayev S., Long J., Girshick R., Guadarrama S., and Darrell T (2014) Caffe: Convolutional architecture for fast feature embedding In Proceedings of the 22nd ACM international conference on Multimedia, pp 675–678 [155] Zintgraf L.M., Cohen T.S., Adel T., and Welling M (2017) Visualizing deep neural network decisions: Prediction difference analysis CoRR , abs/1702.04595 [156] Lan L.T., Nam H.V., Hai V., and Hai T.T.T (2014) Vietnamese medicinal plant retrieval system for android In The 17th nationalconference: selected problems about IT and Telecommunication, Taynguyen, Vietnam ISBN 978-604-67-04263 [157] Loi D.T (2014) The medicinal plants and herbal formulation in Vietnam Medical Publishing House (in Vietnamese) 123 PUBLICATIONS [1] Thi-Lan Le, Duong-Nam Duong, Van-Toi Nguyen, Hai Vu, Van-Nam Hoang and Thi Thanh-Nhan Nguyen, (2015) Complex Background Leaf-based Plant Identification Method Based on Interactive Segmentation and Kernel Descriptor, Proceedings of the 2nd International Workshop on Environmental Multimedia Retrieval, ISBN: 978-1-4503-3274-3, pp.3-8 [2] Thi-Lan Le, Duong-Nam Duong, Hai Vu and Thanh-Nhan Nguyen (2015) Mica at lifeclef 2015: Multi-organ plant identification, In CEUR-WS.org/Vol-1391CLEF2015 Working note proceedings, ISSN: 1613-0073, vol 1391 [3] Thi Thanh Nhan Nguyen, Van Tuan Le, Thi Lan Le, Hai Vu, Natapon Pantuwong and Yasushi Yagi (2016), Flower species identification using deep convolutional neural networks, AUN/SEED-Net Regional Conference on Computer and Information Engineering 2016, Yangon, Myanmar, ISBN: 978-99971-0-231-7, pp.51-56 [4] Thi Thanh-Nhan Nguyen, Thi-Lan Le, Hai Vu, Huy-hoang Nguyen and VanSam Hoang (2017), A combination of Deep Learning and Hand-Designed Feature for Plant Identification Based on Leaf and Flower, In Asian Conference on Intelligent Information and Database Systems, Studies in Computational Intelligence, volume 710, Springer, ISBN: 978-3-319-56659-7, pp 223-233 [5] Nguyen Thi Thanh Nhan, Do Thanh Binh, Nguyen Huy Hoang, Vu Hai, Tran Thi Thanh Hai, Thi-Lan Le (2018), Score-based Fusion Schemes for Plant Identification from Multi-organ Images, VNU Journal of Science: Computer Science and Communication Engineering, Vol 34, No.2, ISSN 2588-1086, pp.1-15 [6] Thi Thanh Nhan Nguyen, Thi-Lan Le, Hai Vu, Van-Sam Hoang, Thanh-Hai Tran (2018), Crowdsourcing for botanical data collection towards to automatic plant identification: A review, Journal of Computers and Electronics in Agriculture (SCIE), vol 155, ISSN:0168-1699, pp.412-425 [7] Nguyen Thi Thanh Nhan, Le Thi Lan, Vu Hai, Hoang Van Sam (2018), Automatic Plant Organ Detection from Images using Convolutional Neural Networks, Journal of Research and Development on Information and Communication Technology (in Vietnamese), vol V-1, No 39, ISSN: 1859-3526, pp 17-25 [8] Thi Thanh-Nhan Nguyen, Thi-Lan Le, Hai Vu, Van-Sam Hoang (2019), Towards an automatic plant identification system without dedicated dataset International Journal of Machie Learning and Computing (Scopus), vol 9, No.1, ISSN: 2010-3700, pp.26-34 APPENDIX A Table A.1 Details of 50 species used in chapter ID Species name #Leaf #Flower #Branch #Entire 54 Cotinus coggygria Scop 91 38 43 18 55 Pistacia lentiscus L 119 33 51 12 151 Daucus carota L 22 157 23 64 326 Ilex aquifolium L 103 55 49 23 329 Hedera helix L 244 46 58 44 333 Aristolochia clematitis L 40 31 46 26 505 Bidens pilosa L 39 43 35 24 702 Erigeron sumatrensis Retz 56 36 37 42 1212 Silybum marianum (L.) Gaertn 46 54 48 44 10 1321 Alnus glutinosa (L.) Gaertn 60 74 91 14 11 1325 Betula pendula Roth 88 34 64 38 12 1328 Carpinus betulus L 67 47 66 14 13 1329 Corylus avellana L 158 65 61 21 14 1837 Buddleja davidii Franch 39 77 47 24 15 1845 Opuntia ficus-indica (L.) Mill 64 55 23 29 16 1966 Lonicera xylosteum L 48 50 75 14 17 1968 Sambucus nigra L 74 78 37 20 18 1972 Viburnum opulus L 40 39 46 13 19 1973 Viburnum tinus L 179 109 64 16 20 2631 Diospyros kaki L.f 73 49 30 19 21 2648 Arbutus unedo L 58 84 72 24 22 2776 Amorpha fruticosa L 56 57 54 20 23 2841 Ceratonia siliqua L 49 29 50 31 24 2918 Gleditsia triacanthos L 40 30 87 16 25 3124 Robinia pseudoacacia L 89 90 56 20 26 3279 Castanea sativa Mill 52 55 66 17 27 3283 Quercus coccifera L 59 40 71 22 28 3288 Quercus ilex L 162 106 80 49 29 3435 Geranium robertianum L 33 80 44 55 125 30 3465 Aesculus hippocastanum L 81 105 51 26 31 3506 Juglans regia L 53 36 62 27 32 3750 Laurus nobilis L 156 84 78 17 33 3797 Liriodendron tulipifera L 82 40 48 42 34 3798 Magnolia grandiflora L 71 85 60 14 35 3801 Alcea rosea L 43 77 48 23 36 3831 Malva sylvestris L 63 200 80 32 37 3846 Broussonetia papyrifera (L.) Vent 76 39 104 17 38 3849 Morus alba L 104 19 52 113 39 3942 Fraxinus angustifolia Vahl 47 45 100 20 40 3956 Olea europaea L 92 32 47 41 41 4026 Chelidonium majus L 58 83 42 63 42 4074 Pittosporum tobira (Thunb.) W.T.Aiton 127 75 52 41 43 4094 Plantago lanceolata L 39 64 10 82 44 4109 Platanus x hispanica Mill ex Mă unchh 99 42 47 33 45 4111 Platanus orientalis L 57 27 35 21 46 4721 Crataegus monogyna Jacq 135 151 79 21 47 5522 Veronica persica Poir 14 114 56 60 48 14869 Ulex europaeus L 66 93 44 50 49 14887 Verbascum thapsus L 94 110 50 80 50 30131 Euphorbia amygdaloides L 65 47 55 65 Total 3870 3309 2774 1661 126 ...Hanoi − 2020 HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Nguyen Thi Thanh Nhan INTERACTIVE AND MULTI- ORGAN BASED PLANT SPECIES IDENTIFICATION Major: Computer Science Code: 9480101... dissertation entitled, ? ?Interactive and multi- organ based plant species identification”, and the work presented in it is my own I confirm that:  This work was done wholly or mainly while in candidature for... Therefore, multi- organ plant identification is also studied in this thesis In this thesis, multiorgan plant identification will be formulated as a late fusion problem: the multi- organ plant results