Develop an android application for enhancing lowlight images

VIETNAM NATIONAL UNIVERSITY - HCMC UNIVERSITY OF TECHNOLOGY FACULTY OF COMPUTER SCIENCE & ENGINEERING UNDERGRADUATE THESIS DEVELOP AN ANDROID APPLICATION FOR ENHANCING LOWLIGHT IMAGES MAJOR: COMPUTER SCIENCE COUNCIL: COMPUTER SCIENCE INSTRUCTOR: DR NGUYEN HO MAN RANG REVIEWER: M.SC TRAN NGOC BAO DUY -o0o STUDENT 1: TRAN THI NGOC DIEP (1827005) STUDENT 2: LUONG THANH NHAN (1820047) HO CHI MINH CITY, 08/2021 VIETNAM NATIONAL UNIVERSITY - HCMC UNIVERSITY OF TECHNOLOGY FACULTY OF COMPUTER SCIENCE & ENGINEERING UNDERGRADUATE THESIS DEVELOP AN ANDROID APPLICATION FOR ENHANCING LOWLIGHT IMAGES MAJOR: COMPUTER SCIENCE COUNCIL: COMPUTER SCIENCE INSTRUCTOR: DR NGUYEN HO MAN RANG REVIEWER: M.SC TRAN NGOC BAO DUY -o0o STUDENT 1: TRAN THI NGOC DIEP (1827005) STUDENT 2: LUONG THANH NHAN (1820047) HO CHI MINH CITY, 08/2021 - - KHOA:KH & KT Máy tính KHMT ình TR MSSV: 1827005 1820047 NGÀNH: - Tìm hi THÀNH NHÂN cơng trình liên quan i sáng - ùh - Hi ki - Hi kho ó th ài ên thi ã ng Android giúp c oogle Play ch 01/03/2021 23/07/2021 1) TS NGUY Ngày thán TS Nguy _ _ _ sáng upload KHOA KH & KT MÁY TÍNH Tr MSSV: 1827005 - 1820047 -Ngày 06 tháng 08 2021 Thành Nhân Ngành (chuyên ngành): Phát tri n ki n th ý cơng trình liên quan ài tốn c - Sinh viên ã hi mơ hình t hi thi HDRNet, EnlightenGAN, ZeroDCE T ó nhóm ã hình c i HDRNet b cách k êm m àm loss cơng trình khác - Th ánh giá t mơ hình c tính mơ hình g ên t p AdobeFiveK SICE - Sinh viên hi thành công gi ên thi di ên Android) minh h - Mơ hình ịn h  ên t OL   m: 9.5 /10 Ký tên (ghi rõ h TRƯỜNG ĐẠI HỌC BÁCH KHOA KHOA KH & KT MÁY TÍNH -Ngày tháng năm CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM Độc lập - Tự - Hạnh phúc PHIẾU CHẤM BẢO VỆ LVTN (Dành cho người hướng dẫn) Họ tên SV: Trần Thị Ngọc Diệp MSSV: 1827005 Ngành (chuyên ngành): Khoa học Máy tính Họ tên SV: Lương Thành Nhân MSSV: 1820047 Ngành (chuyên ngành): Khoa học Máy tính Đề tài: Phát triển ứng dụng thiết bị Android để cải thiện ảnh chụp điều kiện thiếu ánh sáng Họ tên người hướng dẫn/phản biện: TRẦN NGỌC BẢO DUY Tổng quát thuyết minh: Số trang: Số chương: Số bảng số liệu Số hình vẽ: Số tài liệu tham khảo: Phần mềm tính tốn: Hiện vật (sản phẩm) Tổng qt vẽ: - Số vẽ: Bản A1: Bản A2: Khổ khác: - Số vẽ vẽ tay Số vẽ máy tính: Những ưu điểm LVTN: - Chỉ rõ vấn đề cần giải thực tế, đề tài có tính ứng dụng cao - Biết đề xuất mơ hình học sâu để giải toán, khảo sát nhiều phương pháp từ đánh giá ưu khuyết điểm, chọn mơ hình mà nhóm dựa để cải tiến phù hợp - Đã xây dựng ứng dụng Android đơn giản để triển khai thực tế Những thiếu sót LVTN: - Sinh viên trình bày q trình phát triển ứng dụng cịn sơ xài, chưa xây dựng kịch kiểm thử ứng dụng nhiều cấp độ - Sinh viên chưa có so sánh kết mơ hình học sâu so với phương pháp xử lý ảnh truyền thống cách rõ ràng Đề nghị: Được bảo vệ ☑ Bổ sung thêm để bảo vệ □ Không bảo vệ □ câu hỏi SV phải trả lời trước Hội đồng: a Khả mở rộng hệ thống sang tảng nào? Đánh giá yếu tố khách quan chủ quan để nhóm chọn thực tảng Android b Theo nhóm, kịch ứng dụng cần phải kiểm thử để triển khai ứng dụng thực tế? c Trình bày ảnh hưởng việc sử dụng loss function đề xuất kết mơ hình? Loss function đem lại hiệu mơ hình? 10 Đánh giá chung (bằng chữ: giỏi, khá, TB): Giỏi Ký tên (ghi rõ họ tên) Điểm : 8.8 /10 i Declaration We declare that this thesis, not including references to other publications, has been composed by us and that this work has not been submitted for any other degree of professional qualification Acknowledgments We would like to express out special thanks of gratitude to Dr Nguyen Ho Man Rang who gave us this opportunity to work on this exciting project on the topic Develop an Android application for enhancing low-light images Thank you very much for your coaching and encouragement which helps us tremendously to complete this thesis We would like to thank individuals who helped us a lot with their knowledge Dr Le Thanh Sach suggested us how to improve our neural network model which leads to a much better result Mr Tran Cong Thanh - AI Engineer at VinAI who helped us build the SDK and Android application We appreciate that very much We would like to thank all the teachers of Faculty of Computer Science and Engineering - Ho Chi Minh University of Technology for all the wonderful knowledge you have given to us Finally, we would like to thank our family, our friends who supported us with all your best Abstract Camera quality is one of the most concerned features when a customer decides to buy a smartphone nowadays While high-end smartphones own better camera hardware (e.g sensors and lens), which makes them easier to produce a good photo, smartphones in the low-end market need to rely more on the software ability to enhance their photos taken from cheaper camera hardware In this thesis, we research on algorithms which enhance low-light image quality and introduce an improved version of HDR-Net [10], which is an neural network architecture inspired by bilateral grid processing and local affine color transforms We also apply the proposed neural network to build an Android application to help low-end smartphones produce better images captured in low-light condition Contents Introduction 1.1 Motivation 1.2 Purpose 1.3 Thesis structure 2 Concepts and Definitions 2.1 2.2 2.3 Low-light Image 2.1.1 Definition 2.1.2 Low-light Image Characteristics 2.1.3 Low-light Image Enhancement Convolutional Neural Network (CNN) 2.2.1 Layer types of CNN Bilateral Grid Slicing Operation 10 2.3.1 Bilateral Grid 10 2.3.2 Basic Usage of a Bilateral Grid 11 Literature Review 13 i ii CONTENTS 3.1 3.2 3.3 Traditional Methods 13 3.1.1 Gray Transformation Method 14 3.1.2 Histogram Equalization Method 15 3.1.3 Retinex Method 16 3.1.4 Frequency-domain Method 18 3.1.5 Image Fusion Method 20 3.1.6 Defogging Model Method 21 Learning Based Methods 21 3.2.1 Physical-modeling-based Methods 21 3.2.2 Image-to-image Translation Methods 22 3.2.3 Reinforcement Learning Methods 22 Evaluation Methods 23 3.3.1 Subjective Evaluation 23 3.3.2 Full-reference Image Quality Assessment 23 3.3.3 No-reference Image Quality Assessment 24 Datasets 26 4.1 Low-Light Image Datasets 26 4.2 Dataset Reviews 27 4.2.1 Adobe FiveK [1] 27 SICE [2] 28 4.3.1 29 4.3 DICM [12] • In Lowlight-SDK, RGB image under byte array will be transformed to a matrix using OpenCV • This OpenCV matrix then will be transformed into Torch tensor, and then finally fed into TorchScript model to produce enhanced image • Do above steps but in reverse order to get the final result as a RGB bitmap enhanced image Figure 7.4: Correlation of Pytorch and LibTorch Pytorch Pytorch is an open source machine learning library written in Python and based on the Torch library, used for application such as computer vision and natural language processing, primarily developed by Facebook’s AI Research lab LibTorch LibTorch is a distribution of Pytorch It is kind of Pytorch C++ API that allows us to program in C++ instead of Python We use LibToch to communicate with TorchScript model in the Lowlight-SDK 50 TorchScript TorchScript is a way to create serializable and optimizable models from PyTorch code With TorchScript we can incrementally transition a model from pure Python program to a TorchScript program that can be run independently from Python, such as in a standalone C++ program This makes it possible to train models in PyTorch using familiar tools in Python and then export the model via TorchScript to a production environment where Python programs may be disadvantageous for performance and multi-threading reasons We convert Pytorch model into TorchScript model for building Android application purpose 7.1.5 Android NDK The Native Development Kit (NDK) is a set of tools that allows you to use C and C++ with Android, and provides platform libraries you can use to manage native activities and access physical device components, such as sensors and touch input We use NDK to compile C and C++ code into a native library and package it into APK using Gradle Then Java code can call functions in Lowlight-SDK through the JNI 7.2 HDR-Cam Application 7.2.1 User Interface and Functionality Home screen This is the home screen of HDR-Cam application with an initialized image On the top, there is a menu bar that includes four buttons: • The flash button: user click the button to enhance the image selected from gallery • The revert button: user click the button to see the original image 51 • The camera button: user click the button to use camera and take a picture • The gallery button: user click the button to open gallery and choose image needed to enhance Figure 7.5: Home screen At the bottom, there is a save button which allows user to save image to the gallery Home screen with enhanced image This is the home screen with enhanced image User can click on the screen or the revert button in menubar to switch between the original image and the enhanced one 52 Figure 7.6: Home screen with enhanced image Camera screen This is the camera interface User can click the circle on the middle bottom of the screen to take a picture and the button next to it to switch to use front camera After taking a picture, it will be automatically enhanced and show to the screen as the figure 7.6 53 Figure 7.7: Camera screen Selecting image from gallery This is the screen that will show up after users click to gallery button in menu bar in figure 7.5 or figure 7.6 The user will choose an image and then the chosen image will be enhanced and show to the screen as figure 7.6 54 Figure 7.8: Select image from gallery screen 7.2.2 Pros and Cons of HDR-Cam Application Pros • The application is very straight forward and easy to use • The application is resuable and upgradable because it was built by modules Cons • The UX (User Experience) and UI (User Interface) are still simple • There is no real-time preview image functionality • The enhancement time is not fast enough 55 Chapter Conclusions and future developments In this thesis, we have reviewed relevant research and datasets on the topic of low-light image enhancement, as well as investigated on image processing and computer vision definitions and concepts, which closely related to our research topic 8.1 Achievements • Low-light Image Enhancement: We proposed a stable method for low-light image enhancement by predicting the transformations from input to output images, our method was derived from HDR-Net [10] • Evaluating our model on different datasets: Our method was experimented on different test sets, using both full-reference and no-reference metrics Our model shows good performance on no-reference metrics, which usually outperform full-reference metrics in terms of agreement with a subjective human quality score • Building Application: Last but not least, we built an Android Application for low-light image enhancement, which takes an image as input and return its enhanced version 8.2 Limits and Future Developments Although we achieved the topic requirements, there are still limits: 56 • Our model does not outperform other state-of-the-art methods There’s still more to improve about HDR-Net • Bilateral Slicing is the model’s bottleneck, which slows down the inference process even the model’s number of parameters is not quite big • The application haven’t worked in real-time yet We propose several future developments as follows: • Split the model into low, high and mix-exposed and train them separately to achieve the best results in all cases • Implement the bilateral slicing layer in OpenGL Shader to obtain real-time performance on mobile devices 57 Bibliography [1] Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand Learning photographic global tonal adjustment with a database of input / output image pairs In The Twenty-Fourth IEEE Conference on Computer Vision and Pattern Recognition, 2011 [2] Jianrui Cai, Shuhang Gu, and Lei Zhang Learning a deep single image contrast enhancer from multi-exposure images IEEE Transactions on Image Processing, 27(4):2049–2062, 2018 [3] A T Celebi, R Duvar, and O Urhan Fuzzy fusion based high dynamic range imaging using adaptive histogram separation IEEE Transactions on Consumer Electronics, 61(1):119–127, 2015 [4] C Chen, Q Chen, J Xu, and V Koltun Learning to see in the dark In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3291–3300, 2018 [5] Jiawen Chen, Sylvain Paris, and Frédo Durand Real-time edge-aware image processing with the bilateral grid ACM Trans Graph., 26:103, 07 2007 [6] Oleg Chornenko Camera2 preview https://github.com/ochornenko/ Camera2GLPreview, 2018 [7] Frederic Drago, Karol Myszkowski, Thomas Annen, Norishige Chiba, Pere Brunet, and Dieter Fellner Adaptive logarithmic mapping for displaying high contrast scenes EUROGRAPHICS 2003 (EUROGRAPHICS-03) : the European Association for Computer Graphics, 24th Annual Conference, Blackwell, 419-426 (2003), 22, 09 2003 [8] N Faramarzpour, M J Deen, S Shirani, Q Fang, L W C Liu, F de Souza Campos, and J W Swart Cmos-based active pixel for low-light-level detection: Analysis and measurements IEEE Transactions on Electron Devices, 54(12):3229–3237, 2007 [9] Xueyang Fu, Delu Zeng, Yue Huang, Xiao-Ping Zhang, and Xinghao Ding A weighted 58 variational model for simultaneous reflectance and illumination estimation In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 27822790, 2016 [10] Michaăel Gharbi, Jiawen Chen, Jonathan T Barron, Samuel W Hasinoff, and Frédo Durand Deep bilateral learning for real-time image enhancement ACM Transactions on Graphics (TOG), 36(4):118, 2017 [11] Tony B Gines and Michael W Davidson stacks Emission fingerprinting with lambda http://zeiss-campus.magnet.fsu.edu/tutorials/spectralimaging/ lambdastack/indexflash.html [12] Chunle Guo Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong Zero-reference deep curve estimation for low-light image enhancement In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 1780–1789, June 2020 [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun Deep residual learning for image recognition, 2015 [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015 [15] Yuanming Hu, Hao He, Chenxi Xu, Baoyuan Wang, and Stephen Lin Exposure: A whitebox photo post-processing framework CoRR, abs/1709.09602, 2017 [16] S Huang, F Cheng, and Y Chiu Efficient contrast enhancement using adaptive gamma correction with weighting distribution IEEE Transactions on Image Processing, 22(3):1032–1041, 2013 [17] Sergey Ioffe and Christian Szegedy Batch normalization: Accelerating deep network training by reducing internal covariate shift CoRR, abs/1502.03167, 2015 [18] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang Enlightengan: Deep light enhancement without paired supervision arXiv preprint arXiv:1906.06972, 2019 [19] Jing Li, S Z Li, Quan Pan, and Tao Yang Illumination and motion-based video enhancement for night surveillance In 2005 IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, pages 169–175, 2005 [20] Yi Han Jinming Zou and Sung-Sau So Overview of artificial neural networks Artificial Neural Networks, 458, 2009 59 [21] D J Jobson, Z Rahman, and G A Woodell A multiscale retinex for bridging the gap between color images and the human observation of scenes IEEE Transactions on Image Processing, 6(7):965–976, 1997 [22] D J Jobson, Z Rahman, and G A Woodell Properties and performance of a center/surround retinex IEEE Transactions on Image Processing, 6(3):451–462, 1997 [23] Justin Johnson Deep learning for computer vision University of Michigan, 2020 [24] Robin Kent Energy Managementin Plastics Processing 2018 [25] M Kim and M G Chung Recursively separated and weighted histogram equalization for brightness preservation and contrast enhancement IEEE Transactions on Consumer Electronics, 54(3):1389–1397, 2008 [26] M Kim, D Park, D K Han, and H Ko A novel framework for extremely low-light video enhancement In 2014 IEEE International Conference on Consumer Electronics (ICCE), pages 91–92, 2014 [27] Diederik P Kingma and Jimmy Ba Adam: A method for stochastic optimization, 2015 [28] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton Imagenet classification with deep convolutional neural networks In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12, page 1097–1105, Red Hook, NY, USA, 2012 Curran Associates Inc [29] Edwin Land and John McCann Lightness and retinex theory Journal of the Optical Society of America, 61:1–11, 02 1971 [30] H.-S Le and H Li Fused logarithmic transform for contrast enhancement Electronics Letters, 44:19 – 20, 02 2008 [31] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi Photo-realistic single image superresolution using a generative adversarial network CoRR, abs/1609.04802, 2016 [32] H Lee, S Yang, and J Sim Color preserving contrast enhancement for low light level images based on retinex In 2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA), pages 884–887, 2015 [33] Runde Li, Jinshan Pan, Zechao Li, and Jinhui Tang Single image dehazing via conditional generative adversarial network In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8202–8211, 2018 60 [34] Yihao Liu, Jingwen He, Xiangyu Chen, Zhengwen Zhang, Hengyuan Zhao, Chao Dong, and Yu Qiao Very lightweight photo retouching network with conditional sequential modulation CoRR, abs/2104.06279, 2021 [35] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik No-reference image quality assessment in the spatial domain IEEE Transactions on Image Processing, 21(12):4695–4708, 2012 [36] Anish Mittal, R Soundararajan, and A Bovik Making a “completely blind” image quality analyzer IEEE Signal Processing Letters, 20:209–212, 2013 [37] Venkatanath N, Praneeth D, Maruthi Chandrasekhar Bh, Sumohana S Channappayya, and Swarup S Medasani Blind image quality evaluation using perception based features In 2015 Twenty First National Conference on Communications (NCC), pages 1–6, 2015 [38] J A Ojo, I D Solomon, and S A Adeniran Contrast enhancement algorithm for colour images In 2015 Science and Information Conference (SAI), pages 555–559, 2015 [39] Jongchan Park, Joon-Young Lee, Donggeun Yoo, and In So Kweon Distort-and-recover: Color enhancement using deep reinforcement learning CoRR, abs/1804.04450, 2018 [40] Georg Petschnigg, Richard Szeliski, Maneesh Agrawala, Michael Cohen, Hugues Hoppe, and Kentaro Toyama Digital photography with flash and no-flash image pairs ACM Transactions on Graphics (TOG), 23:664–672, 08 2004 [41] Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu Attentive generative adversarial network for raindrop removal from a single image CoRR, abs/1711.10098, 2017 [42] Zia-ur Rahman, Daniel Jobson, and Glenn Woodell Retinex processing for automatic image enhancement J Electronic Imaging, 13:100–110, 01 2004 [43] Felix Richter Smartphones wipe out 40 years of camera industry growth Dec 2020 [44] Cristiano Rafael Steffens, Lucas Ricardo Vieira Messias, Paulo Drews-Jr, and Silvia S d C Botelho Contrast enhancement and image completion: A cnn based model to restore ill exposed images In 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), volume 1, pages 226–232, 2019 [45] L Wang, G Fu, Z Jiang, G Ju, and A Men Low-light image enhancement with attention and multi-level feature fusion In 2019 IEEE International Conference on Multimedia Expo Workshops (ICMEW), pages 276–281, 2019 61 [46] Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia Underexposed photo enhancement using deep illumination estimation In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6842– 6850, 2019 [47] W Wang, X Wu, X Yuan, and Z Gao An experiment-based review of low-light image enhancement methods IEEE Access, 8:87884–87917, 2020 [48] W Wang, X Wu, X Yuan, and Z Gao An experiment-based review of low-light image enhancement methods IEEE Access, 8:87884–87917, 2020 [49] Wencheng Wang, Zhenxue Chen, Xiaohui Yuan, and Xiaojin Wu Adaptive image enhancement method for correcting low-illumination images Information Sciences, 496, 05 2019 [50] Z Wang, E P Simoncelli, and A C Bovik Multiscale structural similarity for image quality assessment In The Thrity-Seventh Asilomar Conference on Signals, Systems Computers, 2003, volume 2, pages 1398–1402 Vol.2, 2003 [51] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu Deep retinex decomposition for low-light enhancement CoRR, abs/1808.04560, 2018 [52] Qing Xu, Hailin Jiang, Riccardo Scopigno, and Mateu Sbert A novel approach for enhancing very dark image sequences Signal Processing, 103:309–330, 10 2014 [53] Masaya Yamakawa and Yasunori Sugita Image enhancement using retinex and image fusion techniques Electronics and Communications in Japan, 101, 06 2018 [54] Yang, Wenhan and Yuan, Ye and Ren, Wenqi and Liu, Jiaying and Scheirer, Walter J and Wang, Zhangyang and Zhang, and et al Advancing image understanding in poor visibility environments: A collective benchmark study IEEE Transactions on Image Processing, 29:5737–5752, 2020 [55] Yu Wang, Qian Chen, and Baeomin Zhang Image enhancement based on equal area dualistic sub-image histogram equalization method IEEE Transactions on Consumer Electronics, 45(1):68–75, 1999 [56] Hui Zeng, Jianrui Cai, Lida Li, Zisheng Cao, and Lei Zhang Learning image-adaptive 3d lookup tables for high performance photo enhancement in real-time IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020 [57] Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo Kindling the darkness: A practical 62 low-light image enhancer In Proceedings of the 27th ACM International Conference on Multimedia, MM ’19, pages 1632–1640, New York, NY, USA, 2019 ACM 63 Appendix A Source code repository This is our source code related to this thesis For privacy, these repositories’ visibility is private If you are not teachers or reviewers, please contact one of us via email: nhan.luongthanh@hcmut.edu.vn or diep.trantn147@hcmut.edu.vn for permission • HDR-Net: https://github.com/nhanluongoe/hdrnet-pytorch-1 • Model conversion: https://github.com/v-diepttn147/model-conversion • HDR-Cam application: https://github.com/nhanluongoe/hdrcam 64 ... UNDERGRADUATE THESIS DEVELOP AN ANDROID APPLICATION FOR ENHANCING LOWLIGHT IMAGES MAJOR: COMPUTER SCIENCE COUNCIL: COMPUTER SCIENCE INSTRUCTOR: DR NGUYEN HO MAN RANG REVIEWER: M.SC TRAN NGOC BAO DUY... topic Develop an Android application for enhancing low-light images Thank you very much for your coaching and encouragement which helps us tremendously to complete this thesis We would like to thank... on an algorithm to enhance low-light image using deep learning and build a low-light image enhancement application for Android phones Our approach will be compared to other related works for