Develop a low cost thermal camera using 2d camera and thermal sensor

Vietnam National University Ho Chi Minh City HCM University of Technology Faculty of Computer Science and Engineering Graduation Thesis Develop A Low-cost Thermal Camera Using 2D Camera and Thermal Sensor Major: Computer Engineering Committee: Council Supervisor: Pham Hoang Anh, Ph.D Reviewer: Le Trong Nhan, Ph.D -o0o Student 1: Tran Dinh Tien 1752541 Student 2: Nguyen Viet Khoa 1752295 Student 3: Nhan Ngoc Thien - 1752508 Ho Chi Minh City, Oct 2021 ĐẠI HỌC QUỐC GIA TP.HCM -TRƯỜNG ĐẠI HỌC BÁCH KHOA KHOA:KH & KT Mỏy tớnh _ THUT MỗY TờNH B MễN: K _ CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM Độc lập - Tự - Hạnh phúc NHIỆM VỤ LUẬN ÁN TỐT NGHIỆP Chú ý: Sinh viên phải dán tờ vào trang thuyết trình HỌ VÀ TÊN: MSSV: 1752541 Trần Đ“nh Tiến Nguyễn Việt Khoa HỌ VÀ TÊN: MSSV: 1752295 Nhan Ngọc Thiện HỌ VÀ TÊN: MSSV: 1752508 Computer Engineering NGÀNH: _ LỚP: Đầu đề luận án: Develop A Low-cost Thermal Camera Using 2D Camera and Thermal Sensor Nhiệm vụ (yêu cầu nội dung số liệu ban đầu): - Study principal operation of existing thermal cameras - Investigate the related works combining a 2D-camera and a thermal sensor to detect human face in an image and estimate body temperature - Investigate existing hardware devices to propose a solution and detail design - Develop a prototype and software applications for evaluation the proposed approach Ngày giao nhiệm vụ luận án: Ngày hoàn thành nhiệm vụ: Họ tên giảng viên hướng dẫn: Phần hướng dẫn: 100% TS Phạm Hoˆng Anh 1) 2) 3) Nội dung yêu cầu LVTN thông qua Bộ môn Ngày tháng năm CHỦ NHIỆM BỘ MƠN GIẢNG VIÊN HƯỚNG DẪN CHÍNH (Ký ghi rõ họ tên) (Ký ghi rõ họ tên) PGS.TS Phạm Quốc Cường PHẦN DÀNH CHO KHOA, BỘ MÔN: Người duyệt (chấm sơ bộ): Đơn vị: _ Ngày bảo vệ: _ Điểm tổng kết: Nơi lưu trữ luận án: TS Phạm Hoˆng Anh TRƯỜNG ĐẠI HỌC BÁCH KHOA KHOA KH & KT MÁY TÍNH CỘNG HỊA XÃ HỘI CHỦ NGHĨA VIỆT NAM Độc lập - Tự - Hạnh phúc -Ngày 06 tháng 08 năm 2021 PHIẾU CHẤM BẢO VỆ LVTN (Dành cho người hướng dẫn/phản biện) Họ tên SV: Trần Đình Tiến, Nguyễn Việt Khoa, Nhan Ngọc Thiện MSSV: 1752541, 1752295, 1752508 Ngành (chuyên ngành): Kỹ thuật Máy tính Đề tài: Develop A Low-cost Thermal Camera Using 2D Camera and Thermal Sensor Họ tên người hướng dẫn/phản biện: Phạm Hoàng Anh Tổng quát thuyết minh: Số trang: Số chương: Số bảng số liệu Số hình vẽ: Số tài liệu tham khảo: Phần mềm tính tốn: Hiện vật (sản phẩm) Tổng quát vẽ: - Số vẽ: Bản A1: Bản A2: Khổ khác: - Số vẽ vẽ tay Số vẽ máy tính: Những ưu điểm LVTN: - The students have demonstrated their excellent capability in self-studying and investigating related works, and then they apply new knowledge and suitable techniques to implement the proposed system - The students have performed various experiments to evaluate the system, and the experimental results have shown that the proposed solution meets all requirements in this thesis’s scope - The students have also developed a full IoT-based system that is ready and applicable to deploy the system in practice, including hardware, embedded software, IoT server, and Webbased application - The proposed solution has been accepted and presented at an international conference where its proceeding will be indexed in Scopus - The report has been well organized and written Những thiếu sót LVTN: - The students should improve their presentation Đề nghị: Được bảo vệ o Bổ sung thêm để bảo vệ o Câu hỏi SV phải trả lời trước Hội đồng: 10 Đánh giá chung (bằng chữ: giỏi, khá, TB): Không bảo vệ o Điểm : 10/10 Ký tên (ghi rõ họ tên) Phạm Hoàng Anh Declaration We commit that our topic “Develop A Low-cost Thermal Camera Using 2D Camera and Thermal Sensor” is our thesis We declare that this topic is conducted under our effort, time, and the recommendation from Dr Pham Hoang Anh as our advisor All of the research results are conducted by ourselves and not copied from any other sources If there is any evidence of plagiarism, we will be responsible for all consequences Ho Chi Minh City, 2021, Tran Dinh Tien, Nguyen Viet Khoa, Nhan Ngoc Thien Acknowledgement First and foremost, we would like to show our deep and grateful gratitude to Dr Pham Hoang Anh as an adviser for our thesis and as a beloved teacher We want to show all of our respect for the precious knowledge, advice, and all the necessary equipment that he has given to us We sincerely appreciate his kindness, patience, and continuous support throughout this thesis Next, we would like to show our appreciation to Dr Le Trong Nhan for giving us advice on which devices and components to choose that are most suitable for the scope of our thesis We also want to show appreciation to Assoc Prof Quan Thanh Tho for all of his advice and comments about our thesis proposal Moreover, we wish to express our appreciation to all our faculty lecturers for teaching the authors all the fundamental knowledge and necessary skills and knowledge in each field, without whom we would not have enough knowledge and skills to complete this thesis Last but not least, we would like to acknowledge the support from our family and classmates in many aspects without hesitation—all of your assistance in helping us Despite our commitment, we are aware that the project is still inadequate and contains inevitable errors Therefore, we are looking forward to hearing feedback from lecturers to remove all the mistakes and improve the project furthermore i With sincere gratitude, the authors would like to wish health, gratitude, and best wishes to the lecturers in the Department of Computer Science and Engineering - Ho Chi Minh City University of Technology - Viet Nam National University Ho Chi Minh City Students ii Abstract The Covid-19 pandemic is spreading worldwide, the demand for controlling and screening fever is increasing very rapidly Some old methods like using thermometers or medical checking cannot handle many people in crowded areas such as airports, train stations, or buildings Some thermographic systems or thermal imaging systems control the people’s ins and outs These kinds of systems are helping a lot for the government of any nation to decrease and prevent the spread of viruses by screening human temperature based on their radiation of skin Despite all of the advantages this technology has had, the exceptionally high cost for this system is one of the reasons to prevent some organizations from applying it Despite all of the advantages this technology has had, this system’s very high cost is one reason to prevent some organizations from applying it despite the high potential of image processing, artificial intelligence, and the Internet of Things This paper will present a system combining a low-cost thermal camera and a standard RGB camera for screening fever combined with attendance checking The experimental results show that this proposed system can be applied in many areas, from universities to buildings, with the minimum cost and acceptable performance iii Contents Declaration Acknowledgement i Abstract iii List of Figures xi List of Table xii Terms xiii Introduction 1.1 Purpose and Motivation 1.2 Scope and Objectives 1.3 Thesis report structure Related Works 2.1 Investigation of Similar Studies 2.1.1 2.1.2 4 Skin temperature extraction using facial landmarks detection and thermal imaging for comfort assessment Mobile-platform for Automatic Fever Screening System based on Infrared Forehead Temperature iv Contents 2.1.3 2.2 Contactless Vital Signs Measurement System Using RGBThermal Image Sensors and Its Clinical Screening Test on Patients with Seasonal Influenza Background and Methodologies 2.2.1 Thermal Camera Principles 2.2.2 Face Detection Methods 2.2.3 Facial Landmarks Detection 11 2.2.4 Camera Registration 11 2.2.5 Object Tracking 14 2.2.6 Thermal Camera Calibration 2.2.7 Face Recognition Methods 16 System Design 14 21 3.1 Solution Description 21 3.2 Proposed System Architecture 22 3.3 Devices and Components 26 3.4 3.5 3.3.1 Hardware Components 26 3.3.2 Software Components 32 Database Design 33 3.4.1 Requirement 33 3.4.2 Entity Relation Diagram 34 3.4.3 Relational Model 35 3.4.4 Database Data Type & Constraint 36 Software Application Design 38 3.5.1 Device Application 38 3.5.2 Web Application 47 Implementation and Evaluation 60 v A.5 Device A.5.4 WiFi Connect The WiFi configuration can be accessed with the WiFi icon Users can change the WiFi connection in this setting 172 A.5 Device A.5.5 Setting Device A.5.6 Calibrate Device 173 A.6 Web Application A.5.7 Check Connection A.6 Web Application For best experience, it is recommended to access the web application on laptop or personal computer Go to the website link https://tktinc.azurewebsites.net to access the home page Click on the "Log in" button at the top right corner to log in 174 A.6 Web Application Click on the "Don’t have an account yet?" to redirect to the registration page and create a new account 175 A.6 Web Application Input Username, Password and Email to create a new account −→ Click Register to verify the information If the account is valid, the website will redirect you to the dash board There will be a notification to add new buildings to add new devices to those buildings 176 A.6 Web Application In the Building Management −→ Click Add Building Input all the required information about the building and the locations of that building −→ Click Add New 177 A.6 Web Application In the Device Management tab on the left navigation bar −→ Click Add Device Input all the required information about the device −→ Click Activate 178 A.6 Web Application 10 To monitor all live and history records directly, click on the Record Management tab on the left navigation bar 179 A.6 Web Application 180 A.6 Web Application 11 To add users to the buildings, click on the User Management tab on the left navigation bar 181 A.6 Web Application 12 In the Dashboard tab, you can view the statistic of the day, devices’ status, and number of entries in a period in each building 182 A.6 Web Application 13 In the My Profile tab, you can view the basic information about your account and the number of buildings, devices, and users 183 Bibliography [1] Angelo Carfì, Roberto Bernabei, Francesco Landi, and for the Gemelli Against COVID-19 Post-Acute Care Study Group Persistent Symptoms in Patients After Acute COVID-19 JAMA, 324(6):603–605, 08 2020 [2] B F Jones and P Plassmann Digital Infrared Thermal Imaging of Human Skin IEEE Engineering in Medicine and Biology Magazine, 21(6):41–48, 2002 [3] E F J Ring The Historical Development of Thermal Imaging in Medicine Rheumatology, 43(6):800–802, 2004 [4] Ana C Pérez de Diego, Pedro J Sánchez-Cordón, Miriam Pedrera, Beatriz Martínez-López, José C Gómez-Villamandos, and José M Sánchez-Vizcno The Use of Infrared Thermography as A Non-invasive Method for Fever Detection in Sheep Infected with Bluetongue Virus The Veterinary Journal, 198(1):182–186, 2013 [5] Carson, Rob “How Thermal Imaging Cameras Work.” 05 February 2017 www thermalcameras.guide/how-thermal-imaging-cameras-work/ [6] Viola, Paul, and Michael Jones “Rapid Object Detection Using a Boosted Cascade of Simple Features.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001 https://ieeexplore.ieee.org/document/ 990517 [7] Thaware, Rajeev “Real-Time Face Detection and Recognition with SVM and HOG Features.” 28 May 2018 [8] Rosebrock, Adrian “Facial landmarks with dlib, OpenCV, and Python.” April 2017, https://www.pyimagesearch.com/2017/04/03/ facial-landmarks-dlib-opencv-python/ 184 Bibliography [9] DeTone, Daniel, et al “Deep Image Homography Estimation.” 13 June 2016, https://https://www.researchgate.net/publication/305881252_Deep_ Image_Homography_Estimation [10] Mallick, Satya “Image Alignment (Feature Based) using OpenCV.” 11 March 2018, https://www.learnopencv.com/ image-alignment-feature-based-using-opencv-c-python/ [11] Wenhai Sun, Nan Sun, Baolong Guo, Wenyan Jia, and Mingui Sun An auxiliary gaze point estimation method based on facial normal Pattern Analysis and Applications, 19, 08 2014 [12] “FLIR LEPTON® Engineering Datasheet.” June https://groupgets-files.s3.amazonaws.com/lepton/Resources/ 500-0659-00-09Rev204LeptonEngineeringDatasheet.pdf 2018 [13] Schroff, Florian, et al “FaceNet: A Unified Embedding for Face Recognition and Clustering.” IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2015, 17 June 2015, https://arxiv.org/pdf/1503.03832.pdf [14] Baid, Ujjwal R “Image Registration and Homography Estimation.” July 2015, https://www.researchgate.net/publication/320805489_Image_ Registration_and_Homography_Estimation [15] Aryal, Ashrant, and Burcin Becerik-Gerber “Skin Temperature Extraction Using Facial Landmark Detection and Thermal Imaging for Comfort Assessment.” the 6th ACM International Conference[USA], November 2019, https://www.researchgate.net/publication/337040711_Skin_Temperature_ Extraction_Using_Facial_Landmark_Detection_and_Thermal_Imaging_ for_Comfort_Assessment [16] Armote Somboonkaew, Panintorn Prempree, Sirajit Vuttivong, Jutaphet Wetcharungsri, Supanit Porntheeraphat, Sataporn Chanhorm, Prasit Pongsoon, Ratthasart Amarit, Yuttana Intaravanne, Kosom Chaitavon, and Sarun Sumriddetchkajorn Mobile-platform for Automatic Fever Screening System based on Infrared Forehead Temperature pages 1–4, 07 2017 [17] Negishi, Toshiaki, et al “Contactless Vital Signs Measurement System Using RGB-Thermal Image Sensors and Its Clinical Screening Test on Patients with 185 Bibliography Seasonal Influenza.” Sensors 2020, 13 April 2020, https://www.mdpi.com/ 1424-8220/20/8/2171/htm [18] “Jetson Nano Developer Kit User Guide.” October 2020, https://developer nvidia.com/embedded/downloads#?search=Jetson%20Nano [19] “Infrared Array Sensor Grid-EYE (AMG88).” April 2020, https://cdn.sparkfun com/assets/4/1/c/0/1/Grid-EYE_Datasheet.pdf [20] “Camera Module camera-module-v2/ V2.” https://www.raspberrypi.org/products/ [21] “About OpenCV.” https://opencv.org/about/ [22] Priya, Jaya “Microsoft Azure Cloud Services” Medium, 2019, https://medium com/gangboard/microsoft-azure-cloud-services-e18b29649304 [23] Liu, Wei, et al “SSD: Single Shot MultiBox Detector.” Computer Vision and Pattern Recognition, 29 December 2016, https://arxiv.org/abs/1512.02325 [24] Rosebrock, Adrian “Face recognition with OpenCV, Python, and deep learning.” June 2018, https://www.pyimagesearch.com/2018/06/18/ face-recognition-with-opencv-python-and-deep-learning/ [25] Mikel Ariz, Arantxa Villanueva, and Rafael Cabeza Robust and accurate 2d-tracking-based 3d positioning method: Application to head pose estimation Computer Vision and Image Understanding, 180:13–22, 2019 [26] Abdullah Talha Kabakus An experimental performance comparison of widely used face detection tools 8:5, 09 2019 [27] Rubén Usamentiaga, Pablo Venegas, Jon Guerediaga, Laura Vega, Julio Molleda, and Francisco G Bulnes Infrared thermography for temperature measurement and non-destructive testing Sensors, 14(7):12305–12348, 2014 [28] Dinh-Tien Tran, Viet-Khoa Nguyen, Ngoc-Thien Nhan, Duc-Hiep Nguyen, and Hoang-Anh Pham A Low-Cost High-Accuracy thermal camera using off-theshelf hardware devices In 2021 9th International Conference on Information and Communication Technology (ICoICT) (ICoICT 2021), August 2021 186 ... no “feature extraction” is taking place [8] 2.2.4 Camera Registration The thermal camera and RGB camera are placed at different angles; each camera has a different Field of View (FOV) and resolution... temperature accurately by extracting the facial landmarks from an RGB camera image and projecting those coordinates into a high-resolution thermal camera image A method is called ROI Temperature... Anh Declaration We commit that our topic ? ?Develop A Low- cost Thermal Camera Using 2D Camera and Thermal Sensor? ?? is our thesis We declare that this topic is conducted under our effort, time, and