Phát hiện một số sự kiện bất thường dựa trên hình ảnh sử dụng mô hình phân cấp

BỘ THÔNG TIN VÀ TRUYỀN THÔNG HỌC VIỆN CÔNG NGHỆ BƯU CHÍNH VIỄN THƠNG - PHÁT HIỆN MỘT SỐ SỰ KIỆN BẤT THƯỜNG DỰA TRÊN HÌNH ẢNH SỬ DỤNG MƠ HÌNH PHÂN CẤP LUẬN ÁN TIẾN SĨ KỸ THUẬT HÀ NỘI - 2023 BỘ THÔNG TIN VÀ TRUYỀN THÔNG HỌC VIỆN CƠNG NGHỆ BƯU CHÍNH VIỄN THƠNG - PHÁT HIỆN MỘT SỐ SỰ KIỆN BẤT THƯỜNG DỰA TRÊN HÌNH ẢNH SỬ DỤNG MƠ HÌNH PHÂN CẤP CHUN NGÀNH: KỸ THUẬT MÁY TÍNH MÃ SỐ: 9.48.01.06 LUẬN ÁN HÀ NỘI - 2023 LỜI CAM ĐOAN Tôi xin cam đoan luận án tốt nghiệp riêng tôi, hướng dẫn PGS.TS– Học Viện Cơng Nghệ Bưu Chính Viễn Thơng Tất kết số liệu luận án trung thực có từ nghiên cứu mà thầy hướng dẫn thực trình làm luận án Hà Nội, ngày tháng năm 2023 Nghiên cứu sinh i LỜI CẢM ƠN Để hồn thành luận án này, trước hết, tơi xin bày tỏ lòng biết ơn sâu sắc tới thầy hướng dẫn tơi PGS.TS hướng dẫn tận tình, lời khun, lắng nghe khích lệ q trình thực luận án Thầy khơng truyền cho kiến thức chuyên môn mà cịn giúp tơi cải thiện nhiều kỹ nghiên cứu khoa học sống Tôi không quên khoảng thời gian dài thầy thực địa thu thập liệu, thực thử nghiệm, mô triển khai hệ thống Các buổi thảo luận thường xuyên vào tối với nhóm nghiên cứu Thầy định hướng, hướng dẫn giúp chỉnh sửa báo thảo luận án tiến sĩ Tôi học hỏi nhiều điều, lần nữa, xin gửi lời cảm ơn chân thành tới thầy hướng dẫn Tôi xin chân thành cảm ơn Lãnh đạo, thầy, cô giáo Khoa Đào tạo Sau đại học, Học viện Công nghệ Bưu Viễn thơng hướng dẫn, giúp đỡ, tạo điều kiện thuận lợi cho thời gian học tập, nghiên cứu thực luận án Tôi xin gửi lời cảm ơn đặc biệt đến Quỹ đổi sáng tạo VINGROUP (VINIF), Viện nghiên cứu liệu lớn (VINBIGDATA), chấp nhận hồ sơ ứng tuyển tài trợ học bổng đào tạo Tiến sĩ nước cho Đây nguồn kinh phí thiết thực, giúp tơi tập trung vào cơng việc nghiên cứu hồn thành hạn chương trình nghiên cứu sinh Tơi xin chân thành cảm ơn Lãnh đạo đồng nghiệp Khoa Công nghệ thông tin 1, Học viện Công nghệ Bưu Viễn thơng, giúp đỡ, tạo điều kiện cơng tác thuận lợi, giúp tơi tập trung hồn thành luận án Cuối cùng, xin cảm ơn gia đình ln bên cạnh giúp tơi vượt qua khó khăn, thách thức suốt trình làm luận án Hà Nội, ngày tháng năm 2023 Nghiên cứu sinh ii MỤC LỤC LỜI CAM ĐOAN i LỜI CẢM ƠN ii DANH MỤC TỪ VIẾT TẮT v DANH MỤC KÝ HIỆU vii DANH MỤC HÌNH VẼ ix DANH MỤC BẢNG xi MỞ ĐẦU CHƯƠNG TỔNG QUAN VỀ GIÁM SÁT SỰ KIỆN BẤT THƯỜNG BẰNG THỊ GIÁC MÁY TÍNH VÀ HỌC MÁY 11 1.1 Tổng quan kiện bất thường 11 1.2 Dữ liệu cho phát kiện bất thường 15 1.2.1 Bộ liệu phát kiện bất thường tĩnh 16 1.2.2 Bộ liệu phát kiện bất thường động 19 1.3 Học máy cho phát kiện bất thường 22 1.4 Các nghiên cứu liên quan 24 1.4.1 Phát kiện bất thường tĩnh 25 1.4.2 Phát kiện bất thường động 28 1.5 Kết luận chương 33 CHƯƠNG ĐỀ XUẤT MƠ HÌNH PHÁT HIỆN SỰ KIỆN BẤT THƯỜNG TĨNH SỬ DỤNG MẠNG PHÂN CẤP 35 2.1 Mô hình hệ thống 36 2.2 Phát hố sụt ảnh nhiệt thu từ UAV 38 2.2.1 Phát hố sụt mơ hình mạng phân cấp 40 2.2.2 Bám vết hố sụt thuật toán Hungary 49 2.2.3 Kết thực nghiệm 51 2.3 Phát đường sạt lở ảnh thu từ UAV 63 2.3.1 Phân đoạn đường 65 2.3.2 Phát sạt lở mơ hình phân cấp 66 2.3.3 Kết thực nghiệm 71 2.4 Kết luận chương 83 iii CHƯƠNG ĐỀ XUẤT MƠ HÌNH PHÁT HIỆN SỰ KIỆN BẤT THƯỜNG ĐỘNG SỬ DỤNG MẠNG PHÂN CẤP 86 3.1 Mơ hình hệ thống 86 3.2 Phát lửa sử dụng mơ hình mạng phân cấp 89 3.2.1 Trích xuất đặc trưng đối tượng nghi ngờ lửa khung ảnh 92 3.2.2 Trích xuất đặc trưng thời gian đối tượng sử dụng mơ hình BiLSTM 96 3.2.3 Bộ liệu lửa video FirePTIT 98 3.2.4 Kết thực nghiệm 102 3.3 Kết luận chương 112 KẾT LUẬN 114 DANH MỤC CƠNG TRÌNH ĐÃ CƠNG BỐ 117 TÀI LIỆU THAM KHẢO 118 iv DANH MỤC TỪ VIẾT TẮT Từ viết tắt Nghĩa tiếng Anh Nghĩa tiếng Việt 1DCNN Dimensional CNN Mạng CNN chiều 2DCNN Dimensional CNN Mạng CNN hai chiều 3DCNN Dimensional CNN Mạng CNN ba chiều AI Artificial Intelligence Trí tuệ nhân tạo ANN Artificial Neural Network Mạng nơ ron nhân tạo BoVW Bag-of-Visual-Word Túi từ trực quan hóa CNN Convolutional Neural Network Mạng nơ ron tích chập CRF Conditional Random Field Trường ngẫu nhiên có điều kiện DBN Deep Belief Network Mạng niềm tin sâu DEM Digital Elevation Models Mơ hình độ cao số DRN Deep Recurrent Network Mạng lặp lại sâu GIS Geographic Information System Hệ thống thông tin địa lý GPS Global Positioning System Hệ thống định vị toàn cầu HA Hungarian Algorithm Thuật toán Hungary HM Hierarchical Model Mơ hình phân cấp HMM Hidden Markov Model Mơ hình Markov ẩn HN Hierarchical Network Mạng phân cấp IoT Internet of Things Internet vạn vật ISTL Incremental Spatiotemporal Learner Bộ học không-thời gian gia tăng LiDAR Light Detection and Ranging Vùng phát ánh sáng LSTM Long Short Term Memory Mạng nhớ dài ngắn hạn v NAIP National Agriculture Imagery Program RCNN Region Based Convolutional Neural Networks Chương trình ảnh vệ tinh nơng nghiệp quốc gia Mạng nơ ron tích chập dựa vùng RGB Red Green Blue Đỏ - xanh - xanh da trời RNN Recurrent Neural Network Mạng nơ ron hồi quy ROI Region of Interest Vùng quan tâm SVM Support Vector Machine Máy vector hỗ trợ TL Transfer Learning Học chuyển tiếp UAV Unmmaned Aerial Vehicle Thiết bị bay không người lái vi DANH MỤC KÝ HIỆU 𝐴𝑃 Chỉ số độ xác trung bình 𝐴𝑅 Chỉ số recall trung bình 𝐴(") Năng lượng nhấp nháy tích lũy 𝑐 (") Trạng thái tế bào mạng LSTM thời điểm t 𝑑 𝐷(") 𝐷𝑖𝑐𝑒 ( ) Giá trị chi phí thuật tốn Hungary Độ sáng điểm ảnh thời điểm t Hàm số chồng lấn đối tượng nhãn 𝐸 (") Năng lượng nhấp nháy điểm ảnh thời điểm t 𝐹𝑖𝑟𝑒 Giá trị dự đoán đối tượng lửa hay lửa 𝑓 (") Giá trị cổng quên tế bào mạng LSTM thời điểm t ℎ$ (.) Hàm lọc tầng ℎ% (.) Hàm lọc tầng ℎ&'' (.) Hàm mơ hình học sâu CNN ℎ() (.) Hàm mơ hình học sâu MobileNet toán phát sạt lở ℎ* Độ dài tenor (Đối tượng liên kết Connected components) ℎ+,)- (.) Hàm lọc theo luật ℎ+-.'-" Hàm mơ hình học sâu Resnet tốn phát lửa 𝐻/! Ngưỡng kênh màu H 𝐻/" Ngưỡng trên kênh màu H ℎ(") Giá trị trạng thái ẩn tế bào mạng LSTM thời điểm t 𝐼𝑜𝑈( ) 𝑖 (") Hàm số phần giao phần hợp đối tượng Giá trị lớp cổng vào tế bào mạng LSTM thời điểm t vii 𝐼τ Tập hợp tensor đầu vào 𝐽(.) Hàm Jaccard 𝐿! Giá trị mát ứng với số Jaccard 𝑁 Số lượng tensor tập hợp 𝑂 Tập hợp tensor đầu 𝑜 (") Giá trị lớp cổng tế bào mạng LSTM thời điểm t 𝑝 Xác suất điểm ảnh điểm ảnh thuộc đối tượng bất thường 𝑅/ Ngưỡng kênh màu R 𝑠* Diện tích tenor (Đối tượng liên kết Connected components) 𝑠𝑜𝑓𝑡𝑚𝑎𝑥( ) Hàm softmax 𝑠𝑤𝑖𝑠ℎ( ) 𝑡𝑎𝑛ℎ(.) Hàm Swish Hàm 𝑡0/ Ngưỡng chồng lấn để xác định tỉ lệ phát xác đối tượng bất thường 𝑤* Độ rộng tenor (Đối tượng liên kết Connected components) (") 𝑋&'' 𝑋12"&3 (") 𝑋+-.'-" Tập hợp vector đặc trưng đối tượng có khả đối tượng bất thường thời điểm t Cụm vector đặc trưng bước thời gian khứ trích xuất từ mơ hình Resnet Tập hợp vector đặc trưng đối tượng có khả đối tượng bất thường thời điểm t trích xuất từ mơ hình Resnet 𝑌 Đầu dự đốn mơ hình mạng nơ ron hồi quy RNN 𝛼 Hệ số tích lũy 𝜎(.) Hàm sigmoid τ Tensor viii [27] L Lopez-Fuentes, J van de Weijer, M González-Hidalgo, H Skinnemoen, and A D Bagdanov, “Review on computer vision techniques in emergency situations,” Multimed Tools Appl., vol 77, no 13, pp 17069–17107, Jul 2018, doi: 10.1007/s11042-017-5276-7 [28] W Sun, P Bocchini, and B D Davison, “Applications of artificial intelligence for disaster management,” Nat Hazards, vol 103, no 3, pp 2631–2689, Sep 2020, doi: 10.1007/s11069-020-04124-3 [29] G F Shidik, E Noersasongko, A Nugraha, P N Andono, J Jumanto, and E J Kusuma, “A Systematic Review of Intelligence Video Surveillance: Trends, Techniques, Frameworks, and Datasets,” IEEE Access, vol 7, pp 170457–170473, 2019, doi: 10.1109/ACCESS.2019.2955387 [30] M Yu et al., “Spatiotemporal event detection: a review,” Int J Digit Earth, vol 13, no 12, pp 1339–1365, Dec 2020, doi: 10.1080/17538947.2020.1738569 [31] A Adam, E Rivlin, I Shimshoni, and D Reinitz, “Robust Real-Time Unusual Event Detection using Multiple Fixed-Location Monitors,” IEEE Trans Pattern Anal Mach Intell., vol 30, no 3, pp 555–560, Mar 2008, doi: 10.1109/TPAMI.2007.70825 [32] G Chen et al., “NeuroAED: Towards Efficient Abnormal Event Detection in Visual Surveillance With Neuromorphic Vision Sensor,” IEEE Trans Inf Forensics Secur., vol 16, pp 923–936, 2021, doi: 10.1109/TIFS.2020.3023791 [33] D Giordan, A Manconi, F Remondino, and F Nex, “Use of unmanned aerial vehicles in monitoring application and management of natural hazards,” Geomat Nat Hazards Risk, vol 8, no 1, pp 1–4, Jan 2017, doi: 10.1080/19475705.2017.1315619 [34] F Outay, H A Mengash, and M Adnan, “Applications of unmanned aerial vehicle (UAV) in road safety, traffic and highway infrastructure management: Recent advances and challenges,” Transp Res Part Policy Pract., vol 141, pp 116–129, Nov 2020, doi: 10.1016/j.tra.2020.09.018 [35] A Billi, L De Filippis, P P Poncia, P Sella, and C Faccenna, “Hidden sinkholes and karst cavities in the travertine plateau of a highly-populated geothermal seismic territory (Tivoli, central Italy),” Geomorphology, vol 255, pp 63–80, Feb 2016, doi: 10.1016/j.geomorph.2015.12.011 121 [36] D Bloomquist, R L Shrestha, and C Slatton, Early sinkhole detection and verification using airborne laser and infrared technologies University of Florida, Department of Civil and Coastal Engineering, 2005 [37] J C Shannon, D Moore, Y Li, and C Olsen, “LiDAR-based Sinkhole Detection and Mapping in Knox County, Tennessee,” Purs J Undergrad Res Univ Tenn., vol 9, no 1, p 3, 2019 [38] E Karantanellis, V Marinos, E Vassilakis, and B Christaras, “Object-Based Analysis Using Unmanned Aerial Vehicles (UAVs) for Site-Specific Landslide Assessment,” Remote Sens., vol 12, no 11, p 1711, May 2020, doi: 10.3390/rs12111711 [39] D Turner, A Lucieer, and S de Jong, “Time Series Analysis of Landslide Dynamics Using an Unmanned Aerial Vehicle (UAV),” Remote Sens., vol 7, no 2, pp 1736– 1757, Feb 2015, doi: 10.3390/rs70201736 [40] S Ham, Y Oh, K Choi, and I Lee, “SEMANTIC SEGMENTATION AND UNREGISTERED BUILDING DETECTION FROM UAV IMAGES USING A DECONVOLUTIONAL NETWORK,” Int Arch Photogramm Remote Sens Spat Inf Sci., vol XLII–2, pp 419–424, May 2018, doi: 10.5194/isprs-archives-XLII-2419-2018 [41] T Beji et al., “RABOT2012–Presentation of the Multi-compartment Full-Scale (‘Rabot’) Fire Tests,” presented at the 7th International Seminar on Fire and Explosion Hazards, 2013, pp 67–76 [42] N Grammalidis, K Dimitropoulos, and E Cetin, “Firesense Database Of Videos For Flame And Smoke Detection.” Zenodo, Jul 31, 2017 doi: 10.5281/ZENODO.836749 [43] P Foggia, A Saggese, and M Vento, “Real-time fire detection for video-surveillance applications using a combination of experts based on color, shape, and motion,” IEEE Trans Circuits Syst Video Technol., vol 25, no 9, pp 1545–1556, 2015 [44] S Robles-Serrano, G Sanchez-Torres, and J Branch-Bedoya, “Automatic Detection of Traffic Accidents from Video Using Deep Learning Techniques,” Computers, vol 10, no 11, p 148, Nov 2021, doi: 10.3390/computers10110148 122 [45] A P Shah, J.-B Lamare, T Nguyen-Anh, and A Hauptmann, “CADP: A Novel Dataset for CCTV Traffic Camera based Accident Analysis,” in 2018 15th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Auckland, New Zealand: IEEE, Nov 2018, pp 1–9 doi: 10.1109/AVSS.2018.8639160 [46] W Sultani, C Chen, and M Shah, “Real-world Anomaly Detection in Surveillance Videos,” 2018, doi: 10.48550/ARXIV.1801.04264 [47] F.-H Chan, Y.-T Chen, Y Xiang, and M Sun, “Anticipating Accidents in Dashcam Videos,” in Computer Vision – ACCV 2016, S.-H Lai, V Lepetit, K Nishino, and Y Sato, Eds., in Lecture Notes in Computer Science, vol 10114 Cham: Springer International Publishing, 2017, pp 136–153 doi: 10.1007/978-3-319-54190-7_9 [48] I González-Díaz, T Martínez-Cortés, A Gallardo-Antolín, and F Díaz-de-María, “Temporal segmentation and keyframe selection methods for user-generated video search-based annotation,” Expert Syst Appl., vol 42, no 1, pp 488–502, Jan 2015, doi: 10.1016/j.eswa.2014.08.001 [49] D T Nguyen, F Ofli, M Imran, and P Mitra, “Damage Assessment from Social Media Imagery Data During Disasters,” in Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, Sydney Australia: ACM, Jul 2017, pp 569–576 doi: 10.1145/3110025.3110109 [50] M Rahnemoonfar, T Chowdhury, A Sarkar, D Varshney, M Yari, and R R Murphy, “FloodNet: A High Resolution Aerial Imagery Dataset for Post Flood Scene Understanding,” IEEE Access, vol 9, pp 89644–89654, 2021, doi: 10.1109/ACCESS.2021.3090981 [51] P V K Borges and E Izquierdo, “A Probabilistic Approach for Vision-Based Fire Detection in Videos,” IEEE Trans Circuits Syst Video Technol., vol 20, no 5, pp 721–731, May 2010, doi: 10.1109/TCSVT.2010.2045813 [52] J Chen, Y He, and J Wang, “Multi-feature fusion based fast video flame detection,” Build Environ., vol 45, no 5, pp 1113–1122, May 2010, doi: 10.1016/j.buildenv.2009.10.017 [53] S Rinsurongkawong, M Ekpanyapong, and M N Dailey, “Fire detection for early fire alarm based on optical flow video processing,” in 2012 9th International 123 Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology, Phetchaburi, Thailand: IEEE, May 2012, pp 1–4 doi: 10.1109/ECTICon.2012.6254144 [54] P Barmpoutis, K Dimitropoulos, and N Grammalidis, “Smoke detection using spatio-temporal analysis, motion modeling and dynamic texture recognition,” presented at the 2014 22nd European Signal Processing Conference (EUSIPCO), IEEE, 2014, pp 1078–1082 [55] P V K Borges, J Mayer, and E Izquierdo, “A probabilistic model for flood detection in video sequences,” in 2008 15th IEEE International Conference on Image Processing, San Diego, CA, USA: IEEE, 2008, pp 13–16 doi: 10.1109/ICIP.2008.4711679 [56] C L Lai, J C Yang, and Y H Chen, “A Real Time Video Processing Based Surveillance System for Early Fire and Flood Detection,” in 2007 IEEE Instrumentation & Measurement Technology Conference IMTC 2007, Warsaw, Poland: IEEE, May 2007, pp 1–6 doi: 10.1109/IMTC.2007.379190 [57] M Andriluka et al., “Vision based victim detection from unmanned aerial vehicles,” in 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, Taipei: IEEE, Oct 2010, pp 1740–1747 doi: 10.1109/IROS.2010.5649223 [58] B Soni and A Sowmya, “Classifier ensemble with incremental learning for disaster victim detection,” in 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO), Guangzhou, China: IEEE, Dec 2012, pp 446–451 doi: 10.1109/ROBIO.2012.6491007 [59] B Soni and A Sowmya, “Victim detection and localisation in an urban disaster site,” presented at the 2013 IEEE international conference on robotics and biomimetics (ROBIO), IEEE, 2013, pp 2142–2147 [60] A Haar, “Zur Theorie der orthogonalen Funktionensysteme: Erste Mitteilung,” Math Ann., vol 69, no 3, pp 331–371, Sep 1910, doi: 10.1007/BF01456326 [61] B U Töreyin, Y Dedeoğlu, U Güdükbay, and A E Çetin, “Computer vision based method for real-time fire and flame detection,” Pattern Recognit Lett., vol 27, no 1, pp 49–58, Jan 2006, doi: 10.1016/j.patrec.2005.06.015 124 [62] S S Beauchemin and J L Barron, “The computation of optical flow,” ACM Comput Surv., vol 27, no 3, pp 433–466, Sep 1995, doi: 10.1145/212094.212141 [63] Z Xu and J Xu, “Automatic Fire Smoke Detection Based on Image Visual Features,” in 2007 International Conference on Computational Intelligence and Security Workshops (CISW 2007), Harbin, Heilongjiang, China: IEEE, Dec 2007, pp 316– 319 doi: 10.1109/CISW.2007.4425500 [64] C Yu, Z Mei, and X Zhang, “A Real-time Video Fire Flame and Smoke Detection Algorithm,” Procedia Eng., vol 62, pp 891–898, 2013, doi: 10.1016/j.proeng.2013.08.140 [65] F Yuan, “Video-based smoke detection with histogram sequence of LBP and LBPV pyramids,” Fire Saf J., vol 46, no 3, pp 132–139, Apr 2011, doi: 10.1016/j.firesaf.2011.01.001 [66] C Piciarelli, C Micheloni, and G L Foresti, “Trajectory-Based Anomalous Event Detection,” IEEE Trans Circuits Syst Video Technol., vol 18, no 11, pp 1544–1554, Nov 2008, doi: 10.1109/TCSVT.2008.2005599 [67] T Wang and H Snoussi, “Histograms of Optical Flow Orientation for Visual Abnormal Events Detection,” in 2012 IEEE Ninth International Conference on Advanced Video and Signal-Based Surveillance, Beijing, China: IEEE, Sep 2012, pp 13–18 doi: 10.1109/AVSS.2012.39 [68] W Ye, J Zhao, S Wang, Y Wang, D Zhang, and Z Yuan, “Dynamic texture based smoke detection using Surfacelet transform and HMT model,” Fire Saf J., vol 73, pp 91–101, Apr 2015, doi: 10.1016/j.firesaf.2015.03.001 [69] Y M Lu and M N Do, “Multidimensional Directional Filter Banks and Surfacelets,” IEEE Trans Image Process., vol 16, no 4, pp 918–931, Apr 2007, doi: 10.1109/TIP.2007.891785 [70] H.-L Chen, M J Tsai, and C C Chan, “A Hidden Markov Model-based approach for recognizing swimmer’s behaviors in swimming pool,” in 2010 International Conference on Machine Learning and Cybernetics, Qingdao, China: IEEE, Jul 2010, pp 2459–2465 doi: 10.1109/ICMLC.2010.5580797 [71] Eng, Toh, Kam, Wang, and Yau, “An automatic drowning detection surveillance system for challenging outdoor pool environments,” in Proceedings Ninth IEEE 125 International Conference on Computer Vision, Nice, France: IEEE, 2003, pp 532– 539 vol.1 doi: 10.1109/ICCV.2003.1238393 [72] J Shin, H Park, and J Paik, “Fire Recognition Using Spatio-Temporal Two-Stream Convolutional Neural Network with Fully Connected Layer-Fusion,” in 2018 IEEE 8th International Conference on Consumer Electronics - Berlin (ICCE-Berlin), Berlin, Germany: IEEE, Sep 2018, pp 1–3 doi: 10.1109/ICCE-Berlin.2018.8576218 [73] Q Zhang, J Xu, L Xu, and H Guo, “Deep Convolutional Neural Networks for Forest Fire Detection,” in Proceedings of the 2016 International Forum on Management, Education and Information Technology Application, Guangzhou, China: Atlantis Press, 2016 doi: 10.2991/ifmeita-16.2016.105 [74] S Frizzi, R Kaabi, M Bouchouicha, J.-M Ginoux, E Moreau, and F Fnaiech, “Convolutional neural network for video fire and smoke detection,” in IECON 2016 42nd Annual Conference of the IEEE Industrial Electronics Society, Florence, Italy: IEEE, Oct 2016, pp 877–882 doi: 10.1109/IECON.2016.7793196 [75] S Khan, K Muhammad, S Mumtaz, S W Baik, and V H C de Albuquerque, “Energy-Efficient Deep CNN for Smoke Detection in Foggy IoT Environment,” IEEE Internet Things J., vol 6, no 6, pp 9237–9245, Dec 2019, doi: 10.1109/JIOT.2019.2896120 [76] P Hristov, “Real-time Abnormal Human Activity Detection Using 1DCNN-LSTM for 3D Skeleton Data,” in 2021 12th National Conference with International Participation (ELECTRONICA), Sofia, Bulgaria: IEEE, May 2021, pp 1–4 doi: 10.1109/ELECTRONICA52725.2021.9513696 [77] R Vrskova, R Hudec, P Kamencay, and P Sykora, “A New Approach for Abnormal Human Activities Recognition Based on ConvLSTM Architecture,” Sensors, vol 22, no 8, p 2946, Apr 2022, doi: 10.3390/s22082946 [78] R Vrskova, R Hudec, P Kamencay, and P Sykora, “Recognition of Human Activity and Abnormal Behavior using Deep Neural Network,” in 2022 ELEKTRO (ELEKTRO), Krakow, Poland: IEEE, May 2022, pp 1–4 doi: 10.1109/ELEKTRO53996.2022.9803355 [79] X Wang, W Xie, and J Song, “Learning Spatiotemporal Features With 3DCNN and ConvGRU for Video Anomaly Detection,” in 2018 14th IEEE International 126 Conference on Signal Processing (ICSP), Beijing, China: IEEE, Aug 2018, pp 474– 479 doi: 10.1109/ICSP.2018.8652354 [80] G A Martínez-Mascorro, J R Abreu-Pederzini, J C Ortiz-Bayliss, and H Terashima-Marín, “Suspicious behavior detection on shoplifting cases for crime prevention by using 3D convolutional neural networks,” ArXiv Prepr ArXiv200502142, 2020 [81] S Majhi, R Dash, and P K Sa, “Temporal Pooling in Inflated 3DCNN for Weaklysupervised Video Anomaly Detection,” in 2020 11th International Conference on Computing, Communication and Networking Technologies (ICCCNT), Kharagpur, India: IEEE, Jul 2020, pp 1–6 doi: 10.1109/ICCCNT49239.2020.9225378 [82] Z Li, G Wu, Y Liu, Y Shuai, and L Wang, “Video Abnormal Event Detection Based on Optical Flow and 3DCNN,” J Phys Conf Ser., vol 1881, no 2, p 022022, Apr 2021, doi: 10.1088/1742-6596/1881/2/022022 [83] H V Nguyen, T X Pham, and C N Le, “Real-time long short-term glance-based fire detection using a CNN-LSTM neural network,” Int J Intell Inf Database Syst., vol 14, no 4, p 349, 2021, doi: 10.1504/IJIIDS.2021.118545 [84] B Kim and J Lee, “A Video-Based Fire Detection Using Deep Learning Models,” Appl Sci., vol 9, no 14, p 2862, Jul 2019, doi: 10.3390/app9142862 [85] T Fernando, S Denman, S Sridharan, and C Fookes, “Soft + Hardwired attention: An LSTM framework for human trajectory prediction and abnormal event detection,” Neural Netw., vol 108, pp 466–478, Dec 2018, doi: 10.1016/j.neunet.2018.09.002 [86] J Gao, P Gu, Q Ren, J Zhang, and X Song, “Abnormal Gait Recognition Algorithm Based on LSTM-CNN Fusion Network,” IEEE Access, vol 7, pp 163180–163190, 2019, doi: 10.1109/ACCESS.2019.2950254 [87] L Xia and Z Li, “A new method of abnormal behavior detection using LSTM network with temporal attention mechanism,” J Supercomput., vol 77, no 4, pp 3223–3241, Apr 2021, doi: 10.1007/s11227-020-03391-y [88] Z.-G Zhou, B Chen, Z Li, and C Li, “The Use of LSTM-Based RNN and SVM Models to Detect Ludian Coseismic Landslides in Time Series Images,” J Phys Conf Ser., vol 1631, no 1, p 012085, Sep 2020, doi: 10.1088/1742-6596/1631/1/012085 127 [89] H Li, Q Xu, Y He, X Fan, H Yang, and S Li, “Temporal detection of sharp landslide deformation with ensemble-based LSTM-RNNs and Hurst exponent,” Geomat Nat Hazards Risk, vol 12, no 1, pp 3089–3113, Jan 2021, doi: 10.1080/19475705.2021.1994474 [90] Y Li, X Pu, Y Qiao, and H Wang, “A Hybrid Recurrent Model Based on LSTM and Statistical Evaluation to Predict Soil Displacement for Landslide Monitoring,” in Proceedings of 6th International Conference on Harmony Search, Soft Computing and Applications, S M Nigdeli, J H Kim, G Bekdaş, and A Yadav, Eds., in Advances in Intelligent Systems and Computing, vol 1275 Singapore: Springer Singapore, 2021, pp 307–315 doi: 10.1007/978-981-15-8603-3_27 [91] Y Ding, Y Zhu, J Feng, P Zhang, and Z Cheng, “Interpretable spatio-temporal attention LSTM model for flood forecasting,” Neurocomputing, vol 403, pp 348– 359, Aug 2020, doi: 10.1016/j.neucom.2020.04.110 [92] J S Almeida, C Huang, F G Nogueira, S Bhatia, and V H C de Albuquerque, “EdgeFireSmoke: A Novel Lightweight CNN Model for Real-Time Video Fire– Smoke Detection,” IEEE Trans Ind Inform., vol 18, no 11, pp 7889–7898, Nov 2022, doi: 10.1109/TII.2021.3138752 [93] I V S Tech, “IVS Tech,” IVS Tech http://www.ivstech.co.kr? (accessed Aug 24, 2023) [94] “Artificial Intelligence | Samsung Research,” Artificial Intelligence | Samsung Research https://research.samsung.com/artificial-intelligence (accessed Aug 24, 2023) [95] G Sreenu and S Durai, “Intelligent video surveillance: a review through deep learning techniques for crowd analysis,” J Big Data, vol 6, no 1, pp 1–27, 2019 [96] G Cheng, L Guo, T Zhao, J Han, H Li, and J Fang, “Automatic landslide detection from remote-sensing imagery using a scene classification method based on BoVW and pLSA,” Int J Remote Sens., vol 34, no 1, pp 45–59, Jan 2013, doi: 10.1080/01431161.2012.705443 [97] A Zeggada, S Benbraika, F Melgani, and Z Mokhtari, “Multilabel Conditional Random Field Classification for UAV Images,” IEEE Geosci Remote Sens Lett., vol 15, no 3, pp 399–403, Mar 2018, doi: 10.1109/LGRS.2018.2790426 128 [98] F Guede-Fernández, L Martins, R V de Almeida, H Gamboa, and P Vieira, “A Deep Learning Based Object Identification System for Forest Fire Detection,” Fire, vol 4, no 4, p 75, Oct 2021, doi: 10.3390/fire4040075 [99] J Zhang, H Zhu, P Wang, and X Ling, “ATT Squeeze U-Net: A Lightweight Network for Forest Fire Detection and Recognition,” IEEE Access, vol 9, pp 10858– 10870, 2021, doi: 10.1109/ACCESS.2021.3050628 [100] S Y Wong, C W C Choe, H H Goh, Y W Low, D Y S Cheah, and C Pang, “Power Transmission Line Fault Detection and Diagnosis Based on Artificial Intelligence Approach and its Development in UAV: A Review,” Arab J Sci Eng., vol 46, no 10, pp 9305–9331, Oct 2021, doi: 10.1007/s13369-021-05522-w [101] K He, X Zhang, S Ren, and J Sun, “Deep Residual Learning for Image Recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA: IEEE, Jun 2016, pp 770–778 doi: 10.1109/CVPR.2016.90 [102] H Li, Y Shi, B Zhang, and Y Wang, “Superpixel-Based Feature for Aerial Image Scene Recognition,” Sensors, vol 18, no 2, p 156, Jan 2018, doi: 10.3390/s18010156 [103] A M Boroujerdian, M Saffarzadeh, H Yousefi, and H Ghassemian, “A model to identify high crash road segments with the dynamic segmentation method,” Accid Anal Prev., vol 73, pp 274–287, Dec 2014, doi: 10.1016/j.aap.2014.09.014 [104] T R Martha, N Kerle, C J van Westen, V Jetten, and K V Kumar, “Segment Optimization and Data-Driven Thresholding for Knowledge-Based Landslide Detection by Object-Based Image Analysis,” IEEE Trans Geosci Remote Sens., vol 49, no 12, pp 4928–4943, Dec 2011, doi: 10.1109/TGRS.2011.2151866 [105] U Seidaliyeva, M Alduraibi, L Ilipbayeva, and N Smailov, “Deep residual neural network-based classification of loaded and unloaded UAV images,” in 2020 Fourth IEEE International Conference on Robotic Computing (IRC), Taichung, Taiwan: IEEE, Nov 2020, pp 465–469 doi: 10.1109/IRC.2020.00088 [106] S L Ullo et al., “A New Mask R-CNN Based Method for Improved Landslide Detection,” 2020, doi: 10.48550/ARXIV.2010.01499 129 [107] K He, G Gkioxari, P Dollár, and R Girshick, “Mask R-CNN,” 2017, doi: 10.48550/ARXIV.1703.06870 [108] J Redmon and A Farhadi, “YOLO9000: Better, Faster, Stronger,” 2016, doi: 10.48550/ARXIV.1612.08242 [109] J Redmon and A Farhadi, “YOLOv3: An Incremental Improvement,” 2018, doi: 10.48550/ARXIV.1804.02767 [110] K He, X Zhang, S Ren, and J Sun, “Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition,” in Computer Vision – ECCV 2014, D Fleet, T Pajdla, B Schiele, and T Tuytelaars, Eds., in Lecture Notes in Computer Science, vol 8691 Cham: Springer International Publishing, 2014, pp 346–361 doi: 10.1007/9783-319-10578-9_23 [111] C Huang, Z Wu, J Wen, Y Xu, Q Jiang, and Y Wang, “Abnormal Event Detection Using Deep Contrastive Learning for Intelligent Video Surveillance System,” IEEE Trans Ind Inform., vol 18, no 8, pp 5171–5179, Aug 2022, doi: 10.1109/TII.2021.3122801 [112] W Ullah, T Hussain, Z A Khan, U Haroon, and S W Baik, “Intelligent dual stream CNN and echo state network for anomaly detection,” Knowl.-Based Syst., vol 253, p 109456, Oct 2022, doi: 10.1016/j.knosys.2022.109456 [113] K K Santhosh, D P Dogra, and P P Roy, “Anomaly detection in road traffic using visual surveillance: A survey,” ACM Comput Surv CSUR, vol 53, no 6, pp 1–26, 2020 [114] B Ko, K.-H Cheong, and J.-Y Nam, “Early fire detection algorithm based on irregular patterns of flames and hierarchical Bayesian Networks,” Fire Saf J., vol 45, no 4, pp 262–270, Jun 2010, doi: 10.1016/j.firesaf.2010.04.001 [115] I Serrano Gracia, O Deniz Suarez, G Bueno Garcia, and T.-K Kim, “Fast fight detection,” PloS One, vol 10, no 4, p e0120448, 2015 [116] V M Arceda, K F Fabián, P L Laura, J R Tito, and J G Cáceres, “Fast face detection in violent video scenes,” Electron Notes Theor Comput Sci., vol 329, pp 5–26, 2016 [117] L Auria and R Moro, “Advantages and disadvantages of support vector machines,” Credit Risk Assess Revisit Methodol Issues Pract Implic., pp 49–68, 2007 130 [118] C Ding, S Fan, M Zhu, W Feng, and B Jia, “Violence detection in video by using 3D convolutional neural networks,” presented at the International symposium on visual computing, Springer, 2014, pp 551–558 [119] S Sudhakaran and O Lanz, “Learning to detect violent videos using convolutional long short-term memory,” in 2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), Lecce, Italy: IEEE, Aug 2017, pp 1–6 doi: 10.1109/AVSS.2017.8078468 [120] F Maiorano and A Petrosino, “Granular trajectory based anomaly detection for surveillance,” in 2016 23rd International Conference on Pattern Recognition (ICPR), Cancun: IEEE, Dec 2016, pp 2066–2072 doi: 10.1109/ICPR.2016.7899940 [121] P Liu, P Yang, C Wang, K Huang, and T Tan, “A Semi-Supervised Method for Surveillance-Based Visual Location Recognition,” IEEE Trans Cybern., vol 47, no 11, pp 3719–3732, Nov 2017, doi: 10.1109/TCYB.2016.2578639 [122] M.-C Chang, Y Wei, N Song, and S Lyu, “Video Analytics in Smart Transportation for the AIC’18 Challenge,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT: IEEE, Jun 2018, pp 61–617 doi: 10.1109/CVPRW.2018.00016 [123] W Luo, W Liu, and S Gao, “A Revisit of Sparse Coding Based Anomaly Detection in Stacked RNN Framework,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice: IEEE, Oct 2017, pp 341–349 doi: 10.1109/ICCV.2017.45 [124] M X Ma, H Y T Ngan, and W Liu, “Density-based Outlier Detection by Local Outlier Factor on Largescale Traffic Data,” Electron Imaging, vol 28, no 14, pp 1– 4, Feb 2016, doi: 10.2352/ISSN.2470-1173.2016.14.IPMVA-385 [125] Y Li, T Guo, R Xia, and W Xie, “Road Traffic Anomaly Detection Based on Fuzzy Theory,” IEEE Access, vol 6, pp 40281–40288, 2018, doi: 10.1109/ACCESS.2018.2851747 [126] W Luo, W Liu, and S Gao, “Remembering history with convolutional LSTM for anomaly detection,” in 2017 IEEE International Conference on Multimedia and Expo (ICME), Hong Kong, Hong Kong: IEEE, Jul 2017, pp 439–444 doi: 10.1109/ICME.2017.8019325 131 [127] H Li, Y Zhang, M Yang, Y Men, and H Chao, “A rapid abnormal event detection method for surveillance video based on a novel feature in compressed domain of HEVC,” in 2014 IEEE International Conference on Multimedia and Expo (ICME), Chengdu, China: IEEE, Jul 2014, pp 1–6 doi: 10.1109/ICME.2014.6890212 [128] X Qi and J Ebert, “A computer vision based method for fire detection in color videos,” Int J Imaging, vol 2, no S09, pp 22–34, 2009 [129] P Huang, J Su, Z Lu, and J Pan, “A Fire-Alarming Method Based on Video Processing,” in 2006 International Conference on Intelligent Information Hiding and Multimedia, Pasadena, CA, USA: IEEE, Dec 2006, pp 359–364 doi: 10.1109/IIHMSP.2006.265017 [130] Z Teng, J.-H Kim, and D.-J Kang, “Fire detection based on hidden Markov models,” Int J Control Autom Syst., vol 8, no 4, pp 822–830, Aug 2010, doi: 10.1007/s12555-010-0414-2 [131] B C Ko, K.-H Cheong, and J.-Y Nam, “Fire detection based on vision sensor and support vector machines,” Fire Saf J., vol 44, no 3, pp 322–329, Apr 2009, doi: 10.1016/j.firesaf.2008.07.006 [132] A E Gunawaardena, R M M Ruwanthika, and A G B P Jayasekara, “Computer vision based fire alarming system,” in 2016 Moratuwa Engineering Research Conference (MERCon), Moratuwa, Sri Lanka: IEEE, Apr 2016, pp 325–330 doi: 10.1109/MERCon.2016.7480162 [133] I Chakraborty and T Kr Paul, “A Hybrid Clustering Algorithm for Fire Detection in Video and Analysis with Color Based Thresholding Method,” in 2010 International Conference on Advances in Computer Engineering, Bangalore, Karnataka, India: IEEE, Jun 2010, pp 277–280 doi: 10.1109/ACE.2010.12 [134] T Çelik and H Demirel, “Fire detection in video sequences using a generic color model,” Fire Saf J., vol 44, no 2, pp 147–158, Feb 2009, doi: 10.1016/j.firesaf.2008.05.005 [135] G Marbach, M Loepfe, and T Brupbacher, “An image processing technique for fire detection in video images,” Fire Saf J., vol 41, no 4, pp 285–289, Jun 2006, doi: 10.1016/j.firesaf.2006.02.001 132 [136] T Celik, “Fast and Efficient Method for Fire Detection Using Image Processing,” ETRI J., vol 32, no 6, pp 881–890, Dec 2010, doi: 10.4218/etrij.10.0109.0695 [137] N M Dung and S Ro, “Algorithm for Fire Detection using a Camera Surveillance System,” in Proceedings of the 2018 International Conference on Image and Graphics Processing - ICIGP 2018, Hong Kong, Hong Kong: ACM Press, 2018, pp 38–42 doi: 10.1145/3191442.3191450 [138] J Sharma, O.-C Granmo, and M Goodwin, “Deep CNN-ELM Hybrid Models for Fire Detection in Images,” in Artificial Neural Networks and Machine Learning – ICANN 2018, V Kůrková, Y Manolopoulos, B Hammer, L Iliadis, and I Maglogiannis, Eds., in Lecture Notes in Computer Science, vol 11141 Cham: Springer International Publishing, 2018, pp 245–259 doi: 10.1007/978-3-030-014247_25 [139] P Li and W Zhao, “Image fire detection algorithms based on convolutional neural networks,” Case Stud Therm Eng., vol 19, p 100625, Jun 2020, doi: 10.1016/j.csite.2020.100625 [140] A Ullah, J Ahmad, K Muhammad, M Sajjad, and S W Baik, “Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features,” IEEE Access, vol 6, pp 1155–1166, 2018, doi: 10.1109/ACCESS.2017.2778011 [141] J Donahue et al., “Long-term Recurrent Convolutional Networks for Visual Recognition and Description,” 2014, doi: 10.48550/ARXIV.1411.4389 [142] C Hu, P Tang, W Jin, Z He, and W Li, “Real-Time Fire Detection Based on Deep Convolutional Long-Recurrent Networks and Optical Flow Method,” in 2018 37th Chinese Control Conference (CCC), Wuhan: IEEE, Jul 2018, pp 9061–9066 doi: 10.23919/ChiCC.2018.8483118 [143] J Munkres, “Algorithms for the assignment and transportation problems,” J Soc Ind Appl Math., vol 5, no 1, pp 32–38, 1957 [144] O Ronneberger, P Fischer, and T Brox, “U-Net: Convolutional Networks for Biomedical Image Segmentation,” in Medical Image Computing and ComputerAssisted Intervention – MICCAI 2015, N Navab, J Hornegger, W M Wells, and A F Frangi, Eds., in Lecture Notes in Computer Science, vol 9351 Cham: Springer International Publishing, 2015, pp 234–241 doi: 10.1007/978-3-319-24574-4_28 133 [145] A Howard et al., “Searching for MobileNetV3,” 2019, doi: 10.48550/ARXIV.1905.02244 [146] N Otsu, “A Threshold Selection Method from Gray-Level Histograms,” IEEE Trans Syst Man Cybern., vol 9, no 1, pp 62–66, Jan 1979, doi: 10.1109/TSMC.1979.4310076 [147] A H Murphy, “The Finley affair: A signal event in the history of forecast verification,” Weather Forecast., vol 11, no 1, pp 3–20, 1996 [148] H N Vu, T A Tran, N I Seop, and S H Kim, “Extraction of Text Regions from Complex Background in Document Images by Multilevel Clustering:,” Int J Networked Distrib Comput., vol 4, no 1, p 11, 2016, doi: 10.2991/ijndc.2016.4.1.2 [149] J Deng, W Dong, R Socher, L.-J Li, Kai Li, and Li Fei-Fei, “ImageNet: A largescale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL: IEEE, Jun 2009, pp 248–255 doi: 10.1109/CVPR.2009.5206848 [150] H W Kuhn, “The Hungarian method for the assignment problem,” Nav Res Logist Q., vol 2, no 1‐2, pp 83–97, 1955 [151] J E Espinosa, S A Velastin, and J W Branch, “Vehicle Detection Using Alex Net and Faster R-CNN Deep Learning Models: A Comparative Study,” in Advances in Visual Informatics, H Badioze Zaman, P Robinson, A F Smeaton, T K Shih, S Velastin, T Terutoshi, A Jaafar, and N Mohamad Ali, Eds., in Lecture Notes in Computer Science, vol 10645 Cham: Springer International Publishing, 2017, pp 3– 15 doi: 10.1007/978-3-319-70010-6_1 [152] K Simonyan and A Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” 2014, doi: 10.48550/ARXIV.1409.1556 [153] A Krizhevsky, I Sutskever, and G E Hinton, “ImageNet classification with deep convolutional neural networks,” Commun ACM, vol 60, no 6, pp 84–90, May 2017, doi: 10.1145/3065386 [154] D Reynolds, “Gaussian Mixture Models,” in Encyclopedia of Biometrics, S Z Li and A K Jain, Eds., Boston, MA: Springer US, 2015, pp 827–832 doi: 10.1007/978-14899-7488-4_196 134 [155] S Matteoli, T Veracini, M Diani, and G Corsini, “A Locally Adaptive Background Density Estimator: An Evolution for RX-Based Anomaly Detectors,” IEEE Geosci Remote Sens Lett., vol 11, no 1, pp 323–327, Jan 2014, doi: 10.1109/LGRS.2013.2257670 [156] P K Gadosey et al., “SD-UNet: Stripping down U-Net for Segmentation of Biomedical Images on Platforms with Low Computational Budgets,” Diagnostics, vol 10, no 2, p 110, Feb 2020, doi: 10.3390/diagnostics10020110 [157] G Neuhold, T Ollmann, S R Bulo, and P Kontschieder, “The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes,” in 2017 IEEE International Conference on Computer Vision (ICCV), Venice: IEEE, Oct 2017, pp 5000–5009 doi: 10.1109/ICCV.2017.534 [158] Y Wang et al., “LEDNet: A Lightweight Encoder-Decoder Network for Real-Time Semantic Segmentation,” 2019, doi: 10.48550/ARXIV.1905.02423 [159] M Tan and Q V Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” 2019, doi: 10.48550/ARXIV.1905.11946 [160] C Kyrkou and T Theocharides, “Deep-Learning-Based Aerial Image Classification for Emergency Response Applications Using Unmanned Aerial Vehicles.,” presented at the CVPR Workshops, 2019, pp 517–525 [161] K Avazov, M Mukhiddinov, F Makhmudov, and Y I Cho, “Fire Detection Method in Smart City Environments Using a Deep-Learning-Based Approach,” Electronics, vol 11, no 1, p 73, 2021 [162] K Dimitropoulos, P Barmpoutis, and N Grammalidis, “Spatio-temporal flame modeling and dynamic texture analysis for automatic video-based fire detection,” IEEE Trans Circuits Syst Video Technol., vol 25, no 2, pp 339–351, 2014 [163] Y Li, A Vodacek, R L Kremens, A Ononye, and C Tang, “A hybrid contextual approach to wildland fire detection using multispectral imagery,” IEEE Trans Geosci Remote Sens., vol 43, no 9, pp 2115–2126, 2005 135