An efficient on demand charging for wrsns using fuzzy logic and q learning

HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY Master’s Thesis in Data Science and Artificial Intelligence An Efficient, On-demand Charging for WRSNs Using Fuzzy Logic and QLearning La Van Quan Quan.LV202335M@sis.hust.edu.vn Supervisor: Dr Nguyen Phi Le Department: Department of Software engineering Institute: School of Information and Communication Technology Hanoi, 2022 Declaration of Authorship and Topic Sentences Personal information Full name: La Van Quan Phone number: 039 721 1659 Email: Quan.LV202335M@sis.hust.edu.vn Major: Data Science and Artificial Intelligence Topic An Efficient, On-demand Charging for WRSNs Using Fuzzy Logic and Q-Learning Contributions • Propose a Fuzzy logic-based algorithm that determines the energy level to be charged to the sensors • Introduce a new method that optimizes the optimal charging time at each charging location to maximize the number of alive sensors • Propose Fuzzy Q-charging, which uses Q-learning in its charging scheme to guarantee the target coverage and connectivity Declaration of Authorship I hereby declare that my thesis, titled An Efficient, On-demand Charging for WRSNs Using Fuzzy Logic and Q-Learning , is the work of myself and my supervisor Dr Nguyen Phi Le All papers, sources, tables, and so on used in this thesis have been thoroughly cited Supervisor confirmation Ha Noi, April 2022 Supervisor Dr Nguyen Phi Le i Acknowledgments I would like to thank my supervisor, Dr Nguyen Phi Le, for her continued support and guidance throughout the course of my Masters’ studies She has been a great teacher and mentor for me since my undergraduate years, and I am proud to have completed this thesis under her supervision I want to thank my family and my friends, who have given me their unconditional love and support to finish my Masters’ studies Finally, I would like to again thank Vingroup and the Vingroup Innovation Foundation, who have supported my studies through their Domestic Master/Ph.D Scholarship program Parts of this work were published in the paper Q-learning-based, Optimized On-demand Charging Algorithm in WRSN by La Van Quan, Phi Le Nguyen, Thanh-Hung Nguyen, and Kien Nguyen in the Proceedings of the 19th IEEE International Symposium on Network Computing and Applications, 2020 La Van Quan was funded by Vingroup Joint Stock Company and supported by the Domestic Master/Ph.D Scholarship Programme of Vingroup Innovation Foun-dation (VINIF), Vingroup Big Data Institute, code VINIF.2020.ThS.BK.03 ii Abstract In recent years, Wireless Sensor Networks (WSNs) have attracted great attention worldwide WSNs consist of sensor nodes deployed on an surveillance area to monitor and control the physical environment In WSNs, every sensor node needs to perform several important tasks, two of which are sensing and communication Every time the above tasks are performed, the sensor’s energy will be lost over time Therefore some sensor nodes may die A sensor node is considered dead when it runs out of energy Correspondingly, the lifetime of WSNs is defined as the time from the start of operation until a sensor dies [1] Thus, one of the important issues to improve the quality of WSNs is to maximize the life of the network In classical WSNs, sensor nodes have fixed energy and always degrade over time The limited battery capacity of the sensor is always a "bottleneck" that greatly af-fects the life of the network To solve this problem, Wireless Rechargeable Sensor Networks (WRSNs) were born WRSNs include sensors equipped with battery charg-ers and one or more mobile chargers (Mobile Chargers (MC)) responsible for adding power to the sensors In WRSNs, MCs move around the network, stopping at spe-cific locations (called charging sites) and charging the sensors Thus, it is necessary to find a charging route for MC to improve the lifetime of WRSNs [2], [3] Keywords: Wireless Rechargeable Sensor Network, Reinforcement Learning, Q-Learning, Network Lifetime Fuzzy Author La Van Quan iii Logic, Contents List of Figures List of Tables vi vii Introduction 1.1 Problem overview 1.2 Thesis contributions 1.3 Thesis structure Theoretical Basis 2.1 Wireless Rechargeable Sensor Networks 2.2 Q-learning 2.3 Fuzzy Logic Literature Review 10 3.1 Related Work 3.2 Problem definition Fuzzy Q-charging algorithm 10 11 13 4.1 4.2 4.3 4.4 Overview State space, action space and Q table Charging time determination Fuzzy logic-based safe energy level determination 4.4.1 Motivation 4.4.2 Fuzzification 4.4.3 Fuzzy controller 4.4.4 Defuzzification 4.5 Reward function 4.6 Q table update Experimental Results 13 13 15 16 16 17 17 18 21 22 24 5.1 Impacts of parameters 5.1.1 Impacts of 5.1.2 Impacts of 25 25 26 iv 5.2 Comparison with existing algorithms 5.2.1 Impacts of the number of sensors 5.2.2 Impacts of the number of targets 5.2.3 Impacts of the packet generation frequency 5.2.4 Non-monitored targets and dead sensors over time Bibliography v 27 27 28 28 30 34 List of Figures 2.1 2.2 2.3 2.4 3.1 A wireless sensor network A sensor structure Network model Q-learning overview Network model 5 12 4.1 The flow of Fuzzy Q-learning-based charging algorithm 14 4.2 4.3 4.4 5.1 Illustration of the Q-table Fuzzy input membership functions Fuzzy output membership function Impact of on the network lifetime 14 18 19 25 5.2 5.3 5.4 5.5 5.6 5.7 Impact of on the network lifetime Network lifetime vs the number of sensors Network lifetime vs the number of targets Network lifetime vs the packet generation frequency Comparison of non-monitored targets over time Comparison of dead sensors over time 26 27 29 29 30 31 vi List of Tables 4.1 Input variables with their linguistic values and corresponding membership function 4.2 Output variable with its linguistic values and membership function 4.3 Fuzzy rules for safe energy level determination 4.4 Inputs of linguistic variables 4.5 Fuzzy rules evaluation 5.1 System parameters vii 18 18 19 19 20 25 Chapter Introduction 1.1 Problem overview Wireless Sensor Networks (WSNs) have found various applications, such as air quality monitoring, environmental management, etc., [4, 5] A WSN typically includes many battery-powered sensor nodes, monitoring several targets, and sending sensed data to a base station for further processing In the WSNs, it is necessary to provide sufficient monitoring quality surrounding the targets (i.e., guarantee-ing target coverage) Moreover, the WSNs need to have adequate capacity for the communication between the sensors and base station (i.e., ensuring connectivity) [6][7][8] The target coverage and connectivity are severely affected by the depletion of the battery on sensor nodes When a node runs out of battery, it becomes a dead node without sensing and communication capability, damaging the whole network in consequence Wireless Rechargeable Sensor Networks (WRSNs) leverages the advan-tages of wireless power transferring technology to solve that critical issue in WSNs A WRSN uses a mobile charger (MC) to wirelessly compensate for a rechargeable battery’s energy consumption on a sensor node, aiming to guarantee both the target coverage and connectivity In a normal operation, the MC moves around the networks and performs charging strategies, which can be classified into the periodic [9][1][10][11][12] or ondemand charging [13][2][14][15] [16][17][18] In the former, the MC, with a predefined trajec-tory, stops at charging locations to charge the nearby sensors’ batteries In the latter, the MC will move and charge upon receiving requests from the sensors, which have the remaining energy below a threshold The periodic strategy is limited since it can-not adapt to the sensors’ energy consumption rate dynamic On the contrary, the on-demand charging approach potentially deals with the uncertainty of the energy consumption rate Since a sensor with a draining battery triggers the on-demand op-eration, the MC’s charging strategy faces a new time constraint challenge The MC needs to handle two crucial issues: deciding the next charging location and staying period at the location Although many, the existing on-demand charging schemes in the literature face two serious problems The first one is the consideration of the same role for the sen-sor nodes in WRSNs That is somewhat unrealistic since, intuitively, several sensors, depending on their locations, significantly impact the target coverage and the connectivity than others Hence, the existing charging schemes may enrich unnecessary sensors’ power while letting necessary ones run out of energy, leading to charging algorithms’ inefficiency It is of great importance to take into account the target coverage and connectivity simultaneously The second problem is about the MC’s charging amount, which is either a full capacity (of sensor battery) or a fixed amount of energy The former case may cause: 1) a long waiting time of other sensors stay-ing near the charging location; 2) quick exhaustion of the MC’s energy In contrast, charging a too small amount to a node may lead to its lack of power to operate until the next charging round Therefore, the charging strategy should adjust the transferred energy level dynamically following the network condition 1.2 Thesis contributions Motivated by the above, this thesis propose a novel on-demand charging scheme for WRSN that assures the target coverage and connectivity and adjusts the energy level charged to the sensors dynamically My proposal, named Fuzzy Q-charging, aims to maximize the network lifetime, which is the time until the first target is not monitored First, this work exploit Fuzzy logic in an optimization algorithm that determines the optimal charging time at each charging location, aiming to maximize the numbers of alive sensors and monitoring targets Fuzzy logic is used to cope with network dynamics by taking various network parameters into account during the determination process of optimal charging time Second, this thesis leverage the Q-learning technique in a new algorithm that selects the next charging location to maximize the network lifetime The MC maintains a Q-table containing the charging locations’ Q-values representing the charging locations’ goodness The Q-values will be updated in a realtime manner whenever there is a new charging request from a sensor I design the Qvalue to prioritize charging locations at which the MC can charge a node depending on its critical role After finishing tasks in one place, the MC chooses the next one, which has the highest Q-value, and determines an optimal charging time The main contributions of the paper are as follows • This thesis propose a Fuzzy logic-based algorithm that determines the energy level to be charged to the sensors The energy level is adjusted dynamically following the network condition • Based on the above algorithm, this thesis introduce a new method that optimizes the optimal charging time at each charging location It considers sev2