Implement step and heading estimation using sensors on smartphones

Vietnam National University Ho Chi Minh City HCM University of Technology Faculty of Computer Science and Engineering ——————– * ——————— Graduate Thesis Implement Step and Heading Estimation Using Sensors on Smartphones Council : Computer Engineering Supervisors : Pham Hoang Anh, Ph.D Le Thanh Van, Ph.D Reviewer : Pham Quoc Cuong, Assoc Prof Ph.D Students : Tran Quang Dai (1552089) Nguyen Tien Loc (1552214) Ho Chi Minh City, 10/2021 TRƯỜNG ĐẠI HỌC BÁCH KHOA KHOA KH & KT MÁY TÍNH CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM Độc lập - Tự - Hạnh phúc -Ngày 06 tháng 08 năm 2021 PHIẾU CHẤM BẢO VỆ LVTN (Dành cho người hướng dẫn/phản biện) Họ tên SV: Trần Quang Đại, Nguyễn Tiến Lộc MSSV: 1552089, 1552214 Ngành (chuyên ngành): Kỹ thuật Máy tính Đề tài: Implement Step and Heading Estimation Using Sensors on Smartphones Họ tên người hướng dẫn/phản biện: Phạm Hoàng Anh, Lê Thanh Vân Tổng quát thuyết minh: Số trang: Số chương: Số bảng số liệu Số hình vẽ: Số tài liệu tham khảo: Phần mềm tính tốn: Hiện vật (sản phẩm) Tổng quát vẽ: - Số vẽ: Bản A1: Bản A2: Khổ khác: - Số vẽ vẽ tay Số vẽ máy tính: Những ưu điểm LVTN: - Basically, the students have successfully studied and implemented an algorithm for step and heading estimation using only built-in sensors on smartphones, which is still a challenge to achieve high accuracy regarding this topic because of device dependency The students have also performed various experiments to evaluate their implementation, and the accuracy errors are acceptable - The students have demonstrated their capability in self-studying and then apply new knowledge and techniques to implement the proposed system - The report has been well written However, the students should revise the details Những thiếu sót LVTN: - The students should investigate and implement various algorithms for having a comparative study, which helps to choose a suitable algorithm for a specific application - The mobile application is just for testing algorithms since it is still simple without UI as an end-user application Đề nghị: Được bảo vệ o Bổ sung thêm để bảo vệ o Câu hỏi SV phải trả lời trước Hội đồng: 10 Đánh giá chung (bằng chữ: giỏi, khá, TB): Không bảo vệ o Điểm : Trần Quang Đại 8.5/10 Nguyễn Tiến Lộc 7.5/10 Ký tên (ghi rõ họ tên) Phạm Hoàng Anh TRƯỜNG ĐẠI HỌC BÁCH KHOA KHOA KH & KT MÁY TÍNH CỘNG HỊA XÃ HỘI CHỦ NGHĨA VIỆT NAM Độc lập - Tự - Hạnh phúc -Ngày 07 tháng năm 2021 PHIẾU CHẤM BẢO VỆ LVTN (Dành cho người phản biện) Trần Quang Đại – 1552089 – Ngành Kỹ thuật Máy tính Nguyễn Tiến Lộc – 1552214 – Ngành Kỹ thuật Máy tính Đề tài: Implementation of Step and Heading Estimation Using Sensors on Smartphones Họ tên người hướng dẫn/phản biện: Phạm Quốc Cường Tổng quát thuyết minh: Số trang: Số chương: Số bảng số liệu Số hình vẽ: Số tài liệu tham khảo: Phần mềm tính tốn: Hiện vật (sản phẩm) Tổng quát vẽ: - Số vẽ: Bản A1: Bản A2: Khổ khác: - Số vẽ vẽ tay Số vẽ máy tính: Những ưu điểm LVTN: - Students proposed an approach to estimating the steps and heading of pedestrians - Students implemented the proposal well and tested the proposed systems with different scenarios - The report is adequately written, although some sections need more details explained Họ tên SV: Những thiếu sót LVTN: - Students need to improve the algorithm so that users can keep their mobile phones in their pockets instead of holding them in their hands - Students should more tests and compare results with other devices (like a smartwatch) or mobile apps to prove the concept Besides, students should conduct testing with different working conditions such as running or biking to verify the application's ability - Teamwork is not defined well when the contributions of students are not balanced Đề nghị: Được bảo vệ o Bổ sung thêm để bảo vệ o Không bảo vệ o câu hỏi SV phải trả lời trước Hội đồng: a Would you please explain why users must hold their phones in their hands while walking to measure steps? b How can the algorithm recognize if users are walking or biking at low velocity? 10 Đánh giá chung (bằng chữ: giỏi, khá, TB): - Trần Quang Đại - Nguyễn Tiến Lộc Điểm: 8.5/10 Điểm: 7.5/10 Ký tên (ghi rõ họ tên) Phạm Quốc Cường Declaration We declare that this thesis represents our own work, except where due reference is made Any contribution made to the research by others, with whom we have worked at Bach Khoa university or elsewhere, is explicitly acknowledged in the thesis We also declare that this thesis has not been previously included in a thesis or dissertation submitted to this or any other institution for a degree, diploma or other qualifications Tran Quang Dai, Nguyen Tien Loc Acknowledgement In order to complete this graduate thesis, our group have received a great deal of support and assistance First, we would like to express our gratitude to our supervisors, PhD Pham Hoang Anh and PhD Le Thanh Van They are the main direct instructors, providing materials as well as monitoring the progress of the topic and providing support when our group is in trouble They kindly supported us and continuous advice went through the process of completion of our thesis We could not forget all the effort and dedication of lecturers in the Department of Computer Science and Engineering in particular, as well as the Ho Chi Minh City University of Technology in general We would particularly thank our form teacher, PhD Pham Quoc Cuong, who has supported our class for the past few years The knowledge received from the teachers is very valuable and useful, greatly supporting us in completing this graduation thesis Lastly, our deepest thanks come to our family, relatives, and friends during the time we struggled with this thesis Their kind help, care, encouragement, and motivation gave us strength and lifted us up both physically and mentally Tran Quang Dai, Nguyen Tien Loc i Abstract Positioning estimation systems play an important role in everyday life The global positioning system (GPS) is one of the most popular positioning systems, which is highly efficient for outdoor environments However, in indoor scenarios, GPS signal reception is weak Therefore, achieving good position estimation accuracy is a challenge To overcome this challenge, it is necessary to utilize position estimation systems for indoor localization This thesis aims to implement a step and heading system (SHS) for estimating position in indoor environments using handheld smartphones SHS is a combination of three components: step detection, step length estimation, and heading estimation ii Contents Declaration Acknowledgement Abstract i ii List of Figures Terms vii viii Introduction 1.1 Topic Introduction 1.2 Thesis Objective and Scope 1.3 Thesis Structure Literature Review Theoretical Basis 10 3.1 3.2 Inertial Measurement Units (IMUs) 10 3.1.1 Accelerometer 11 3.1.2 Gyroscope 11 3.1.3 Magnetometer 12 Pedestrian Dead Reckoning 12 iii Contents 3.3 3.4 Coordinates systems 14 3.3.1 Smartphone/Device Coordinates System 14 3.3.2 Ground/Earth Coordinates System 15 3.3.3 Coordinates Transformation 16 Step Detection 17 3.4.1 Peak Detection 18 3.4.2 Zero-crossing Based Step Detection 18 3.4.3 Peak-trough based step detection 19 3.5 Step Length Estimation 19 3.6 Heading Estimation 22 3.7 3.6.1 Gyroscope based heading estimation 22 3.6.2 Principal Component Analysis for heading estimation 22 Position Estimation 23 Methodology 4.1 4.2 24 Introduction 24 4.1.1 Advantages 25 4.1.2 Limitations 25 4.1.3 Expo framework 4.1.4 Device Motion from Expo 26 25 Planning and Analysis 26 4.2.1 Step detection 27 4.2.2 Step length estimation 30 4.2.3 Heading estmation 30 4.3 Design 31 4.4 Implementation 33 4.4.1 Installation 33 iv Contents 4.4.2 DeviceMotion Implementation 34 4.4.3 Coordinates transformation 35 4.4.4 Data filtering 38 4.4.5 Step and Heading Detection 41 4.4.6 Step Detection 41 4.4.7 Step Length Estimation 42 4.4.8 Heading determination 43 4.4.9 Position Estimation 43 Evaluation 46 5.1 Results 46 5.2 Test Scenarios 50 5.2.1 Test Scenario 50 5.2.2 Test Scenario 51 5.2.3 Test Scenario 52 5.2.4 Test Scenario 53 5.2.5 Test Scenario 54 5.2.6 Evaluation 55 Conclusion and future work 57 6.1 The contribution of the thesis 57 6.2 Result 58 6.3 Future work 58 v Evaluation 5.1 Results Two main screens of our application as in Fig.5.1 and 5.2 46 5.1 Results Figure 5.1: Calibrate Step Length Screen 47 5.1 Results Figure 5.2: Main Screen In these screens: • On/Off Button: Start and Stop application • Export Data Button: Export data to other applications in order to display trajectory 48 5.1 Results • Reset Button: Reset the step, x and y Figure 5.3: Export data to other applications or excel file 49 5.2 Test Scenarios Figure 5.4: Exported Data 5.2 Test Scenarios 5.2.1 Test Scenario • The phone is held parallel to the walking direction • walking with a trajectory of edges of a rectangle, with a length of 8m and a width of 2.5m 50 5.2 Test Scenarios Result Figure 5.5: Result Scenario 5.2.2 Test Scenario • The phone is held vertically • Walking with a trajectory of edges of a rectangle, with a length of 5m and a width of 2.5m Result 51 5.2 Test Scenarios Figure 5.6: Result Scenario 5.2.3 Test Scenario • The phone is held perpendicular to the walking direction (landscape mode) • Walking with a trajectory is edges of a rectangle, with a length of 6m and a width of 1.8m Result 52 5.2 Test Scenarios Figure 5.7: Result Scenario 5.2.4 Test Scenario • The phone is held in an arbitrary orientation • Walking with a trajectory is edges of a rectangle, with length of and width of 1.8m and reverse Result 53 5.2 Test Scenarios Figure 5.8: Result Scenario 5.2.5 Test Scenario • The phone is held in an arbitrary orientation • Walking for more than 100 steps and returning to the initial position Result 54 5.2 Test Scenarios Figure 5.9: Result Scenario The results of those scenarios above prove that our system successfully implemented what we expected in Chapter 1.2 Our system returns position nearly in real-time (once per second), with 115/116 steps recognized (scenario 5) and final position deviation final position from initial position is less than 1m The main errors are caused by the estimation of each step as well as the fact that the device cannot be held completely stable when walking 5.2.6 Evaluation Advantages: • The system is easy to implement • Be able to build on both iOS and Android • The measurements are accurate enough to be used in other systems • Can be combined with other system to increase accuracy • The system can work with any orientation of smartphone 55 5.2 Test Scenarios • The system returns results nearly in real-time Disadvantages: • The system sometimes lags due to the performance of React Native and high re-render frequency • The user interface is not completed yet • Cannot handle the case where the user changes the orientation of their smartphone when walking • It is possible to trick the algorithm of step detection by standing still and accelerating the accelerometer • The accuracy highly depends on step length calibration • Cannot display trajectory in the fixed Earth’s coordinate system 56 Conclusion and future work 6.1 The contribution of the thesis An indoor positioning strategy is presented in this thesis, which relies solely on the built-in inertial sensors within smartphones Sub problems such as step detection, step length estimates, and heading angle estimations have all been explored in this approach The result of this strategy could serve as a complementary data source to improve the stabilization and accuracy of the current positioning system Since the PDR scheme is sourced from path integration, theoretically, the positioning result is subject to cumulative errors and each tiny deviation would be amplified over time This strategy is hardly qualified for long-term positioning and navigation solely Ideally, this IMU-based scheme could cooperate with other communication signals-based schemes On the one hand, other schemes could provide the initial position and calibrate the drifts and cumulative errors from the inertial sensors; on the other hand, IMU can extend navigation into areas where mainstream positioning systems are problematic When the external navigation signal, such as GPS or WLAN, is temporarily unavailable or sometimes unreliable, this strategy can be used to maintain the performance of the whole system independently Due to the invulnerability of the IMU, more reliable results of position tracking can be achieved 57 6.2 Result 6.2 Result Our team has successfully implemented a step and heading PDR system based on a handheld smartphone to support other indoor positioning systems To evaluate the performance of the proposed methodologies, several test scenarios have been established The performance of the proposed method was evaluated by a series of experiments This system satisfies time-constraints and provides a position with acceptable mean error However, there are still other cases that need to be solved to improve the system as well as the feasibility of combining it with a more general system 6.3 Future work • Improving step detection algorithms as well as applying motion mode recognition algorithms to handle different motions of the user • Testing other step length estimation algorithm algorithms and improving our chosen method as it uses a data set from other researchers • Combining with the map matching algorithm or other indoor positioning systems to reduce heading drift and get the initial heading • The altitude related information, such as determining on which floor the user is, or whether climbing stairs or riding an elevator, can be analysed from the barometer With the air pressure the relative altimeter could be reckoned and deployed into our system to get 3-D position • The initial pedestrian position is essential, which is also a necessary prerequisite for the PDR method The initial position is entered by the user in most researches However, this is unrealistic for practical applications Because of this shortcoming, the collaboration between other positioning schemes is highly appreciated, because the initial state can be obtained from other alternative information sources 58 Bibliography [1] Woyano and Feyissa Step and Heading System Algorithms for Estimating Position Using Smartphones, IEEE Explore, 8/2020 [2] X Zhu, Q Li and G Chen, APT: Accurate outdoor pedestrian tracking with smartphones 2013 Proceedings IEEE INFOCOM, 2013, pp 2508-2516, doi: 10.1109/INFCOM.2013.6567057 [3] Sun Yi, An Indoor Positioning Strategy based on Inertial Sensors, 2016 [4] Julang Ying, Kaveh Pahlavan and Liyuan Xu Using Smartphone Sensors for Localization in BAN, Published: February 27th 2019, DOI: 10.5772/intechopen.80472 [5] Weisstein, Eric W Rotation Matrix From MathWorld–A Wolfram Web Resource https://mathworld.wolfram.com/RotationMatrix.html [6] Van Thanh Pham, Duc Anh Nguyen, Nhu Dinh Dang, Hong Hai Pham, Van An Tran, Kumbesan Sandrasegaran, Duc-Tan Tran Highly Accurate Step Counting at Various Walking States Using Low-Cost Inertial Measurement Unit Support Indoor Positioning System, Published: : 20 September 2018 [7] Ling Pei, Ruizhi Chen, Jingbin Liu, Wei Chen, Heidi Kuusniemi, Tomi Tenhunen, Tuomo Kröger, Yuwei Chen, Helena Leppäkoski, Jarmo Takala Motion Recognition Assisted Indoor Wireless Navigation on a Mobile Phone, Published: September 2010 [8] Ken Shoemake Animating Rotation with Quaternion Curves Proc, Published: 1985, pp 245 - 254 59 Bibliography [9] Fuqiang Gu, Kourosh Khoshelham, Jianga Shang, Fangwen Yu, and Zhuo Wei Robust and Accurate Smartphone-Based Step Counting for Indoor Localization, Published: March 2017 [10] Jun Yang Toward Physical Activity Diary: Motion Recognition Using Simple Acceleration Features with Mobile Phones, Published: January 2009, DOI: 10.1145/1631040.1631042 [11] Yi Sun, Huaming Wu, Jochen Hermann Schiller A Step Length Estimation Model for Position Tracking Published: June 2015 [12] DAvid Mizell Using Gravity to Estimate Accelerometer Orientation, Published: January 2003, DOI: 10.1109/ISWC.2003.1241424 [13] Nirupam Roy, He Wang, Romit Roy Choudhury I am a smartphone and I can tell my user’s walking direction, Published: June 2014, DOI: 10.1145/2594368.2594392 [14] Nirupam Roy WalkCompass: Finding Walking Direction Leveraging Smartphone’s Inertial Sensors, Published: 2013 60 ... utilize position estimation systems for indoor localization This thesis aims to implement a step and heading system (SHS) for estimating position in indoor environments using handheld smartphones SHS... combination of three components: step detection, step length estimation, and heading estimation ii Contents Declaration Acknowledgement Abstract i ii List of Figures Terms vii viii Introduction 1.1... Basically, the students have successfully studied and implemented an algorithm for step and heading estimation using only built-in sensors on smartphones, which is still a challenge to achieve high