Weighted pca regression to recover missing markers in human motion data

VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY Kieu Hai Dang Weighted PCA Regression To Recover Missing Markers In Human Motion Data MASTER THESIS Major: Computer Science HANOI - 2021i VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY Kieu Hai Dang Weighted PCA Regression To Recover Missing Markers In Human Motion Data Major: Computer Science Code: 8480101.01 Supervisor: Assoc Prof Dr Le Thanh Ha HANOI - 2021 ABSTRACT “Missing markers problem”, that is, missing markers during a motion capture session, has been raised for many years in Motion Capture field Some algorithms have been proposed but they still show some limitations Recent approaches used Principal Component Analysis (PCA) to create a single basis system from the unmissing frames in the motion sequence However, we are aware that those approaches of PCA have limitations in case of missing multiple gaps with a very long sequence With the rising amount of Motion data, we question how could we use a part of the available Mocapdata to improve the performance of the current state of art algorithms In this study, we aimed to present methods that could utilize the available Mocap dataset to enhance the performance of the current state-of-the-art algorithm We first applied PCA into a collection of samples extracted from the Mocap dataset to gain a set of basis systems Then, We further used least square method to compute a weight vector for the basis systems Finally, interpolation is the synthesis of the basis systems with their weight To test the performance, we analyze our algorithms and compare with original PCA approaches in scenarios of missing gaps in famous Mocap datasets (Carnegie Mellon University Dataset and Motion Capture Database HDM05) Each experiment is conducted over 50 times to guarantee effectiveness Our final results show that my methods outperform the original PCA approaches Index Term: Missing markers problem, MoCap data, principle component analysis i ACKNOWLEDGEMENTS First of all, I would like to express special gratitude to my supervisor – Assoc Prof Dr Le Thanh Ha for his enthusiastic instructions, technical explanations as well as advice during this project I have received a lot of detailed guidance from my supervisor when I faced problems I could haven’t been completed my thesis without the instruction of those from email and weekly meetings I also want to give sincere thanks to Dr Hong Chuan Yu (Bournemouth University, UK) for the instructions as well as the background knowledge for this thesis and the guidance when I worked at Bournemouth University I would like to also thank my colleagues in Human Machine Interaction Lab for their support Last but not least, I want to thank my family and all of my friends for their motivation and support as well They stand by and inspire me whenever I face a tough time ii DECLARATION I hereby declare that the research and experimental results presented in the thesis are my own and under the guidance of Assoc Prof Dr Le Thanh Ha and Dr Hong Chuan Yu I confirm that the work has never been used as any other master thesis or other degrees of any university iii TABLE OF CONTENTS ABSTRACT i ACKNOWLEDGEMENTS ii DECLARATION iii TABLE OF CONTENTS iv LIST OF FIGURES .vi LIST OF TABLES viii TABLE OF ABBREVIATIONS CHAPTER INTRODUCTION 1.1 Introduction 1.2 Contributions and Thesis’s layout CHAPTER RELATED WORK 2.1 Polynomial interpolation 2.2 Low-rank decomposition techniques 2.3 Model-based methods 2.4 PCA method 2.5 Deep learning 11 CHAPTER THE METHOD 13 3.1 Pre-processing 13 iv 3.2 Stacked PCA 14 3.3 Weighted PCA 16 3.4 Locally Weighted PCA 20 CHAPTER EXPERIMENT RESULTS 22 4.1 Experiment setup 22 4.2 Results 24 4.2.1 Missing a single joint 24 4.2.2 Missing multiple markers 27 4.2.3 Performance stability 29 CHAPTER CONCLUSIONS 32 THESIS-RELATED PUBLICATIONS 34 REFERENCES 35 APPENDIX 39 v LIST OF FIGURES Figure 1: Human in Xsens suit and motion analysis Figure 2: Joints in Mocap dataset Figure 3: Examples of a motion sequence with missing marker The missing marker has coordinate (0,0) is indicated in black circle Figure 4: Block diagram of the Tits method [9], probabilistic averaging method of several recovery models .7 Figure 5: Peng et al [28] nonegative matrix factorization pipeline Figure 6: Gloersen's method of using PCA to recovery missing gap 11 Figure 7: : Illustration of bi-directional attention network proposed by Qiongjie Cui [31] 12 Figure 8: Holden et al [32] states the comparision of various techniques in his work 12 Figure 9: Logical flow of proposed approaches .13 Figure 10: Example of missing gaps in a testing sample YELLOW indicates gaps 22 Figure 11: Comparison of the reconstruction for missing single gap at the LKNE marker in the CMU dataset Dotted black lines represent the original trajectory of missing marker, black lines represent the remaining trajectory in the LKNE marker, red lines represent our method reconstruction 25 Figure 12: Comparison of the reconstruction for missing single gap at LSHO and LWA marker in CMU dataset Dotted black lines represent the original trajectory of vi the missing marker, black lines represent the remaining trajectory in the LWA marker, the red lines represent our method reconstruction 26 Figure 13: Boxplot of reconstruction error for missing 3, 6, markers in a testing sample using CMU Dataset 28 Figure 14: Example reconstruction for missing multiple gaps at the same time frame in the right ankle marker .29 Figure 15: Performance stability along with the location of missing gap Reconstruction errors of our methods at missing single marker case for LKNE, LSHO in CMU and HDM dataset .30 Figure 16: Example of missing the marker LWA in motion sequence in the CMU data 33 Figure 17: LWPCA's reconstruction 33 vii LIST OF TABLES Table 4-1: Mean reconstruction error for missing a single gap .24 Table 4-2: Average reconstruction error for missing multiple gaps 27 Table 4-3: Running time per sample (s) 31 viii trajectory at the right ankle marker in Figure 14 It is easy to see that the outputs of our approaches for X and Y coordinate are close to the original one However, for the Z coordinate, there is a long distance between the interpolated position and the ground truth at some middle frames Figure 14: Example reconstruction for missing multiple gaps at the same time frame in the right ankle marker 4.2.3 Performance stability I further explore the performance stability of my approaches through 30 tests by visualizing the recovery errors along with the location of the missing gap in a testing sample It can be noticed from Figure 15 that my methods are affected when the gap starts missing at the first frame in some cases However, there are only slight changes 29 in the reconstruction error when the gap starts missing at the middle of the testing sample In addition, I also inspect the running time, Table shows the mean running time of my approaches It can be noticed that my SPCAs running time is stable when increasing the number of missing markers in a testing sample The WPCA and LWPCA on the other hand, both methods are likely sensitive to the changing number of the missing marker This is because SPCA needs only compute the transition matrix one time for all missing markers while WPCA and LWPCA exploit reconstructing each missing marker separately scheme Figure 15: Performance stability along with the location of missing gap Reconstruction errors of our methods at missing single marker case for LKNE, LSHO in CMU and HDM dataset 30 Table 4-3: Running time per sample (s) PCA SPCA WPCA LWPCA Single 0.07 0.07 0.13 0.136 markers 0.07 0.078 0.477 0.476 markers 0.07 0.076 0.914 0.92 markers 0.07 0.078 1.64 1.648 31 CHAPTER CONCLUSIONS Reconstruction missing trajectories by applying PCA are proved in the recent state-of-the-art papers However, the current state-of-the-art techniques show limitations As mentioned in [5, 23], the reconstruction of missing trajectory relies on the available un-missing frames in sequence motion In our study, we propose methods that could use additional data extracted from a huge Mocap dataset to boost the performance Firstly, our approaches could handle the problem of missing makers in the motion sequence Figure 16 illustrates an example of missing the marker LWA and the reconstruction for the corresponding motion sequence is depicted in Figure 17 Besides, as can be seen from the results, our approaches outperform the current state-of-the-art methods Thank to advance of the additional collection, our methods can obtain a lower reconstruction error Additionally, ours gain a lower variant of error at the missing single marker case than the original PCA method that making our methods are more stable Apart from that, we realize our approaches still encounter a dilemma of selecting the type and number of samples for training data Firstly, selecting style of motion sequences for training affects the performance of our methods Secondly, a large collection of training data will cost the computing time Moreover, the showed results in this study are specific to some datasets For these issues, we will keep studying in the future Furthermore, with the rising ability of computing via GPUs, we strongly believe that we can handle this problem 32 Figure 16: Example of missing the marker LWA in motion sequence in the CMU data The missing marker has coordinate (0,0) Figure 17: LWPCA's reconstruction 33 THESIS-RELATED PUBLICATIONS Li, Z., Yu, H., Kieu, H D., Vuong, T L., & Zhang, J J (2020) PCA-Based Robust Motion Data Recovery IEEE Access, 8, 76980-76990 My contributions in this journal: designing and performing experiments, analyzing algorithms Kieu, H D, Yu, H Locally Weighted PCA Regression To Recover Missing Markers In Human Motion Data (Submitted in PLOS ONE) My contributions in this journal: All 34 REFERENCES Moeslund, Thomas B., Adrian Hilton, and Volker Krüger "A survey of advances in vision-based human motion capture and analysis." Computer vision and image understanding 104.2-3 (2006): 90-126 Hilton, Adrian, Pascal Fua, and Rémi Ronfard "Special issue on modeling people: Vision-based understanding of a Person's shape, appearance, movement, and behaviour." Computer vision and image understanding (Print) 104.2-3 (2006) Rego, Paula, Pedro Miguel Moreira, and Luis Paulo Reis "Serious games for rehabilitation: A survey and a classification towards a taxonomy." 5th Iberian conference on information systems and technologies IEEE, 2010 Zhou, Huiyu, and Huosheng Hu "Human motion tracking for rehabilitation—A survey." Biomedical signal processing and control 3.1 (2008): 1-18 Federolf, Peter Andreas "A novel approach to solve the “missing marker problem” in marker-based motion analysis that exploits the segment coordination patterns in multi-limb motion data." PloS one 8.10 (2013): e78689 Vicon Software https://wwwviconcom/products/software/ 2020 Federolf, Peter Andreas "A novel approach to solve the “missing marker problem” in marker-based motion analysis that exploits the segment coordination patterns in multi-limb motion data." PloS one 8.10 (2013): e78689 Xia, Guiyu, et al "Nonlinear low-rank matrix completion for human motion recovery." IEEE Transactions on Image Processing 27.6 (2018): 3011-3024 Tits, Mickaël, Joëlle Tilmanne, and Thierry Dutoit "Robust and automatic motioncapture data recovery using soft skeleton constraints and model averaging." Plos one 13.7 (2018): e0199744 10 Gløersen, Øyvind, and Peter Federolf "Predicting missing marker trajectories in human motion data using marker intercorrelations." PloS one 11.3 (2016): e0152616 11 Fritsch, Frederick N., and Ralph E Carlson "Monotone piecewise cubic interpolation." SIAM Journal on Numerical Analysis 17.2 (1980): 238-246 35 12 Howarth, Samuel J., and Jack P Callaghan "Quantitative assessment of the accuracy for three interpolation techniques in kinematic analysis of human movement." Computer methods in biomechanics and biomedical engineering 13.6 (2010): 847-855 13 Dorfmüller-Ulhaas, Klaus "Robust optical user motion tracking using a kalman filter." (2007) 14 Li, Lei, et al "Bolero: a principled technique for including bone length constraints in motion capture occlusion filling." (2010) 15 Wu, Qiong, and Pierre Boulanger "Real-time estimation of missing markers for reconstruction of human motion." 2011 XIII Symposium on Virtual Reality IEEE, 2011 16 Wang, Jack M., David J Fleet, and Aaron Hertzmann "Gaussian process dynamical models for human motion." IEEE transactions on pattern analysis and machine intelligence 30.2 (2007): 283-298 17 Li, Lei, et al "Dynammo: Mining and summarization of coevolving sequences with missing values." Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining 2009 18 Huang, Chung-Lin, and Chia-Ying Chung "A real-time model-based human motion tracking and analysis for human-computer interface systems." EURASIP Journal on Advances in Signal Processing 2004.11 (2004): 1-15 19 Liu, Guodong, and Leonard McMillan "Estimation of missing markers in human motion capture." The Visual Computer 22.9 (2006): 721-728 20 Hu, Wenyu, et al "Motion capture data completion via truncated nuclear norm regularization." IEEE Signal Processing Letters 25.2 (2017): 258-262 21 Liu, Xin, et al "Automatic motion capture data denoising via filtered subspace clustering and low rank matrix approximation." Signal Processing 105 (2014): 350362 22 Xia, Guiyu, et al "Human motion recovery jointly utilizing statistical and kinematic information." Information Sciences 339 (2016): 189-205 36 23 Lou, Hui, and Jinxiang Chai "Example-based human motion denoising." IEEE Transactions on Visualization and Computer Graphics 16.5 (2010): 870-879 24 Hou, Junhui, et al "Human motion capture data recovery via trajectory-based sparse representation." 2013 IEEE International Conference on Image Processing IEEE, 2013 25 Tan, Cheen-Hau, Junhui Hou, and Lap-Pui Chau "Human motion capture data recovery using trajectory-based matrix completion." Electronics Letters 49.12 (2013): 752-754 26 Tan, Cheen-Hau, JunHui Hou, and Lap-Pui Chau "Motion capture data recovery using skeleton constrained singular value thresholding." The Visual Computer 31.11 (2015): 1521-1532 27 Lai, Ranch YQ, Pong C Yuen, and Kelvin KW Lee "Motion Capture Data Completion and Denoising by Singular Value Thresholding." Eurographics (Short Papers) 2011 28 Peng, Shu-Juan, et al "Hierarchical block-based incomplete human mocap data recovery using adaptive nonnegative matrix factorization." Computers & Graphics 49 (2015): 10-23 29 Kirk, Adam G., James F O'Brien, and David A Forsyth "Skeletal parameter estimation from optical motion capture data." 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) Vol IEEE, 2005 30 Li, Z., Yu, H., Kieu, H D., Vuong, T L., & Zhang, J J (2020) PCA-Based Robust Motion Data Recovery IEEE Access, 8, 76980-76990 31 Cui, Qiongjie, et al "A Deep Bi-directional Attention Network for Human Motion Recovery." IJCAI 2019 32 Holden, Daniel "Robust solving of optical motion capture data by denoising." ACM Transactions on Graphics (TOG) 37.4 (2018): 1-12 33 Mall, U., Lal, G R., Chaudhuri, S., & Chaudhuri, P (2017) A deep recurrent framework for cleaning motion capture data arXiv preprint arXiv:1712.03380 37 34 Fragkiadaki, K., Levine, S., Felsen, P., & Malik, J (2015) Recurrent network models for human dynamics In Proceedings of the IEEE International Conference on Computer Vision (pp 4346-4354) 38 APPENDIX Applying the weighted least square method to Equation 20: (24) Let * 𝑃#×& = ST\𝐴' − 𝐴(' (𝑔)(𝑈 ( 𝑇' )(𝑔)𝑈 + ] ⊙ 𝑄5 U '," We get 39 (25) (26) Regarding 𝐴𝑥 = 0, the least square solution is the minimal eigenvector 𝐴+ 𝐴 Rewrite the Equation 20: 𝐴' ⊙ 𝑄 = T 𝐴(' (𝑔)\(𝑈 ( 𝑇' )(𝑔)𝑈 + ] ⊙ 𝑄5 , 𝑖 = 1, … , 𝐾 5," Then, embedding the weight W into the Equation 20: (27) Let 40 𝑃3' = T\𝐴(' \𝑈 ( 𝑇3 ](𝑔)𝑈 + ] ⊙ 𝑄5 (28) We may rewrite the Equation 27 as below: (29) Take the first differentiation with respect to 𝛼, (30) Which is rewritten as, (31) 41 For each 𝛼 There are K equations We merge the above linear system into one matrix equation and solve the vector 𝛼 * * 𝛼" T T 𝑋'" (ℎ) 2," '," * + * * ⋯ + 𝛼* T T 𝑋'* (ℎ) 2," '," * = T T 𝑌' (ℎ) (32) 2," '," Where + 𝑋' (ℎ) = \𝑃2' ] 𝑊𝑃3' (33) And + 𝑌' (ℎ) = \𝑃2' ] 𝑊𝐴' ⊙ 𝑄 (34) Again, reorganize the matrix equation into a linear system, 𝛼" 𝐴" + ⋯ + 𝛼* 𝐴* = 𝐵 Where * * 𝐴' = T T 𝑋' (ℎ) 2," '," With ≤ 𝑖 ≤ 𝐾 and 42 (35) * * 𝐵 = T T 𝑌' (ℎ) 2," '," The equation is changed as 𝛼" (𝐴" (𝑖, 𝑗), … , 𝐴* (𝑖, 𝑗)) i … j = 𝐵(𝑖, 𝑗) 𝛼* With ≤ 𝑖, 𝑗 ≤ 𝑛 And then solve 𝛼 by least square method 43 (36) ... of frame that contains missing joints are large 10 Figure 6: Gloersen''s method of using PCA to recovery missing gap 2.5 Deep learning Recovering the missing markers in human motion is stated... for missing multiple gaps CMU Dataset Number of missing markers HDM Dataset PCA SPCA WPCA LWPCA PCA SPCA WPCA LWPCA markers (6%) 16.1621 3.6048 3.1280 3.0099 0.0697 0.0968 0.0468 0.0296 markers. .. reconstruction for missing single gap at the LKNE marker in the CMU dataset Dotted black lines represent the original trajectory of missing marker, black lines represent the remaining trajectory in the LKNE