Motion analysis from encoded video bitstream

VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY NGUYEN MINH HOA MOTION ANALYSIS FROM ENCODED VIDEO BITSTREAM MASTER’S THESIS HA NOI – 2018 VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY NGUYEN MINH HOA MOTION ANALYSIS FROM ENCODED VIDEO BITSTREAM Major: Computer Science MASTER’S THESIS Supervisor: Dr Do Van Nguyen Co-Supervisor: Dr Tran Quoc Long HA NOI - 2018 i AUTHORSHIP “I hereby declare that the work contained in this thesis is of my own and I have not submitted this thesis at any other institution in order to obtain a degree To the best of my knowledge and belief, the thesis contains no materials previously published or written by another person other than those listed in the bibliography and identified as references.” Signature: ……………………………………………… i SUPERVISOR’S APPROVAL “I hereby approve that the thesis in its current form is ready for committee examination as a requirement for the Master of Computer Science degree at the University of Engineering and Technology.” Signature: ……………………………………………… Signature: ……………………………………………… i ACKNOWLEDGMENTS First of all, I would like to express special gratitude to my supervisors, Dr Do Van Nguyen and Dr Tran Quoc Long, for their enthusiasm for instructions, the technical explanation as well as advices during this project I also want to give sincere thanks to Assoc Prof Dr Ha Le Thanh, Assoc Prof Dr Nguyen Thi Thuy for the instructions as well as the background knowledge for this thesis And I would like to also thank my teachers, my friends in Human Machine Interaction Lab for their support Thank my friends, my colleagues in the project "Nghiên Cứu Cơng Nghệ Tóm Tắt Video", and project “Multimedia application tools for intangible cultural heritage conservation and promotion”, project number ĐTDL.CN-34/16 for their working and support Last but not least, I want to thank my family and all of my friends for their motivation and support as well They stand by and inspire me whenever I face the tough time TABLE OF CONTENTS AUTHORSHIP .i SUPERVISOR’S APPROVAL ii ACKNOWLEDGMENTS iii TABLE OF CONTENTS ABBREVIATIONS .3 List of Figures .4 List of Tables .5 INTRODUCTION .6 CHAPTER LITERATURE REVIEW Moving object detection in the pixel domain Moving object detection in the compressed domain 10 1.2.1 Motion vector approaches 11 1.2.2 Size of Macroblock approaches 13 Chapter Summarization .14 CHAPTER METHODOLOGY 15 Video compression standard h264 .15 2.1.1 H264 file structure .15 2.1.2 Macroblock 18 2.1.3 Motion vector 19 Proposed method 21 2.2.1 Process video bitstream .21 2.2.2 Macroblock-based Segmentation 22 2.2.3 Object-based Segmentation 24 2.2.4 Object Refinement .28 Chapter Summarization .28 CHAPTER RESULTS 30 The moving object detection application .30 3.1.1 The process of application 31 3.1.2 The motion information .34 3.1.3 Synthesizing movement information 35 3.1.4 Storing Movement Information 36 Experiments 36 3.2.1 Dataset 36 3.2.2 Evaluation methods .40 3.2.3 Implementations 41 3.2.4 Experimental results 41 Chapter Summarization .44 CONCLUSIONS .45 List of of author’s publications related to thesis .46 REFERENCES 47 ABBREVIATIONS MB Macroblock MV Motion vector NALU Network Abstraction Layer Unit RBSP Raw Byte Sequence Payload SODB String Of Data Bits List of Figures Figure 1.1 The process of moving object detection with data in the pixel domain 10 Figure 1.2 The process of moving object detection with data in the compressed domain .11 Figure 2.1 The structure of a H264 file 15 Figure 2.2 RBSP structure 16 Figure 2.3 Slide structure 18 Figure 2.4 Macroblock structure 18 Figure 2.5 The motion vector of a Macroblock 20 Figure 2.6 The process of moving object detection method 22 Figure 2.7 Skipped Macroblock .23 Figure 2.8 (a) An outdoor and in-door frames (b) The "size-map" of frames, (c) The "motion-map" of frames .24 Figure 2.9 Example about the “consistent” of motion vector 26 Figure 3.1 The implementation process of the approach 33 Figure 3.2 Data struct to storage motion information 35 Figure 3.3 Example frames of test videos 37 Figure 3.4 Example frames and their ground truth 39 Figure 3.5 An example frame of Pedestrians (a) and ground truth image (b) 40 List of Tables Table 2.1 NALU types .16 Table 2.2 Slide types 17 Table 3.1 The information of test videos 38 Table 3.2 The information of test sequences in group 39 Table 3.3 The performance of two approachs with Pedestrians, PETS2006, Highway, and Office 42 Table 3.4 The experimental result of Poppe’s approach on 2nd group .42 Table 3.5 The experimental result of proposed method on 2nd group 43 ... NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY NGUYEN MINH HOA MOTION ANALYSIS FROM ENCODED VIDEO BITSTREAM Major: Computer Science MASTER’S THESIS Supervisor: Dr Do Van Nguyen... vision are taken from scientific articles related to the video analysis problem on the compression domain, determine the motion form on the compression domain of the video The videos for test... server for analysis The video analysis process needs a lot of features to describe different aspects of vision Typically, these features are extracted from the pixel values of each video frame