VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY PHAM THANH NAM REGION-OF-INTEREST 3D VIDEO CODING BASED ON DEPTH IMAGES MASTER THESIS OF INFORMATION TECHNOLOGY Hanoi - 2015 VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF ENGINEERING AND TECHNOLOGY PHAM THANH NAM REGION-OF-INTEREST 3D VIDEO CODING BASED ON DEPTH IMAGES Major: Computer Science Code: 60 48 01 01 MASTER THESIS OF INFORMATION TECHNOLOGY SUPERVISOR: Dr Le Thanh Ha Hanoi - 2015 Originality Statement I hereby declare that this submission is my own work and to the best of my knowledge it contains no materials previously published or written by another person, or substantial proportions of material which have been accepted for the award of any other degree or diploma at University of Engineering and Technology (UET/Coltech) or any other educational institution, except where due acknowledgment is made in the thesis Any contribution made to the research by others, with whom I have worked at UET/Coltech or elsewhere, is explicitly acknowledged in the thesis I also declare that the intellectual content of this thesis is the product of my own work, except to the extent that assistance from others in the project’s design and conception or in style, presentation and linguistic expression is acknowledged Date: Signed: i Abstract Due to characteristics of human visual system, people usually focus more on a specific region named Region-of-Interest (ROI) of a video frame, rather than watch the whole frame In addition, ROI-based video coding can also help to e↵ectively reduce the number of encoding bitrates required for video transmission over networks, especially for the 3D-TV transmissions Therefore, in this work, we propose a novel ROI-based bit allocation method which can adaptively extract and increase the visual quality of ROI while saving a huge number of encoding bitrates for video data In the proposed method, we first detect and extract ROI based on the depth information obtained from 3D-TV video coding sequences Then, based on the extracted ROI, a novel bit allocation scheme is performed to solve the rate-distortion optimization problem, in which the higher priority bitrates are adaptively assigned to ROI while the total encoding bitrates of video frames are kept satisfying all constraints required by the R-D optimization Experimental results show that the proposed method can provide not only higher peak signal-to-noise ratio performance but also save up to 13-26% encoding bitrates compared to other conventional methods ii Acknowledgements First and foremost, I would like to express my deepest gratitude to my supervisor, Dr Le Thanh Ha, for his patient guidance and continuous supports throughout the years He always appears when I need help, and responds to queries so helpfully and promptly I also appreciate Dr Dinh Trieu Duong for giving me some useful advice I would like to give my honest appreciation to my best friends at University of Engineering and Technology for whatsoever they did for me I sincerely acknowledge the Vietnam National University, Hanoi and especially, 102.01-2012.36 project (Coding and communication of multiview video plus depth for 3D Television Systems) for supporting finance to my master study Finally, this thesis would not have been possible without the support and love of my family Thank you! iii Table of contents Originality Statement i Abstract ii Acknowledgements iii Table of contents iv List of figures vi List of tables vii List of abbreviations viii Introduction 1.1 Motivation 1.2 Proposed approach summary 1.3 Thesis structure 1 Related works 2.1 ROI detection 2.1.1 Unsupervised detection method 2.1.2 Supervised detection method 2.2 ROI tracking 2.3 Rate control and ROI video coding 6 8 ROI detection and tracking 3.1 ROI detection 3.1.1 ROI region extraction 3.1.2 The ROI boundary expansion 11 13 14 16 iv TABLE OF CONTENTS 3.2 v 3.1.3 ROI-o↵ problem and solution 20 ROI tracking 21 Rate control model and ROI coding 25 4.1 Bit allocation 26 4.2 Calculating QP 30 Experimental results 32 Conclusion 37 List of Figures 1.1 1.2 1.3 1.4 The general video transmission system Conventional video compression ROI video coding results Example of bitrate fluctuation vs target bitrate 2 3 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 Framework of proposed ROI video coding method A color frame of break-dancers video sequence A depth frame of break-dancers video sequence Illustration of floodfill algorithm in ROI detection The ROI is expanded surrounding its border Color, depth and extracted ROI frames in ballet sequence Color, depth and extracted ROI frames in break-dancers sequence The floor is misunderstood as ROI The correctly extracted ROI Block-matching Spiral searching 12 13 14 16 17 18 19 20 21 22 22 4.1 PSNR comparison between color and depth video of Ballet sequence (view 0) 26 5.1 5.2 5.3 5.4 PSNR PSNR PSNR PSNR comparison of Ballet sequence (view 0) with low target bitrate comparison of Ballet sequence (view 0) of ROI comparison of Ballet sequence (view 0) of ROI comparison of Break-dancers sequence (view 1) vi 34 34 35 35 List of Tables 4.1 Summary of notations 28 5.1 5.2 5.3 Results of ROI detection and tracking 33 QP values for all regions in specific cases 36 Bitrate reduction comparison 36 vii List of abbreviations 3D 3-dimensional 1, 4, 5, 9, 25 dB Decibel 33–35 GOP Group Of Pictures 2, 8, HEVC High Efficiency Video Coding HVS Human Visual System 1, 8, 29 JM Joint Model 10, 32, 34 JVT Joint Video Team 32 MB Macroblock 4, 8, 9, 11, 17, 22, 23, 25, 28–31 MSE Mean Square Error 33 PSNR Peak Signal to Noise Ratio 10, 32–35 QP Quantization Parameter 2, 4, 5, 9, 10, 13, 17, 25, 30–32, 34, 36, 37 RC Rate Control 2, 4–6, 9, 25, 26, 37 RD Rate Distortion ROI Region-of-Interest 1, 2, 4–11, 13, 14, 16, 17, 20, 21, 23–29, 31–34, 36, 37 RQ Rate Quantization SAD Sum of Absolute Di↵erences 22, 23 SRC Scalable Rate Control 25 viii