Nghiên cứu phát triển giải pháp thực thi phần cứng cho bộ mã hóa số học nhị phân thích nghi theo ngữ cảnh ứng dụng trong chuẩn HEVC

MINISTRY OF EDUCATION AND TRANING MINISTRY OF DEFENSE ACADEMY OF MILITARY SCIENCE AND TECHNOLOGY TRAN DINH LAM RESEARCHING ON THE DEVELOPMENT OF HARDWARE IMPLEMENTATION SOLUTION FOR THE CONTEXT-ADAPTIVE BINARY ARITHMETIC CODER IN THE HEVC STANDARD Ph.D THESIS IN ENGINEERING Ha Noi - 2021 MINISTRY OF EDUCATION AND TRANING MINISTRY OF DEFENSE ACADEMY OF MILITARY SCIENCE AND TECHNOLOGY TRAN DINH LAM RESEARCHING ON THE DEVELOPMENT OF HARDWARE IMPLEMENTATION SOLUTION FOR THE CONTEXT-ADAPTIVE BINARY ARITHMETIC CODER IN THE HEVC STANDARD Specialization: Electronic engineering Code: 9 52 02 03 Ph.D THESIS IN ENGINEERING SCIENTIFIC SUPERVISOR: ASSOC PROF DR TRAN XUAN TU Ha Noi - 2021 i STATEMENT OF AUTHORSHIP Except where reference is made in the text of the thesis, this thesis contains no material published elsewhere or extracted in whole or in part from a thesis or any other degree or diploma No other person’s work has been used without acknowledgment in the main text of the thesis This thesis has not been submitted for the award of any degree or diploma in any other tertiary institution Hanoi, Date month year 2021 Author Tran Dinh Lam ii ACKNOWLEDGEMENTS During the research work and fulfill the Ph.D thesis, I have received tremendous support, facilitated from Institute of Electronics, Training Department of Academy of Military Science and Technology I would like to express my sincere thanks to the representatives of the organisations I would like to express my deep gratitude to my supervisor, Assoc Prof., Dr Tran Xuan Tu for his continuous support, encouragement and supervision throughout my research work and the completion of this thesis I would also like to extend my gratitude to the teachers in the Institute of Electronics, the Academy of Military Science and Technology for their valuable suggestions during my research I would like to thank the teachers and other members of VNU Key Labpratory for Smart Integrated Systems (SISLAB) for their comments and supports throughout my PhD work I would like to thank my colleagues and friends, who always support and encourage me during taking the PhD Last but not least, I would like to express my sincere gratitude to my family, relatives who have shared and always encouraged me to overcome difficulties to successfully complete this thesis Hanoi, Date month year 2021 Author Tran Dinh Lam iii CONTENTS Page LIST OF ABBREVIATIONS …………………….…………………….… vi LIST OF TABLES……….………….………….………….………….…… ix LIST OF FIGURES……….………….………….………….………….…… x INTRODUCTION………………………………………………………… 1 Chapter 1 OVERVIEW OF THE CONTEXT ADAPTIVE BINARY ARITHMETIC CODING IN THE HEVC STANDARD 7 1.1 Development history of video encoding standard 7 1.1.1 The necessity of video encoding 7 1.1.2 The evolution of video compression standards 9 1.2 Principle of video encoding in the HEVC standard 10 1.2.1 The architecture of the HEVC encoder .10 1.2.2 The improvements of encoding algorithms in the HEVC standard 11 1.2.3 Principle and architecture of a CABAC encoder for HEVC 19 1.3 Overview on the development of CABAC encoder in HEVC standard 25 1.3.1 Research directions of hardware development towards realizing CABAC in HEVC standard for video applications 26 1.3.2 Solutions to improve encoding throughput 29 1.3.3 High efficient hardware architecture 35 1.3.4 Conclusion on research and development, thesis research orientation 38 1.4 Chapter conclusion 39 Chapter 2 PROPOSE HARDWARE DESIGN SOLUTIONS TO IMPROVE THE EFFICIENCY OF THE CABAC ENCODER IN THE HEVC STANDARD 41 2.1 Proposed funtional block diagram of CABAC encoder architecture .41 2.2 Binarizer 43 2.2.1 Data statistics of Binarizer 43 2.2.2 The structure of residual syntax elements 45 iv 2.2.3 The drawbacks of multi-core syntax element generation architecture 49 2.2.4 The “one scan for multiple syntax element generation” technique 52 2.2.5 The solution for binarization of last_sig_coeff_post syntax element 56 2.3 Binary arithmetic encoding (BAE) module 59 2.3.1 The context-adaptive encoding algorithm .60 2.3.2 Algorithm modification for simultaneously encoding multi bypass bins 63 2.3.3 The hardware solution for four-stage BAE architecture 65 2.3.4 Proposed architecture for hardware savings 67 2.4 Chapter conclusion 71 Chapter 3 SIMULATION, VERIFICATION, IMPLEMENTATION AND EVALUATION OF THE RESEARCH RESULTS 73 3.1 Building the simulation software model 74 3.1.1 HM Test Model 74 3.1.2 Building the software model for verification 75 3.2 Proposing the Hardware-Software co-simulation model for verification the research results .76 3.2.1 CodecVisa tool 76 3.2.2 The experimental model for verification of the results 78 3.3 Simulation and verification of proposed design solutions 84 3.4 Synthesis, simulation and evaluation of parameters of proposed design solutions .88 3.4.1 Report of performance parameters for residual syntax element generation module .89 3.4.2 Report of performance parameters for Binarization module 90 3.4.3 Report of performance parameters for Binary Arithmetic Encoding module .91 3.5 Summary of proposed research results and comparisons with related state-of-the-art results 92 3.6 Chapter conclusion 95 CONCLUSION .96 v LIST OF SCIENTIFIC PUBLICATIONS 99 BIBLIOGRAPHY 100 vi LIST OF ABBREVIATIONS AI All Intra ASIC Application Specific Integrated Circuit AVC Advanced Video Coding BAE Binary Arithmetic Encoder BIBO Block-In-Block-Out CABAC Context Adaptive Binary Arithmetic Coding CALR Coefficient Absolute Level Remaining CALVC Context Adaptive Variable Length Coding CB Coding Block CG Coefficient Group CM Context Modeler CODEC COding DECoding CTB Coding Tree Block CTU Coding Tree Unit CU Coding Unit DCT Discrete Cosine Transform DST Discrete Sine Transform EGk Exponential Golomb k-order FIFO First In First Out FL Fixed Length FSM Finite State Machine vii HDTV High Definition TeleVision HEVC High Efficient Video Coding ISO/IE C International Organization for Standardization/ International ITU-T Electrotechnical Commission International Telecommunication Union – Telecommunication Joint Collaborative Team – Video Coding JCT-VC LD LPS LUT MBBS MC MRSET MPEG MPS MV PB PISO PU RD rLPS RTL SAO Low Delay Least Probably Symbol Look Up Table Multi Bypass Bin Spliting Motion Compensation Multiple Residual Syntax Element Treatment Moving Picture Expert Group Most Probably Symbol Motion Vector Prediction Block Parallel In Serial Out Prediction Unit Random Access range Least Probably Symbol Register Transfer Level Sample Adaptive Offset viii SE Syntax Element TB Transform Block TC Transform Coefficient TU Transform Unit UHD Ultra High Definition UHD-TV Ultra High Definition-TeleVision VHDL Very-high-speed-integrated-circuit Hardware Description Language VLSI Very Large Scale Integration WPP Wave-front Parallel Processing 96 CONCLUSION The HEVC standard is proposed to support effectively encoding high- quality real-time digital video streams Since the standard was published (April 2013), the realization of the HEVC standard into video applications has faced many challenges Researching and proposing technical solutions, especially hardware solutions, are important in applying the standard diversely into various video services In the framework of this doctoral thesis, the Ph.D student focuses on researching and proposing solutions to improve the performance of CABAC encoder hardware in the HEVC standard The thesis achievements contribute to this evolution To sum up, based on the research work has been conducted, this part presents conclusions drawn from the main achievements and contributions of this thesis, including: 1 Research outcomes The thesis “Research and development of hardware implementation solutions for the context-adapted binary arithmetic encoder in the HEVC standard” has contributed to solving the problems in the hardware implementation of the CABAC encoder The proposed solutions aim to effectively apply the HEVC standard in next-generation multimedia services The thesis has studied the hardware architecture design solutions for the CABAC encoder, processing techniques for residual video data Particularly, it focuses on the main component modules of the CABAC encoder such as residual syntax element generation, binarization and binary arithmetic encoder Accordingly, the three main issues that the dissertation solves are: Firstly, the thesis has proposed an efficient algorithm and solution in terms of power consumption in hardware implementation for the residual syntax element generation module The one scan for multiple syntax element generation technique has been applied to reduce the frequency of memory access, thus reducing dynamic power consumption for this module 97 Secondly, an efficient hardware resource usage solution has been proposed for the binarization module The combined-binarization technique was applied to implement the binarization process for several syntax elements on the same hardware architecture As a result, the hardware resource is reduced and the internal datapath is simplified, which contributes to improving the hardware efficiency of the binarization module Thirdly, the thesis has proposed an efficient hardware usage in the implementation of the binary arithmetic encoding module In the four-stage BAE architecture, the proposed “multiple bypass bin processing” solution allows the processing of high throughput video streams Whereas, the proposed pre-multiply and unified datapath techniques have been applied to reduce the number of stage-buffered registers and the number of datapaths for hardware savings in implementation The main research results of the dissertation have been published in three articles published in scientific journals, the proceedings of prestigious scientific conferences The publications are the authors’ own work and are compatible with the thesis proposals The proposed solutions in the thesis were evaluated, modeled at RTL level using VHDL and simulated by software tools The hardware implementation was synthesized on the specialized IC design tool The results of synthesis and simulation were used to evaluate and compare with other related works The implementation hardware designs were tested and evaluated on the HM Test Model and the CodecVisa tools The standard video test sequences recommended by ITU-T for validating hardware implementation solutions were employed in the software model to build test vectors for verification 2 Contributions This thesis has proposed techniques and developed hardware solutions for the CABAC encoder in the HEVC standard towards improving efficiency 98 in hardware resource and energy consumption The thesis has three main contributions that are specificated as follows: - Proposing the “one scanning for multiple syntax element generation” techniques in order to reduce the power consumption of the residual syntax element generation module - Proposing the “combined binarization” technique to save hardware resources in implementing the binarization module - Proposing the “pre-multiply” and “unified datapath” to save hardware resources in implementing the binary arithmetic encoding module 3 Future research works Based on the achieved results of the thesis, future research objectives are expected as follows: - Further simulating the proposed solution of the thesis with different high-quality video sequences for more diverse evaluations - Conducting research work to propose efficient techniques and architectural design solutions for the remaining modules in the CABAC encoder This will contribute to implementing a complete high efficient hardware architecture of the CABAC encoder for video applications on mobile battery-powered devices - Continue to research high-efficiency hardware solutions for other functional modules of the HEVC encoder in the form of IP cores to propose a complete hardware implementation of the HEVC encoder - Based on the completed hardware implementation, several experiments of the HEVC encoder should be carried out on DSP, FPGA hardware platforms Otherwise, a VLSI implementation of an HEVC encoder is expected to be fulfilled for special applications 99 LIST OF SCIENTIFIC PUBLICATIONS 1 Dinh-Lam Tran, Viet-Huong Pham, Kiem Hung Nguyen and Xuan-Tu Tran, “A Survey of High-Efficient CABAC Hardware Implementations in HEVC Standard,” VNU Journal of Computer Science and Communication Engineering, vol 35, no 2, pp 1-21, 2019 2 Quang-Linh Nguyen, Dinh-Lam Tran, Duy-Hieu Bui, Duc-Tho Mai and Xuan-Tu Tran, “Efficient Binary Arithmetic Encoder for HEVC with Multiple Bypass Bin Processing,” in The 7th IEEE International Conference on Integrated Circuits, Design, and Verification, Hanoi, Vietnam, 2017, pp.82-87 DOI: 10.1109/ICDV.2017.8188644 3 Dinh-Lam Tran, Xuan-Tu Tran, Duy-Hieu Bui, Cong-Kha Pham An Efficient Hardware Implementation of Residual Data Binarization in HEVC CABAC Encoder Electronics, vol 9, issue 4, p 684, April 2020 ISSN 20799292 (SCI, IF 2.412, Q1) DOI:10.3390/electronics9040684 100 BIBLIOGRAPHY English [1] M Abeydeera, M Karunaratne, G Karunaratne, K Silva and A Pasqual, “4K Real Time HEVC Decoder on FPGA,” IEEE Transactions on Circuits and Systems for Video Technology, Vol 26, Issue 1, 2015 [2] C.-M Alonso, “Low-power HEVC Binarizer Architecture for the CABAC Block targeting UHD Video Processing,” in In Proceedings of the 30th Symposium on Integrated Circuits and Systems Design (SBCCI), August 28-September 01, 2017, pp 30-35 [3] A Alshin et al., “Coding efficiency improvements beyond HEVC with known tools,” in Applications of Digital Image Processing XXXVIII, 2015, pp 1-14 [4] V Arora and H H Saini, “A Review on Different Video Coding Standards,” International Journal on Recent and Innovation Trends in Computing and Communication, Volume: 3 Issue: 4, Apr 2015 [5] E.-A Ayele and S.-B Dhok, “Review of Proposed High Efficiency Video Coding (HEVC) Standard,” International Journal of Computer Applications, vol 59, no 15, pp 1-9, Dec 2012 [6] F Bossen, B Bross, K Suhring, and D Flynn, “HEVC complexity and implementation analysis,” IEEE Trans on Circuits and Systems for Video Technology, vol 22, pp 1685-1696, 2012 [7] B Bross et al Fraunhofer, Heinrich Hertz Institute [Online] https://vcgit.hhi.fraunhofer/jct-vc/HM [8] B Bross, W.-J Han, G J Sullivan, J.-R Ohm, and T Wiegand, High Efficiency Video Coding (HEVC) Text Specification Draft 9 , document JCTVC-K1003, ITU-T/ISO/IEC Joint Collaborative Team on Video Coding (JCT-VC), Oct 2012 101 [9] Y Chen and V Sze, “A 2014 MBIN/S DEEPLY PIPELINED CABAC DECODER FOR HEVC,” 2014 IEEE International Conference on Image Processing (ICIP), pp 2110-2114, Oct 2014 [10] Y-H Chen and V Sze, “A Deeply Pipelined CABAC Decoder for HEVC Supporting Level 6.2 High-tier Applications,” IEEE Transactions on Circuits and Systems for Video Technology , Volume: 25 Issue: 5, May 2015 [11] C Chen, K Liu, and S Chen, “High-Throughput Binary Arithmetic Encoder Architecture for CABAC in H.265/HEVC,” in In Proceedings of the 13th IEEE International Conference on Solid-State and Integrated Circuit Technology (ICSICT), Hangzhou, China, 2016, pp 1416-1418 [12] C-C Chi et al., “Parallel Scalability and Efficiency of HEVC Parallelization Approaches,” IEEE Transactions on Circuits and Systems for Video Technology, Vol.22, Issue 12, 2012 [13] J.-A Choi and Y.-S Ho, “Efficient residual data coding in CABAC for HEVC lossless video compression,” Signal, Image and Video compression, vol 9, no 5, pp 1055–1066, Sep 2013 [14] (2017, Jan) Codecian Co.Ltd [Online] http://www.codecian.com/index.html [15] G Fan, J Lei, Z Deshan, Z Mengying, and J Zhiping, “Cooperative DVFS for energy-efficient HEVC decoding on embedded CPU-GPU architecture,” in 2017 54th ACM/EDAC/IEEE Design Automation Conference (DAC), Austin, 2017, pp 1-6 [16] P FRÖJDH, R SJÖBERG, and A NORKIN, “Next generation video compression,” The communications technology journal since 1924 - Ericsson, Apr 2013 102 [17] Generic Coding of Moving Pictures and Associated Audio Information— Part 2: Video, ITU-T Rec H.262 and ISO/IEC 13818-2 (MPEG 2 Video), ITU-T and ISO/IEC JTC 1, Nov 1994 [18] Hadar, D Grois, D Marpe, A Mulayoff, B Itzhaky and O “Performance Comparison of H.265/MPEG-HEVC, VP9, and H.264/MPEG-AVC Encoders,” in 2013 Picture Coding Symposium (PCS), 2013, pp 394-397 [19] P Habermann, C-C Chi, M Mesa and B Juurlink, “A Bin- Based Bitstream Partitioning Approach for Parallel CABAC Decoding in Next Generation Video Coding,” Conference on International Parallel and Distributed Processing Symposium, May 2019 [20] J Hahlbeck and B Stabernack, “A 4k capable FPGA based high throughput binary arithmetic decoder for H.265/MPEG-HEVC,” in 2014 IEEE Fourth International Conference on Consumer Electronics Berlin (ICCE-Berlin), Berlin, Germany, 2015, pp 388-390 [21] Q Huangyuan, L Song, Z Luo, X Wang and Y Zhao, “Performance evaluation of H.265/MPEG-HEVC encoders for 4K video sequences,” in Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific, Siem Reap, Cambodia, 2014 [22] ISO/IEC, H.264: Advanced video coding for generic audiovisual services, 2003 [23] ISO/IEC, Information Technology — Coding of Audio-visual Objects — Part 2: Visual, 1999 [24] ISO/IEC(MPEG-1), Coding of Moving Pictures and Associated Audio for Digital Storage Media at up to About 1.5 Mbit/s-Part 2: Video, 1993 [25] ITU-T, H.265: High Efficiency Video Coding, 2013 [26] 1993 IUT-T, Video codec for audiovisual services at p x 64 kbit/s, 103 [27] H Jo, D Gookyi and K Ryoo, “Hardware Architecture of CABAC Binary Arithmetic Encoder for HEVC Encoder,” Advanced Science and Technology Letters, Vol 141, pp 58-63, 2016 [28] H Jo, S Park, and K Ryoo, “High-Throughput Architecture of HEVC CABAC Binary Arithmetic Encoder,” International Journal of Control and Automation Vol 10, No 5, pp 199-208, 2017 [29] C -C Ju et al., “A 0.5nJ/pixel 4K H.265/HEVC codec LSI for multi-format smartphone applications,” IEEE Journal of SolidState Circuits, Vol 1, No 58, pp 1-12, 2015 [30] B.-G Kim, K Psannis, and D.-S Jun, “Special issue on architectures and algorithms of high-efficiency video coding (HEVC) standard for real‐time video applications,” Journal of Real-Time Image Processing, vol 12, no 2, pp 215-218, Aug 2016 [31] D Kim, J Moon, and S Lee, “Hardware Implementation of HEVC CABAC Encoder,” In Proceedings of the 2015 International SoC Design Conference (ISOCC), Gyungju, South Korea Nov 2015, pp 183-184 [32] B.-R Kumar, S.-M Yousuf, and C Madhu, “Six Staged Pipelined Architecture for CABAC in H.265/HEVC,” International Journal of VLSI, Embedded Systems and Signal Processing Vol.2, Issue.2, 2016 [33] S Lee, H Kim, E Hyunmi and N Eum, “Reduced complexity single core based HEVC video codec processor for mobile 4K-UHD applications,” IEEE 6th International Conference on Consumer Electronics – Berlin, 2016 [34] T Nguyen et al., “Transform Coding Techniques in HEVC,” IEEE Journal of Selected Topics in Signal Processing, Vol 7, No 6, pp 978-989, 2013 104 [35] Quang-Linh Nguyen, Dinh-Lam Tran, Duy-Hieu Bui, Duc-Tho Mai, and Xuan-Tu Tran, “Efficient Binary Arithmetic Encoder for HEVC with Multiple Bypass Bin Processing,” In Proceedings of the 7th IEEE International Conference on Integrated Circuits, Design, and Verification (ICDV), Hanoi, Vietnam, 2017, pp 82-87 [36] E Nogues et al., “A DVFS based HEVC decoder for energy- efficient software implementation on embedded processors,” 2015 IEEE International Conference on Multimedia and Expo, Turin, Italy, 29 Jun – 3rd Jul 2015 [37] E Nogues et al., “Efficient DVFS for low power HEVC software decoder,” Journal of Real-Time Image Processing, Volume 13, Issue.1, pp 39-54, Aug 2016 [38] J.-R Ohm, G.-J Sullivan, V Sze, T Wiegand, and M Budagavi, “Introduction to the Special Issue on HEVC Extensions and Efficient HEVC Implementations,” IEEE Transactions on Circuits and Systems for Video Technology, vol 26, no 1, pp 1-3, Jan 2016 [39] G Pastuszak, “High-speed Architecture of the CABAC Probability Modeling for H.265/HEVC Encoders,” 2016 International Conference on Signals and Electronic Systems (ICSES), Krakow, Poland, Sep 5-7 2016, pp 143-146 [40] B Peng, D Ding, X Zhu, and L Yu, “A Hardware CABAC Encoder for HEVC,” 2013 IEEE International Symposium on Circuits and Systems (ISCAS), Beijing, China, 19-23 May 2013, pp 1372-1375 [41] D-H Pham, J Moon, and S Lee, “Hardware Implementation of HEVC CABAC Binarizer,” Journal of IKEEE, Vol 18, No 3, pp 356361, September 2014 [42] D-H Pham, J Moon, D Kim, and S Le, “Hardware Implementation of HEVC CABAC Binary Arithmetic Encoder,” Journal of IKEEE, vol 18, no.4, 2014 105 [43] P Pinol, H Migallon, and O Lopez-Granado, “Slice-based parallel approach for HEVC encoder,” The Journal of Supercomputing, vol 71, no 5, pp 1882–1892, Dec 2015 [44] S Radicke, J.-U Hahn, C Grecos, and Q Wang, “A MultiThread Full-feature HEVC Encoder Based on Wavefront Parallel Processing,” In Proceeding of the 11th International Conference on Signal Processing and Multimedia Application (SIGMAP-2014), Vienna, Austria, 2014, pp 90-98 [45] E Raffin et al., “Low Power HEVC Software Decoder for Mobile Devices,” Journal of Real-Time Image Processing, 2015 [46] F-L-L Ramos, J Goebel, M Porto, and S Bampi, “Low-Power Hardware Design for the HEVC Binary Arithmetic Encoder Targeting 8K Videos,” In Proceedings of the 29th Symposium on Integrated Circuits and Systems Design (SBCCI), Belo Horizonte, Brazil, Aug 29 – Sep 3, 2016, pp 1-6 [47] F-L-L Ramos, B Zatt, M Porto, and S Bampi, “Novel Multiple Bypass Bins Scheme for Low-power UHD Video Processing HEVC Binary Arithmetic Encoder Architecture,” In Proceedings of the 30th Symposium on Integrated Circuits and Systems Design (SBCCI), Fortaleza - Ceará , Brazil, August 28-September 01, 2017, pp 47-52 [48] F-L-L Ramos, B Zatt, M Porto, and S Bampi, “High- Throughput Binary Arithmetic Encoder using Multiple-Bypass Bins Processing for HEVC CABAC.,” In Proceedings of the 2018 IEEE International Symposium on Circuits and Systems (ISCAS), Florence, Italy, 27-30 Sep 2018, pp 1-5 [49] F.-L.-L Ramos, A.-V.-P Saggiorato, B Zatt, M Porto, and S Bampi, “Residual Syntax Elements Analysis and Design Targeting High- Throughput HEVC CABAC,” IEEE Transactions on Circuits and Systems, vol 67, no 2, pp 475-488, 2019 ... ARITHMETIC CODING IN THE HEVC STANDARD Preamble Chapter presents the theoretical basis of video encoding, video encoding standards and the CABAC encoder in HEVC standards This Chapter also reports... the HEVC standard 1.2.1 The architecture of the HEVC encoder As an enhancement of the H.264/AVC standard, the HEVC architecture was designed to encode video in blocks of image data However, HEVC. .. H.265 /HEVC Figure 1.8 Comparison of intra prediction between HEVC and H.264/AVC [17] d) Frequency-Time Domain Transform Intra/Inter Prediction Figure 1.9 Frequency-Time Transform in HEVC [57] In HEVC,