2019 3rd International Conference on Recent Advances in Signal Processing, Telecommunications & Computing (SigTelCom) Complexity Controlled Side Information Creation for Distributed Scalable Video Coding Quang Hoang Van1, Le Dao Thi Hue2, Vien Dinh Du1, Vu Nguyen Hong3, and Xiem HoangVan2 Hanoi University of Industry VNU-University of Engineering and Technology Radio Electronics Association of Vietnam quanghvdt@gmail.com, xiemhoang@vnu.edu.vn Abstract—Distributed scalable video coding (DSVC) has recently been gaining many attentions due to its benefits in terms of computational complexity, error resilience and scalability, which are important for emerging video applications like wireless sensor networks and visual surveillance system (VSS) In DSVC, the side information (SI) creation plays a key role as it directly affects the DSVC compression performance and the encoder/decoder computational complexity However, for many VSS applications, the energy of each VSS node is usually attenuating along the time, making the difficulty in transmitting surveillance video in real time To address this problem, we propose a complexity controlled SI creation solution for the newly DSVC framework To achieve the flexible SI creation, the complexity associated with SI creation process is modeled using a linear model in which the model parameters are estimated from a fitting process To adjust the SI complexity, a user parameter is defined based on the availability of the VSS energy resource Experiments conducted for a rich set of video surveillance data have revealed the benefits of the proposed complexity control solution, notably in both complexity control and compression performance Keywords—Distributed scalable video coding, side information, visual sensor networks I INTRODUCTION Nowadays, video surveillance systems (VSS) have been widely used in many important applications such as public safety and private protection [1] Such a system can provide real-time monitoring and analysis of the observed environment Real-world video surveillance applications, which typically require storing videos without neglecting any part of scenarios for weeks or months In addition, the heterogeneity of devices, networks and environments is also gaining a request for adaptation solutions In this scenario, there is a critical need of a powerful video coding scheme that is featured by high coding efficiency, scalability and low encoding complexity capabilities A VSS typically includes three main parts, the camera nodes, the center and the users as shown in Fig The video is firstly captured and processed at the camera node and sent to the server Such video bitstream can be transcoded or distributed to users with different quality, resolutions At the user side, 978-1-5386-7963-0/19/$31.00 ©2019 IEEE video data can be used for object detection, activity tracking, and/or event analysis TV Camera Camera Monitor Server Camera Camera Router Local PC Internet Portable Decives Remote PC Fig A video surveillance system The recent researches have shown that the distributed scalable video coding [2-5], a newly High Efficiency Video Coding (HEVC) [6] scalable extension [6] can satisfy the mentioned requirements of a VSS [7] However, for many VSS applications, the energy allowed at each VSS node is usually attenuating along the time In this case, the complexity of DSVC should be adjusted depending on the energy situation in each VSS node In DSVC, the SI creation [8] usually consumes the largest percentage of computational complexity [2] In this context, we propose in this paper a novel complexity controlled SI creation solution to adaptively adjust the overall DSVC complexity, notably by a SI complexitymodeling framework In the proposed SI creation solution, the complexity associated with the motion estimation stage is controlled using a user setting parameter Depending on the energy situation of each VSS node, the user parameter is imported to control the complexity of the SI creation process and thus the overall DSVC solution Experiments conducted for a rich set of test surveillance video has shown that the proposed SI creation solution is easy to manage the complexity at both the encoder and decoder 104 2019 3rd International Conference on Recent Advances in Signal Processing, Telecommunications & Computing (SigTelCom) Surveillance Videos WZ frames Enhancement Layer Sequence Splitting Syndrome Creation Syndrome Encoding Syndrome Decoding Syndrome Reconstruction nLSB Xˆ B nLSB Xˆ B Correlation Modeling SI Residue Creation SI Residue Creation SHVC Intra Encoder Key frames Reconstructed EL + Correlation Modeling Sequence Merging SHVC Intra Decoder Reconstructed BL Base Layer HEVC Intra Encoder Xˆ B HEVC Intra Decoder S-DSVC Encoder S-DSVC Decoder Fig Distributed scalable video coding architecture (highlighting the SI creation component) The organization of this paper is as follows Section II briefly introduces the background work on DSVC Section III presents the proposed complexity controlled SI creation while Section IV evaluates and discusses the performance of the method Finally, in Section V, some concluding remarks and future works are outlined II OVERVIEW OF DISTRIBUTED SCALABLE VIDEO CODING Distributed scalable video coding (DSVC), was firstly proposed by X HoangVan et al, in [2-5] The DSVC is in fact an HEVC scalable extension, which follows a layered coding approach with one base layer (BL) and one or several enhancement layers (ELs) Fig illustrates the general architecture of the DSVC in which two layers are presented In DSVC, while the BL is backward compatible with the HEVC standard [6], the EL is processed with the distributed coding structure to achieve the low encoding complexity and error resilience features [2] Since the DSVC EL is compressed with the distributed source coding approach [9], instead of finding the best prediction which usually requires a large number of computations, the EL compresses only a part of original information (called syndrome) which cannot be inferred at the decoder In such a system, the correlation between the original and side information is estimated at both the encoder and decoder sides [3] Hence, the SI creation and the correlation estimation model are critical parts in DSVC The detail of the DSVC working steps can be referred to [2] Following the DSVC solution proposed in [2-5], the SI creation is performed at both encoder and decoder sides While the encoder SI is used to estimate the correlation model, represented through a number of least significant bitplanes ( nLSB ), the decoder SI is used for both correlation estimation and reconstruction Considering the importance of SI, several SI creation techniques have been presented in literatures such as: the motion compensated temporal interpolation (MCTI) [10], the motion compensated temporal filtering (MCTF) [11] and the SI fusion [2] Although these SI creation solutions are able to achieve SI with high quality, the complexity associated with each SI creation component is extremely large [4], thus, leading the difficulty in real-time video transmission III PROPOSED COMPLEXITY CONTROLLED SI CREATION SOLUTION This section describes the proposed complexity controlled SI creation solution Before that, the computational complexity associated with the SI creation processes is analyzed A SI Complexity Analysis Fig shows the proposed SI creation architecture Here, we introduce a complexity controlling factor, gamma – γ, to adjust the SI computational complexity based on the energy allowed at each VSS node 105 complexity control factor MCTI X f E Motion Estimation MV Refinement Motion Compensation SI X Eb Motion Estimation Motion Compensation SI Fusion MCTF XB 1.ME 2.MVR 3.MC Fig Proposed SI creation solution 4.SIF 2019 3rd International Conference on Recent Advances in Signal Processing, Telecommunications & Computing (SigTelCom) The input of the SI creation includes the BL reconstructed frame, Xˆ B and two forward, backward EL decoded frames, Xˆ f , Xˆ b [2] The proposed SI E E creation includes four main stages: the motion estimation (ME), the motion vector refinement (MVR), the motion compensation (MC) and the SI fusion (SIF) To analyze the SI complexity, we measure the SI processing time ( PTSI ), in seconds, for two video surveillance sequences, Bank and Campus Fig shows the percentage of complexity associated with each SI component mentioned above In this case, to control the computational complexity for the SI creation process, especially when the energy of VSS node is degraded along the time, a gamma factor, γ, with (0;1] is defined In this case, γ=1 corresponds to the maximum SR is used, i.e., SR = 32 Therefore, given a gamma factor, the optimal SR ( SRopt ) can be determined as: SRopt PTSI (SR 32) (2) In summary, the proposed SI complexity control solution can be performed as the following procedure: Procedure: SI complexity control - Input: User preferred SI creation complexity, γ (0;1] - Output: SRopt size Perform the SI creation with SR = 32 Compute PTSI (SR 32) Read the complexity control factor, γ Determine the optimal SRopt as in (2) Fig SI creation complexity analysis From the obtained results, it can be concluded that the ME stage usually consumes the highest computational complexity percentage among other SI creation components B Proposed SI complexity controlled model In ME process, the search range (SR) has been confirmed as the main factor, which affects to both SI quality and SI creation complexity [12] To study this relationship, we measure the SI processing time for several SR sizes, e.g., SR = {8; 12; 16; 20; 24; 28; 32} Two training video sequences are examined and experimental results are shown in Fig IV PERFORMANCE EVALUATION This section evaluates the proposed complexity controlled SI creation solution Firstly, the test conditions are presented Secondly, the SI creation complexity with the proposed complexity controlled solution is evaluated Thirdly, the proposed method accuracy and the SI quality with different γ are shown Finally, the overall DSVC compression performance is discussed A Test conditions As usual, four common surveillance video sequences [13] with different motion characteristics and contents are examined The detailed spatial, temporal resolutions, number of coded frames and other factors are provided in TABLE I while the first frame of each tested video is illustrated in Fig Fig SI creation complexity with several SRs From the results obtained in Fig 5, it is able to conclude that the relationship between the SI processing time and the SR size can be modeled as a linear function: (1) PTSI ( SR ) SR Here, { , } are two model parameters, which can be computed by a model fitting as shown in Fig 106 Classover Crossroad Overbridge Office Fig Illustration of the first frame for the tested surveillance videos 2019 3rd International Conference on Recent Advances in Signal Processing, Telecommunications & Computing (SigTelCom) TABLE I Summary of Test Conditions Spatial resolution, 720×576, @30Hz, temporal resolution, 201 frames number of frames GOP size (Key-WZ-Key-…) Quantization QPB = {38;34;30;26} Parameters QPE = QPB – Hardware Processor: Intel® Core™ configuration i7-4800MQ @2.7 GHz RAM: 8.00 GB System: Win 10, 64-bit Environment: Microsoft Visual Studio 2017 Community B SI creation complexity evaluation To evaluate the SI creation complexity controlled with the proposed method, the SI processing time is measured for several gamma values Experimental results are shown in Fig confirm that the user selected gamma factor is proportional to the SI creation complexity For example, when the energy of a VSS node reduces to 75% compared to the beginning, it is able to adjust the SR size of SI creation process using the proposed complexity controlling procedure in section III.B In fact, this complexity management is very important for VSS applications, especially when the energy of each VSS node is reduced along the time Fig SI quality assessment with various gamma values D DSVC compression performance evaluation Finally, it would not be completed without the compression performance evaluation for the proposed DSVC structure In this issue, we examine the DSVC compression performance for four test videos and four quantization parameters as described in TABLE I The common rate-distortion (RD) performance is compared between the proposed DSVC and the related benchmark, SHVC standard [6] Experimental results are shown in Fig Fig DSVC performance evaluation with different SI creation Fig SI creation complexity evaluation with different γ C SI quality evaluation It should be noted that the complexity reduction achieved with the small gamma factor is usually associated with the quality degradation in the created SI Fig illustrates SI quality along frames with several gamma values for four test sequences The experimental results shown that the complexity associated with the SI creation is usually proportional to the quality of SI frame The higher energy can be used for the SI creation process, the better SI quality can be achieved From the obtained results, it can be concluded that the proposed DSVC solution importantly improves the compression performance than SHVC standard In addition, with the complexity control mechanism proposed in Section III.B, the proposed DSVC is even more flexible for many VSS applications V CONCLUSIONS AND FUTURE WORKS Considering the need for a flexible DSVC structure in VSS application, we proposed in this paper a novel complexity controlled SI creation 107 2019 3rd International Conference on Recent Advances in Signal Processing, Telecommunications & Computing (SigTelCom) solution The proposed complexity controlled mechanism is performed based on the relationship between the SI creation processing time and the size of SR in ME stage The relationship between the SI creation complexity and the SR is modeled as a linear function A complexity controlling gamma factor is defined to adjust the SI complexity and also the overall DSVC processing time Experimental results show the benefits of the proposed complexity controlling solution, notably when the energy of VSS node is attenuating along the time REFERENCES [1] [2] [3] [4] [5] M Valera and S Velastin, “Intelligent distributed surveillance systems: A review,” IEE Proceedings Vision, Image and Signal Processing, vol 152, no 2, pp 192–204, Apr 2005 X Hoang Van, J Ascenso, F Pereira, “HEVC backward compatible scalability: A low encoding complexity distributed video coding based approach,” Signal Processing: Image Communication, vol 33 pp 51–70, Apr 2015 X HoangVan, J Ascenso, and F Pereira, “Optimal Reconstruction for a HEVC Backward Compatible Distributed Scalable Video Codec,” IEEE Visual Communication and Image Processing (VCIP), Valletta, Malta, Dec 2014 X HoangVan, J Ascenso, F Pereira, “Adaptive Scalable Video Coding: a HEVC based Framework Combining the Predictive and Distributed Paradigms”, IEEE Transactions on Circuits and Systems for Video Technology, vol 27, no 8, pp 1761-1776, Aug 2017 L Dao Thi Hue, et al “Efficient and low complexity surveillance video compression using distributed scalable 108 video coding” VNU Journal of Science: Comp Science & Com Eng., Vol 2018 [6] G J Sullivan, J.-R Ohm, W.-J Han, and T Wiegand, “Overview of the High Efficiency Video Coding (HEVC) Standard,” IEEE Transactions on Circuits and Systems for Video Technology, vol 22, no 12, pp 1649-1668, Dec 2012 [7] J M Boyce, Y Ye, J Chen, and A K Ramasubramonian, “Overview of SHVC: Scalable Extensions of the High Efficiency Video Coding Standard,” IEEE Transactions on Circuits and Systems for Video Technology, Vol 26, Issue 1, pp 20-34, Sept 2015 [8] F Pereira, C Brites, J Ascenso, M Tagliasacchi “Wyner–Ziv video coding: a review of the early architectures and further developments,” IEEE International Conference on Multimedia and Expo (ICME), Hannover, Germany, June 2008 [9] A D Wyner, J Ziv, “The rate-distortion function for source coding with side information at the decoder,” IEEE Information Theory Society, vol 22, no 1, pp 1–10, 1976 [10] J Ascenso, C Brites, and F Pereira, “Improving frame interpolation with spatial motion smoothing for pixel domain distributed video coding,” 5th EURASIP Conference on Speech and Image Processing, Multimedia Communications and Services, Slovak, Jul 2005 [11] J Ascenso, C Brites, and F Pereira, “A flexible side information generation framework for distributed video coding,” Multimedia Tools and Applications, vol 48, no 9, pp 381-409, 2009 [12] X HoangVan, H Phi Cong, “A novel content adaptive search strategy for low complexity frame rate up conversion,” 2nd International Conference on Recent Advances in Signal Processing, Telecommunications & Computing (SigTelCom), Ho Chi Minh City, Vietnam, Jan 2018 [13] PKU-SVD-A [Online] Available: http://mlg.idm.pku.edu.cn/-resources/pku-svd-a.html ... PROPOSED COMPLEXITY CONTROLLED SI CREATION SOLUTION This section describes the proposed complexity controlled SI creation solution Before that, the computational complexity associated with the SI creation. .. outlined II OVERVIEW OF DISTRIBUTED SCALABLE VIDEO CODING Distributed scalable video coding (DSVC), was firstly proposed by X HoangVan et al, in [2-5] The DSVC is in fact an HEVC scalable extension,... evaluates the proposed complexity controlled SI creation solution Firstly, the test conditions are presented Secondly, the SI creation complexity with the proposed complexity controlled solution