H.264 and MPEG-4 Video Compression phần 9 docx

7 Design and Performance 7.1 INTRODUCTION The MPEG-4 Visual and H.264 standards include a range of coding tools and processes and there is significant scope for differences in the way standards-compliant encoders and decoders are developed. Achieving good performance in a practical implementation requires careful design and careful choice of coding parameters. In this chapter we give an overview of practical issues related to the design of software or hardware implementations of the coding standards. The design of each of the main functional blocks of a CODEC (such as motion estimation, transform and entropy coding) can have a significant impact on computational efficiency and compression performance. We discuss the interfaces to a video encoder and decoder and the value of video pre-processing to reduce input noise and post-processing to minimise coding artefacts. Comparing the performance of video coding algorithms is a difficult task, not least be- cause decoded video quality is dependent on the input video material and is inherently subjec- tive. We compare the subjective and objective (PSNR) coding performance of MPEG-4 Visual and H.264 reference model encoders using selected test video sequences. Compression per- formance often comes at a computational cost and we discuss the computational performance requirements of the two standards. The compressed video data produced by an encoder is typically stored or transmitted across a network. In many practical applications, it is necessary to control the bitrate of the encoded data stream in order to match the available bitrate of a delivery mechanism. We discuss practical bitrate control and network transport issues. 7.2 FUNCTIONAL DESIGN Figures 3.51 and 3.52 show typical structures for a motion-compensated transform based video encoder and decoder. A practical MPEG-4 Visual or H.264 CODEC is required to implement some or all of the functions shown in these figures (even if the CODEC structure is different H.264 and MPEG-4 Video Compression: Video Coding for Next-generation Multimedia. Iain E. G. Richardson. C  2003 John Wiley & Sons, Ltd. ISBN: 0-470-84837-5 DESIGN AND PERFORMANCE • 226 from that shown). Conforming to the MPEG-4/H.264 standards, whilst maintaining good com- pression and computational performance, requires careful design of the CODEC functional blocks. The goal of a functional block design is to achieve good rate/distortion performance (see Section 7.4.3) whilst keeping computational overheads to an acceptable level. Functions such as motion estimation, transforms and entropy coding can be highly com- putationally intensive. Many practical platforms for video compression are power-limited or computation-limited and so it is important to design the functional blocks with these limita- tions in mind. In this section we discuss practical approaches and tradeoffs in the design of the main functional blocks of a video CODEC. 7.2.1 Segmentation The object-based functionalities of MPEG-4 (Core, Main and related profiles) require a video scene to be segmented into objects. Segmentation methods usually fall into three categories: 1. Manual segmentation: this requires a human operator to identify manually the borders of each object in each source video frame, a very time-consuming process that is obviously only suitable for ‘offline’ video content (video data captured in advance of coding and transmission). This approach may be appropriate, for example, for segmentation of an important visual object that may be viewed by many users and/or re-used many times in different composed video sequences. 2. Semi-automatic segmentation: a human operator identifies objects and perhaps object boundaries in one frame; a segmentation algorithm refines the object boundaries (if neces- sary) and tracks the video objects through successive frames of the sequence. 3. Fully-automatic segmentation: an algorithm attempts to carry out a complete segmentation of a visual scene without any user input, based on (for example) spatial characteristics such as edges and temporal characteristics such as object motion between frames. Semi-automatic segmentation [1,2] has the potential to give better results than fully-automatic segmentation but still requires user input. Many algorithms have been proposed for automatic segmentation [3,4]. In general, bettersegmentation performance can be achieved at the expense of greater computational complexity. Some of the more sophisticated segmentation algorithms require significantly more computation than the video encoding process itself. Reasonably accurate segmentation performance can be achieved by spatio-temporal approaches (e.g. [3]) in which a coarse approximate segmentation is formed based on spatial information and is then refined as objects move. Excellent segmentation results can be obtained in controlled environments (for example, if a TV presenter stands in front of a blue background) but the results for practical scenarios are less robust. The output of a segmentation process is a sequence of mask frames for each VO, each frame containing a binary mask for one VOP (e.g. Figure 5.30) that determines the processing of MBs and blocks and is coded as a BAB in each boundary MB position. 7.2.2 Motion Estimation Motion estimation is the process of selecting an offset to a suitable reference area in a previously coded frame (see Chapter 3). Motion estimation is carried out in a video encoder (not in a FUNCTIONAL DESIGN • 227 32×32 block in current frame Figure 7.1 Current block (white border) decoder) and has a significant effect on CODEC performance. A good choice of prediction reference minimises the energy in the motion-compensated residual which in turn maximises compression performance. However, finding the ‘best’ offset can be a very computationally- intensive procedure. The offset between the current region or block and the reference area (motion vector) may be constrained by the semantics of the coding standard. Typically, the reference area is constrained to lie within a rectangle centred upon the position of the current block or region. Figure 7.1 shows a 32 × 32-sample block (outlined in white) that is to be motion-compensated. Figure 7.2 shows the same block position in the previous frame (outlined in white) and a larger square extending ± 7 samples around the block position in each direction. The motion vector may ‘point’ to any reference area within the larger square (the search area). The goal of a practical motion estimation algorithm is to find a vector that minimises the residual energy after motion compensation, whilst keeping the computational complexity within acceptable limits. The choice of algorithm depends on the platform (e.g. software or hardware) and on whether motion estimation is block-based or region-based. 7.2.2.1 Block Based Motion Estimation Energy Measures Motion compensation aims to minimise the energy of the residual transform coefficients after quantisation. The energy in a transformed block depends on the energy in the residual block (prior to the transform). Motion estimation therefore aims to find a ‘match’ to the current block or region that minimises the energy in the motion compensated residual (the difference between the current block and the reference area). This usually involves evaluating the residual energy at a number of different offsets. The choice of measure for ‘energy’ affects compu- tational complexity and the accuracy of the motion estimation process. Equation 7.1, equa- tion 7.2 and equation 7.3 describe three energy measures, MSE, MAE and SAE. The motion DESIGN AND PERFORMANCE • 228 Previous (reference) frame Figure 7.2 Search region in previous (reference) frame compensation block size is N × N samples; C ij and R ij are current and reference area samples respectively. 1. Mean Squared Error: MSE = 1 N 2 N −1  i=0 N −1  j=0 (C ij − R ij ) 2 (7.1) 2. Mean Absolute Error: MAE = 1 N 2 N −1  i=0 N −1  j=0 |C ij − R ij | (7.2) 3. Sum of Absolute Errors: SAE = N −1  i=0 N −1  j=0 |C ij − R ij | (7.3) Example Evaluating MSE for every possible offset in the search region of Figure 7.2 gives a ‘map’ of MSE (Figure 7.3). This graph has a minimum at (+2, 0) which means that the best match is obtained by selecting a 32 × 32 sample reference region at an offset of 2 to the right of the block position in the current frame. MAE and SAE (sometimes referred to as SAD, Sum of Absolute Differences) are easier to calculate than MSE; their ‘maps’ are shown in Figure 7.4 and Figure 7.5. Whilst the gradient of the map is different from the MSE case, both these measures have a minimum at location (+2, 0). SAE is probably the most widely-used measure of residual energy for reasons of computa- tional simplicity. The H.264 reference model software [5] uses SA(T)D, the sum of absolute differences of the transformed residual data, as its prediction energy measure (for both Intra and Inter prediction). Transforming the residual at each search location increases computation but improves the accuracy of the energy measure. A simple multiply-free transform is used and so the extra computational cost is not excessive. The results of the above example indicate that the best choice of motion vector is (+2,0). The minimum of the MSE or SAE map indicates the offset that produces a mini- mal residual energy and this is likely to produce the smallest energy of quantised transform -10 -5 0 5 10 -10 -5 0 5 10 0 1000 2000 3000 4000 5000 6000 MSE map Figure 7.3 MSE map -10 -5 0 5 10 -10 -5 0 5 10 0 10 20 30 40 50 60 MAE map Figure 7.4 MAE map -10 -5 0 5 10 -10 -5 0 5 10 0 1 2 3 4 5 6 x 10 4 SAE map Figure 7.5 SAE map DESIGN AND PERFORMANCE • 230 Centre (0,0) position Initial search location Raster search order Search ‘window’ Figure 7.6 Full search (raster scan) coefficients. The motion vector itself must be transmitted to the decoder, however, and as larger vectors are coded using more bits than small-magnitude vectors (see Chapter 3) it may be useful to ‘bias’ the choice of vector towards (0,0). This can be achieved simply by subtracting a constant from the MSE or SAE at position (0,0). A more sophisticated approach is to treat the choice of vector as a constrained optimisation problem [6]. The H.264 reference model encoder [5] adds a cost parameter for each coded element (MVD, prediction mode, etc) before choosing the smallest total cost of motion prediction. It may not always be necessary to calculate SAE (or MAE or MSE) completely at each off- set location. A popular shortcut is to terminate the calculation early once the previous minimum SAE has been exceeded. For example, after calculating each inner sum of equation (7.3) (  N −1 j=0 |C ij − R ij |), the encoder compares the total SAE with the previous minimum. If the total so far exceeds the previous minimum, the calculation is terminated (since there is no point in finishing the calculation if the outcome is already higher than the previous minimum SAE). Full Search Full Search motion estimation involves evaluating equation 7.3 (SAE) at each point in the search window (±S samples about position (0,0), the position of the current macroblock). Full search estimation is guaranteed to find the minimum SAE (or MAE or MSE) in the search window but it is computationally intensive since the energy measure (e.g. equation (7.3)) must be calculated at every one of (2S + 1) 2 locations. Figure 7.6 shows an example of a Full Search strategy. The first search location is at the top-left of the window (position [−S, −S]) and the search proceeds in raster order FUNCTIONAL DESIGN • 231 Initial search location Spiral search order Search ‘window’ Figure 7.7 Full search (spiral scan) until all positions have been evaluated. In a typical video sequence, most motion vectors are concentrated around (0,0) and so it is likely that a minimum will be found in this region. The computation of the full search algorithm can be simplified by starting the search at (0,0) and proceeding to test points in a spiral pattern around this location (Figure 7.7). If early termination is used (see above), the SAE calculation is increasingly likely to be terminated early (thereby saving computation) as the search pattern widens outwards. ‘Fast’ Search Algorithms Even with the use of early termination, Full Search motion estimation is too computationally intensive for many practical applications. In computation- or power-limited applications, so- called ‘fast search’ algorithms are preferable. These algorithms operate by calculating the energy measure (e.g. SAE) at a subset of locations within the search window. The popular Three Step Search (TSS, sometimes described as N-Step Search) is illustrated in Figure 7.8. SAE is calculated at position (0,0) (the centre of the Figure) and at eight locations ±2 N −1 (for a search window of ±(2 N − 1) samples). In the figure, S is 7 and the first nine search locations are numbered ‘1’. The search location that gives the smallest SAE is chosen as the new search centre and a further eight locations are searched, this time at half the previous distance from the search centre (numbered ‘2’ in the figure). Once again, the ‘best’ location is chosen as the new search origin and the algorithm is repeated until the search distance cannot be subdivided further. The TSS is considerably simpler than Full Search (8N + 1 searches compared with (2 N +1 − 1) 2 searches for Full Search) but the TSS (and other fast DESIGN AND PERFORMANCE • 232 2 1 11 1 11 111 2 2 2 22 2 2 3 3 3 3 3 3 33 Figure 7.8 Three Step Search -10 0 10 -8 -6 -4 -2 0 2 4 6 8 0 1 2 3 4 5 6 7 x10 4 SAE map Figure 7.9 SAE map showing several local minima search algorithms) do not usually perform as well as Full Search. The SAE map shown in Figure 7.5 has a single minimum point and the TSS is likely to find this minimum correctly, but the SAE map for a block containing complex detail and/or different moving components may have several local minima (e.g. see Figure 7.9). Whilst the Full Search will always identify the global minimum, a fast search algorithm may become ‘trapped’ in a local minimum, giving a suboptimal result. FUNCTIONAL DESIGN • 233 0 1 2 3 1 1 1 1 2 2 3 Predicted vector Figure 7.10 Nearest Neighbours Search Many fast search algorithms have been proposed, such as Logarithmic Search, Hierar- chical Search, Cross Search and One at a Time Search [7–9]. In each case, the performance of the algorithm can be evaluated by comparison with Full Search. Suitable comparison criteria are compression performance (how effective is the algorithm at minimising the motion-compensated residual?) and computational performance (how much computation is saved compared with Full Search?). Other criteria may be helpful; for example, some ‘fast’ algorithms such as Hierarchical Search are better-suited to hardware implementation than others. Nearest Neighbours Search [10] is a fast motion estimation algorithm that has low com- putational complexity but closely approaches the performance of Full Search within the frame- work of MPEG-4 Simple Profile. In MPEG-4 Visual, each block or macroblock motion vector is differentially encoded. A predicted vector is calculated (based on previously-coded vec- tors from neighbouring blocks) and the difference (MVD) between the current vector and the predicted vector is transmitted. NNS exploits this property by giving preference to vectors that are close to the predicted vector (and hence minimise MVD). First, SAE is evaluated at location (0,0). Then, the search origin is set to the predicted vector location and surrounding points in a diamond shape are evaluated (labelled ‘1’ in Figure 7.10). The next step depends on which of the points have the lowest SAE. If the (0,0) point or the centre of the diamond have the lowest SAE, then the search terminates. If a point on the edge of the diamond has the lowest SAE (the highlighted point in this example), that becomes the centre of a new diamond-shaped search pattern and the search continues. In the figure, the search terminates after the points marked ‘3’ are searched. The inherent bias towards the predicted vector gives excellent compression performance (close to the performance achieved by full search) with low computational complexity. Sub-pixel Motion Estimation Chapter 3 demonstrated that better motion compensation can be achieved by allowing the offset into the reference frame (the motion vector) to take fractional values rather than just integer values. For example, the woman’s head will not necessarily move by an integer number of pixels from the previous frame (Figure 7.2) to the current frame (Figure 7.1). Increased [...]... 7.26 Office: encoded and decoded, MPEG-2 Video (close-up) Figure 7.27 Office: encoded and decoded, MPEG-4 Simple Profile (close-up) Figure 7.28 Office: encoded and decoded, H.264 Baseline Profile (close-up) Figure 7. 29 Grasses: encoded and decoded, MPEG-2 Video (close-up) Figure 7.30 Grasses: encoded and decoded, MPEG-4 Simple Profile (close-up) Figure 7.31 Grasses: encoded and decoded, H.264 Baseline Profile... rate-constrained encoder control and a comparison of H.264 performance with H.263, MPEG-2 Video and MPEG-4 Visual is given in [38] 7.4.4 Computational Performance MPEG-4 Visual and (to a lesser extent) H.264 provide a range of optional coding modes that have the potential to improve compression performance For example, MPEG-4 s Advanced Simple Profile is designed to offer greater compression efficiency than... prediction and hence compression efficiency • 244 DESIGN AND PERFORMANCE Figure 7. 19 Distortion introduced by MPEG-4 Visual encoding (lower half) Figure 7.20 Distortion introduced by H.264 encoding (lower half) Block transform-based CODECs introduce characteristic types of distortion into the decoded video data The lower half of Figure 7. 19 shows typical distortion in a frame encoded and decoded using MPEG-4. .. of MPEG4 and H.264 performance are presented here Figure 7.32 and Figure 7.33 compare the performance of MPEG4 (Simple Profile) and H.264 (Baseline Profile, 1 reference frame) for the ‘Office’ and ‘Grasses’ sequences Note that ‘Office’ is easier to compress than ‘Grasses’ (see above) and at a given bitrate, the PSNR of ‘Office’ is significantly higher than that of ‘Grasses’ H.264 out-performs MPEG4 compression. .. Post-processing Video compression algorithms that incorporate quantisation (such as the core algorithms of MPEG-4 Visual and H.264) are inherently lossy, i.e the decoded video frames are not identical to the original The goal of any practical CODEC is to minimise distortion and maximise compression efficiency It is often possible to reduce the actual or apparent distortion in the decoded video sequence... 7.2.3 DCT/IDCT The Discrete Cosine Transform is widely used in image and video compression algorithms in order to decorrelate image or residual data prior to quantisation and compression (see Chapter 3) The basic FDCT and IDCT equations (equations (3.4) and (3.5)), if implemented directly, require a large number of multiplications and additions It is possible to exploit the structure of the transform... arithmetic coder adopted for H.264 Main Profile are discussed in Chapter 6 and a detailed overview of the CABAC scheme is given in [30] 7.3 INPUT AND OUTPUT 7.3.1 Interfacing Figure 7.17 shows a system in which video frames are encoded, transmitted or stored and decoded At the input to the encoder (A) and the output of the decoder (D), data are in the • 242 DESIGN AND PERFORMANCE A video frames C B encode... moderate motion and in this case the source is QCIF resolution at 30 frames per second Four sets of results are compared, two from MPEG-4 and two from H.264 The first two are MPEG-4 Simple Profile (first frame coded as an I-picture, subsequent frames coded as P-pictures) and MPEG-4 Advanced Simple Profile (using two Bpictures between successive P-pictures, no other ASP tools used) The second pair are H.264 Baseline... MPEG-2 Video CODEC, an MPEG-4 Simple Profile CODEC and the H.264 Reference Model CODEC (operating in Baseline Profile mode, using only one reference picture for motion compensation) In each case, the first frame was encoded as an I-picture The remaining frames were encoded as Figure 7.24 Office: original frame • 248 DESIGN AND PERFORMANCE Figure 7.25 Grasses: original frame P-pictures using the MPEG-4 and H.264. .. Figure 7.12 8 × 8 block in boundary MB forward and inverse transforms [ 19] describes alternative implementations using (i) a series of shifts and additions (‘shift and add’), (ii) a flowgraph algorithm and (iii) matrix multiplications Some platforms (for example DSPs) are better-suited to ‘multiply-accumulate’ calculations than to ‘shift and add’ operations and so the matrix implementation (described in . Nearest Neighbours Search Many fast search algorithms have been proposed, such as Logarithmic Search, Hierar- chical Search, Cross Search and One at a Time Search [7 9] . In each case, the performance of. the diamond has the lowest SAE (the highlighted point in this example), that becomes the centre of a new diamond-shaped search pattern and the search continues. In the figure, the search terminates after. The next step depends on which of the points have the lowest SAE. If the (0,0) point or the centre of the diamond have the lowest SAE, then the search terminates. If a point on the edge of the