Multimed Tools Appl DOI 10.1007/s11042-016-3392-4 A new depth image quality metric using a pair of color and depth images Thanh-Ha Le1 · Seung-Won Jung2 · Chee Sun Won3 Received: 17 July 2015 / Revised: 17 February 2016 / Accepted: 23 February 2016 © Springer Science+Business Media New York 2016 Abstract Typical depth quality metrics require the ground truth depth image or stereoscopic color image pair, which are not always available in many practical applications In this paper, we propose a new depth image quality metric which demands only a single pair of color and depth images Our observations reveal that the depth distortion is strongly related to the local image characteristics, which in turn leads us to formulate a new distortion assessment method for the edge and non-edge pixels in the depth image The local depth distortion is adaptively weighted using the Gabor filtered color image and added up to the global depth image quality metric The experimental results show that the proposed metric closely approximates the depth quality metrics that use the ground truth depth or stereo color image pair Keywords Depth image · Image quality assessment · Reduced reference · Quality metric Seung-Won Jung swjung83@dongguk.edu Thanh-Ha Le ltha@vnu.edu.vn Chee Sun Won cswon@dongguk.edu University and Engineering and Technology, Vietnam National University, Hanoi, Vietnam Department of Multimedia Engineering, Dongguk University, Pildong-ro 1gil, Jung-gu, Seoul 100-715, Korea Department of Electronics and Electrical Engineering, Dongguk University, Pildong-ro 1gil, Jung-gu, Seoul 100-715, Korea Multimed Tools Appl Introduction Depth images play a fundamental role in many 3-D applications [6, 17, 24, 25] For example, depth images can be used to generate arbitrary novel viewpoint images by interpolating or extrapolating the images at the available viewpoints In addition, high quality depth images open up opportunities to solve challenging problems in computer vision [13] The depth image can be obtained either by matching a rectified color image pair, i.e., stereoscopic image, or by using depth cameras In stereo matching techniques [19, 29], inaccurate depth images are often produced because of occlusion, repeated patterns, and large homogeneous regions Although the inherent difficulty of stereo matching can be solved using the depth camera [27], an inevitable sensor noise problem remains Owing to the widespread use of depth images, the quality assessment of depth images becomes essential One simple method of the depth image quality assessment is to compare the depth image to be tested with its ground truth depth image [21] This method corresponds to the full reference depth quality metric (FR-DQM), which can precisely measure the accuracy of the depth image However, the ground truth depth image is not always attainable in most practical applications An alternative method is to evaluate the quality of the reconstructed color image obtained using the depth image For example, the right-viewpoint image can be rendered by using the left-view image and depth image, and the rendered image can be compared with the original right-viewpoint image [12] However, such a color image pair is not always obtainable in the depth-image-based rendering (DIBR) applications [1, 5] In this paper we introduce a new depth quality metric, which requires only a pair of color and depth images and, thus, is a reduced reference DQM (RR-DQM) Here, we consider the color image as the reduced reference for the depth quality assessment To formulate the depth quality metric we investigate the effects of various sources of depth distortions and come up with a local measurement using the Gabor filter [2] and the smallest univalue segment assimilating nucleus (SUSAN) detector [23] The experimental results demonstrate that the proposed RR-DQM closely approximates the conventional DQMs that use the ground truth depth information or stereo image pair This paper is an extended version of our conference paper [20] Compared to [20], more detailed description of the proposed metric is provided with extensive experimental verification Moreover, the proposed metric is applied to the depth image post-processing technique to show its usefulness The rest of the paper is organized as follows The proposed RR-DQM is described in Section The experimental results and conclusions are given in Section and Section 4, respectively Proposed depth quality metric A new depth image quality metric is designed for the case when the depth image is not used in a stand-alone fashion but in a combined fashion with the color image The combination of color and depth images is often required in multi-view and 3-D video applications, where the depth image is frequently used to render or synthesize color images at novel viewpoints In such applications, since the same local distortion of the depth image does not equally affect the resultant color images, we need to consider the local distortion of the depth image jointly with the local characteristics of the color image For example, a pair of simple Multimed Tools Appl Fig (a)-(b) Synthetic image pair and (c) ground truth depth image synthetic grayscale images of the size 400×400 is shown in Fig 1a and b Here, the square region of Fig 1a including horizontal, vertical, and two diagonal edges is left-shifted by 50 pixels as shown in Fig 1b In other words, the pixels inside the square have the same horizontal disparity value as shown in Fig 1c The black background is located in the infinite distance, i.e disparity value is zero Note that the other directional disparities can be ignored when the two images are rectified [14] To analyze the effect of depth distortion, we change the disparity values inside the square as shown in Fig 2a and b Precisely, one noisy row is generated using the zero-mean uniform random distribution with a variance of 10 The length of the row is the same as the width of the square, and the generated noisy row is added to all the rows in the square of the depth image, resulting Fig 2a This can simulate depth distortion along the horizontal direction Note that the depth values should be the same along the vertical direction The depth image with distortion along vertical direction can be produced in a similar manner as shown in Fig 2b Given a pair of color and depth images, the stereoscopic image can be obtained Specifically, the pixels in the one viewpoint image can be found from the pixels in the other viewpoint image in which the pixel positions are determined according to the disparity values in the depth image From the synthesized color image, we can analyze the influence of depth distortions From Fig 2c and d, one can notice that the horizontal image edges are not seriously deteriorated Since only the horizontal disparity is assumed, different directional distortions can change only the start and end positions of the horizontal edges In other words, the local distortion of the depth image in the horizontal edge regions does not have a significant impact on the quality of the rendered image It can be also found that the distortion in the rendered images is prominent when the depth value varies along the image edges For example, the vertical image edges are severely damaged when the depth image has distortion along vertical direction as shown in Fig 2b and d From the above observations, it is found that the effect of local depth distortion is strongly dependent on the local image characteristics Thus, the relation between the depth distortion and image characteristics should be exploited to measure the quality of the depth image Figure shows the flowchart of the proposed RR-DQM in which the Gabor filter [2] is used to weight differently according to the local image structures In addition, the SUSAN edge detector [23] is employed to attain the edge information of the image In particular, the SUSAN detector is known to robustly estimate image edges and their edge direction Of course, other edge detectors [15, 18] can be employed Multimed Tools Appl Fig Distorted depth images and rendered grayscale images: (a) depth image with distortion along horizontal direction, (b) depth image with distortion along vertical direction, (c) rendered left-view image obtained using Fig 1b and a, g rendered left-view image obtained using Fig 1b and b The Gabor filter is a close model of the receptive fields [7, 28] and widely applied to image processing applications Let g denote the kernel of the Gabor filter defined as follows: g(x, y) = exp − where xr2 + γ yr2 2σ cos 2π xr +φ , λ (1) xr = x cos θ + y sin θ (2) yr = −x sin θ + y cos θ In (1) and (2), γ , σ , λ, φ and θ represent the aspect ratio, standard deviation, preferred wavelength, phase offset, and orientation of the normal to the parallel stripes, respectively [4] Since the Gabor filter can be simply viewed as a sinusoidal plane wave multiplied by the Multimed Tools Appl Fig Flowchart of the proposed depth image quality metric Gaussian envelope, especially for the edge and bar detection, antisymmetric and symmetric versions of the Gabor filter [10] can be defined as xr2 + γ yr2 2σ sin 2π xr , λ (3) xr2 + γ yr2 2σ cos 2π xr λ (4) gedge (x, y) = exp − gbar (x, y) = exp − The edges and bars of the image, Iedge and Ibar , are obtained by convolving the original image I with gedge and gbar , respectively Here, the mean value of gbar is subtracted to compensate for the DC component The filtered outputs are combined into a single quantity Iθ , called the Gabor engergy, as follows: Iθ (x, y) = (x, y) + I (x, y) Ibar edge (5) This Gabor energy approximates a specific type of orientation selective neuron in the primary visual cortex [9] Figure shows the Gabor energy results on Fig 1a with various θ values In this example, γ and σ are set to 0.5 and 1.69 according to the default settings [4] In addition, λ is adjusted to and the Gabor energy outputs are scaled for the visualization As can be seen, the four directional components are successfully decomposed and the perceptually sensitive regions are distinguished Thus, the Gabor energy of the image can be exploited to adaptively weight the local distortion of the depth image In Fig 2, we found that the influence of local depth distortion is strongly related to edge direction To this end, the SUSAN detector is used to extract edges and their directions Detailed description and analysis of the SUSAN operator can be found in [23] Let EbI and EdI denote the edge map and edge direction map of the image I obtained using the SUSAN detector, respectively For simplicity, EdI is quantized to represent only the horizontal, vertical, left diagonal, and right diagonal directions At the non-edge pixels, local depth distortion is measured by the average difference of the depth values in the local neighborhood On the other hand, at the edge pixels, depth variation along edge direction is measured to consider edge distortion or deformation Multimed Tools Appl Fig Gabor filtered results on Fig 1a: (a) I0◦ , (b) I90◦ , (c) I135◦ , (d) I45◦ Let denote the depth distortion map obtained using the binary edge map EbI and the depth image D: ⎧1 |D(x, y)−D(x +u, y +v)| ; if EbI (x, y) = ⎨8 (u,v)∈N (x, y) = , (6) ⎩ D(x, y)− (D(x +x , y +y )+D(x +x , y +y )) ; otherwise 1 2 where N8 represents the 8-neighborhood At the non-edge pixels, the mean absolute difference (MAD) is used to measure local depth distortion Meanwhile, at the edge pixels, the average of the two adjacent depth values along the edge direction is differentiated with the center pixel’s depth value In other words, (xi , yi ) is determined according to the edge direction For example, (x1 , y1 ) = (1, 0) and (x2 , y2 ) = (−1, 0) for the horizontal edge Note that the central difference can distinguish an abrupt change from a gradual change Thus, a Multimed Tools Appl natural change of depth values along edge direction caused by slanted surfaces is excluded in the computation of local depth distortion When the depth image is captured by the depth camera, saturated pixels often appear in highly reflective regions Such saturated depth pixels have invalid depth values, and thus we consider those pixels as outliers Many stereo matching algorithms [21, 26] also identify the outlier pixels without estimating their depth values In the proposed method, if the neighboring pixel in (6) belongs to the outlier pixels, the corresponding position is excluded from N8 In a similar manner, for the edge positions, only one neighboring depth value is used if one of two neighbors is outlier Distortion estimation is not performed if both are outliers As the depth discontinuities along color image edges are the major source of local depth distortion, both the local image characteristics (Iθ ) and local depth distortion ( ) are used to obtain the global distortion map To this end, is defined by merging Iθ and as follows: (x, y) = αθ · Iθ (x, y) · (x, y), (7) θ∈ where = {0◦ , 45◦ , 90◦ , 135◦ } and αθ is the weight of the direction θ Figure shows the resultant distortion maps obtained using Fig 2a and b, where αθ = {1, 0.5, 0, 0.5} for the four directions in By comparing Figs and 5, it can be seen that the distortion maps are highly correlated with visual geometric degradation caused by depth distortion Finally, the RR-DQM is defined by pooling all the distortion values in except for the outlier regions, ⎛ ⎞ ⎝ (8) (x, y) ⎠, DQMRR = n(ϒ1 ) (x,y)∈ϒ1 where ϒ1 is a set of all pixels excluding the outlier pixels and n(ϒ1 ) is the cardinality of ϒ1 Note that the proposed RR-DQM requires only one pair of color and depth images Fig Global distortion maps corresponding to (a) Figs 2a and (b) 2b Multimed Tools Appl Fig Example on the Cones image: (a) original left-view image, (b) ground truth depth image, (c) ground truth occlusion map, (d) LPCD distorted depth image with max =2, (e) compensated left-view image, (f) error image excluding occluded region Multimed Tools Appl Fig Scatter plot of the RR-DQM versus depth distortion for random noise: (a) Cones, (b) T eddy, (c) T sukuba, (d) V enus Experimental results The proposed RR-DQM mainly consists of the Gabor filer and the SUSAN detector with some parameters In the Gabor filter, γ and σ were set to 0.5 and 1.69, respectively (these are the default values in typical applications of Gabor filters [4]) In addition, λ was empirically determined as by using test color-depth image pairs available in the Middlebury database [21] The brightness threshold and the kernel radius in the SUSAN operator were chosen to 15 and 3, respectively, according to [23] In order to validate the proposed RR-DQM, the RR-DQM was compared with the conventional metrics To this end, we used the Middlebury dataset, where the ground truth depth image and stereo image pair are available The two different types of the depth distortion were simulated First, the uniformly distributed random noise was added to the ground truth depth image since the noisy depth images can approximate the depth images obtained by Multimed Tools Appl Fig Scatter plot of the RR-DQM versus prediction error for random noise: (a) Cones, (b) T eddy, (c) T sukuba, (d) V enus the depth camera Second, the geometric distortion, local permutation with cancelation and duplication (LPCD) [3], was applied to the depth image D by D(x, y) = Dgt (x + h (x, y), y + (9) w (x, y)), where Dgt denotes the ground truth depth image, h and w are the i.i.d integer random variables uniformly distributed in the interval [− max , max ], and max controls the amount of distortion This local geometric distortion can simulate the inaccurate depth values in the object boundaries, where the stereo matching techniques usually find difficulty in estimating depth values Given the degraded and ground truth depth images, the FR-DQM measures the difference between two depth images as follows: D(x, y) − Dgt (x, y) DQMF R = (x,y)∈ϒ2 n(ϒ2 ) , (10) Multimed Tools Appl Fig Scatter plot of the RR-DQM versus depth distortion for the LPCD distortion: (a) Cones, (b) T eddy, (c) T sukuba, (d) V enus where ϒ2 denotes a set of the ground truth depth pixels Alternatively, the depth accuracy can be measured by warping one image using the depth image and comparing the warped image with the original image In our work, the right-view image of the stereo pair is warped and the prediction error, DQMP red , is measured by I l (x, y) − I r (x + D(x, y), y) DQMP red = (x,y)∈ϒ3 n(ϒ3 ) , (11) where I l and I r denote the left-view and right-view images, respectively Here, ϒ3 indicates a set of pixels in the left-view image except for the occluded pixels Figure illustrates the above two metrics for the Cones image In this example, the LPCD distortion with max of was applied to the ground truth depth image in Fig 6b Then, the right-view image and the distorted depth image in Fig 6d were used to reconstruct the left-view image shown in Fig 6e DQMF R is the amount of the difference between Figs 6b and d in the non-outlier region, a white region in Fig 6c Similarly, the prediction error of the left-view image within the non-outlier region, shown in Fig 6f, is used to Multimed Tools Appl Fig 10 Scatter plot of the RR-DQM versus prediction error for the LPCD distortion: (a) Cones, (b) T eddy, (c) T sukuba, (d) V enus compute DQMP red These two metrics, DQMF R and DQMP red , can accurately assess the quality of the depth image by using the ground truth depth image and the stereo image pair, respectively Thus, our objective is to show strong correlation between the proposed RR-DQM and these metrics Figure shows the scatter plot of DQMRR versus DQMF R for the noisy depth images using four stereo test images, Cones, T eddy, T sukuba, and V enus In this test, 100 noisy depth images were obtained by increasing the variance of the added noise from to 100 As can be seen, DQMRR has almost perfect linear relationship with DQMF R Thus, the proposed metric can accurately assess the amount of the noise in the depth image The relation between DQMRR and DQMP red for the same test images is shown in Fig Since the prediction error depends on the characteristics of the image, the correlation is not as strong as that in Fig However, there still exists strong linear relationship, where the Pearson’s correlation coefficient, R , approaches to The relationships of DQMRR with DQMF R and DQMP red for the LPCD distortion are shown in Figs and 10, respectively Here, 25 distorted depth images were generated by increasing max from to 25 As can be seen, the proposed technique provides the similar quality metric without requiring the additional information of the ground truth depth or the stereo image pair Multimed Tools Appl Fig 11 The sensitivity of the proposed metric with various λ values The R scores are measured between (a) DQMRR and DQMF R for random noise, (b) DQMRR and DQMP red for random noise, (c) DQMRR and DQMF R for LPCD distortion, and (d) DQMRR and DQMP red for LPCD distortion, respectively As aforementioned, we used the default parameter settings from [4, 23], and thus λ in (1) and αθ in (7) are only empirically chosen parameters Figure 11 shows the sensitivity of the RR-DQM with respect to λ values Here we set αθ as {1, 0.5, 0, 0.5} and extracted R scores by the same manner as Figs 7-10 The resultant R score curves were generally smooth and had a peak around Similarly, Fig 12 shows the sensitivity of the RR-DQM with respect to α0o values Here we set α45o =0.5, α90o =0, α135o =0, and λ=3.0 and extracted the R scores We observed that the R scores tend to converge when α0o is around Similar results were obtained when α45o =0.5, α90o =0.0, and α135o =0.5, respectively We then applied the RR-DQM to more general outdoor images in the KITTI database [8] in which the disparity images estimated using Displets method [11] are used as the depth ground truth Figure 13a depicts the color image 10 and Fig 13b depicts its estimated depth Table demonstrates that the proposed depth quality metric is strongly correlated with the conventional metrics requiring the ground truth depth image for both depth distortion and prediction error Thus, the proposed metric can be used to assess the depth image quality when such information is not attainable Multimed Tools Appl Fig 12 The sensitivity of the proposed metric with various α0o values The R scores are measured between (a) DQMRR and DQMF R for random noise, (b) DQMRR and DQMP red for random noise, (c) DQMRR and DQMF R for LPCD distortion, and (d) DQMRR and DQMP red for LPCD distortion, respectively Moreover, the proposed RR-DQM can be worked with the depth refinement algorithm In [31], an iterative depth refinement algorithm was proposed by using bilateral filtering In each iteration, the cost volume representing the depth probability is modified by bilateral filtering and the depth image is updated Since no specific depth quality metric is employed, the update process can be terminated when the number of iteration reaches the predefined number or the change of the depth image is negligible If the RR-DQM is used as a termination criterion, the update process can be finished when the amount of the quality improvement is saturated Figure 14 shows the RR-DQM results for each iteration using the Cones and T eddy images In this test, in order to simulate the low resolution depth image, the original depth image was down-sampled by a factor of and then up-scaled to the original resolution with the nearest neighborhood interpolation Then, the LPCD distortion with max of was applied to induce the depth distortion As can be seen, the RR-DQM converges in a few iterations and thus the change of the RR-DQM is effectively used as the termination criterion of the depth refinement algorithm Multimed Tools Appl Fig 13 A test image pair from KITTI database: (a) Color image and (b) its estimated disparity using [11] Lastly, we applied the proposed RR-DQM to evaluate the performance of depth sensors To this end, we used the Kinect dataset [16, 22, 30, 32], which includes aligned pairs of color and depth images captured by different depth sensors In particular, 1449 and 3485 test image pairs captured by the Kinect v1 and v2 sensors were used for RR-DQM measurement, respectively Figure 15 shows the first three test image pairs in the dataset Because all the Table The R scores for 10 image pairs in the KITTI database Image ID Random noise LDPC distortion depth distortion prediction error depth distortion prediction error 10 0.999 0.992 0.978 0.990 10 0.999 0.996 0.983 0.985 10 0.999 0.993 0.983 0.996 10 0.999 0.992 0.975 0.997 10 0.998 0.989 0.983 0.964 10 0.999 0.988 0.977 0.993 10 0.998 0.978 0.978 0.972 15 10 0.999 0.992 0.981 0.992 17 10 0.999 0.991 0.980 0.988 19 10 0.997 0.991 0.985 0.981 Multimed Tools Appl Fig 14 The RR-DQM results of the depth refinement algorithm: (a) Cones and (b) T eddy images were captured in indoor environments and the type and size of captured objects are very similar, we can estimate the performance of the depth sensor by comparing the average RR-DQM scores The average RR-DQM scores of the Kinect v1 and v2 sensors were obtained as 14.676 and 13.016, respectively, which illustrate the superiority of the Kinect v2 sensor over the Kinect v1 Fig 15 The first three image pairs of the Kinect dataset [16, 22, 30, 32] (a) Color (left) and depth (right) image pairs captured by the Kinect v1 sensor, (b) color (left) and depth (right) image pairs captured by the Kinect v2 sensor Multimed Tools Appl Conclusion In this paper, we proposed a depth quality assessment technique that does not require the ground truth depth image or the stereo image pair Based on the analysis using the synthetic image, the strong correlation between local depth distortion and the local image characteristic is verified Then, the depth distortion is measured depending on the edge directions In addition, the Gabor filter is used to adaptively weight local depth distortion The experimental results show that the proposed metric closely approximates the conventional depth quality metrics that necessitate the additional information Since the color image is usually captured together with the depth image in the depth camera applications, the proposed quality metric can be used to assess the performance of the depth camera Also, depth image refinement algorithms can adopt the proposed metric for the termination criterion of the refinement In depth based image rendering, the proposed metric can be employed to predict the quality of the image to be rendered Acknowledgments Dr Thanh-Ha Le’s work was supported by the basic research projects in natural science in 2012 of the National Foundation for Science & Technology Development (Nafosted), Vietnam (102.01-2012.36, Coding and communication of multiview video plus depth for 3D Television Systems) Prof Seung-Won Jung’s research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Science, ICT & Future Planning (NRF-2014R1A1A2057970) References Chai B-B, Sethuraman S, Sawhney HS, Hatrack P (2004) Depth map compression for real-time viewbased rendering Pattern Recogn Lett 25(7):755–766 Daugman J (1985) Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters Opt Soc Amer J A: Opt Image Sci 2:1160–1169 D’Angelo A, Barni M, Menegaz G (2007) Perceptual quality evaluation of geometric distortions in images In: Proceedings SPIE, Human Vision and Electronic Imaging XII, pp 1–12 D’Angelo A, Zhaoping L, Barni M (2010) A full-reference quality metric for geometrically distorted images IEEE Trans Image Process 19(4):867–881 Fehn C (2004) Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3d-TV In: Proceedings of SPIE, pp 93–104 Fehn C, De la Barr´e R, Pastoor S (2006) Interactive 3-DTV-concepts and key technologies Proc IEEE 94(3):524–538 Gallant J, Braun J, Essen DV (1993) Selectivity for polar, hyperbolic, and cartesian gratings in macaque visual cortex Science 259(5091):100–103 Geiger A, Lenz P, Urtasun R (2012) Are We ready for Autonomous Driving? The KITTI Vision Benchmark Suite In: Conference on Computer Vision and Pattern Recognition Grigorescu C, Petkov N, Kruizinga P (2002) Comparison of texture features based on Gabor filters IEEE Trans Image Process 11(10):1160–1167 10 Grigorescu C, Petkov N, Westenberg MA (2003) Contour detection based on nonclassical receptive field inhibition IEEE Trans Image Process 12(7):729–739 11 Guney F, Geiger A (2015) Displets: Resolving Stereo Ambiguities using Object Knowledge In: Conference on Computer Vision and Pattern Recognition 12 Ha K, Bae S.-H., Kim M (2013) An objective no-reference perceptual quality assessment metric based on temporal complexity and disparity for stereoscopic video IEIE Trans Smart Signal Comput 2(5):255– 265 13 Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with microsoft kinect sensor: a review IEEE Trans Cybern 43(5):1318–1334 14 Hartley RI (1999) Theory and practice of projective rectification Int J Comput Vis 35(2):115–127 Multimed Tools Appl 15 Jain M, Gupta M, Jain NK (2014) The design of the IIR differintegrator and its application in edge detection J Inf Process Syst 10(2):223–239 16 Janoch A, Karayev S, Jia Y, Barron JT, Fritz M, Saenko K, Darrell T (2011) Acategory-level 3-d object dataset: putting the kinect to work In: Proceedings of ICCV Workshop on Consumer Depth Cameras for Computer Vision, pp 1168–1174 17 Jung S-W (2014) Image contrast enhancement using color and depth histograms IEEE Signal Process Lett 21(4):382–385 18 Khongkraphan K (2014) An efficient color edge detection using the Mahalanobis distance J Inf Process Syst 10(4):589–601 19 Klaus A, Sormann M, Karner K (2006) Segment-based stereo matching using brief propagation and a self-adapting dissimilarity measure In: Proceedings of IEEE Conference on Pattern Recongnition, pp 15–18 20 Le T-H, Lee S, Jung S-W, Won CS (2015) Reduced reference quality metric for depth images In: Proceedings Advanced Multimedia and Ubiquitous Engineering, pp 117–122 21 Scharstein D, Szeliski R (2002) A taxonomy and evaluation of dense two-frame stereo correspondence algorithms Int J Comput Vis 47(1-3):7–42 22 Silberman N, Hoiem D, Kohli P, Fergus R (2012) Indoor segmentation and support inference from rgbd images In: Proceedings of European Conference on Computer Vision, pp 746–760 23 Smith SM, Brady JM (1997) SUSAN-A new approach to low level image processing Int J Comput Vis 23(1):45–78 24 Smolic A, Mueller K, Stefanoski N, Ostermann J, Gotchev A, Akar GB, Triantafyllidis G, Koz A (2007) Coding algorithms for 3DTV-a survey IEEE Trans Circ Syst Video Technol 17(11):1606–1621 25 Smolic A, Mueller K, Merkle P, Kauff P, Wiegand T (2009) An overview of available and emerging 3D video formats and depth enhanced stero as efficient generic solution In: Proceedings of 27th Conference on Picture Coding Symposium, pp 389–392 26 Strecha C, Fransens R, Gool L V (2006) Combined depth and outlier estimation in multi-view stereo In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp 2394–2401 27 Um G.-M., Kim KY, Ahn C, Lee K H (2005) Three-dimensional scene reconstruction using multiview images and depth camera, pp 271–280 28 Vipparthi S, Nagar S (2014) Color directional local quinary patterns for content based indexing and retrieval Human-centric Comput Inf Sci 4(6):1–13 29 Wang Z-F, Zheng Z-G (2008) A region based stereoo matching algorithm using cooperative optimization In: Proceedings of IEEE Conference on Computer Vision and Pattern Recongnition, pp 1–8 30 Xiao J, Owens A, Torralba A (2013) SUN3d: A database of big spaces reconstructed using SfM and object labels In: Proceedings International Conference on Computer Vision, pp 1–8 31 Yang Q, Yang R, Davis J, Nister D (2007) Spatial-depth super resolution for range images In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pp 1–8 32 Zhou B, Lapedriza A, Xiao J, Torralba A, Oliva A (2014) Learning deep features for scene recognition using places database In: Proceedings of Advances in Neural Information Processing Systems, pp 1–9 Thanh-Ha Le received the B.S and M.S degrees in information technology from College of Technology, Vietnam National University, in 2005 He received the Ph.D degree from the Department of Electronics Engineering at Korea University He is now a researcher with the Faculty of Information Technology, University of Engineering and Technology, Vietnam National University, Hanoi, Vietnam His research interests are video processing, image processing, robot vision, and robotics Multimed Tools Appl Seung-Won Jung received the B.S and Ph.D degrees in electrical engineering from Korea University, Seoul, Korea, in 2005 and 2011, respectively He was a Research Professor with the Research Institute of Information and Communication Technology, Korea University, from 2011 to 2012 He was a Research Scientist with the Samsung Advanced Institute of Technology, Yongin-si, Korea, from 2012 to 2014 He is currently an Assistant Professor at the Department of Multimedia Engineering, Dongguk University, Seoul, Korea He has published over 40 peer-reviewed articles in international journals His current research interests include image enhancement, image restoration, video compression, and computer vision Chee-Sun Won received the B.S degree in electronics engineering from Korea University, Seoul, in 1982, and the M.S and Ph.D degrees in electrical and computer engineering from the University of Massachusetts, Amherst, in 1986 and 1990, respectively From 1989 to 1992, he was a Senior Engineer with GoldStar Co., Ltd (LG Electronics), Seoul, Korea In 1992, he joined Dongguk University, Seoul, Korea, where he is currently a Professor in the Division of Electronics and Electrical Engineering He was a Visiting Professor at Stanford University, Stanford, CA, and at McMaster University, Hamilton, ON, Canada His research interests include MRF image modeling, image segmentation, robot vision, image resizing, stereoscopic 3D video signal processing, and image watermarking ... not always obtainable in the depth- image- based rendering (DIBR) applications [1, 5] In this paper we introduce a new depth quality metric, which requires only a pair of color and depth images and, ... image is not always attainable in most practical applications An alternative method is to evaluate the quality of the reconstructed color image obtained using the depth image For example, the right-viewpoint... right-viewpoint image can be rendered by using the left-view image and depth image, and the rendered image can be compared with the original right-viewpoint image [12] However, such a color image pair is