Đang tải... (xem toàn văn)
This paper proposes a Histogram Based Exposure Time Selection (HBETS) method to automatically adjust proper exposure time of each lens for different scenes. It guarantees at least two valid reference values for HDR image processing. Adopting the proposed weighting function restrains random distributed noise caused by micro-lens and produces a high quality HDR image.
Nghiên cứu khoa học công nghệ A HIGH DYNAMIC RANGE IMAGING ALGORITHM AND ITS IMPLEMENTATION FOR A BY CAMERA ARRAY Vu Hong Son*, Pham Ngoc Thang Abstract: Camera specification becomes smaller and smaller accompanied with great strides in technology and thinner product demands, which leads to some challenges and problems One of those problems is that micro lens captures less light than normal lens, which makes low quality noise-image Moreover, current image sensor cannot preserve whole dynamic range in real world High Dynamic Range (HDR) image with multi-exposure images overcomes the problems mentioned above Choosing good exposure time is a seldom-discussed but important issue in HDR imaging technology This paper proposes a Histogram Based Exposure Time Selection (HBETS) method to automatically adjust proper exposure time of each lens for different scenes It guarantees at least two valid reference values for HDR image processing Adopting the proposed weighting function restrains random distributed noise caused by micro-lens and produces a high quality HDR image An integrated tone mapping methodology, which keeps all details in bright and dark parts when compressing the HDR image to Low Dynamic Range (LDR) image for being displayed on monitors is also proposed We first align these images to the same plane, and then adopt the proposed methods The result image has extended the dynamic range, i.e comprehensive information is provided Eventually, we implement the entire system on Adlink MXC-6300 platform that can reach 10fps to demonstrate the feasibility of the proposed technology Keywords: Auto-exposure; HDR image; Tone mapping INTRODUCTION In recent years, HDR imaging technology becomes more and more popular An HDR imaging system for micro camera array is composed of many stages such as auto-exposure control, HDR generation, and tone mapping HDR imaging requires multiple exposed photographs to reproduce higher quality and clearer images One kind of these methods is using bracketing [1]–[5], which captures the different-exposure image sequence by adjusting Exposure Value (EV) The other kind of methods is brute force, which photographs lots of different-exposure images with no pixels over-exposed and underexposed Benjamin Guthier et al [6] exploited pre-established HDR radiance histogram to derive the exposure time, which satisfies the user-defined shape of LDR histogram O Gallo et al [7] also proposed an approach to estimate HDR histogram of the scene, and selected the appropriate exposure times for LDR images HDR image processing technology can mainly be classified into two methods, i.e exposure fusion and recovering high dynamic range radiance maps Both approaches require multiple exposed photographs to reproduce higher quality and clearer images Image fusion technologies [8], [14]–[16] have been developed for several years, which include depth-of-field extension [10], image enhancement [11], and multi-resolution image fusion [12] Fusion of multi-exposure images [15] proposed by Goshtasby is a famous approach to reproduce high quality image, but it cannot handle the boundary of objects perfectly Exposure fusion technology proposed by Mertens et al [8] generates an ideal image by preserving the perfect portion of the multiple different exposure images Fusion process technique described in [8] inspired by Burt and Adelson [13] transforms the domain of image, and adopts multiple resolutions generated by pyramidal image decomposition Debevec et al [9] proposed a method, which uses differently exposed Tạp chí Nghiên cứu KH&CN quân sự, Số Đặc san ACMEC, 07 - 2017 191 Điều khiển – Cơ điện tử - Truyền thông photographs to recover the camera response function and blends multiple exposed images into a single high dynamic range radiance map The final stage of the HDR imaging system is the tone mapping that is required to compress HDR image to LDR one The tone mapping approaches can be classified into two categories, i.e local tone mapping and global tone mapping Fattal et al [17] proposed a local tone mapping method, called gradient domain HDR compression This method is based on the changes of luminance in high dynamic range image It uses diffident levels of attenuation to compress high dynamic range according to the magnitude of the gradient Reinhard et al [18] proposed a global tone mapping method, called linear mapping approach In this paper, we develop a high dynamic range imaging algorithm and its implementation for a by camera array with more implementation details and additional experimental results than our previous work [20] The rest of this paper is organized as follows: Section describes the proposed algorithm that combines a histogram based exposure time selection, new weighting function and integrated tone mapping Section presents experimental results and performance analysis Finally, the conclusion is given in Section PROPOSED ALGORITHM In order to achieve a high quality and high dynamic range imaging system, we propose a system that can deal with higher noises of the low dynamic range images captured by using micro camera arrays In addition, by using the proposed algorithm, all details in the extreme scene can be completely preserved The design flow of the overall system is shown in Figure Figure The design flow of the proposed HDR system As shown in Figure 1, the proposed algorithm is composed of many stages for different purposes The upper part represents the initialization of system, and the others indicate multi-exposure high dynamic range imaging generation and tone mapping In the histogram based exposure time selection stage, appropriate images are chosen for generating high quality HDR images Then, the new weighting function is used in HDR generation stage Eventually, those pixel values of high dynamic range image over 255 must be compressed through the tone mapping stage for display The details of each stage of the proposed work are presented in the following paragraphs A Image Alignment An image alignment consists of the mathematical relationships that map pixel coordinates from source images to target image, is used due to each camera in camera array has its own viewpoint A feature-based method is adopted to accomplish image alignment, which is described in the following The feature point, which has information 192 V H Son, P N Thang, “A high dynamic range imaging algorithm… by camera array.” Nghiên cứu khoa học công nghệ about the position and its descriptor, is extracted from images We can recognize the similarity among these features in different images by the feature descriptors In order to calibrate images to the same coordinate system, the homography matrix, which is a three by three coordinate transformation matrix, is adopted Only eight elements are needed in light of a two-dimensional image, as shown in (1) The relationship between the original coordinate and the objective coordinate is represented by (2) and (3) x' h11 h12 y' h 21 h 22 z' h 31 h 32 x' h13 x h 23 y h11 x h12 y h13 , h x h22 y h23 y ' 21 h31 x h32 y h31 x h32 y (1) (2) x' , Y y' , Z (3) z' z' B Histogram Based Exposure Time Selection (HBETS) We propose a method called Histogram Based Exposure Time Selection (HBETS) to choose suitable source images to generate the HDR images The flow of HBETS is shown in Figure and Figure in [20] Figure shows the generated HDR image by using the source images captured by the proposed HBETS method Comparing Figure with Figure 2, we can see that the red box region in Figure has higher performance than that in Figure after adopting HBETS to guarantee two effective pixel values, one of which is a redundant pixel value to reckon as a remedy to suppress the noise effect, and construct a higher quality HDR image X Figure The HDR result image with distortion Figure The HDR resulting image generated by using the source images chosen by the proposed HBETS C HDR Generation for Image Continuity The camera response function curve g(x) has intense slope near the maximum and minimum pixel values, so g(x) is considered to be less smooth and more inaccurate near these two sides To overcome this, Debevec et al [10] proposed the triangle weighting function that highlighted the importance in the middle of pixel values In the case of different exposure, short exposure images generally have larger noise than long exposure images The micro camera array composed of small lens receives less amount of light than common cameras The ISO value of micro camera should be increased for enhancement However, noise is also amplified After applying the process of the Debevec’s weighting function, the noise dominates the pixel value Hence, this resulting pixel value is not the realistic luminance In order to overcome the drawbacks mentioned above, we propose a new weighting function to enhance HDR image quality as shown in [20] This weighting Tạp chí Nghiên cứu KH&CN quân sự, Số Đặc san ACMEC, 07 - 2017 193 Điều khiển – Cơ điện tử - Truyền thông function can suppress more noises than Debevec’s weighting function does in the result image as shown in Figure 4, where Figure 4(a) uses the weighting function proposed by Debevec, and Figure 4(b) uses the proposed weighting function (b) (a) Figure Comparison of adopting two different weighting functions (a) HDR image using Debevec’s weighting function, (b) HDR image using the proposed weighting function Moreover, we utilize Gaussian filter and Laplacian filter to denoise and enhance image for further improving the image quality The method of applying Gaussian filter is to obtain a smoother image through a convolution of image with a normal Gaussian distribution model A three by three Gaussian kernel is used to achieve this target as shown in Table I(a) Then, we adopt Laplacian filter to further enhance the image quality by strengthening the region changing rapidly such as edges and making image clearer, as shown in Table I(b) Table I Two Different Enhancement Kernels Adopted in The Proposed Algorithm (a) Gaussian Kernel (b) Laplacian Kernel (a) Gaussian kernel (b) Laplacian kernel 0.0751 0.1238 0.0751 -1 0.1238 0.2042 0.1238 -1 -1 0.0751 0.1238 0.0751 -1 In addition, we consider that the pixel having large value in short exposure than the one in corresponding long exposure has a higher chance of noise by reason of noise characteristics Consequently, there is a correction on the problematic pixel value The average of eight pixels which are the neighborhood of the problematic pixel in short exposure image is calculated, and used to replace the problematic pixel As shown in Figure 5, the noise (i.e red dot in the Figure 5(a)) is eliminated by the proposed method of pixel correction (a) Before denoising (b) After denoising Figure Denoised image by applying the proposed method D Integrated Tone Mapping There are two major kinds of tone mapping techniques, i.e global tome mapping and local tone mapping The global tone mapping technique such as photographic compression uses a fixed formula for each pixel in compressing HDR image into LDR image This approach is relatively fast, but it loses details in high luminance regions On the other 194 V H Son, P N Thang, “A high dynamic range imaging algorithm… by camera array.” Nghiên cứu khoa học công nghệ hand, the local tone mapping technique such as gradient domain compression refers to nearby pixel values before compression As a result, all details can be retained, but it takes a lot of computation time Since both kinds of tone mapping methods have pros and cons, this motivate us to propose a new tone mapping approach that can preserve details in bright regions accompanying with lower computation time Figure 6(a) to Figure 6(d) show four input images captured respectively with exposure time 0.33 ms, 2.10 ms, 10.49ms, and 66.23 ms, selected by the proposed HBETS method Meanwhile, Figure 6(e) demonstrates photographic tone mapping, and Figure 6(f) to Figure 6(h) are images used in the proposed algorithm with the scaling parameters 0.8, 0.5, and 0.2, respectively Photographic tone mapping lost details in bright regions (e.g the shape of lamp and the word near the lamp) In the proposed tone mapping method, large scaling parameter leads to discontinuity and small scaling parameter causes the unclear details Hence, some corrections are put on (4) (a) (b) (c) (d) (e) (i) (g) (h) (f) Figure (a) to (d) show four input images captured respectively with exposure time 0.33 ms, 2.10 ms, 10.49ms, and 66.23 ms, selected by the proposed HBETS method (e) demonstrates the result by using photographic tone mapping (f) to (h) are images used in the proposed algorithm with the scaling parameters 0.8, 0.5, and 0.2, respectively (i) is the final result of the proposed tone mapping Our idea is to blend two lower exposure source images first, which preserves details and also adjusts the brightness for image continuity, and then use the same equation to gain the result image, as shown in (5) I result (x, y) (1 - ) * I photograph (x, y) * I ' source (x, y) (4) I ' source (x, y) (1 ) * I source1 ( x, y ) * I source ( x, y ) (5) where α is shown in (6), and β is also a Gaussian-like function as illustrated in (7) * exp I photograph ic u v * exp ( x , y ) 255 255 I threshold I source ( x, y ) 2552 255 Tạp chí Nghiên cứu KH&CN quân sự, Số Đặc san ACMEC, 07 - 2017 (6) (7) 195 Điều khiển – Cơ điện tử - Truyền thông where u is a constant value which dominates the weighting to the two image’s pixel values, v is a scaling parameter and Isource2 is the second low exposure image’s luminance By using the proposed tone mapping method as shown in (5), the result image, which retains details in the brightness regions and keeps color continuity, can be acquired as shown in Figure 6(i) In our experiment, the setting of γ as 0.5, u as 0.1, and v as 0.4 obtains a better result Comparing Figure 6(i) with Figure 6(e) to Figure 6(h), the word and the texture of the lamp in the scene by using the proposed algorithm can be preserved comprehensively EXPERIMENTAL RESULTS AND PERFORMANCE ANALYSIS We have implemented the proposed 4-CAM HDR system on Adlink MXC-6300 platform that can reach VGA video @10 fps The camera array consists of four Logitech webcams is shown in Figure Figure Four Logitech webcams to form a 4x1 camera array (a) (b) Table II Computation Time Analysis of The Proposed Algorithm Execution time (ms) Functions Before After optimization optimization Alignment 31.01 19.10 HDR generation 61.48 28.51 Tone Mapping 44.99 39.27 HBETS and data 23.03 14.62 type conversions Total 181.47 101.50 (c) (d) (e) (f) Figure Experimental results of the proposed HDR algorithm preserving more details in dark regions (a) to (d) are input images, (e) is the result image of easyHDR, and (f) is the result image of the proposed method Table II shows a detailed computational complexity analysis of code optimization for the proposed algorithm From Table II, we can see that the proposed design achieves 1.79 times faster processing speed after code optimization Besides, Table II also shows that tone mapping and HDR generation are the two most computation intensive computations which occupy near 70% computation time of the proposed HDR algorithm To further 196 V H Son, P N Thang, “A high dynamic range imaging algorithm… by camera array.” Nghiên cứu khoa học công nghệ illustrate the validity of the proposed algorithm, we show the high dynamic range imaging results processed by the proposed algorithm, and compare those to a well-known commercial software tool called easyHDR [19] in the following (a) (b) (c) (d) (f) (e) Figure Experimental results of the proposed HDR algorithm achieving more saturated scene (a) to (d) are input images, (e) is the result image of easyHDR, and (f) is the result image of the proposed method The exposure times of input images are chosen by the proposed HBETS approach with a four by camera array, which means four source images are used to generate an HDR image Figure demonstrates the proposed HDR result image, where Figure 8(a) to Figure 8(d) are input images; Figure 8(e) is the result image of easyHDR; and Figure 8(f) is the result image of the proposed algorithm Experimental results shown in Figure indicate that the proposed algorithm preserves more details in dark region than easyHDR The texture and the words inside the box are able to be viewed clearly, and vibrant scene is given in the result image by applying the proposed method Comparing easyHDR result image shown in Figure 9(e) and the result image adopting the proposed method shown in Figure 9(f), we can observe that using the proposed algorithm generates more saturated image in HDR CONCLUSION The proposed HDR system has been implemented on a four-by-one micro-camera array so that the four source images can be used to generate HDR image The proposed histogram-based adaptive exposure time selection, which conquers the problem of extreme environment that auto-exposure system cannot afford, not only enhances image contrast but also keeps the image details in light-regions, and it also reduces the noise effect HDR system in camera array records comprehensive details at extremely low-light scene, which could be applied on car event recorders, surveillance systems, HDR movies, smart phones, and etc This work solves severe conditions such as night vision, and provides better visibility of the video at night Moreover, for entertainment applications, movie filmed with this technique will produce realistic videos for human visual perception Higher quality in back-lighted scene can also be achieved by the proposed design The proposed HDR system makes cameras achieving high dynamic range close to that of human eyes REFERENCES [1] Reinhard, E., Pouli, T., and Cunningham, D., “Image Statistics: From Data Collection to Applications in Graphics,” Proc ACM SIGGRAPH, no 6, 2010 Tạp chí Nghiên cứu KH&CN quân sự, Số Đặc san ACMEC, 07 - 2017 197 Điều khiển – Cơ điện tử - Truyền thông [2] M A Robertson, S Borman, and R L Stevenson, “Estimation-theoretic approach to dynamic range enhancement using multiple exposures,” J Electronic Imaging, pp 219–285, Apr 2003 [3] R.C Bilcu,A Burian, A Knuutila, M Vehvilainen, “High Dynamic Range Imaging on Mobile Devices,” Proc IEEE International Conference on Electronics, Circuits and Systems, pp 1312–1315, 2008 [4] M A Robertson, S Borman, and R L Stevenson, “Dynamic Range Improvement through Multiple Exposures,” Proc International Conference Image Processing, pp 159–163, 1999 [5] W C Kao, C C Hsu, C C Kao, and S H Chen, “Adaptive Exposure Control and Real-time Image Fusion for Surveillance Systems,” Proc IEEE International Symposium on Circuits and Systems, pp 935–938, 2006 [6] B Guthier, K Ho, S Kopf, and W Effelsberg, “Determining exposure values from HDR histograms for smartphone photography,” Proc International Conference on Multimedia, pp 425–426, 2013 [7] Gallo O., Tico, M., Manduchi, R., Gelfand, N., and Pulli, K., “Metering for Exposure Stacks,” Proc Computer Graphics Forum, pp 479–488, 2012 [8] T Mertens, J Kautz, and F V Reeth, “Exposure fusion,” Proc Pacific Graphics, pp 382–390, 2007 [9] P Debevec and J Malik, “Recovering High Dynamic Range Radiance Maps from Photographs,” Proc ACM SIGGRAPH, pp 369–378, 1997 [10] E Eisemann and F Durand, “Flash photography enhancement via intrinsic relighting,” Proc ACM SIGGRAPH, pp 673–678, 2004 [11] R Raskar, A Ilie and J Yu, “Image fusion for context enhancement and video surrealism,” Proc International Symposium Non-Photorealistic Animation and Rendering, pp 85–94, 2004 [12] T Stathaki, “Image Fusion: Algorithms and Applications,” Academic 2008 [13] P J Burt and E H, Adelson, “The Laplacian Pyramid as a Compact Image Code,” IEEE Trans Commun., vol Com-31, no 4, pp 532–540, 1983 [14] M Tico, N Gelfand, and K Pulli, “Motion blur free exposure fusion,” Proc IEEE International Conference on Image Processing, pp 26–29, 2010 [15] A A Goshtasby, “Fusion of multi-exposure images,” Image and Vision Computing, vol 23, no 6, pp 611–618, Jun 2005 [16] Kong, J et al., “A Novel Fusion Approach of Multi-exposure Image,” Proc International Conference on “Computer as a Tool”, pp 163–169, 2007 [17] R Fattal, D Lischinski and M Werman, “Gradient domain high dynamic range compression,” Proc ACM SIGGRAPH, pp 249–256, 2002 [18] E Reinhard, M Stark, P Shirley and J Ferwerda, “Photographic tone reproduction for digital images,” Proc ACM SIGGRAPH, pp 267–276, 2002 [19] easyHDR, “High Dynamic Range photography made easy”, http://www.easyhdr.com/ [20] P H Huang, Y H Maio, and J I Guo, “High Dynamic Range Imaging Technology for Micro Camera Array,” Proc 2014 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, pp.1–4, 2014 198 V H Son, P N Thang, “A high dynamic range imaging algorithm… by camera array.” Nghiên cứu khoa học cơng nghệ TĨM TẮT MỘT THUẬT TỐN TẠO ẢNH DẢI ĐỘNG CAO VÀ SỰ THỰC HIỆN CỦA NÓ CHO MỘT MẢNG CAMERA 4x1 Đặc điểm camera trở lên nhỏ kèm theo tiến vượt bậc công nghệ nhu cầu sản phẩm mỏng dẫn tới số thách thức vấn đề Một vấn đề thấu kính nhỏ thu ánh sáng yếu so với thấu kính thơng thường, tạo ảnh - nhiễu chất lượng thấp Ngoài ra, cảm biến ảnh khơng thể bảo tồn tồn dải động giới thực Ảnh HDR với nhiều hình ảnh phơi sáng có khả khắc phục vấn đề đề cập Lựa chọn thời gian phơi sáng tốt vấn đề thảo luận lại vấn đề quan trọng kỹ thuật tạo ảnh HDR Bài báo đề xuất phương pháp lựa chọn thời gian phơi sáng vào biểu đồ để tự động điều chỉnh thời gian phơi sáng phù hợp thấu kính cho ngữ cảnh khác Nó đảm bảo hai giá trị tham chiếu hợp lệ cho xử lý ảnh HDR Thông qua hàm trọng số đề xuất để hạn chế nhiễu phân phối ngẫu nhiên sinh thấu kính nhỏ tạo ảnh HDR chất lượng cao Một phương pháp tích hợp ánh xạ sắc để giữ tất chi tiết phần tối sáng nén ảnh HDR cho ảnh dải động thấp để hiển thị hình đề xuất Đầu tiên ảnh mặt phẳng, sau thơng qua phương pháp đề xuất Ảnh kết mở rộng dải động, tức thơng tin tồn diện cung cấp Cuối chúng tơi thực tồn hệ thống Adlink MXC-6300, đạt 10 khung hình giây để chứng minh cho tính khả thi kỹ thuật đề xuất Từ khóa: Phơi sáng tự động; Hình ảnh HDR; Ánh xạ sắc Received date, 02nd May, 2017 Revised manuscript, 10th June, 2017 Published, 20th July, 2017 Author affiliations: Hung Yen University of Technology and Education *Corresponding author: hongson.ute@gmail.com Tạp chí Nghiên cứu KH&CN quân sự, Số Đặc san ACMEC, 07 - 2017 199 ... Proc 20 14 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, pp .1 4, 20 14 19 8 V H Son, P N Thang, A high dynamic range imaging algorithm by camera array. ”... 2002 [19 ] easyHDR, High Dynamic Range photography made easy”, http://www.easyhdr.com/ [20] P H Huang, Y H Maio, and J I Guo, High Dynamic Range Imaging Technology for Micro Camera Array, ” Proc... the other 1 94 V H Son, P N Thang, A high dynamic range imaging algorithm by camera array. ” Nghiên cứu khoa học công nghệ hand, the local tone mapping technique such as gradient domain compression