New strategies for generating panoramic images for imperfect image series 4

Chapter Dual-Homography Image Stitching 3.1 Introduction As presented in the previous chapters, although image mosaicing has received a great deal of attention over the years, stitching a sequence of images that not meet the imaging assumptions (i.e. an imperfect images series) is still a challenging problem. Due to the parallax effect, the traditional image mosaicing pipeline which uses single homography transformation model usually results in misalignments in the stitched images. As previously discussed, these misalignments are then hidden via post-processing techniques, such as image blending or seam cutting, to varying degrees of success, often producing mosaics that appear seamless at a quick glance, only to reveal breaks and tears on closer inspection. In this chapter, we focus on one common type of scene that is particularly troublesome for existing methods: a scene containing both a distant plane and a ground plane that sweeps out from the camera’s location. Figure 3.1 shows an illustration of this type of scene. There are two main reasons that we choose this 33 CHAPTER 3. Dual-Homography Image Stitching Distant Plane Ground Plane Figure 3.1: A scene containing two dominant planes targeted by our mosaicing approach. specific scene as our input series. Firstly, this type of scene is commonly found at tourist locations where panoramic imaging is desired. When the photographer is standing on a flat ground and targeting on the landscape far away, the taken image series usually contains these two dominant planes. The other reason is that the parallax of this type of scene is between this two continuous plane. Without abrupt structures that cross the scene, it is possible to smoothly align these two planes respectively and leave the misalignment to the unnoticeable part of the joint region. The work presented here describes how to address this type of panoramic scene before the post-processing stage by focusing on how to improve the image alignment. This is achieved by estimating two homographies per image-pair and 34 3.2. System Overview blending these homographies to align the images. As demonstrated by our results, this approach can produce more seamless results than those obtained using current state-of-the-art methods that rely on a single homography per image pair. While a seemingly straight-forward idea, given that our scene has two dominate planes, there are several issues addressed in this chapter that must be considered to make this approach work. These issues include estimating the dual homographies from matched points, weights for blending the homographies, extending the nonlinear warping to concatenate adjacent images, and post-processing to blend the images and to reduce undesired curvilinear artifacts introduced by the nonlinear warping. 3.2 System Overview Our dual-homography stitching system follows the traditional image mosaicing pipeline. When a series of input images is given, the first step of our system is to detect the feature points and compute transformation between each pair of neighboring images. Unlike the traditional single homography transformation model, we use our dual-homography model to provide a more flexible transformation which tries to align the two dominant planes at same time. After we obtain the dual-homography transformation for each pair of the images, a full stitched panorama result is then generated by concatenating the images one by one onto the virtual canvas. Similar as the traditional image mosaicing pipeline, seam-cut is applied to the aligned result as a post-processing step which tries to remove the misalignment artifacts. Unique to our system, since the dual-homography may introduce curvilinear artifacts due to the interpolation of two homographies, a content-aware straightening process is applied to achieve a visually appealing 35 CHAPTER 3. Dual-Homography Image Stitching result which remove these artifacts. Based on the challenges of each step in our dual-homography pipeline, the following of this chapter is organized as follows: Section 3.3 describes our dual-homography estimation and blending for an input image pair and its extension for multiple image concatenation; Section 3.4 describes post-processing procedures to reduce visual seams and rectify curvilinear artifacts; Section 3.5 demonstrates results obtained using our approach followed by a discussion and summary of our work in Section 3.6. 3.3 Dual-Homography Alignment We first describe how to compute the two homographies for a pair of overlapping images, and show how to use the dual-homographies to perform nonlinear alignment. We then describe how to extend this warping to adjacent images in the mosaic. For the sake of simplicity, we assume that each image in our panorama has at most two overlapping regions with images on either side of its borders. 3.3.1 Dual-Homography Estimation The idea of blending homographies is to find a smooth interpolating warping field to register the pair of input images. As we know, for the objects which are lying on the same plane in the target scene, a single homography model can represent the transformation between the planes. Thus, to register two dominant planes inside the scene with parallax, a natural solution is to compute a homography for each plane and assign the ambiguous regions a smooth interpolating transformations. Following this idea, our approach models the relationship between two overlapping images in our panorama using a blending of two homographes, H g and Hd , 36 3.3. Dual-Homography Alignment Input image pair and clustered match points Distant plane homography Hd Ground plane homography Hg Weight map ωij Dual homography Hij = ωij Hg+(1-ωij)Hd Figure 3.2: Work flow of our dual-homography computation. Candidate points are first clustered into groups from which two homographies are estimated using RANSAC. A per-pixel weight map ωi j is computed to control the blending of the two homographies. 37 CHAPTER 3. Dual-Homography Image Stitching expressed as: Hi j = ωi j H g + (1 − ωi j )Hd , (3.1) where H g and Hd represents the ground plane and distant plane homographies respectively, and ωi j is a per pixel weight that controls the contribution of each homography at the pixel location (i, j). As described in the Chapter 2, our approach begins with a set of corresponding points matched between the input images. We use SIFT features [42] to establish correspondences as done in [16]. Since we are computing two homographies per image pair, our first step is to cluster the correspondences into two groups G g and Gd based on their spatial location in the image. This is done using K-means clustering, with 2D seed points taken to be: cg = n i n xi T ,0 , cd = n i n xi T ,h , (3.2) where h is the height of input image. These seed points are selected to ensure that the final clusters are oriented about the top and bottom of the image, since we expect that the distant plane and ground plane will most likely reside at the top and bottom of the image respectively. We note that this initial step does not need to be too accurate as outliers will be discarded in the homography estimation procedure. This approach also assumes images are taken with the camera held upright. After the feature points are grouped, RANSAC [24] is used to robustly estimate the homography for each group. In each trial, we select four matched feature points to compute the homography. If the consensus reaches 95%, the estimated homography H g and Hd is considered as the transform of its corresponding group of features. After RANSAC is performed, outlier SIFT features are removed to form 38 3.3. Dual-Homography Alignment new groups G g and Gd . The two resulting homographies can partially align the image as shown in Figure 3.2. The goal now is to assign a weight to each pixel location pi j to determine how much of each homography should be used. A natural choice is to weight the warping based on spatial proximity, i.e.: ωi j = d g /(d g + dd ), (3.3) ) to the closest feature points where d g and dd are the inverse squared distances (||·||−2 in the sets G g and Gd respectively. Figure 3.2 shows an example of this weighting map and results from this weighted dual-homography warping. Dual Homography vs. Two Homographies with Explicit Segmentation After the dual-homography stitching model is proposed, a straight-forward question can be raised up: can we use two homographies as the transformation model for each dominant plane, and give the scene a explicit segmentation? Figure 3.3 shows a synthesis case of a scene which contains exactly two explicit dominant planes. Both the dual homography result and the two homography result are provided. From the result we can see that, in ideal cases, since the accurate segmentation between the two planes is known, the two homography result can achieve a fully alignment for each plane, while our dual homography method have mis-alignment artifacts in the middle “vague” region of the scene. This is because our dual homography method does not know the explicit segmentation of the scene. However, things change when it comes to a more realistic case. Figure 3.4 shows a synthesis example of a scene that contains stair structures between two dominant planes. As we can see that, these stair structures breaks the assumption of only two dominant 39 CHAPTER 3. Dual-Homography Image Stitching (a) Synthesized Input images in a scene with two planes (b) Stitched result using two homographies with explicit segmentation (c) Stitched result using dual-homography Figure 3.3: A synthesized example of an ideal scene of two plains and its correspondent results of two homographies with explicit segmentation method and dual-homography method. 40 3.3. Dual-Homography Alignment (a) Synthesized Input images in a scene with two planes and stair structure in the middle (b) Stitched result using two homographies with explicit segmentation (c) Stitched result using dual-homography Figure 3.4: A synthesized example of a more realistic scene of two plains with stair structures in the middle and its correspondent results of two homographies with explicit segmentation method and dual-homography method. 41 CHAPTER 3. Dual-Homography Image Stitching Two homographies with segmentation - Accurate alignment in ideal case Pros - No curvilinear artifacts in ideal case Dual Homography - Do not need the explicitly segmentation - Seamlessly blends in non-ideal case - Need to perform explicit segmentation - Curvilinear artifacts Cons - Breaks for more complex geometry - Not geometrically correct Table 3.1: Pros and cons of the compared two homographies with segmentation method and dual-homography method. planes. In such case, the composite results of both two homography with explicit segmentation result and our dual homography result will have misalignment at the overlapping region. However, for non-overlapping region, since the stair structure not belongs to either of the predominant planes, the two homographies with segmentation method will have this noticeable breaking artifacts at the joint of Pros: Pros: the- Accurate segmentation. The dual-homography on the hand,segment provides a registration in ideal case - method Do not need theother explicitly smooth interpolation in this Tablethe 3.1images has lists the pros and cons of these - No curvilinear artifacts in region. ideal case - Seamlessly blends in non-ideal case two method. From the table we can see that, although using two homographies Cons: - Need to perform explicit segmentation - Curvilinear artifacts ideal case, it simply breaks when the underlying geometry is more complex. Fur- Breaks another for moremain complex geometry Not homographies geometrically with correct thermore, drawback of the- two segmentation with segmentation has the advantages that it can achieve accurate alignment in Cons: method is that it requires explicit segmentation process. Therefore, we believe that our dual-homography method is more practiced. 42 3.3. Dual-Homography Alignment Figure 3.5: This figure illustrates how to concatenate multiple images into the virtual image plane, IV using dual-homographies. Refer to the text in Section 3.2 for details. 3.3.2 Extending to Multiple Images Unlike single homography approaches that can easily concatenate multiple homographies into a single matrix, our nonlinear warping requires more care. Figure 3.5 is used to help illustrate the concatenation procedure using three overlapping images I0 , I1 , and I2 that we wish to map to the virtual mosaic image plane IV . I0 is used as the root image of the mosaic and undergoes no warping. The first dual-homography used to map I1 to I0 is denoted as H1→0 , dropping the subscripts, ij, for clarity. This dual-homography is computed as previously described and is the only mapping required to place I1 in the virtual image: i.e. H1→V = H1→0 . 43 CHAPTER 3. Dual-Homography Image Stitching The task now is to add another dual-homography pair, H2→V , that maps I2 to I1 in the virtual image plane. This is achieved by first computing the dualhomography, H2→1 , to aligned I2 to I1 . This is computed as described in the prior section. Note that this estimation is done agnostic of the relationship between I1 and I0 . However, in the virtual image, IV , image I1 is expressed as H1→0 (I1 ). Thus, in the overlapped region between I2 and I1 , we can relate I2 to the virtual image by H1→0 (H2→1 (I2 )). The question now is what we about the non-overlapped region of I2 with I1 . Unlike single homography approaches where the mapping between homography can be applied to points outside the overlapping region, our warping function H1→0 is not defined for points outside I1 . To ‘virtually extend’ the reach of H1→0 , we use a weighted blending of the boundary points of H1→V , resulting in:     H1→V (H2→1 (p)) , if p ∈ I1 ∩ H2→1 (I2 )       2→V H (p) =      1→V 2→1     q∈ΩB ζq Hq (H (p)) , otherwise, (3.4) where ΩB is the boundary of I1 that overlaps with I2 . The scalar ζq = 1/ H2→1 (p) − q is computed using the Euclidean distance from point p to all the boundary points q on ΩB. Note that these weights are normalized such that ζq = 1. While this procedure has been illustrated using only three images, it is straightforward to derive a recursive expression from Eq. to concatenate further images. 44 3.4. Post-Processing the Mosaic 3.4 Post-Processing the Mosaic This section describes two post-processing techniques, seam blending and straightening, that are used to enhance the aesthetic appearance of our final results. Seam-blending aims to reduce the visible seams between overlapping images, this method was previously described in Section 2.4.2. Content-aware straightening is unique to our mosaicing procedure and is applied to reduce curvilinear effects introduced by the dual-homography warping. 3.4.1 Global Straightening The linear interpolation of the homographies makes the images warp in a quadratic fashion. This can introduce a curved effect that may be aesthetically unappealing for the final result. To remove this effect, we employ a content-aware warping method similar to that used in [69, 71]. In particular, our final virtual image, IV is tesselated into a polygon mesh, where each unit, Q, is a quad. This is done by firstly tessellating each input image into a unit square mesh. For each of the vertices inside these meshes, we compute the position that after dual-homography warping is applied. After that, for each image’s mesh we collect all the vertices that have a neighboring vertex lie in the other image (which means these vertices are along the seam cut). Then line up these vertices into a polyline. Similarly, on the other side of the seam there would be another polyline. Connect these two polylines, a strap-like polygon is achieved. At last, the two meshes are stitched together by triangulating this strap-like polygon. An example of this polygonal mesh is shown in Figure 3.6. Each polygon is composed of vertices, vi , that can be deformed to a new set of 45 CHAPTER 3. Dual-Homography Image Stitching polygons, Q , with the following two distortion energies: Ds (Q, Q ) = s s(vi ) − vi , (3.5) n where vi is the target warping position of vi and s( ·) is a similar transformation function which has a general form:         x   c −d   x   tx             +   , v =   . s(v) =         d c   y t  y y (3.6) This energy attempts to constrain the deformation on all the points vi such that the polygons, Q, only undergo a similarity transformation. The other distortion energy is according to the observation in [16] that people rarely twist the camera relative to the horizon when shooting the panorama images. Hence, we give each point pair v1 and v2 in the vertical edge E a bending cost energy defined as follows: Dl (E, E ) = (v1 − v2 ) · ˆj , (3.7) where ˆj = (0, 1)T is a unit horizontal vector. This energy constrains all the vertical edges in the source image more to maintain a vertical position in the final result. Combining these two energies we can obtain the straightened result as the points set V = {vi } which minimizes the following energy: µ(Q) s Ds + µl D= Q Dl . (3.8) E allows this deformation to be flexible. This weight is Here, the weight µ(Q) s 46 3.4. Post-Processing the Mosaic Figure 3.6: Before and after the application of content-aware straightening. determined by the underlying image content contained within each polygon, Q, defined as: µ(Q) = s n ∇IV (x, y) , ∀IV (x, y) ∈ Q, n (3.9) where ∇IV is the image gradient inside a polygon. This weight polygons with higher image energy to resist deformation, while polygons with lower energy can be deformed more. To enforce the vertical edge constraint, the weight µl we give a comparatively large number which equals to 20 times of max{µ(Q) s } to force the lines straight. This approach is similar to that proposed by [69, 71] for content-aware image resizing, however, we have utilized it in a completely different manner, exploiting its ability to linearize the mesh using Equation 3.8. This system is directly computed using an over-determined linear solver. Figure 3.6 shows examples of our result before and after this straightening procedure. 47 CHAPTER 3. Dual-Homography Image Stitching Closed-form solution for Equation 3.8 According to Equation 3.6, for each polygon, Q = {vi (xi , yi )|1 ≤ i ≤ n} (n = for triangles and n = for quadrilaterals) , it undergoes a similarity transformation s(·) which contains four parameters xs = [c d tx t y ]T . Inside these parameters, c and d are the general rotation and scale parameters and tx , t y represent the translations of x and y direction. Thus, for each vertex in Q, we have   xi −yi  s(vi ) − vi =   y x i i     x       xs −  i  .    yi  (3.10) Then we substitute the error term in Equation 3.5, and we get Ds (Q, Q ) = |AQ xs − yQ |2 (3.11) s where         AQ =        x1 −y1 y1 . x1 . . . xn −yn yn xn                 , and         yQ =        x1 y1 . xn yn          .        (3.12) As we can see, AQ is a 2n × matrix which contains all the position information of the vertices in Q, and yQ that is the desired target coordinates of the vertices after warping. Then the optimal xs can be substituted in closed form by solving AQ xs = yQ in a least square sense: xs = (ATQ AQ )−1 ATQ yQ . Now the error definition can be written as: 48 3.4. Post-Processing the Mosaic Ds (Q, Q ) = |AQ (ATQ AQ )−1 ATQ − I)yQ |2 = |CQ yQ |2 , (3.13) Thus, the polygon similarity error depends on the result position yQ . On the other hand, computing the bending energy in Equation 3.7 is actually to compute the displacement of y-axis along any vertical edge. Thus,    −1   y       Dl (E, E ) = |y1 − y2 |2 =  −1   y 2     (3.14) Similarly to Equation 3.13, we have a form of bending energy depends on the result position yE = {y1 , y2 }. With the constraints for each vertex represented by Equation 3.5 and Equation 3.7, we can construct a large sparse linear system which has the general form Ax = 0, where x = [x1 , y1 , x2 , y2 , . . . , xn , yn ]T is the result coordinates of the vertices. To avoid the zero solution, we let each vertex to have a small bias to the original position. This is done by adding a A = diag( )2n to A and the right of the linear system equation is now b = ( x1 , y1 , . . . , xn , yn )T User mark-up constrains for global straightening While the nonlinear straightening is intended to fix artifacts introduced by the dual-homography warp, it can sometimes introduce noticeable bending on image content. As such, we have a simple UI to allow the user to specify image regions whose shape cannot be changed as shown in Figure 3.7. This can be achieved by increasing the µ(Q) for the s polygons falling within the user specified regions. Figure 3.7 shows an example of how user-markup can be used to protect regions in this straightening process. 49 CHAPTER 3. Dual-Homography Image Stitching Figure 3.7: An example where the user can specify regions that should remain intact in the straightening process. 3.5 Results Our results are compared against three state-of-the-art softwares, AutoStitch [1], Photoshop CS5’s mosaicing feature [50] and Microsoft’s ICE [2]. AutoStitch [1] is based on the work in [15], while Photoshop and ICE appear to combine elements of [15] and seam-cutting [3]. All of these techniques utilize single homography alignment. Images were taken at locations that fit our targeted scene description. When imaging these scenes, we purposely moved the camera in an arc fashion to violate the common center of projection assumption for the input images. Figure 3.9 and Figure 3.10 shows results of our algorithm on a panorama sequence containing five input images. These results are compared with AutoStitch and Photoshop CS5. While all the results appear good at first glance, we highlight 50 3.5. Results Figure 3.8: A failure case that is caused by a large structure that violates our two plane assumption. a few notable regions where results from AutoStitch and Photoshop exhibit tears and misalignments. Our results, however, remain seamless in these regions. Misalignments in the results produced by Photoshop are often the hardest to detect since they use seam-cutting to stitch together images. While not shown here, ICE produced results similar to those by Photoshop on these examples, with slightly more noticeable seams. For the example in Figure 3.9, the user markup shown in section 3.4 was used. Figure 3.11 shows a result comparing our approach with all three softwares. Again, at first glance each approach appears to have performed well, on close inspection we can see artifacts due to alignment issues not exhibited in our results. Note that most of the errors happen in the bottom part of the panoramic image. This implies that the single-homography based approaches are using the points in the distant plane region to compute their homographies, resulting in misalignments in 51 CHAPTER 3. Dual-Homography Image Stitching Figure 3.9: Example of our approach compared with AutoStitch and Photoshop CS5. Example regions with noticeable tears in the single homography-based techniques are shown in the zoomed insets. the ground plane region. We also note that our lack of color balancing of the input images is sometimes noticeable when compared with the results from Photoshop CS5. 52 3.5. Results Figure 3.10: Another example comparing our approach with AutoStitch and Photoshop CS5. Tears in the ground plane are highlighted. Our approach fails when the scene contains a relatively large structure that does not belong to either the ground plane nor distant plane. In such case, the parallax effect will occur since there are more than two homographies involved. Figure 3.8 shows an example of this failure case. 53 CHAPTER 3. Dual-Homography Image Stitching Figure 3.11: Example of our approach compared with AutoStitch, Photoshop, and Microsoft ICE. Regions with noticeable alignment artifacts are shown in the zoom insets. Our results, however, are able to maintain a seamless alignment over the panorama. 54 3.6. Summary The run time of our approach is around 5∼10 seconds for the homography estimation stage, 5∼10 seconds for the seam cut stage and 5∼10 seconds for straightening stage (for a typical five images input). Compare to other softwares, which is around 10∼15 seconds for both AutoStitch and Photoshop, our approach does need a comparatively longer time to process, mainly due to the complex warping procedure. 3.6 Summary In this chapter, we have focused on the image-alignment step of the image mosaicing pipeline to solve the imperfect image stitching problem. The dual-homography warping approach tries to use a more sophisticated model to better simulate the cases when the input image series has two dominant planes. It can be seen from the result that the alignment is improved by using this model. Post-processing techniques were applied to further acquire a seamless and distortion-less result. In the content-aware straightening process, since the straightening warping is based on the visual saliency for each region, sometimes desired result cannot be generated due to the patch-based saliency descriptor. Hence, a user interactive phase is introduced to let the user guide the warping. Limitations and potential future work are discussed in Chapter 6. 55 CHAPTER 3. Dual-Homography Image Stitching 56 [...]... three images, it is straightforward to derive a recursive expression from Eq 4 to concatenate further images 44 3 .4 Post-Processing the Mosaic 3 .4 Post-Processing the Mosaic This section describes two post-processing techniques, seam blending and straightening, that are used to enhance the aesthetic appearance of our final results Seam-blending aims to reduce the visible seams between overlapping images, ... seconds for straightening stage (for a typical five images input) Compare to other softwares, which is around 10∼15 seconds for both AutoStitch and Photoshop, our approach does need a comparatively longer time to process, mainly due to the complex warping procedure 3.6 Summary In this chapter, we have focused on the image- alignment step of the image mosaicing pipeline to solve the imperfect image stitching... examples of our result before and after this straightening procedure 47 CHAPTER 3 Dual-Homography Image Stitching Closed-form solution for Equation 3.8 According to Equation 3.6, for each polygon, Q = {vi (xi , yi )|1 ≤ i ≤ n} (n = 3 for triangles and n = 4 for quadrilaterals) , it undergoes a similarity transformation s(·) which contains four parameters xs = [c d tx t y ]T Inside these parameters, c... multiple images into the virtual image plane, IV using dual-homographies Refer to the text in Section 3.2 for details 3.3.2 Extending to Multiple Images Unlike single homography approaches that can easily concatenate multiple homographies into a single matrix, our nonlinear warping requires more care Figure 3.5 is used to help illustrate the concatenation procedure using three overlapping images I0... techniques utilize single homography alignment Images were taken at locations that fit our targeted scene description When imaging these scenes, we purposely moved the camera in an arc fashion to violate the common center of projection assumption for the input images Figure 3.9 and Figure 3.10 shows results of our algorithm on a panorama sequence containing five input images These results are compared with AutoStitch... of content-aware straightening determined by the underlying image content contained within each polygon, Q, defined as: µ(Q) = s where n IV (x, y) , ∀IV (x, y) ∈ Q, n (3.9) IV is the image gradient inside a polygon This weight polygons with higher image energy to resist deformation, while polygons with lower energy can be deformed more To enforce the vertical edge constraint, the weight µl we give a... the virtual mosaic image plane IV I0 is used as the root image of the mosaic and undergoes no warping The first dual-homography used to map I1 to I0 is denoted as H1→0 , dropping the subscripts, ij, for clarity This dual-homography is computed as previously described and is the only mapping required to place I1 in the virtual image: i.e H1→V = H1→0 43 CHAPTER 3 Dual-Homography Image Stitching The... final virtual image, IV is tesselated into a polygon mesh, where each unit, Q, is a quad This is done by firstly tessellating each input image into a unit square mesh For each of the vertices inside these meshes, we compute the position that after dual-homography warping is applied After that, for each image s mesh we collect all the vertices that have a neighboring vertex lie in the other image (which... 3.6 Each polygon is composed of vertices, vi , that can be deformed to a new set of 45 CHAPTER 3 Dual-Homography Image Stitching polygons, Q , with the following two distortion energies: Ds (Q, Q ) = min s s(vi ) − vi 2 , (3.5) n where vi is the target warping position of vi and s( ·) is a similar transformation function which has a general form:           c −d   x   tx       ... vertical j edges in the source image more to maintain a vertical position in the final result Combining these two energies we can obtain the straightened result as the points set V = {vi } which minimizes the following energy: µ(Q) Ds + µl s D= Q Dl (3.8) E Here, the weight µ(Q) allows this deformation to be flexible This weight is s 46 3 .4 Post-Processing the Mosaic Figure 3.6: Before and after the application . illustrated using only three images, it is straight- forward to derive a recursive expression from Eq. 4 to concatenate further images. 44 3 .4. Post-Processing the Mosaic 3 .4 Post-Processing the Mosaic This. transformation for each pair of the images, a full stitched panorama result is then generated by concatenating the images one by one onto the virtual canvas. Similar as the traditional image. homographies for a pair of overlap- ping images, and show how to use the dual-homographies to perform nonlinear alignment. We then describe how to extend this warping to adjacent images in the mosaic. For