Chapter Seam-Driven Stitching 4.1 Introduction and Motivation Our previous chapter focused on addressing image mosaicing in the face of imperfect image images by allowing for more flexible warping. This approach worked well for scenes with two dominate planes, however, there are many cases where a single homography can align large portions of the image, however, due to parallax there are still artifacts. In such cases, artifact hiding using the post-processing step based on seam-cutting becomes very crucial. Matter of fact, our observation is that at times the seam-cutting step may be the most crucial step. This led us to consider promoting the role of the seam-cut in a method we term seam-driven image stitching. Instead of selecting homographies between two image pairs based on the best geometric fit to matched feature points, a set of potential transforms are evaluated based on the perceptual quality of the resulting seam-cut. The idea is that a potential homography may give a better perceptual seam-cut even though the transformation is considered sub-optimal. Figure 4.1 shows an example. We 57 CHAPTER 4. Seam-Driven Stitching (B) (A) Figure 4.1: (A) A traditional image-stitching result where the images are first aligned using a geometric transform computed based on the best overall fit of 96 matched points followed by seam-cutting. (B) Our seam-driven image stitching result where the geometric transform is less optimal in terms of geometric fit (only 26 matched points used), but the transform gives a perceptually better seam-cut. The insets show the points used to compute the transforms as well as the seam-cut with our computed seam error. demonstrate that this approach can sometimes produce better results than current state-of-the-art methods, especially for challenging cases where the scene geometry or imaging conditions are not ideal. 4.2 Seam-Driven Image Stitching Figure 4.2 shows an example of the traditional image stitching pipeline (top) and our modification to perform seam-driven image stitching (bottom). In the traditional method, feature points such as SIFT [42] are computed and matched between image pairs resulting in a set of feature correspondences. Due to the non-planar scene geometry and erroneous matches, random sample consensus (RANSAC) [24] is used to robustly estimate the homography with the best geometric fit. Specifically, a set of homographies H1 , H2 , . . . are randomly hypothesized and ranked based on their goodness-of-fit (measured in terms of consensus size). The best 58 4.2. Seam-Driven Image Stitching Traditional image stitching Feature extraction and matching 𝐻1 𝐻2 𝐻4 𝐻5 𝐻3 Choose H with best geometric fit Apply seam-cut Robust estimation to find candidate homographies Final result Seam-driven image stitching Feature extraction and matching 𝐻1 𝐻2 𝐻4 𝐻5 Choose result with best seam-cut 𝐻3 Robust estimation to find candidate homographies Apply seam-cut to all possible transformations Final result Figure 4.2: A comparison between the traditional image stitching process and our seam-driven process. homography is used to warp and align the images. Seam-cutting is then applied to produce the final result. Our seam-driven approach operates in a similar fashion but evaluates and selects the homography differently. This is achieved by modifying the hypothesizeand-test loop in RANSAC, whereby we generate candidate homographies in a manner that encourages diversity, and applying seam-cut to the aligned images by using all candidates. Next, we evaluate the different seam-cuts based on our novel ranking metric that favors results with good perceptual quality. The details to our approach are outlined in the following sections. 59 CHAPTER 4. Seam-Driven Stitching 4.2.1 Generating Homography Candidates RANSAC generates model hypotheses by fitting a homography on randomly sampled minimal subsets of the data, where each subset contains four correspondences [16]. For each hypothesis, the correspondences that are consistent with it (i.e. the inliers of the hypothesis) are recorded. This random sampling process is repeated n times and the hypothesis with the highest consensus size is chosen. The number n rarely corresponds to an exhaustive sampling of all four-subsets of the data. Moreover, usually there is a dominant plane in the scene (e.g. the background plane) that contributes a large number of similar candidate homographies. Traditional image-stitching methods favor the dominant plane as it gives the best consensus. Since we desire more diverse homographies (to yield more varied seam-cuts), we use a simple filtering strategy to avoid resampling the dominant plane. For each iteration, RANSAC instances each with a smaller n (50 in our implementation) is applied on the current matched points set. Such a small replication number lowers the probability that the RANSAC process keep retrieving the dominant homography of the current matching point set. After this, we remove the inliers of the dominant plane which are achieved by applying standard RANSAC with a comparatively large n (50 in our implementation). This process repeats until the number of points in current data set is less than a threshold τ (10 in our system). This helps to find homographies that align other smaller planar structures which may result in better seam-cuts. 60 4.2. Seam-Driven Image Stitching Figure 4.3: An illustration of randomly generated homographies with respective seam cuts. . 4.2.2 Computing the Seam-Cut We used the seam-cut method described in Section 2.4.2 to stitch each neighboring images. For each image pair with the candidate homography, we denote I1 and I2 as the overlapping regions. Then the seam-cut is a global labeling problem on an MRF and can be solved by using the work in [13] . 4.2.3 Evaluating the Cut While the seam-cut energy minimizes image gradient between I1 and I2 , it is ineffective when used to rank the perceptual quality of different cuts. We introduce a more discriminating error measurement. For each pixel, p, along the seam, we 61 CHAPTER 4. Seam-Driven Stitching Seam-cut result I1 s s - s ( - s) s I2 Search for the most similar patch in the overlapping region of source images Figure 4.4: This illustrates how seams are evaluated. For each pixel along the seam, we search for the most similar patch about this pixel from either of two overlapping image and use the squared difference as the mismatch error. estimate an error value, E(p) by extracting a 17 × 17 patch, P, centered at p, and searching for its most similar patch in either I1 or I2 . This can be expressed as: E(p) = ||P − Si ||2 , high Si ∈I1 ,I2 (4.1) where Si represents all image patches in the overlapping regions of I1 and I2 . The idea here is that if the patches along the seam resemble a patch found in either I1 or I2 , the patch about the seam is perceptually plausible. If a patch along the seam cannot be found in either source images, it is more likely to be an artifact low −4 The total error −3 and therefore assigned error. of size −3m, is 𝐸 a=larger 9.706𝑒 𝐸 = 5.365𝑒 along a𝐸seam = 6.150𝑒 computed as E = 62 n p E(p)/m. 4.3. Results and Discussion high low 𝐸 = 9.706𝑒 −4 𝐸 = 5.365𝑒 −3 𝐸 = 6.150𝑒 −3 Figure 4.5: This comparison shows a hot map of energy from different cuts using our error metric. The accumulated seam error, E is used to rank the cuts. Figure 4.4 and Figure 4.5 shows a diagram of this procedure as well as an example with three different cuts with the corresponding per pixel error, E(p) shown as a hot map and the total error E. The minimum error is indicative of our perceptual ranking. 4.3 Results and Discussion Figure 4.6 shows our results and those obtained using and Photoshop CS6 on a panorama sequence of four input images. The images contain parallax and therefore are difficult to align. Photoshop uses a traditional image-stitching pipeline that selects a best fit transform. There are still noticeable artifacts even after seamcutting. In contrast, our seam-driven strategy is able to obtain perceptually better results (also see supplemental material). Noticeably, inside our results there exist 63 CHAPTER 4. Seam-Driven Stitching artifacts which are not geometrically corrected (e.g. duplicated mountain texture in the first example and omitting trees in the second example). However, these artifacts are perceptually not noticeable. Compare to the result generated by Photoshop [50], our results are more visually appealing. Detailed intermediate results of Figure 4.6 are provided in Appendix A This work have present a new sight of image mosaicing approaches by let the post-processing technique determine the transformation. As demonstrated, our seam driven image stitching work can generate better result for imperfect shot images. Limitations and potential future work related to this seam-drive image stitching method are presented in Chapter 6. 64 4.3. Results and Discussion Photoshop result Our result Photoshop result Our result Figure 4.6: Comparison of panoramas constructed based on a traditional image stitching pipeline (Adobe Photoshop CS6) and our seam-driven approach. There are noticeable visual artifacts in the Photoshop’s results, while our results appear more seamless. 65 CHAPTER 4. Seam-Driven Stitching 66 . sequence of four input images. The images contain parallax and there- fore are difficult to align. Photoshop uses a traditional image- stitching pipeline that selects a best fit transform. There are still. transformation. As demonstrated, our seam driven image stitching work can generate better result for imperfect shot images. Limitations and potential future work related to this seam-drive image stitching. previous chapter focused on addressing image mosaicing in the face of imper- fect image images by allowing for more flexible warping. This approach worked well for scenes with two dominate planes,