Chapter Introduction 1.1 Motivation Constructing panoramic images from a series of overlapping input images is wellstudied topic in the computer vision and computer graphics research communities. The work has reached a level of maturity that there are now several commercially successful editing tools like Adobe Photoshop [50] and smart phone applications such as AutoStitch [1] as well as other established academic tools such as Microsoft ICE [2]. The successful proliferation of these image stitching tools may lead to the impression that image stitching is solved, but, in fact, many tools fail to give convincing results when given non-ideal data. The goal of image stitching can be described as follows. Given a series of photographs of a real world scene taken from different but overlapping view points, image stitching techniques attempt to geometrically align the image series and then map them onto a common canvas resulting in single wide-field-of-view panorama (see Figure 1.1). This provides a way to capture very wide-field-of-views without CHAPTER 1. Introduction Figure 1.1: An example of image mosaicing technique. The panorama is constructed by using the four images on the top. the need for specialized lenses and high resolution single image capture. When constructing panorama images, not any arbitrary pair of images with overlapping view points are able to be stitched together. Traditional image stitching methods have certain requirement for the input images. The vast majority of methods rely on perspective planar transformations (also called homographies) to align the images. This type of transformations is valid under two restrictive imaging conditions: one is that the camera must be strictly rotated about its center of projection; the other condition is that the target scene is planar or far away enough that it can be treated as planar. When these two conditions are violated, the images cannot always be stitched due to the parallax effect. Figure 1.2 gives an illustration of these scenarios. From Figure 1.2(a, b) we can see that all target objects can be projected onto a single virtual plane, while Figure 1.2(c,d) show 1.1. Motivation ν ν (a) Camera is rotated along its projective center (b) All objects are lying on the same plane ν (c) Parallax effect (project object onto different position) (d) An example of image pairs with parallax Figure 1.2: An illustration of traditional photo taking assumption for constructing panorama. (a) and (b) show two valid assumptions for image mosaicing. We can see that any point on the object can be projected onto a unique position on the virtual plane ν. Intuitively, each single image the camera takes can be considered as partial content on the virtual plane. On the contrary, as illustrated in (c), the object is projected onto different positions on ν due to the moving of camera. Thus, although the images are targeting on the same objects, they cannot be fully registered due to the parallax. (d) shows a real example of parallax effect. an example of the parallax effect. Both of the two input image assumptions are not easy to achieve by using a hand-held camera in daily imaging. As a result, noticeable artifacts usually exist in the stitched results. In this thesis, we use the term “imperfect image series” to indicate an input image series which was not taken under ideal assumptions. In practice, most input series are imperfect. As a result, image stitching approaches strive to provide the most optimal alignment, where the notion of goodness is measured by how well CHAPTER 1. Introduction matched feature points between overlapping images align. Since imperfect image series violate the imaging assumptions, even under the geometrically optimal alignment there will still be misaligned regions resulting in undesirable visual artifacts. This problem is so well known that virtually all image mosaicing techniques employ a post-processing step to hide the artifacts after transforming the images. This post-processing step is either in the form of image blending or most recently performed by estimating a seam-cut between overlapping images that minimizes perceptual artifacts. In fact, image mosaicing algorithms have become so reliant on this postprocessing stage that it is now accepted that the panoramic construction process is a two step procedure: i.e. first, perform alignment; second, hide misalignment artifacts. It is worth noting that under this two step strategy, the goal is not to produce a geometrically accurate panorama, but instead to produce a perceptually seamless panoramic image. While the post-processing is critical in producing a final output, these techniques are limited and cannot always remove all visual artifacts. The aim of the work in this thesis is to revisit the traditional image mosaicing pipeline and develop new strategies for stitching imperfect image series which provide more visually appealing result than the existing works. 1.2 Current Image Mosaicing Process Research work on image mosaicing for consumer level camera arose in mid 1990s [44, 65, 20], and get further development in the past decade [70, 15, 16, 61]. As we discussed in Section 1.1, for a typical image mosaicing pipeline, two distinctive steps are usually applied in tandem to generate the final result: The first step is to com4 1.2. Current Image Mosaicing Process Image Mosaicing Pipeline Input Images Step1: Register and warp images Step2: Seam-cut to provide smooth stitches Final Result Figure 1.3: An illustration of traditional image mosaicing pipeline. pute the geometry transformation to align the images and the second step is to apply a post-processing to achieve a seamless composite result. Figure 1.3 shows an illustration of the traditional image mosaicing pipeline. In the first step, classical image mosaicing approaches use a perspective planar transform, as called a Input images Aligned result homography, as the transformation between each pair of image. This is because the camera motion can be fully modeled by using a series of × matrices when the input images are taken under the ideal assumptions. A homography can be estimated by using a set of reliable registering correspondences inside image pairs. The purpose of the post-processing step is to hide the misalignment artifacts. To achieve such goal, blending techniques [17, 16, 49, 36] that provide smooth tranFinal result Seam-cut result sition between image pairs have been applied in early days. In recent years, a seam-cut approach [38, 3], which stitches the images by finding the least noticeable seam between image pairs has been adopted in many works [50, 2]. Figure 1.4 shows an example of traditional image mosaicing. We can see that the seam-cut approach is able to remove most of the artifacts when the overlapped images are misaligned. Chapter provides technical details of some of these techniques for traditional image mosaicing systems. Image Mosaicing Pipeline Input Images Step1: Register and warp images Step2: Seam-cut to provide smooth stitches Final Result CHAPTER 1. Introduction Input images Final result Aligned result Seam-cut result Figure 1.4: An example result of image stitching result using traditional image mosaicing pipeline. As we can see, traditional image mosaicing system which is established on the homography model is not tailored for imperfect image series stitching. We believe this system can be further ameliorated to generate better stitched result for imperfect images. 1.3 Objectives The goal of our work is to attempt to construct visually plausible panoramas from input images that violate the conventional imaging assumption. That is, we want to improve upon the current image mosaicing processing given an imperfect 1.4. Contributions image series. While our work cannot correct all possible imperfect image series, we have examined the current image stitching pipeline and found several places where contributions can be made. Specifically, we identified the following issues to address: • Virtually all state-of-the-art image stitching softwares [1, 50, 2] use × homographies to model the transformations between pairs of the input images. We examine how allowing a more flexible warping method can improve the results for particular scenes. • Traditional image mosaicing first computes an alignment based on the best geometrical fit of match points. However, given that the goal of image mosaicing is to produce a perceptually seamless result over a geometrically correct result for an imperfect image series, we consider how promoting the role of the seam-cut step may be beneficial in selecting he alignment, i.e. a sub-optimal geometric alignment may provide a better perceptual seam-cut. • In some cases, an imperfect input series cannot be stitched together in an automatic fashion without some noticeable artifacts. In such cases, we want to explore how to design an editing framework to expediate users in making manual corrections. 1.4 Contributions In this thesis, three distinct works are proposed which aim to provide solutions to the issues in Section 1.3. These three works correspond to Chapters 3, 4, in this thesis. Figure 1.5 shows an illustration on which part of the conventional image CHAPTER 1. Introduction Image Mosaicing Pipeline Input Images Step1: Register and warp images Dual-Homography Warping Using dual-homography to provide better registering and alignment result. Chapter Step2: Seam-cut to provide smooth stitches Seam Driven Stitching Exploit the performance of the seam-cut to find a better warp. Chapter Final Result Interactive Correction Enable user to manipulate the result after the automatic system. Chapter Figure 1.5: An illustration of our works with corresponding targeting part of the traditional image mosaicing pipeline. mosaicing pipeline each work targets. Specifically, the contributions of these works can be summarized as follows: • A dual-homography framework is proposed that focuses on the imperfect image series stitching problem in the image alignment stage. A new registration model is developed for the case which the target scene has two dominant planes. A smoothly varying homography interpolation method is developed to achieve more accurate alignment between the image pairs and is extended further to multiple images. Results show our framework can generate more visually appealing results than existing commercial softwares. This work has been published in CVPR’2011 [27]. • A seam-driven image stitching system is introduced that targets how the geometric transformation is selected. In particular, instead of selecting homo8 1.5. Road Map graphies based on the best geometric fit of matched feature points, potential transforms are evaluated based on the perceptual quality of the resulting seam-cut. Along with the seam-evaluation pipeline, we propose a simple, yet effective, method to evaluate the seam cuts produced with different transforms. We demonstrate that this method can produce better results than current state-of-the-art methods. This work has been published as a short paper in EuroGraphics’2013 [34]. • Finally, a software for interactive editing of flawed stitched panoramas is proposed. Specifically, we have developed a framework that allows the user to locally correct the visual artifacts arising from both alignment and/or a bad seam-cut. Our tool allows the user to locally recompute the seam-cuts based on simple user markup. In addition, we provide a content-aware local warp tool that helps the user find the best matching for two overlapping layers while warping. We demonstrate that these editing tools can significantly speed up the manual editing time of flawed panoramas over conventional image-editing tools. This work has been published as a technical brief in SIGGRAPH ASIA’2012 [26]. 1.5 Road Map The rest of this thesis is organized as follows: Chapter provides background on traditional image mosaicing techniques and gives details to those most related to our works. Chapter presents our dual-homography work targeting more flexible geometric alignment. Chapter presents the seam-driven image stitching pipeline and Chapter proposes our interactive tools for post correction of the CHAPTER 1. Introduction panorama. Finally, Chapter concludes the thesis with a discussions on future research directions. 10 . to com- 4 1 .2. Current Image Mosaicing Process Step1: Register and warp images Step2: Seam-cut to provide smooth stitches Final ResultInput Images Image Mosaicing Pipeline Input images Aligned. traditional image mosaicing pipeline and develop new strategies for stitching imperfect image series which provide more visually appealing result than the existing works. 1 .2 Current Image Mosaicing. panorama images, not any arbitrary pair of images with overlapping view points are able to be stitched together. Traditional image stitching methods have certain requirement for the input images.