New strategies for generating panoramic images for imperfect image series 7

Chapter Conclusion This chapter provides an assessment of the work presented in this thesis as well as a discussion of its limitations. At the same time, we also describe potential areas of future work. 6.1 Assessment This thesis has examined the current image mosaicing pipeline and identified three areas that can be improved in the case of non-ideal input images. These areas pertain to more flexible image alignment, selection of alignment transformations, and post-processing in the event of a flawed mosaic. Improvement in each of these areas has been demonstrated. Specifically we note the following summaries: • The current reliance on a single homographic transformation to align overlapping image pairs is often too restrictive and is easily violated. We have examined a typical case found when imaging outdoor scenes that are dominated by two planes, i.e. a ground plane and distant place. We have shown 81 CHAPTER 6. Conclusion that for these input cases that using a weighted combination of two homographies (i.e. dual homography) we can produce better results than obtainable by current image mosaicing techniques. • The post-processing step used by image mosaicing techniques to hide misalignment artifacts is often critical in producing a seamless mosaic. For some input series the post-processing plays a more important role than the alignment. Based on this observation, we have introduced a method to select potential alignment transformations not based on how well they perform geometric alignment, but instead on how perceptually optimal the resulting post-processing step is. We have shown that in some cases this seam-driven image stitching can produce better results than obtainable by the current image mosaicing techniques. • There are some imperfect image series that no existing algorithm can seamlessly stitch. Existing commercial software packages offer no customized tools to assist in manually adjusting the results. Users must rely on standard image editing tools which can be tedious and time consuming. We have proposed an interactive post-processing framework that is tailored for adjusting flawed panoramic images. From the results presented in the previous chapters, we present the following findings of this thesis: • A weighted combination of two homographies can be used to align panoramic scenes containing a dominant distant and ground plane. We have presented a method to estimate this dual-homography transformation and demonstrated 82 6.1. Assessment how to apply it within the context of stitching several images onto a single panoramic canvas. While this approach requires more effort when handling multiple images than single-homography approaches, the additional processing is offset by the ability to produce seamless images. Combined with standard post-processing approaches (seam cutting and blending), together with a straightening procedure, this approach is able to stitch together input series that other methods cannot. • We have demonstrated that for some imperfect image series, relying on the homographic transformation that is considered to be the best geometric alignment (e.g. contains the most matched SIFT point consensus between image pairs) may not best the transformation in terms of producing an perceptually optimal seam-cut used to merge the two images. We have shown that it is possible to compute a seam-cut ranking criterion by examining if image patches along the seam-cut is similar to imagery found within either of the two overlapping images. Based on this perceptual ranking we are able to select from a set of potential geometric transforms the transformation that results in the most perceptually seamless result thus producing a better result. • Finally, in cases where all methods fail to produce a perceptually seamless mosaic, we have introduced an interactive software that provides two tools targeting editing panoramic images. The first is an interactive tool to adjust the seam cut result in a local fashion, while the second is an interactive local warping tool to adjust small misalignments by considering the content of the two overlapped images. We have demonstrated that our approach can not only achieve a satisfactory correction faster than existing tools, but is also 83 CHAPTER 6. Conclusion preferred by users. 6.2 Discussion and Limitations In this section, we provide some discussions and limitations related to the three contributions made: • The dual-homography work tries to remove the breaking and tearing artifacts by using a non-linear warping at the alignment step. A trade-off of the non-linear warping techniques is that it sacrifices the property of single homography which preserves the straight lines after transformation. This sometimes raises the concern that is it worth exchanging breaking artifacts with a different type of visual distortion artifacts. We believe that achieving a seamless alignment is the primary goal of the imperfect image stitching system. This is because the introduced bending artifacts can be further corrected by using our content-aware straightening process, however, beyond seam-cutting there is limited avenues for hiding breaking artifacts. • The seam-driven stitching work relies on an enumerating estimation process to generates the possible homographies. One limitation of this work is that there lacks of a mechanism to guarantee that the computation converges to an perceptual optimal homography. Therefore, a time-consuming repetitive process is currently necessary in the system. A more sophisticated algorithm which is able to provide a converging solution may be left as a future work. Another issue is that traditional image stitching methods perform a bundle adjustment step where the collection of homographies are adjusted to provide 84 6.3. Future Work a global fit to the matching feature points. Applying bundle adjustment within our framework is not as straight forward and our approach is therefore limited to being applied in an incremental fashion. • The interactive correction provides a user-assisted solution for the misalignment artifacts of the panorama results. Dealing with another kind of stitching artifacts, which is the color inconsistency for each image in the stitched results, is not in the scope of this work. Although some existing works [36, 16] have developed some color correction method for stitched panoramic images, but none of these approaches work in an interactive manner. We believe it would be more flexible to have a user in the color tuning process. 6.3 Future Work In addition to the potential future works discussed in Section 6.2 which target on the limitation of the proposed works, we also have interesting extensions of our works: • A natural extension of the dual-homography work is to consider a warping model which contains more than two planes. This extension can quickly drive itself towards full 3D scene understanding and ultimately proxy geometry estimation in a shape from motion setting. Determining the minimum number of planes to maintain a seamless panorama in a given scene is an promising avenue for future work. • The reason we use seam-cut as the post-processing technique in the seamdriven stitching work is that it is currently state-of-the-art for image mosaic85 CHAPTER 6. Conclusion ing, yet the overall idea of the post-processing driven stitching mechanism can be applied to any strategy that has a two-steps procedure. Therefore, for imperfect image stitching, it can be further extended to any post-processing techniques such as blending or image completion. Also, as previously noted this approach needs to be applied to one pair of images at a time, determining a global method analogous to bundle adjustment is an interesting avenue for future work. • In the interactive tools work, the warping of the content-aware snapping tool is only a distance based homogeneous warping method. It is inevitable to have local distortion while warping the image. This can be possibly improved by introducing the non-linear content-aware warping idea, such as the global straightening in the dual-homography work, into the warping phase. In that case, how to simplify the warping process to have a real-time interactive response may be left for future work. 86 . is able to stitch together input series that other methods cannot. • We have demonstrated that for some imperfect image series, relying on the homographic transformation that is considered to. inconsistency for each image in the stitched result- s, is not in the scope of this work. Although some existing works [36, 16] have developed some color correction method for stitched panoramic images, . some cases this seam-driven image stitching can produce better results than obtainable by the current image mosaicing techniques. • There are some imperfect image series that no existing algorithm