New strategies for generating panoramic images for imperfect image series 7

dual homography we can produce better results than obtainable by current image mosaicing techniques.. • The post-processing step used by image mosaicing techniques to hide mis-alignment

This chapter provides an assessment of the work presented in this thesis as well as

a discussion of its limitations At the same time, we also describe potential areas

of future work

6.1 Assessment

This thesis has examined the current image mosaicing pipeline and identified three areas that can be improved in the case of non-ideal input images These areas pertain to more flexible image alignment, selection of alignment transformations, and post-processing in the event of a flawed mosaic Improvement in each of these areas has been demonstrated Specifically we note the following summaries:

• The current reliance on a single homographic transformation to align over-lapping image pairs is often too restrictive and is easily violated We have examined a typical case found when imaging outdoor scenes that are domi-nated by two planes, i.e a ground plane and distant place We have shown

CHAPTER 6 Conclusion

that for these input cases that using a weighted combination of two homo-graphies (i.e dual homography) we can produce better results than obtainable

by current image mosaicing techniques

• The post-processing step used by image mosaicing techniques to hide mis-alignment artifacts is often critical in producing a seamless mosaic For some input series the post-processing plays a more important role than the align-ment Based on this observation, we have introduced a method to select potential alignment transformations not based on how well they perform geometric alignment, but instead on how perceptually optimal the resulting post-processing step is We have shown that in some cases this seam-driven image stitching can produce better results than obtainable by the current image mosaicing techniques

• There are some imperfect image series that no existing algorithm can seam-lessly stitch Existing commercial software packages offer no customized tools to assist in manually adjusting the results Users must rely on standard image editing tools which can be tedious and time consuming We have pro-posed an interactive post-processing framework that is tailored for adjusting flawed panoramic images

From the results presented in the previous chapters, we present the following findings of this thesis:

• A weighted combination of two homographies can be used to align

panoram-ic scenes containing a dominant distant and ground plane We have presented

a method to estimate this dual-homography transformation and demonstrated

how to apply it within the context of stitching several images onto a single panoramic canvas While this approach requires more effort when handling multiple images than single-homography approaches, the additional pro-cessing is offset by the ability to produce seamless images Combined with standard post-processing approaches (seam cutting and blending), together with a straightening procedure, this approach is able to stitch together input series that other methods cannot

• We have demonstrated that for some imperfect image series, relying on the homographic transformation that is considered to be the best geometric align-ment (e.g contains the most matched SIFT point consensus between image pairs) may not best the transformation in terms of producing an perceptu-ally optimal seam-cut used to merge the two images We have shown that

it is possible to compute a seam-cut ranking criterion by examining if im-age patches along the seam-cut is similar to imim-agery found within either of the two overlapping images Based on this perceptual ranking we are able

to select from a set of potential geometric transforms the transformation that results in the most perceptually seamless result thus producing a better result

• Finally, in cases where all methods fail to produce a perceptually seamless mosaic, we have introduced an interactive software that provides two tools targeting editing panoramic images The first is an interactive tool to adjust the seam cut result in a local fashion, while the second is an interactive local warping tool to adjust small misalignments by considering the content of the two overlapped images We have demonstrated that our approach can not only achieve a satisfactory correction faster than existing tools, but is also

CHAPTER 6 Conclusion

preferred by users

6.2 Discussion and Limitations

In this section, we provide some discussions and limitations related to the three contributions made:

• The dual-homography work tries to remove the breaking and tearing arti-facts by using a non-linear warping at the alignment step A trade-off of the non-linear warping techniques is that it sacrifices the property of single homography which preserves the straight lines after transformation This sometimes raises the concern that is it worth exchanging breaking artifacts with a different type of visual distortion artifacts We believe that achieving

a seamless alignment is the primary goal of the imperfect image stitching system This is because the introduced bending artifacts can be further cor-rected by using our content-aware straightening process, however, beyond seam-cutting there is limited avenues for hiding breaking artifacts

• The seam-driven stitching work relies on an enumerating estimation process

to generates the possible homographies One limitation of this work is that there lacks of a mechanism to guarantee that the computation converges to

an perceptual optimal homography Therefore, a time-consuming repetitive process is currently necessary in the system A more sophisticated algorithm which is able to provide a converging solution may be left as a future work Another issue is that traditional image stitching methods perform a bundle adjustment step where the collection of homographies are adjusted to provide

a global fit to the matching feature points Applying bundle adjustment within our framework is not as straight forward and our approach is therefore limited to being applied in an incremental fashion

• The interactive correction provides a user-assisted solution for the misalign-ment artifacts of the panorama results Dealing with another kind of stitching artifacts, which is the color inconsistency for each image in the stitched

result-s, is not in the scope of this work Although some existing works [36,16] have developed some color correction method for stitched panoramic images, but none of these approaches work in an interactive manner We believe it would

be more flexible to have a user in the color tuning process

6.3 Future Work

In addition to the potential future works discussed in Section6.2 which target on the limitation of the proposed works, we also have interesting extensions of our works:

• A natural extension of the dual-homography work is to consider a warping model which contains more than two planes This extension can quickly drive itself towards full 3D scene understanding and ultimately proxy geom-etry estimation in a shape from motion setting Determining the minimum number of planes to maintain a seamless panorama in a given scene is an promising avenue for future work

• The reason we use cut as the post-processing technique in the seam-driven stitching work is that it is currently state-of-the-art for image

mosaic-CHAPTER 6 Conclusion

ing, yet the overall idea of the post-processing driven stitching mechanism can be applied to any strategy that has a two-steps procedure Therefore, for imperfect image stitching, it can be further extended to any post-processing techniques such as blending or image completion Also, as previously noted this approach needs to be applied to one pair of images at a time, determining

a global method analogous to bundle adjustment is an interesting avenue for future work

• In the interactive tools work, the warping of the content-aware snapping tool

is only a distance based homogeneous warping method It is inevitable to have local distortion while warping the image This can be possibly improved

by introducing the non-linear content-aware warping idea, such as the global straightening in the dual-homography work, into the warping phase In that case, how to simplify the warping process to have a real-time interactive response may be left for future work