NEW STRATEGIES FOR GENERATING PANORAMIC IMAGES FOR IMPERFECT IMAGE SERIES GAO JUNHONG NATIONAL UNIVERSITY OF SINGAPORE 2013 NEW STRATEGIES FOR GENERATING PANORAMIC IMAGES FOR IMPERFECT IMAGE SERIES GAO JUNHONG (B.Sc., Sichuan University of China, 2008) A THESIS SUBMITTED FOR THE DEGREE OF Doctor of Philosophy in SCHOOL OF COMPUTING NATIONAL UNIVERSITY OF SINGAPORE SINGAPORE, 2013 c 2013, GAO Junhong Declaration I hereby declare that this thesis is my original work and it has been written by me in its entirety. I have duly acknowledged all the sources of information which have been used in the thesis. This thesis has also not been submitted for any degree in any university previously. Signature: Date: To my parents, my wife and my coming son. Acknowledgements First of all, I own my deepest gratitude to my advisor Dr. Michael S. Brown for his patient guidance during my Ph.D. candidature. I am, and will always be, thankful for all of his encouraging advice, shared experiences and technical/financial support. All the things that he taught me I’ll treasure for the rest of my career. I am thankful to my co-authors: Seon Joo Kim, Tat-Jun Chin and Li Yu for their great contribution to my research work, for all their valuable suggestions and auxiliary works that contributed to this thesis. It is always my pleasure to collaborate with these brilliant people and to work on exiting research topics together. I am grateful to my colleagues in the Computer Vision Lab: Lin Haiting, Deng Fanbo, Cheng Dongliang, Cheng Yuan etc. , for every colorful day I spent at the National University of Singapore (NUS). I really enjoyed the life in Singapore with these warmhearted and reliable friends. At last, I would like to thank my parents for their unconditional support and encouragement. A special thank to my father, Gao Rongting, who has contributed at least two sets of photo data to this thesis. I owe my deep appreciation to my wife, Zhang Lingyan, for her sincere love and all the wonderful things she has done for me. May this thesis be my best present for her and our coming baby. Abstract The success of commercial image stitching tools often leads to the impression that image stitching is a ”solved problem”. The reality, however, is that many tools give unconvincing results when the input data violates fairly rigid imaging assumptions; the main two assumptions being that the photos correspond to views that differ purely by rotation or that the imaged scene is planar. In fact, many existing image stitching tools depend critically on post-processing routines to hide misalignment artifacts. As a result, the defacto image-stitching pipeline involves two distinct steps: 1) a geometric alignment step; and 2) a post-processing step that computes optimal seam-cuts between overlapping images to hide misalignment artifacts. In this thesis, we re-examine this established panoramic image construction pipeline and introduce three strategies that are useful when the input images are less than ideal. First, we introduce a method to compute a more flexible geometric alignment when the imaged scene contains two dominant planes. We show that this method, termed dualhomography alignment, is able to stitch photographs that current stateof-the-art methods cannot. Second, we show that instead of selecting an alignment based on the geometric fit of matched points between images, we can use the seam-cut step to evaluate several possible geometric transformations and select the alignment that gives the most “perceptually optimal” seam. We demonstrate several cases where this seamdriven image stitching is able to produce better results than existing methods. Lastly, for examples when the automatic image stitching procedure fails, we introduce an interactive method that provides the user tools to manually edit the seam-cuts and/or geometric alignment in a local manner. These three works collectively provide solutions for image stitching using imperfect input images while targeting on different parts of the image stitching pipeline. Contents List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Introduction 1.1 Motivation . . . . . . . . . . . . . 1.2 Current Image Mosaicing Process 1.3 Objectives . . . . . . . . . . . . . 1.4 Contributions . . . . . . . . . . . 1.5 Road Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Background and Preliminaries 2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2 Feature Registering . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2.1 SIFT Feature Matching . . . . . . . . . . . . . . . . . . . . 2.2.2 RANSAC Process . . . . . . . . . . . . . . . . . . . . . . . 2.3 Transformation Estimation . . . . . . . . . . . . . . . . . . . . . . 2.3.1 Homography Model Estimation . . . . . . . . . . . . . . 2.3.2 Rotation Model Estimation . . . . . . . . . . . . . . . . . 2.3.3 Cylinder Mapping . . . . . . . . . . . . . . . . . . . . . . 2.4 Post-Processing Techniques . . . . . . . . . . . . . . . . . . . . . 2.4.1 Blending . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4.2 Seam-cut . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Challenges for Imperfect Image Stitching and Its Related Works 2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 . . . . . . . . . . . . . 11 11 12 14 14 17 17 18 19 21 21 22 27 31 i CONTENTS Dual-Homography Image Stitching 3.1 Introduction . . . . . . . . . . . . . . . 3.2 System Overview . . . . . . . . . . . . 3.3 Dual-Homography Alignment . . . . 3.3.1 Dual-Homography Estimation 3.3.2 Extending to Multiple Images . 3.4 Post-Processing the Mosaic . . . . . . 3.4.1 Global Straightening . . . . . . 3.5 Results . . . . . . . . . . . . . . . . . . 3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . 33 33 35 36 36 43 45 45 50 55 . . . . . . 57 57 58 60 61 61 63 . . . . . . . 67 67 69 69 70 72 74 78 Conclusion 6.1 Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Discussion and Limitations . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 81 84 85 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Seam-Driven Stitching 4.1 Introduction and Motivation . . . . . . . . . . 4.2 Seam-Driven Image Stitching . . . . . . . . . 4.2.1 Generating Homography Candidates 4.2.2 Computing the Seam-Cut . . . . . . . 4.2.3 Evaluating the Cut . . . . . . . . . . . 4.3 Results and Discussion . . . . . . . . . . . . . Interactive Post-Processing Tools for Correction 5.1 Introduction . . . . . . . . . . . . . . . . . . 5.2 System . . . . . . . . . . . . . . . . . . . . . 5.2.1 Pipeline . . . . . . . . . . . . . . . . 5.2.2 Local seam-editing . . . . . . . . . . 5.2.3 Content-aware snapping . . . . . . . 5.3 Results . . . . . . . . . . . . . . . . . . . . . 5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Intermediate Result of Seam Driven Stitching Approach ii . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 CONTENTS B Interface of the Interactive Panorama Correction Tool 91 C Input Images for All Examples used in This Thesis 95 Bibliography 98 iii CONTENTS iv List of Figures 1.1 1.2 1.3 1.4 1.5 2.1 2.2 2.3 2.4 2.5 2.6 2.7 3.1 3.2 3.3 An example of panorama construction. . . . . . . . . . . . . . . . . An illustration of traditional photo taking assumption for constructing panorama . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . An illustration of traditional image mosaicing pipeline. . . . . . . . An example result of image stitching result using traditional image mosaicing pipeline. . . . . . . . . . . . . . . . . . . . . . . . . . . . . An illustration of our works with corresponding targeting part of the traditional image mosaicing pipeline. . . . . . . . . . . . . . . . . . . . . A comparison of the performance of different feature descriptors. . . An example of SIFT feature detection and RANSAC filtering . . . . . An example of homography warping and cylinder warping . . . . . An example of post-processing techniques of image mosaicing. Multiband blending and seam-cut results are compared. . . . . . . . . . . . An illustration of seam-cut theory. . . . . . . . . . . . . . . . . . . . . Seam cut example with blending . . . . . . . . . . . . . . . . . . . . . An illustration of graph cut labeling . . . . . . . . . . . . . . . . . . . 13 15 20 23 24 25 27 A scene containing two dominant planes targeted by our mosaicing approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Work flow of our dual-homography computation . . . . . . . . . . . 37 A synthesized example of an ideal scene of two plains and its correspondent results of two homographies with explicit segmentation method and dual-homography method. . . . . . . . . . . . . . . . . . 40 v LIST OF FIGURES 3.4 A synthesized example of a more realistic scene of two plains with stair structures in the middle and its correspondent results of two homographies with explicit segmentation method and dualhomography method. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Illustration of concatenate multiple images . . . . . . . . . . . . . . 3.6 An example of content-aware straightening . . . . . . . . . . . . . . 3.7 An example where the user can specify regions that should remain intact in the straightening process. . . . . . . . . . . . . . . . . . . . 3.8 A failure case of dual-homography . . . . . . . . . . . . . . . . . . . 3.9 A comparison of our dual-homography stitching approach with commercial softwares. . . . . . . . . . . . . . . . . . . . . . . . . . 3.10 A comparison of our dual-homography stitching approach with commercial softwares. . . . . . . . . . . . . . . . . . . . . . . . . . 3.11 A comparison of our dual-homography stitching approach with commercial softwares. . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 4.2 4.3 4.4 4.5 4.6 5.1 5.2 5.3 5.4 5.5 5.6 5.7 vi An illustration of our approach and traditional image stitching approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A comparison between the traditional image stitching process and our seam-driven process. . . . . . . . . . . . . . . . . . . . . . . . . . An illustration of randomly generated homographies with respective seam cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . An illustration of seam-cut evaluating. . . . . . . . . . . . . . . . . . A comparison of energy for different seam cut. . . . . . . . . . . . . Comparison of panoramas constructed based on a traditional image stitching pipeline (Adobe Photoshop CS6) and our method. . . . . An example of the usage of our interactive tool. . . . . Overview of our approach. . . . . . . . . . . . . . . . . Illustration of seam adjustment. . . . . . . . . . . . . . . Illustration of content-aware warping. . . . . . . . . . . A comparison example with other softwares . . . . . . Example results generated by our interactive approach( Example results generated by our interactive approach( . . . . . . . . . . 1) 2) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 . 43 . 47 . 50 . 51 . 52 . 53 . 54 . 58 . 59 . 61 . 62 . 63 . 65 . . . . . . . 68 69 71 72 74 75 76 LIST OF FIGURES 5.8 5.9 Example results generated by our interactive approach( 3) . . . . . . 77 User-study result of comparing our panorama correction tool and Photoshop CS5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 5.10 An example where Photoshop (and AutoStitch and ICE) fails to generate a panoramic image. . . . . . . . . . . . . . . . . . . . . . . . 79 A.1 Input images used in Chapter . . . . . . . . . . . . . . . . . . . . . . 87 A.2 Intermediate results for image set 1. . . . . . . . . . . . . . . . . . . 88 A.3 Intermediate results for image set 2. . . . . . . . . . . . . . . . . . . . 89 B.1 The control dialog for the interactive panorama correction tools. . . . 91 B.2 An example of the display window in seam-editing mode. . . . . . . 92 B.3 An example of the display window in warp mode. . . . . . . . . . . . 93 C.1 C.2 C.3 C.4 C.5 C.6 C.7 Input image series for Figure 3.9 Input image series for Figure 3.10 Input image series for Figure 3.11 Input image series for Figure 5.1 Input image series for Figure 5.6 Input image series for Figure 5.7 Input image series for Figure 5.8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 95 96 96 96 96 97 vii LIST OF FIGURES viii List of Tables 2.1 An illustration of different types of 2D motions . . . . . . . . . . . . . 18 3.1 Pros and cons of the compared two homographies with segmentation method and dual-homography method. . . . . . . . . . . . . . . 42 5.1 Processing time of the examples in Figure 5.6, 5.7, 5.8 . . . . . . . . . 75 ix LIST OF TABLES x [...]... 92 B.3 An example of the display window in warp mode 93 C .1 C.2 C.3 C.4 C.5 C.6 C.7 Input image series for Figure 3.9 Input image series for Figure 3 .10 Input image series for Figure 3 .11 Input image series for Figure 5 .1 Input image series for Figure 5.6 Input image series for Figure 5.7 Input image series for Figure 5.8 ... 1 1 4 6 7 9 11 11 12 14 14 17 17 18 19 21 21 22 27 31 i CONTENTS 3 4 5 6 Dual-Homography Image Stitching 3 .1 Introduction 3.2 System Overview 3.3 Dual-Homography Alignment 3.3 .1 Dual-Homography Estimation 3.3.2 Extending to Multiple Images 3.4 Post-Processing the Mosaic 3.4 .1 Global Straightening 3.5 Results... Panorama Correction Tool 91 C Input Images for All Examples used in This Thesis 95 Bibliography 98 iii CONTENTS iv List of Figures 1. 1 1. 2 1. 3 1. 4 1. 5 2 .1 2.2 2.3 2.4 2.5 2.6 2.7 3 .1 3.2 3.3 An example of panorama construction An illustration of traditional photo taking assumption for constructing panorama An illustration of traditional image mosaicing pipeline... Photoshop (and AutoStitch and ICE) fails to generate a panoramic image 79 A .1 Input images used in Chapter 4 87 A.2 Intermediate results for image set 1 88 A.3 Intermediate results for image set 2 89 B .1 The control dialog for the interactive panorama correction tools 91 B.2 An example of the display window in seam-editing... ix 1 2 Introduction 1. 1 Motivation 1. 2 Current Image Mosaicing Process 1. 3 Objectives 1. 4 Contributions 1. 5 Road Map Background and Preliminaries 2 .1 Overview 2.2 Feature Registering 2.2 .1 SIFT Feature... 55 57 57 58 60 61 61 63 67 67 69 69 70 72 74 78 Conclusion 6 .1 Assessment 6.2 Discussion and Limitations 6.3 Future Work 81 81 84 85 Seam-Driven Stitching 4 .1 Introduction and Motivation 4.2 Seam-Driven Image Stitching 4.2 .1 Generating Homography... our interactive approach( 1) 2) 41 43 47 50 51 52 53 54 58 59 61 62 63 65 68 69 71 72 74 75 76 LIST OF FIGURES 5.8 5.9 Example results generated by our interactive approach( 3) 77 User-study result of comparing our panorama correction tool and Photoshop CS5 78 5 .10 An example where Photoshop (and AutoStitch... 2.3 Transformation Estimation 2.3 .1 Homography Model Estimation 2.3.2 Rotation Model Estimation 2.3.3 Cylinder Mapping 2.4 Post-Processing Techniques 2.4 .1 Blending 2.4.2 Seam-cut 2.5 Challenges for Imperfect Image Stitching and Its Related... commercial softwares 2 3 .11 A comparison of our dual-homography stitching approach with commercial softwares 3 4 .1 4.2 4.3 4.4 4.5 4.6 5 .1 5.2 5.3 5.4 5.5 5.6 5.7 vi An illustration of our approach and traditional image stitching approach A comparison between the traditional image stitching process and our seam-driven process... illustration of traditional image mosaicing pipeline An example result of image stitching result using traditional image mosaicing pipeline An illustration of our works with corresponding targeting part of the traditional image mosaicing pipeline 2 3 5 6 8 A comparison of the performance of different feature descriptors An example of SIFT feature detection . NEW STRATEGIES FOR GENERATING PANORAMIC IMAGES FOR IMPERFECT IMAGE SERIES GAO JUNHONG NATIONAL UNIVERSITY OF SINGAPORE 2 013 NEW STRATEGIES FOR GENERATING PANORAMIC IMAGES FOR IMPERFECT IMAGE. Input image series for Figure 3 .10 . . . . . . . . . . . . . . . . . . . . . 95 C.3 Input image series for Figure 3 .11 . . . . . . . . . . . . . . . . . . . . . 96 C.4 Input image series for Figure. . ix 1 Introduction 1 1 .1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Current Image Mosaicing Process . . . . . . . . . . . . . . . . . . . . . 4 1. 3 Objectives