Series Editors: Gérard Medioni, University of Southern California Sven Dickinson, University of Toronto Ellipse Fitting for Computer Vision Implementation and Applications Kenichi Kanatani, Okayama University Yasuyuki Sugaya, Toyohashi University of Technology Yasushi Kanazawa, Toyohashi University of Technology KANATANI • SUGAYA • KANAZAWA Series ISSN: 2153-1056 Because circular objects are projected to ellipses in images, ellipse fitting is a first step for 3-D analysis of circular objects in computer vision applications For this reason, the study of ellipse fitting began as soon as computers came into use for image analysis in the 1970s, but it is only recently that optimal computation techniques based on the statistical properties of noise were established These include renormalization (1993), which was then improved as FNS (2000) and HEIV (2000) Later, further improvements, called hyperaccurate correction (2006), HyperLS (2009), and hyper-renormalization (2012), were presented Today, these are regarded as the most accurate fitting methods among all known techniques This book describes these algorithms as well implementation details and applications to 3-D scene analysis We also present general mathematical theories of statistical optimization underlying all ellipse fitting algorithms, including rigorous covariance and bias analyses and the theoretical accuracy limit The results can be directly applied to other computer vision tasks including computing fundamental matrices and homographies between images This book can serve not simply as a reference of ellipse fitting algorithms for researchers, but also as learning material for beginners who want to start computer vision research The sample program codes are downloadable from the website: https://sites.google com/a/morganclaypool.com/ellipse-fitting-for-computer-vision-implementation-and-applications ELLIPSE FITTING FOR COMPUTER VISION About SYNTHESIS store.morganclaypool.com MORGAN & CLAYPOOL This volume is a printed version of a work that appears in the Synthesis Digital Library of Engineering and Computer Science Synthesis books provide concise, original presentations of important research and development topics, published quickly, in digital and print formats Ellipse Fitting for Computer Vision Implementation and Applications Kenichi Kanatani Yasuyuki Sugaya Yasushi Kanazawa Ellipse Fitting for Computer Vision Implementation and Applications Synthesis Lectures on Computer Vision Editors Gérard Medioni, University of Southern California Sven Dickinson, University of Toronto Synthesis Lectures on Computer Vision is edited by Gérard Medioni of the University of Southern California and Sven Dickinson of the University of Toronto e series publishes 50- to 150 page publications on topics pertaining to computer vision and pattern recognition e scope will largely follow the purview of premier computer science conferences, such as ICCV, CVPR, and ECCV Potential topics include, but not are limited to: • Applications and Case Studies for Computer Vision • Color, Illumination, and Texture • Computational Photography and Video • Early and Biologically-inspired Vision • Face and Gesture Analysis • Illumination and Reflectance Modeling • Image-Based Modeling • Image and Video Retrieval • Medical Image Analysis • Motion and Tracking • Object Detection, Recognition, and Categorization • Segmentation and Grouping • Sensors • Shape-from-X • Stereo and Structure from Motion • Shape Representation and Matching iii • Statistical Methods and Learning • Performance Evaluation • Video Analysis and Event Recognition Ellipse Fitting for Computer Vision: Implementation and Applications Kenichi Kanatani, Yasuyuki Sugaya, and Yasushi Kanazawa 2016 Background Subtraction: eory and Practice Ahmed Elgammal 2014 Vision-Based Interaction Matthew Turk and Gang Hua 2013 Camera Networks: e Acquisition and Analysis of Videos over Wide Areas Amit K Roy-Chowdhury and Bi Song 2012 Deformable Surface 3D Reconstruction from Monocular Images Mathieu Salzmann and Pascal Fua 2010 Boosting-Based Face Detection and Adaptation Cha Zhang and Zhengyou Zhang 2010 Image-Based Modeling of Plants and Trees Sing Bing Kang and Long Quan 2009 Copyright © 2016 by Morgan & Claypool All rights reserved No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means—electronic, mechanical, photocopy, recording, or any other except for brief quotations in printed reviews, without the prior permission of the publisher Ellipse Fitting for Computer Vision: Implementation and Applications Kenichi Kanatani, Yasuyuki Sugaya, and Yasushi Kanazawa www.morganclaypool.com ISBN: 9781627054584 ISBN: 9781627054980 paperback ebook DOI 10.2200/S00713ED1V01Y201603COV008 A Publication in the Morgan & Claypool Publishers series SYNTHESIS LECTURES ON COMPUTER VISION Lecture #8 Series Editors: Gérard Medioni, University of Southern California Sven Dickinson, University of Toronto Series ISSN Print 2153-1056 Electronic 2153-1064 Ellipse Fitting for Computer Vision Implementation and Applications Kenichi Kanatani Okayama University, Okayama, Japan Yasuyuki Sugaya Toyohashi University of Technology, Toyohashi, Aichi, Japan Yasushi Kanazawa Toyohashi University of Technology, Toyohashi, Aichi, Japan SYNTHESIS LECTURES ON COMPUTER VISION #8 M &C Morgan & cLaypool publishers ABSTRACT Because circular objects are projected to ellipses in images, ellipse fitting is a first step for 3D analysis of circular objects in computer vision applications For this reason, the study of ellipse fitting began as soon as computers came into use for image analysis in the 1970s, but it is only recently that optimal computation techniques based on the statistical properties of noise were established ese include renormalization (1993), which was then improved as FNS (2000) and HEIV (2000) Later, further improvements, called hyperaccurate correction (2006), HyperLS (2009), and hyper-renormalization (2012), were presented Today, these are regarded as the most accurate fitting methods among all known techniques is book describes these algorithms as well implementation details and applications to 3-D scene analysis We also present general mathematical theories of statistical optimization underlying all ellipse fitting algorithms, including rigorous covariance and bias analyses and the theoretical accuracy limit e results can be directly applied to other computer vision tasks including computing fundamental matrices and homographies between images is book can serve not simply as a reference of ellipse fitting algorithms for researchers, but also as learning material for beginners who want to start computer vision research e sample program codes are downloadable from the website: https://sites.google.com/a/morganclaypool.com/ellipse-fitting-forcomputer-vision-implementation-and-applications/ KEYWORDS geometric distance minimization, hyperaccurate correction, HyperLS, hyperrenormalization, iterative reweight, KCR lower bound, maximum likelihood, renormalization, robust fitting, Sampson error, statistical error analysis, Taubin method vii Contents Preface xi Introduction 1.1 Ellipse Fitting 1.2 Representation of Ellipses 1.3 Least Squares Approach 1.4 Noise and Covariance Matrices 1.5 Ellipse Fitting Approaches 1.6 Supplemental Note Algebraic Fitting 11 2.1 Iterative Reweight and Least Squares 11 2.2 Renormalization and the Taubin Method 12 2.3 Hyper-renormalization and HyperLS 13 2.4 Summary 15 2.5 Supplemental Note 16 Geometric Fitting 19 3.1 Geometric Distance and Sampson Error 19 3.2 FNS 20 3.3 Geometric Distance Minimization 21 3.4 Hyperaccurate Correction 23 3.5 Derivations 24 3.6 Supplemental Note 28 Robust Fitting 31 4.1 Outlier Removal 31 4.2 Ellipse-specific Fitting 32 4.3 Supplemental Note 34 viii Ellipse-based 3-D Computation 37 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 Experiments and Examples 55 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 Ellipse Fitting Examples 55 Statistical Accuracy Comparison 56 Real Image Examples 59 Robust Fitting 59 Ellipse-specific Methods 59 Real Image Examples 61 Ellipse-based 3-D Computation Examples 62 Supplemental Note 64 Extension and Generalization 67 7.1 7.2 7.3 Intersections of Ellipses 37 Ellipse Centers, Tangents, and Perpendiculars 38 Perspective Projection and Camera Rotation 40 3-D Reconstruction of the Supporting Plane 43 Projected Center of Circle 44 Front Image of the Circle 45 Derivations 47 Supplemental Note 50 Fundamental Matrix computation 67 7.1.1 Formulation 67 7.1.2 Rank Constraint 70 7.1.3 Outlier Removal 71 Homography Computation 72 7.2.1 Formulation 72 7.2.2 Outlier Removal 76 Supplemental Note 77 Accuracy of Algebraic Fitting 79 8.1 8.2 8.3 8.4 8.5 Error Analysis 79 Covariance and Bias 80 Bias Elimination and Hyper-renormalization 82 Derivations 83 Supplemental Note 90 114 ANSWERS is a square of a linear expression in y If B AC > 0, it has the form Âp à BD AE B AC y C f0 p : B AC us, we obtain the following factorization: ! p p By C Df0 / C B AC y C f0 BD AE/= B AC A x A ! p p By C Df0 / B AC y f0 BD AE/= B AC x D 0: A Equation (5.56) results from this 5.4 If n2 Ô 0, we substitute y = n1 x C n3 f0 /=n2 into Eq (5.1) to obtain Ax 2Bx.n1 x C n3 f0 /=n2 C C.n1 x C n3 f0 /2 =n22 C2f0 Dx E.n1 x C n3 f0 /=n2 / C f02 F D 0: Expanding this, we obtain the following quadratic equation in x : An22 2Bn1 n2 C C n21 /x C 2f0 Dn22 C C n1 n3 C.C n23 2En2 n3 C F n22 /f02 D 0: Bn2 n3 En1 n2 /x If we let xi , i = 1, 2, be the two roots, yi is given by yi = n1 xi C n3 f0 /=n2 If n2 we substitute x = n2 y C n3 f0 /=n1 into Eq (5.1) to obtain 0, A.n2 x C n3 f0 /2 =n21 2Bx.n2 y C n3 f0 /=n1 C Cy C2f0 D.n2 y C n3 f0 /=n1 C Ey/ C f02 F D 0: Expanding this, we obtain the following quadratic equation in y : An22 2Bn1 n2 C C n21 /y C 2f0 En21 C An2 n3 C.An23 2Dn1 n3 C F n21 /f02 D 0: Bn1 n3 Dn1 n2 /y If we let yi , i = 1, 2, be the two roots, xi is given by xi = n2 xi C n3 f0 /=n1 In actual computation, in stead of considering if n2 Ô or n2 0, we compute the former solution whenjn2 j jn1 j and the latter solution when jn2 j < jn1 j If the quadratic equation has imaginary roots, the line does not intersect the ellipse If it has a double root, we obtain the tangent point 5.5 e normal to the line n1 x C n2 y C n3 f0 = is given by n1 ; n2 /> e normal to a curve F x; y/ = at x0 ; y0 /> is given up to scale by rF = @F=@x; @F=@y/ Hence, the normal n1 ; n2 /> to the curve of Eq (5.1) at x0 ; y0 / is given up to scale by n1 D Ax0 C By0 C f0 D; n2 D Bx0 C Cy0 C f0 E: ANSWERS 115 > e line passing through x0 ; y0 / having normal n1 ; n2 / is given by n1 x x0 / C n2 x x0 / C n3 f0 D 0: After expansion using Ax02 C 2Bx0 y0 C Cy02 C 2f0 Dx0 C Ey0 / C f02 F = 0, we obtain the line n1 x C n2 y C n3 f0 = given by Eq (5.8) 5.6 We only need to read x˛ ; y˛ / in Procedure 3.2 as a; b/ and remove the step of updating the ellipse Hence, we obtain the following procedure Represent the ellipse by the 6-D vector θ as in Eq (1.8), and let J0 = (a sufficiently large number), aO = a, bO = b , and aQ = bQ = Compute the normalized covariance matrix V0 ŒξO obtained by replacing xN ˛ and yN˛ of V0 Œξ˛ in Eq (1.21) by aO and bO , respectively Compute B B B B ξ DB B B @ aO C 2aO aQ Q 2.aO bO C bO aQ C aO b/ Ob C 2bO bQ 2f0 aO C a/ Q O Q 2f0 b C b/ f0 Update aQ , bQ , aO , and bO as follows:  aQ bQ à 2.ξ ; θ / θ ; V0 ŒξOθ /  Â1 Â2 Â2 Â3 Â4 Â5 à aO @ bO A ; f0 C C C C C: C C A aO a a; Q bO b Q b: Compute J D aQ C bQ : If J O and stop Else, let J0 J0 , return a; O b/ J , and go back to Step 5.7 Transposition of AA = I on both side gives A /> A> = I is means that A /> is the inverse of A> , i.e., A /> = A> / 5.8 Since x; y; f /> is the direction of the line of sight, we can write X = cx , Y = cy , Z = cf for some c is point is on the supporting plane n1 X C n2 Y C n3 Z = h, so c.n1 x C n2 y C n3 f / = h holds Hence, c = h=.n1 x C n2 y C n3 f /, and X , Y , and Z are given by Eq (5.57) 116 ANSWERS 5.9 e unit vector along the Z -axis is k = 0; 0; 1/> Let ˝ be the angle made by n and k (positive for rotating n toward k screw-wise) It is computed by ˝ D sin kn kk: e unit vector orthogonal to both n and k is given by l D N Œn k: e rotion around l by angle ˝ screw-wise is given by the matrix RD cos ˝ C l12 cos ˝/ @ l2 l1 cos ˝/ C l3 sin ˝ l3 l1 cos ˝/ l2 sin ˝ l1 l2 cos ˝/ l3 sin ˝ cos ˝ C l22 cos ˝/ l3 l2 cos ˝/ C l1 sin ˝ l1 l3 cos ˝/ C l2 sin ˝ l2 l3 cos ˝/ l1 sin ˝ A : cos ˝ C l32 cos ˝/ 5.10 Let I.i; j / be the value of the pixel with integer coordinates i; j / For non-integer coordinates x; y/, let i; j / be their integer parts, and ; Á/ (= x i; y j /) their fraction parts We compute I.x; y/ by the following bilinear interpolation: I.x; y/ D /.1 Á/I.i; j / C Á/I.i C 1; j / C.1 /ÁI.i; j C 1/ C ÁI.i C 1; j C 1/: is means that we first linear interpolation in the j -direction, combining I.i; j / and I.i; j C 1/ in the ratio Á W Á, combining I.i C 1; j / and I.i; j C 1/ in the ratio Á W Á, and then combining these in the i -direction in the ratio W We obtain the same result if we change the order, linearly interpolating first in the i -direction and then in the j -direction Chapter 6.1 Let  be the angle made by u and v (Fig 6.12) e length of the projection of v onto the line in the direction of the unit vector u is given by kv k cos  D kukkv k cos  D u; v /: Hence, v has the component u; v /u in the u direction It follows that the projection of v onto the plane is given by u u; v /u D u uu> v D I uu> /v D Pu v : 6.2 We see that Pu2 D I uu> /.I uu> / D I uu> uu> C uu> uu> D I D I 2uu> C uu> D I uu> D Pu : 2uu> C u.u; u/u> ANSWERS 117 Chapter 7.1 (1) Let F DU@ 0 0 A V >; > 0; / be the singular value decomposition of the matrix F Using the identities trŒA> A = kAk2 and trŒAB = trŒBA about the matrix trace and noting that U and V are orthogonal matrices, we obtain the following expression: 1 0 0 1 kF k2 D trŒF > F D trŒV @ 0 A U >U @ 0 A V > 2 0 0 3 2 0 0 1 2 A V > D trŒ@ D trŒV @ 0 A V >V 2 2 0 0 3 0 D tr @ 0 A 2 0 D C 2 C 3: (2) We first compute the singular value decomposition of F in the form of the above / and replace the smallest eigenvalue by to make det F = en, we change the scale so that kF k = In other words, we correct F to q 2 = C 0 B C > q C F UB = C AV : @ 2 7.2 From Eq (7.20), we see that x ξ 1/ y ξ 2/ = ξ 3/ Hence, the three equations of Eq (7.21) are related by x ξ 1/ ; θ / y ξ 2/ ; θ / = ξ 3/ ; θ /, meaning that if the first and the second equations are satisfied, the third is automatically satisfied 118 ANSWERS Chapter 8.1 From the rule of matrix multiplication, we observe that 10 AD u1 ur B @ :: : r v1> r C B :: C X A@ : A D iD1 vr > i ui vi : 8.2 is is immediately obtained from the above result 8.3 Since fui g and fvi g are orthonormal sets, ui ; uj / = ıij and vi ; vj / = ıij hold, where ıij is the Kronecker delta (1 for i = j and otherwise) Hence, we observe that AA A D D A AA D D r X r X > i ui vi iD1 r X j D1 i k i;j;kD1 r X iD1 r X i i;j;kD1 j j vj uj> r X kD1 r X > j uj vj j D1 i k > k uk vk ıij ıj k ui vk> D vi u> i j r X > i ui vi iD1 r X kD1 r X ıij ıj k vi u> k D D k iD1 i i k ui vi ; vj /.uj ; uk /vk> j i;j;kD1 D A; vk u> k D r X r X j i;j;kD1 i k vi ui ; uj /.vj ; vk /u> k vi u> i DA : 8.4 We see that AA D A AD r X iD1 > i ui vi r X vj uj> D r X j j D1 i;j D1 r r X X > u v D vi u> j j j i i j D1 i;j D1 r X iD1 i j j i ui vi ; vj /uj> D vi ui ; uj /vj> D r X i i;j D1 r X i;j D1 j j i ıij ui uj> D ıij vi vj> D r X ui u> i ; iD1 r X vi vi> : iD1 Hence, AA uk D A Avk D r X iD1 r X iD1 ui ui ; uk / D vi vi ; vk / D r X iD1 r X iD1 ıi;j ui D ıi;j vi D uk vk 1ÄkÄr ; r