Feature detectors and descriptors

Feature detectors and descriptors Fei-Fei Li Feature Detection Feature Description Matching / Indexing / Recognition local descriptors – (invariant) vectors detected points – (~300) coordinates, neighbourhoods database of local descriptors e.g. DoG e.g. SIFT e.g. Mahalanobis distance + Voting algorithm [Mikolajczyk & Schmid ’01] Some of the challenges… • Geometry – Rotation – Similarity (rotation + uniform scale) – Affine (scale dependent on direction) valid for: orthographic camera, locally planar object • Photometry – Affine intensity change (I → a I + b) Detector Descrip- tor Intensity Rotation Scale Affine Harris corner 2 nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2 nd moment(s) Tuytelaars, ‘00 2 nd moment(s) Lowe ’99 (DoG) SIFT, PCA- SIFT Kadir & Brady, 01 Matas, ‘02 others others Detector Descriptor Intensity Rotation Scale Affine Harris corner 2 nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2 nd moment(s) Tuytelaars, ‘00 2 nd moment(s) Lowe ’99 (DoG) SIFT, PCA- SIFT Kadir & Brady, 01 Matas, ‘02 others others An introductory example: Harris corner detector C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988 Harris Detector: Basic Idea “flat” region: no change in all directions “edge”: no change along the edge direction “corner”: significant change in all directions Harris Detector: Mathematics [ ] 2 , (,) (, ) ( , ) (, ) xy Euv wxy Ix uy v Ixy= + +− ∑ Change of intensity for the shift [u,v]: Intensity Shifted intensity Window function or Window function w(x,y) = Gaussian1 in window, 0 outside Harris Detector: Mathematics [ ] (,) , u Euv uv M v  ≅   For small shifts [u,v] we have a bilinear approximation: 2 2 , (, ) x xy xy xy y I II M wxy II I  =    ∑ where M is a 2×2 matrix computed from image derivatives: Harris Detector: Mathematics [ ] (,) , u Euv uv M v  ≅   Intensity change in shifting window: eigenvalue analysis λ 1 , λ 2 – eigenvalues of M direction of the slowest change direction of the fastest change (λ max ) -1/2 (λ min ) -1/2 Ellipse E(u,v) = const [...]... (x L , Σ I , L , Σ D , L ) = M L Σ I , L = tM L and µ (x R , Σ I , R , Σ D , R ) = M R −1 Σ I , R = tM R Σ D , L = dM L −1 −1 Σ D , R = dM R −1 Then by normalising: − x′L → M L 1/ 2 x L We get: and − x′R → M R 1/ 2 x R x′L → Rx′R so the normalised regions are related by a pure rotation See also [Lindeberg & Garding ’97] and [Baumberg ’00] Interest point detectors Harris-Affine [Mikolajczyk & Schmid ’02]... Interest point detectors Harris-Affine [Mikolajczyk & Schmid ’02] • Adds invariance to affine image transformations • Initial locations and isotropic scale found by Harris-Laplace • Affine invariant neighbourhood evolved iteratively using the 2nd moment matrix μ: g (Σ ) = 1 2π xT Σ −1x exp(− ) 2 Σ L(x, Σ) = g (Σ) ⊗ I (x) µ (x, Σ I , Σ D ) = g (Σ I ) ⊗ ((∇L(x, Σ D ))(∇L(x, Σ D ))T ) Interest point detectors. ..Harris Detector: Mathematics Classification of image points using eigenvalues of M: λ2 “Edge” λ2 >> λ1 “Corner” λ1 and λ2 are large, λ1 ~ λ2 ; E increases in all directions λ1 and λ2 are small; E is almost constant in all directions “Flat” region “Edge” λ1 >> λ2 λ1 Harris Detector: Mathematics Measure of corner response: = det M − k ( trace M )... others Yes Yes Scale Affine No No Interest point detectors Harris-Laplace [Mikolajczyk & Schmid ’01] • Adds scale invariance to Harris points – Set si = λsd – Detect at several scales by varying sd – Only take local maxima (8-neighbourhood) of scale adapted Harris points – Further restrict to scales at which Laplacian is local maximum Interest point detectors Harris-Laplace [Mikolajczyk & Schmid ’01]... Harris Detector: Some Properties • Quality of Harris detector for different scale changes Repeatability rate: # correspondences # possible correspondences C.Schmid et.al “Evaluation of Interest Point Detectors IJCV 2000 Detector Descriptor Intensity Rotation Harris corner 2nd moment(s) Mikolajczyk & Schmid ’01, ‘02 2nd moment(s) Tuytelaars, ‘00 2nd moment(s) Lowe ’99 (DoG) SIFT, PCASIFT Kadir & Brady,... moment(s) Lowe ’99 (DoG) SIFT, PCASIFT Kadir & Brady, 01 Matas, ‘02 others others Affine Invariant Detection • Take a local intensity extremum as initial point • Go along every ray starting from this point and stop when extremum of function f is reached I (t ) − I 0 f f (t ) = t 1 points along the ray t ∫ I (t ) − I 0 dt o • We will obtain approximately corresponding regions T.Tuytelaars, L.V.Gool “Wide . Feature detectors and descriptors Fei-Fei Li Feature Detection Feature Description Matching / Indexing / Recognition local descriptors – (invariant) vectors detected. E(u,v) = const Harris Detector: Mathematics λ 1 λ 2 “Corner” λ 1 and λ 2 are large, λ 1 ~ λ 2 ; E increases in all directions λ 1 and λ 2 are small; E is almost constant in all directions “Edge”. others An introductory example: Harris corner detector C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988 Harris Detector: Basic Idea “flat” region: no change in all directions “edge”: no