Xử lý ảnh trong cơ điện tử machine vision chapter 9 camera calibration and 3d reconstruction

TRƯỜNG ĐẠI HỌC BÁCH KHOA HÀ NỘI XỬ LÝ ẢNH TRONG CƠ ĐIỆN TỬ Machine Vision Giảng viên: TS Nguyễn Thành Hùng Đơn vị: Bộ môn Cơ điện tử, Viện Cơ khí Hà Nội, 2021 Chapter Camera Calibration and 3D Reconstruction Camera calibration Robot Camera Calibration Pose estimation Stereo vision Chapter Camera Calibration and 3D Reconstruction Camera calibration Robot Camera Calibration Pose estimation Stereo vision Camera calibration ❖ Geometric camera calibration, also referred to as camera resectioning, estimates the parameters of a lens and image sensor of an image or video camera ❖ You can use these parameters to correct for lens distortion, measure the size of an object in world units, or determine the location of the camera in the scene ❖ These tasks are used in applications such as machine vision to detect and measure objects They are also used in robotics, for navigation systems, and 3-D scene reconstruction Camera calibration Camera calibration ❖ Camera parameters include intrinsics, extrinsics, and distortion coefficients ❖ To estimate the camera parameters, you need to have 3-D world points and their corresponding 2-D image points ❖ You can get these correspondences using multiple images of a calibration pattern, such as a checkerboard Using the correspondences, you can solve for the camera parameters Camera calibration ❖ After you calibrate a camera, to evaluate the accuracy of the estimated parameters, you can: ➢ Plot the relative locations of the camera and the calibration pattern ➢ Calculate the reprojection errors ➢ Calculate the parameter estimation errors Camera calibration ❖ Pinhole Camera Model ➢ A pinhole camera is a simple camera without a lens and with a single small aperture Camera calibration ❖ Pinhole Camera Model Camera calibration ❖ Pinhole Camera Model ▪ The calibration algorithm calculates the camera matrix using ➢ The intrinsic parameters ➢ The extrinsic parameters Scale factor Image points Intrinsic matrix Extrinsics Rotation and translation World points 10 Camera Pose Estimation ❖ Algorithm ➢ The algorithm for determining pose estimation is based on the Iterative Closest Point algorithm The main idea is to determine the correspondences between 2D image features and points on the 3D model curve (a) Reconstruct projection rays from the image points (b) Estimate the nearest point of each projection ray to a point on the 3D contour (c) Estimate the pose of the contour with the use of this correspondence set (d) goto (b) 31 Camera Pose Estimation with OpenCV 32 Chapter Camera Calibration and 3D Reconstruction Camera calibration Robot Camera Calibration Pose estimation Stereo vision 33 Stereo Vision ❖ What is stereo correspondence? ➢ only have 2D information when capture images ➢ the depth information is lost ➢ our brain takes two images and builds a 3D map using stereo vision ➢ capture two photos of the same scene using different viewpoints, and then match the corresponding points to obtain the depth map of the scene → stereo vision algorithms 34 Stereo Vision ❖ What is stereo correspondence? Left images Right images 35 Stereo Vision ❖ What is stereo correspondence? ➢ The absolute difference between d1 and d2 is greater than the absolute difference between d3 and d4 ➢ The camera moved by the same amount, there is a big difference between the apparent distances between the initial and fnal positions → This happens because we can bring the object closer to the camera; the apparent movement decreases when you capture two images from different angles ➢ This is the concept behind stereo correspondence: we capture two images and use this knowledge to extract the depth information from a given scene 36 Stereo Vision ❖ What is epipolar geometry? 37 Stereo Vision ❖ What is epipolar geometry? Epipolar lines Epipolar lines 38 Stereo Vision ❖ What is epipolar geometry? ➢ Our goal is to match the keypoints in these two images to extract the scene information ➢ The way we this is by extracting a matrix that can associate the corresponding points between two stereo images → the fundamental matrix ➢ The point at which the epipolar lines converge is called epipole ➢ If you match the keypoints using SIFT, and draw the lines towards the meeting point on the left and right images, they will look like this: 39 Stereo Vision ❖ What is epipolar geometry? 40 Stereo Vision ❖ What is epipolar geometry? ➢ If two frames are positioned in 3D, then each epipolar line between the two frames must intersect the corresponding feature in each frame and each of the camera origins ➢ This can be used to estimate the pose of the cameras with respect to the 3D environment ➢ We will use this information later on, to extract 3D information from the scene 41 Stereo Vision ❖ Building the 3D map 42 Stereo Vision ❖ Building the 3D map ➢ The frst step is to extract the disparity map between the two images ➢ Once we fnd the matching points between the two images, we can fnd the disparity by using epipolar lines to impose epipolar constraints 43 Stereo Vision ❖ Building the 3D map ➢ x and x′ are the distance between points in image plane corresponding to the scene point 3D and their camera center ➢ B is the distance between two cameras (which we know) and f is the focal length of camera (already known) ➢ So in short, above equation says that the depth of a point in a scene is inversely proportional to the difference in distance of corresponding image points and their camera centers ➢ So with this information, we can derive the depth of all pixels in an image 44 Stereo Vision ❖ Building the 3D map 45 .. .Chapter Camera Calibration and 3D Reconstruction Camera calibration Robot Camera Calibration Pose estimation Stereo vision Chapter Camera Calibration and 3D Reconstruction Camera calibration. .. https://visp-doc.inria.fr/doxygen/visp-daily/classvpHandEyeCalibration.html#a68ab673312cc 098 bf8ed4f22b2d04d16 ➢ https://github.com/zhixy/SolveAXXB 28 Chapter Camera Calibration and 3D Reconstruction Camera calibration Robot Camera Calibration Pose estimation Stereo vision 29 Camera. .. goto (b) 31 Camera Pose Estimation with OpenCV 32 Chapter Camera Calibration and 3D Reconstruction Camera calibration Robot Camera Calibration Pose estimation Stereo vision 33 Stereo Vision ❖ What