Single lens multi ocular stereovision using prism 1 4

ACKNOWLEDGEMENT The author would like to express his most sincere appreciation to: Associate Professor Kah Bin LIM, the supervisor of my Ph.D. study, for giving me such an interesting and fruitful project to improve and demonstrate my ability, and for his continuous supervision and valuable foresight and insight on this project. Mr. Voon Pong LEE, for his excellent early contribution on single-lens stereovision using mirrors and initiation on single-lens stereovision using biprism (2F-filter); and Mr. Raymond Lye Choon NG, for his cooperation on the preliminary discussion on binocular stereovision using biprism. Mr. Yee, Mrs. Ooi, Ms. Tshin, Miss Hamidah and Mr. Zhang and all the staff in Control and Mechatronics Laboratory of the Mechanical Engineering Department, for their kind support. All colleagues and friends in Control and Mechatronics Laboratory, with whom this project has become such a meaningful and memorable experience. i TABLE OF CONTENT ACKNOWLEDGEMENT I TABLE OF CONTENT II SUMMARY V LIST OF TABLES VII LIST OF FIGURES VIII LIST OF SYMBOLS X CHAPTER 1. INTRODUCTION CHAPTER 2. LITERATURE REVIEW 2.1 CONVENTIONAL TWO CAMERA STEREOVISION TECHNIQUE 2.1.1 STEREOVISION USING TWO CAMERAS 2.1.2 A REVIEW ON CAMERA CALIBRATION TECHNIQUE 2.2 THE SINGLE-LENS STEREOVISION TECHNIQUE 2.2.1 SINGLE-LENS STEREOVISION SYSTEMS USING OPTICAL DEVICES 2.2.2 SINGLE-LENS STEREOVISION SYSTEM USING KNOWN CUES 11 CHAPTER 3. CAMERA CALIBRATION 15 CHAPTER 4. SINGLE-LENS BINOCULAR STEREO-VISION 20 4.1 ANALYSIS OF VIRTUAL STEREOVISION SYSTEM 21 ii 4.1.1 FORMATION OF VIRTUAL CAMERAS 21 4.1.2 DETERMINING THE VIRTUAL CAMERAS BASED ON CALIBRATION 24 4.1.3 DETERMINING THE VIRTUAL CAMERAS BASED ON GEOMETRICAL ANALYSIS OF RAY SKETCHING 27 4.2 EXPERIMENT 35 4.2.1 EXPERIMENTAL SETUP 35 4.2.2 EXPERIMENTAL PROCEDURES 37 4.2.3 EXPERIMENT RESULTS 39 4.3 DISCUSSION 41 4.3.1 FIELD OF VIEW: CONVERGENCE AND DIVERGENCE 41 4.3.2 ERROR ANALYSIS FOR THE GEOMETRICAL ANALYSIS BASED APPROACH 45 4.4 SUMMARY 49 CHAPTER 5. SINGLE-LENS TRINOCULAR STEREO-VISION 50 5.1 VIRTUAL CAMERA GENERATION 53 5.1.1 DETERMINING THE VIRTUAL CAMERAS BY CALIBRATION 56 5.1.2 DETERMINING THE VIRTUAL CAMERAS BY GEOMETRICAL ANALYSIS OF RAY SKETCHING 60 5.1.2.1 The basic idea 61 5.1.2.2 Detailed description 65 5.2 EXPERIMENT AND DISCUSSION 89 5.3 SUMMARY 92 CHAPTER 6. SINGLE-LENS MULTI-OCULAR STEREOVISION 94 6.1 VIRTUAL CAMERA GENERATION 96 6.1.1 DETERMINING THE VIRTUAL CAMERAS BY CALIBRATION 99 iii 6.1.2 DETERMINING THE VIRTUAL CAMERA BY GEOMETRICAL ANALYSIS OF RAY SKETCHING 104 6.2 EXPERIMENT AND DISCUSSION 109 6.3 SUMMARY 113 CHAPTER 7. CONCLUSION 115 CHAPTER 8. FUTURE WORK 118 BIBLIOGRAPHY 120 APPENDICES 129 A EPIPOLAR CONSTRAINTS 129 A SIMPLE CALIBRATION TECHNIQUE 138 GEOMETRY STUDY OF 3F FILTER 141 B C                         iv SUMMARY This thesis investigated a passive single-lens stereovision system using prism (filter). Each image captured by this system is split into multiple different subimages and these sub-images are taken as images simultaneously captured by one group of virtual cameras which are generated by the prism. Hence this system is able to obtain multiple different views of the same scene using a single camera in one shoot. The differences among these views, called disparities, are exploited to perform depth recovery. This system can also be called a virtual stereovision system corresponding to a virtual camera concept. According to the numbers of virtual cameras generated, binocular stereovision system, trinocular stereovision system and multi-ocular stereovision system are discussed separately. Two different approaches are developed to understand and model this system: one based on a camera calibration technique and another based on geometrical analysis of ray sketching. The latter approach requires no complex camera calibration, thus saving a large implementation effort without compromising accuracy. One real system is implemented and experiments are designed and conducted to test this concept. The result shows that both approaches are effective. While this stereovision system has the advantages of low cost, compactness, simultaneous image capturing, no camera synchronization problem, etc, it has the limitation of small baseline due to the dimension of prisms used. Hence this system is more suitable for close-range stereovision. To our knowledge, the approaches developed in this thesis to study and implement the single-lens binocular stereovision system are novel. Furthermore, the designs of the single-lens trinocular and multi-ocular stereovision systems and the v approaches used to understanding these two systems that are reported in this thesis are novel. Parts of this thesis have been previously published in papers. vi LIST OF TABLES TABLE 4.1 RECOVERED DEPTH BY BINOCULAR STEREOVISION, λ = 40MM 41 TABLE 5.1 RECOVERED DEPTH BY TRINOCULAR STEREOVISION, λ = 40MM 93 TABLE 6.1 RECOVERED DEPTH BY MULTI-OUCLAR STEREOVISION, FACE FILTER, λ=45MM 113 vii LIST OF FIGURES FIGURE 2.1 MODELING OF TWO-CAMERA STEREOVISION SYSTEM . FIGURE 2.2 THE CONCEPT OF EPIPOLAR LINE AND EPIPOLAR PLANE FIGURE 2.3 A SINGLE-LENS STEREOVISION SYSTEM USING A GLASS PLATE . FIGURE 2.4 A SINGLE-LENS STEREOVISION SYSTEM USING THREE MIRRORS 10 FIGURE 2.5 A SINGLE-LENS STEREOVISION SYSTEM USING TWO MIRRORS . 11 FIGURE 3.1 CAMERA CALIBRATION MODELING . 16 FIGURE 4.1 SINGLE-LENS STEREOVISION SYSTEM USING A BIPRISM 22 FIGURE 4.2 GENERATION OF VIRTUAL CAMERAS USING A BIPRISM (TOP VIEW) . 23 FIGURE 4.3 RAY MAP OF VIRTUAL-CAMERA CONFIGURATION . 31 FIGURE 4.4 SYSTEM SETUP 36 FIGURE 4.5 CALIBRATION BOARD . 36 FIGURE 4.6 CALIBRATION OF REAL CAMERA . 37 FIGURE 4.7 CALIBRATION OF VIRTUAL CAMERA . 38 FIGURE 4.8 DISPARITY INFORMATION . 39 FIGURE 4.9 FIELD OF VIEW: CONVERGENT SYSTEM (ω′1 < γ) 43 FIGURE 4.10 FIELD OF VIEW: DIVERGENT SYSTEM (ω′1 > 2γ) . 43 FIGURE 4.11 FIELD OF VIEW: DIVERGENT SYSTEM (γ < ω′1 < 2γ) . 44 FIGURE 4.12 A CASE OF CONVERGENT FIELD OF VIEW . 44 FIGURE 5.1 POSITIONING A 3F FILTER IN FRONT OF A CCD CAMERA 55 FIGURE 5.2 ONE IMAGE CAPTURED BY THE SINGLE-LENS TRINOCULAR SYSTEM . 55 FIGURE 5.3 POSITION RELATIONSHIP BETWEEN REAL CAMERA AND 3F FILTER . 64 FIGURE 5.4 SYMBOLIC ILLUSTRATION OF VIRTUAL CAMERA MODELING USING GEOMETRICAL ANALYSIS65 FIGURE 5.5 WORKFLOW OF DETERMINING THE VIRTUAL CAMERA VIA GEOMETRICAL ANALYSIS 68 FIGURE 5.6 PLANE PMN . 71 FIGURE 5.7 TEMPORARY COORDINATE SYSTEM T AND T′ USED IN FINDING LINE MN . 73 FIGURE 5.8 PLANE LNM . 76 FIGURE 5.9 TEMPORARY COORDINATE SYSTEM R AND R′ USED IN FINDING LINE NL 77 viii FIGURE 5.10 PLANE KJS . 79 FIGURE 5.11 ILLUSTRATION OF THE SHORTEST SEGMENT CONNECTING TWO NON-INTERSECTING, AND NON-PARALLEL LINES 82 FIGURE 5.12 PLANE F′P′K′ . 84 FIGURE 5.13 CALIBRATION OF VIRTUAL CAMERAS . 91 FIGURE 6.1 SYMBOLIC ILLUSTRATIONS OF MULTI-FACE FILTERS WITH AND FACES . 97 FIGURE 6.2 ONE IMAGE CAPTURED BY THE SINGLE-LENS MULTI-OCULAR SYSTEM (4 FACES) . 99 FIGURE 6.3 CALIBRATION OF VIRTUAL CAMERAS (4 FACES FILTER USED) . 112 FIGURE A. EPIPOLAR CONSTRAINT . 130 FIGURE A. EPIPOLAR CONSTRAINT (USING DIFFERENT CAMERA MODE) . 131 FIGURE A. ILLUSTRATIONS OF EPIPOLAR CONSTRAINTS IN TRINOCULAR STEREOVISION 136 FIGURE A. A SIMPLE PIN-HOLE CAMERA MODEL (SIDE VIEW) . 138 FIGURE A. A SIMPLE PIN-HOLE CAMERA MODEL WITH TWO CROSSING OBJECT LINES . 139 FIGURE A. SYMBOLIC ILLUSTRATION OF 3F FILTER STRUCTURE 142 FIGURE A. 3F FILTER 3D STRUCTURE, WITH FRONT AND SIDE VIEW . 143 ix LIST OF SYMBOLS λ = BASELINE, I.E. THE DISTANCE BETWEEN THE TWO CAMERA OPTICAL CENTRES γ = THE ANGLE BETWEEN TWO CAMERA OPTICAL AXES f = EFFECTIVE REAL CAMERA FOCAL LENGTH f′ = EFFECTIVE REAL VIRTUAL CAMERA FOCAL LENGTH NR = REFLECTIVE INDEX OF PRISM NCX = NUMBER OF COLUMNS OF SENSOR ELEMENTS IN X-DIRECTION IN THE CCD NCY = NUMBER OF COLUMNS OF SENSOR ELEMENTS IN Y-DIRECTION IN THE CCD NFX = NUMBER OF PIXELS IN A LINE AS SAMPLED BY THE COMPUTER IN X-DIRECTION NFY = NUMBER OF PIXELS IN A LINE AS SAMPLED BY THE COMPUTER IN Y-DIRECTION dX = DISTANCE BETWEEN ADJACENT CCD ELEMENTS IN X-DIRECTION dY = DISTANCE BETWEEN ADJACENT CCD ELEMENTS IN Y-DIRECTION ρ = YAW ANGLE (ROTATION ABOUT Y AXIS) ν = PITCH ANGLE (ROTATION ABOUT X AXIS) ζ = TILT ANGLE (ROTATION ABOUT Z AXIS) G = TRANSLATION MATRIX R = ROTATION MATRIX P = TRANSFORMATION MATRIX (xw,yw,zw) = WORLD COORDINATE SYSTEM (xcam,ycam,zcam) = CAMERA COORDINATE SYSTEM k1 , k2 = CAMERA LENS DISTORTION COEFFICIENTS x (x L − x R )sin 2γ f' cos γ − x L x R sin γ − f ' ZV = λ (x L − x R )cos 2γ f' + x L x R sin 2γ + f ' . ( ) (4.24) The depth of the object measured from the left (right) virtual optical center, ZV, is given by equation (4.24). In this equation, λ is the baseline, γ is panning angle and is equal to φ2. The values of xR and xL can be determined from the image plane using equation (4.3). If we let the required depth measured from the optical center of real camera be Z, then Z = ZV − d . (4.25) In the proceeding equation d is the distance between the real and virtual optical centers in the Z-direction and has been determined using (4.17). Now, f and f′, the only parameters that are not known after the system is established (as it is assumed that the real camera is not calibrated in this approach), can be found by a field point test. Fieled point test is devised in our work to determine system parameters using the known coordinates of points. In this case, after the system has been set up, a point with known coordinates with respect to the world coordinate system is identified. A pair of corresponding points of the known point is then identified from the two sub-views of the stereo images. The depth of the known point Z is known and d can be determined from equation (4.17). In addition, γ and λ are known. Hence using equations (4.24) and (4.25) can be solved for f′. Then f = f′ from equation (4.20), thus f and f′ are determined. The calculated values of f and f′ can be used for depth recovery of other points of interest. This recovered f and f′ actually is a scale factor between the system and the external world. Nevertheless, this recovered f and f′ also pick up or compensate 34 the non-idealities introduced by the approximation of pinhole camera model and some other errors, such as, errors caused in system setup, to a certain extent. Thus a steady depth recovery performance can be expected from the proceeding approach. On the other hand, the depth performance calibration based system modeling approach may yield poorer depth recovery result. This is due to the fact that in the experiments the value of λ (usually, 30-50mm) and the disparities between stereo image pairs obtained are small due to hardware constraints. This results in making the system more sensitive to camera calibration errors. The proceeding analysis suggests that a complicated calibration process including the preparation of calibration software/equipments and the calibration operation itself can all be replaced by an alignment procedure between camera and prism and a field point test, which obviously results in a much easier system implementation. 4.2 Experiment 4.2.1 Experimental Setup The camera used in this system is a PULNIX TM-6CN camera (its calibrated focal length is about 25-26mm in the experiments), and the main bi-prism (2F filter) used has a corner angle of 6.4°, its refraction index is 1.48 and its thickness is 5.55mm. A mechanical stand is designed to hold the camera and bi-prism so that their relative positions can be adjusted easily with acceptable accuracy, as shown in Figure 4.4 which shows the system setup. 35 Figure 4.4 System setup Figure 4.5 Calibration board The calibration technique described in Chapter is implemented using C++ under NT environment. A calibration board is designed as shown in Figure 4.5. The relative positions between the circle centers are known, and when these circles are projected onto camera image planes, the centers of the captured circles are calculated by the software developed (the calculation bases on one threshold value of the circle boundary and one parameter which defines the automatic circle range search zoom, and the mass center of the circles are automatically searched and found), and the 36 calculated values are used as the coordinate values in the computer image coordinate system. 4.2.2 Experimental Procedures The experimental procedures are designed to test and verify the feasibility and accuracy of the two approaches developed for this single-lens binocular stereovision system. They can be divided into three main steps. The first step is to calibrate the real camera, and the second step is to determine the left and the right virtual cameras either by calibration or by geometrical analysis. The main purpose for the first step is to determine the intrinsic and extrinsic parameters of the real camera, and the second step is to determine and adjust the relative position between the camera and the bi-prism which defines the positions of virtual cameras. Figure 4.6 Calibration of real camera 37 Figure 4.6 is an image captured during the calibration of the real camera and Figure 4.7 is an image captured during the calibration of the virtual cameras. The centers of the labeled circles are the points to be used in the calibration. One parameter worth mentioning here is the threshold values used to binarize the captured image before calculating the circle centers, as different threshold values will result in different sizes of the circle and will hence noticeably affect the calculated circle centers. The final step is the depth recovery. This step aims to verify the two developed approaches by comparing the recovered depth against the actual depth. Figure 4.8 shows an image captured by two virtual cameras of the same scene (a mechanical stand), from which obvious view angle and view zone difference between the left half image and right half image can be seen. In the experiments the correspondence searching ends at pixel accuracy and does not go into sub-pixels. Figure 4.7 Calibration of virtual camera 38 Figure 4.8 Disparity information 4.2.3 Experiment Results The biprism and camera described in section 4.2.1 are used in this experiment, and one typical system setup is: the distance between camera optical center and biprism apex a is set to 365 mm and the value of the distance between two virtual camera optical centers λ is about 40mm. The recovered depth error increases as depth increases. In our experiments, only the x coordinates need to be determined as the y coordinates of the corresponding points are approximately the same. This is because φ2, the angle between the virtual camera image plane and real camera image plane is about 3.08° according to geometrical analysis of ray sketching or between 3.1° – 3.4° according to the calibration result (see Figure 4.2). Therefore it is reasonable to assume that the two virtual camera image planes are coplanar when doing depth 39 recovery, and it can then be assumed that the epipolar lines are horizontally positioned. Simple calculation shows that the shift in the y-coordinate of correspondence caused by this approximation is less than pixel size for the depth recovery situation encountered in our experiments. This minor error is even less significant compared with the error caused by the system setup such as the alignment of the prism with respect to the real camera. Obviously this angle (the angle between the virtual camera image plane and real camera image plane) should not be neglected when analyzing the field of view. Note that Lee, et al. [23][24] directly concluded that the image pairs captured by this system are coplanar and the corresponding points in the stereo image pairs lie on the same scanline without such a discussion. Encouraging results are obtained from these two different approaches of modeling the virtual stereovision system. It is shown that for the depth ranged from 1.3m to 2.5m the geometrical analysis based approach can give an absolute depth recovery error of about 0.93% in average using a typical setup in the experiment (λ = 60mm) (see equations (4.24) and (4.25)), while the calibration based approach can give an error of about 0.96% under the same condition (see equations (4.6) and (4.7)). Therefore we believe that both approaches are capable to determine this system sufficiently accuracy for many stereovision based applications, such as autonomous robot indoor navigation. The detailed depth recovery result using the typical setup in experiment is given in Table 4.1. Please note in Table 4.1 the actual depth is defined as the projection distance between the optical centre of the real camera and the object of interested along the optical axis (z-direction). This definition also applies to the other experiment results in the rest of the thesis. 40 Actual Depth (mm) Correspondence Pixel Pairs (from Computer Screen) Recovered Depth (mm, Calibrationbased Approach) Absolute Error in Percentage (%) 1300 457,194 523,260 1283.4 1288.2 1.28 0.91 Recovered Depth (mm, Geometrical Analysis-based Approach) 1315.5 1316.9 586,324 604,328 1279.8 1494.5 1.55 0.37 1302.2 1504.2 0.20 0.28 551,275 494,218 1485.5 1476.3 0.97 1.58 1507.3 1507.4 0.49 0.49 484,198 535,248 1662.8 1698.7 2.19 0.08 1695.5 1718.2 0.30 1.21 598,311 565,270 1717.3 1912.5 1.02 0.66 1714.8 1911.2 0.99 0.75 502,207 451,155 1885.7 1892.0 0.75 0.42 1911.4 1933.8 0.76 2.25 622,321 540,239 2136.6 2087.0 1.74 0.62 2081.2 2088.9 1.26 0.74 480,178 450,142 2086.4 2284.8 0.65 0.66 2118.4 2331.3 1.23 2.09 529,222 580,273 2303.8 2344.4 0.17 1.93 2301.4 2299.3 0.09 0.05 467,155 517,205 2470.2 2517.8 1.19 0.71 2506.8 2513.8 0.45 0.92 589,278 2540.4 1.62 0.96 2465.4 2.30 0.93 1500 1700 1900 2100 2300 2500 AVG Absolute Error in Percentage (%) 1.19 1.54 Table 4.1 Recovered depth by binocular stereovision (λ λ = 40mm) 4.3 Discussion 4.3.1 Field of View: Convergence and Divergence According to different types of field of view (FOV is defined as the region that is visible to both virtual cameras) this system can be further categorized into two groups: Convergent Type and Divergent Type. If the virtual stereovision system has the characteristic that the angle between the virtual camera optical axes produced by the bi-prism are further increased along the positive direction of z-axis, the outer 41 FOV boundaries of the two virtual cameras generated diverge and the resulting configuration is known as a divergent virtual stereovision system. Otherwise, a convergent virtual stereovision system is generated. In this thesis, our main interest is on the divergent system. The reason is that although the convergent system yields a larger baseline λ (unlike the divergent system, which is constrained by the width of the biprism) which gives better accuracy (as will be explained in the next section), it has the disadvantage of a closed and narrow field of view (FOV) which renders it impractical for most usages. Figure 4.9 shows the convergent system with a closed FOV. In Figure 4.10, a divergent system with an open FOV is illustrated, which is the most ideal divergent situation while Figure 4.11 shows another divergent system. These different scenarios are caused by different ω′1 and γ values which can be varied by choosing different α values (corner angle of the bi-prism). Figure 4.12 shows a pair of images captured by a setup with convergent configuration from which two completely excluded scenes can be captured if the object is not located within the closed FOV. Convergence or divergence of FOV provides one useful way to characterize the binocular single-lens stereovision system. The configuration in Figure 4.2 and Figure 4.3 illustrate the geometrical analysis of virtual cameras is the divergent case. Similar analysis can be applied to a convergent systems, with the difference that the bi-prism apex is a singularity point in the geometrical analysis of ray sketching, the ray aligning with real camera principle axis and passing through this point should bend towards right (left) when left (right) virtual camera is analyzed. 42 Closed FOV γ Left virtual image plane Right virtual image plane ω’1 Figure 4.9 Field of view: convergent system (ω ω′1 < γ) Open FOV γ ω’1 Left virtual image plane Right virtual image plane Figure 4.10 Field of view: divergent system (ω ω′1 > 2γγ) 43 FOV γ Left virtual image plane ω’1 Right virtual image plane Figure 4.11 Field of view: divergent system (γγ < ω′1 < 2γγ) Figure 4.12 A case of convergent field of view 44 4.3.2 Error Analysis for the Geometrical Analysis based Approach This section describes a simple error analysis for the approach based on the geometrical analysis of ray sketching. Taking reciprocal on both sides of equation (4.24) and decomposing the right hand side into partial fractions gives: λ Z = xR cos γ − f sin γ x cos γ + f sin γ − L − xR sin γ − f cos γ xL sin γ − f cos γ (4.26) Defining the disparity as | δ | = | xL - xR | and supposing a point is selected on the left image while an error is incurred in the process of finding the corresponding point on the right image, it follows that the disparity error is: dδ = − dx R = dx R (4.27) when xL is kept constant. Taking the first derivative of equation (4.26) with respect to xR gives − λ dZ δ Z dx R = ( − x R sin γ − f cos γ )(cos γ ) − ( x R cos γ − f sin γ )( − sin γ ) ( − x R sin γ − f cos γ ) dZ δ dZ δ fZ = = dδ dx R λ ( x R sin γ + f cos γ ) dZ δ = fZ dδ λ ( x R sin γ + f cos γ ) (4.28) Since ( x R sin γ ) is very small compared to f cos γ , it can be ignored, thus: 45 Z2 dZ δ ≈ dδ fλ cos γ (4.29) Now, suppose an error of one pixel width is incurred when finding a corresponding point in the right image, the disparity error is equal to | dδ | = Horizontal distance between adjacent sensor elements of the camera = 0.0086 mm (for the camera used) If, for instance, the stereo-system setup is such that f = 27mm, γ = 3o and λ = 40mm, then, assuming the disparity error in corresponding point searching is onepixel, the error incurred in determining the depth at 2.5m is dZ δ ≈ 2500 × 0.0086 ≈ 49.9 mm 27 × 40 cos We can see that equation (4.29) gives a quantitative guide on how much depth recovery error is caused by the disparity error, and our experience shows that this is a good guide for the experimental setup. For example, the setup used in experiment gives a theoretical relative absolute error of about 1% - 2% for the range of the depth from 1.3m to 2.5m if the correspondence error of one pixel size is made, and the experimental results shows that most errors are located about within this range (see Table 4.1). In another word, people can use equation (4.29) to make the system design including system setup and acquiring the hardware based on their requirement on system depth recovery accuracy. From equation (4.24) it can be seen that the calculation of depth also involves the values of λ, γ and f′. Hence, errors that are incurred during the experimental process of obtaining these parameters also contribute to the total error of the system. From equation (4.24), the first order derivative with respect to λ is: 46 (xL − xR )sin 2γ f' cos γ − x L x R sin γ − f ' dZ λ = dλ (x L − x R )cos 2γ f' + x L x R sin 2γ + f ' ( ) (4.30) When finding dZ γ and dZ f , it is assumed that (x L − x R )sin 2γ , and cos γ − x L x R sin γ − f ' A= f' ( ) B= f' + x L x R sin 2γ + f ' (x L − x R )cos 2γ , so that Z λ = A , then the B derivative with respect to γ is: [ ] B− f' sin 2γ − x L x R sin 2γ − f ' ( x L − x R ) cos 2γ B dZ γ = λ dγ ( x L − x R ) sin 2γ A 2( f '+ x L x R ) cos 2γ − f ' − B2 [ ] (4.31) and the dirivative with resepect to f′ is: [ ] B− f' sin 2γ − x L xR sin 2γ − f '( xL − x R ) cos 2γ B2 dZ f ' = λ df ' A 2( f ' + x L xR ) cos 2γ − f '( xL − xR ) sin 2γ − B2 [ ] (4.32) The total error in the calculation of depth can be defined as dZ T = dZ δ + dZ λ + dZ f ' + dZ γ (4.33) Experiments show that the disparity error is the most significant source of error once the setup is sufficiently accurate, which implies that: 47 Z2 dZ T ≈ dZ δ ≈ dδ f' λ cos γ (4.34) From the proceeding analysis, it can be seen that the performance of this binocular single-lens stereovision system can be improved in several ways. From equation (4.29) or (4.34), it can be seen that in order to reduce dZ δ , which is the depth error caused by the disparity error, a smaller dδ (camera CCD pixel size) or larger γ and λ values should be used: 1) to decrease dδ , a camera with a smaller sensor element size should be employed; 2) to increase λ, which is the length of the baseline between the two virtual cameras, larger values of α (the corner angle of bi-prism), nr or a (the distance between the prism apex and the real camera lens center) should be chosen. And γ can also be increased by a larger value of α. However, increasing α may not be a good idea, as a sufficiently large biprism corner angle will generate a convergent system as described in section 4.3.1, which is not desirable for most of the applications. Another reason is that increasing α may cause chromatic problems, such as, chromatic dispersion, which means that the refracted rays will split into its various color components and the resulted blurriness of object will affect object detection and corresponding matching. Hence increasing a is a relatively better way to achieve a larger value of λ. Further experiments are conducted to investigate the effect of different values of λ on the system performance. The result shows that a larger λ will obviously result in a better depth recovery. For example, under the same situation described in section 4.2.3, if two different values of a are used, say, 456mm and 273mm, which 48 give the value of λ of 50mm and 30mm respectively, the system can give an overall relative absolute error (error in percentage) of 0.54% and 1.61% in contrast with the error of 0.93% given in Table 4.1 for the geometrical analysis based approach. 4.4 Summary This chapter analyzes a single-lens binocular stereovision system employing a bi-prism. The stereo-image acquired by the proposed camera system is equivalent to the two images obtained by two symmetrically located virtual cameras. Two different approaches have been described to determine this system: one of them based on camera calibration and the other based on geometrical analysis of ray sketching. As the model based on geometrical analysis does not require complex calibration procedure which normally cannot be avoided by the normal two or multiple camera stereovision system, it presents the significant advantage of a greatly simplified setup process. In addition, its accuracy is good enough for many close-range stereovision applications as shown in the experiment. Furthermore, as the corresponding points in the stereo-image pairs automatically and approximately lie on the same horizontal scan-lines, or in another words, the epipolar lines can approximately be taken as horizontally positioned on the camera image plane, correspondence searching is greatly facilitated. Field of view and different error factors were also analyzed for the consideration of more thorough implementation. 49 [...]... the single- lens binocular stereovision system is presented, followed by the description on trinocular stereovision system which combines the advantages of the single- lens stereovision and the trinocular stereovision Finally, the understanding of this single- lens trinocular stereovision is generalized and a singlelens multi- ocular stereovision system is created 2 The advantages of this single- lens stereovision. .. organized as follows: Chapter 2 gives a literature review on single- lens stereovision technique; Chapter 3 describes the theory of calibration technique which is used by this system; Chapter 4 describes the single- lens binocular stereovision system; Chapter 5 and 6 describe the single- lens trinocular stereovision system and the single- lens multi- ocular stereovision system, respectively; finally Chapter 7 and... is explained further in section 4. 2 This equation is: yl = yr (4. 5) If this condition (equation (4. 5)) is used, by re-arranging the equation (4. 4) and (4. 5) following equation can be obtained: Aε = B , (4. 6) 25 where rl1 rl 2 rl 3 0 rl 4 rl 5 rl 6 1 X ul fl 0 rl 7 rl 8 rl 9 0 1 rr1 rr 2 rr 3 0 0 rr 4 rr 5 rr 6 0 0 0 X ur 0 − fr 1 0 rr 7 rr 8 rr 9 0 0 1 0 0 0 0 1 0 1 0 ε = [x w yw yr zr ] − Trx... cameras which are generated by the prism The disparities among these views are exploited to perform depth recovery like usual stereovision systems This system can be further categorized into three types according to different numbers of virtual cameras generated: single- lens binocular stereovision system, single- lens trinocular stereovision system and single- lens multi- ocular stereovision system which will... test the single- lens binocular, trinocular and multi- ocular stereovision systems and to verify the validity of this system The results can prove the effectiveness of the both approaches used to model these systems We believe that most of the work presented in this thesis, including the way of modeling the single- lens binocular system and the design of the single- lens trinocular and multi- ocular systems... camera) does not locate at the virtual image plane center but at its boundary (see Figure 4. 2) α Left virtual optical axis Left virtual image plane 4 φ3 4 φ3' f' Prism 1 f Real image plane 1 1 1' φ2' φ2 Real optical axis Prism apex Figure 4. 2 Generation of virtual cameras using a biprism (top view) 23 4 .1. 2 Determining the Virtual Cameras Based on Calibration The calibration technique introduced... thesis In this thesis the newest approaches of understanding this kind of binocular stereovision and also designs of trinocular and multi- ocular stereovision system are presented, which are believed to be novel Lee and Kweon et al [23][ 24] proposed a single- lens stereo system using one biprism which has a similar setup of the binocular system that is presented in this thesis, but in the aspect of the approaches... α Left virtual camera 1 CCD camera Common View Zone Left virtual optical axis Right virtual camera B -prism Real optical axis Figure 4 .1 Single- lens stereovision system using a biprism If the following conditions hold: 1) the real image plane of camera has consistent properties; 22 2) the bi -prism is exactly symmetrical with respect to its apex line; 3) the projection of bi -prism apex line on camera... the Appendices 4 CHAPTER 2 LITERATURE REVIEW This section firstly gives a brief review on the theories of computer stereovision and camera calibration that are the basic concepts used through the thesis, and then presents a detailed review on the single- lens stereovision techniques 2 .1 Conventional Two Camera Stereovision Technique 2 .1. 1 Stereovision Using Two Cameras A conventional stereovision process... Figure 2.3 A single- lens stereovision system using a glass plate Mirrors are often used to assist in achieving single- lens stereovision effect Teoh and Zhang [9] described a single- lens stereo-camera system which employs three mirrors as shown in Figure 2 .4 Two mirrors are positioned to at a 45 ° relative to the optical axis of the camera, and a third mirror is positioned in front of the 9 camera lens and . Optical axis Optical axis P l P r Epipolar plane 8 , 1 or 44 4 342 41 343 332 31 242 322 21 1 41 3 1 211 4 3 2 1                         =             = w w w h h h h hh z y x aaaa aaaa aaaa aaaa c c c c PRGwc . generated: single- lens binocular stereovision system, single- lens trinocular stereovision system and single- lens multi- ocular stereovision system which will be discussed separately. Firstly the single- lens. system; Chapter 4 describes the single- lens binocular stereovision system; Chapter 5 and 6 describe the single- lens trinocular stereovision system and the single- lens multi- ocular stereovision