FISITA nof2008 08 118 v2

FISITA World Automotive Congress 2008 14th -18th September 2008, ICM Munich, Germany 16th Sep.2008, S-7 Advanced Safety Systems II Pedestrian Recognition by Single Camera for Driver Assistance Hirotomo Muroi, Ikuko Shimizu Pongsathorn Raksincharoensak* and Masao Nagai Faculty of Engineering Tokyo University of Agriculture and Technology Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Contents ▶ ▶ ▶ ▶ ▶ ▶ Motivation and Objectives of the Research Recognition Algorithm of Pedestrian near Crosswalk Crosswalk Detection Algorithm Pedestrian Classification Algorithm Experimental Results Summary and Conclusion Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide Motivation ( Pedestrian accidents ) ▶ International comparison of distribution of traffic accident death classified by condition (2000) Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide Motivation Car drivers and passengers ▶Statistics data about traffic accidents in JapanPedestrians Pedestrian accidents (2007) 20000 2000 15000 1500 10000 1000 5000 500 Cross Nearby walk cross walk Pedestrian During crossing crossing another bridge places 35.1% 33.9% Playing Operating Oncoming Rearward on road on road Traffic accident fatalities 5,732 13.0% Bicyclists 8.2% 9.8% Motorcycle Small-scaleriders motorcycle riders Statistical data of traffic accidents in Japan (2007) ・ Pedestrian collision accidents mostly occur near crosswalk  Pedestrian recognition near crosswalk is effective for pedestrian protection Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide State of the Art ▶ Characteristics of sensors for pedestrian recognition Sensor Environment Information amount Cost performance Algorithmic simplicity Visible camera Day ○ ○ × Infrared camera Day/Night △ × △ Millimeter Wave radar Day/Night × △ ○ ▶ Current driver assistance systems for pedestrian protection Stereo camera Millimeter Wave radar Headlight with Infrared beams ▶Infrared stereo camera (Honda Night Vision) ▶Stereo camera with Millimeter Wave radar (Toyota) ▶Stereo camera (Subaru) Toyota LEXUS LS460 Pre-Crash Safety System Using the multipurpose single camera is still in development process Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide Multiple functions of single camera Front vehicle detection Lane marker detection Traffic sign recognition for speed pilot Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide Pedestrian detection Objectives Development of a new recognition algorithm for detection of pedestrians near crosswalk by on-vehicle single camera Pedestrian Pedestrian crosswalk Vehicle Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide Strategy of Pedestrian Recognition Focusing on pedestrian (or other moving objects) on the crosswalk ▶The computational results of two algorithms are integrated to detect pedestrians : Crosswalk detection Pedestrian classification Crosswalk Crosswalk detection Integration Pedestrian classification Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide Pedestrian near a crosswalk Crosswalk Detection Using Cross Ratio ▶ Perspective projection: the relation between the real world and its image The cross ratio of parallel lines are preserved under the perspective projection Definition of cross ratio ▶ [ ABCD ] = AC BD ⋅ BC AD AB ,CD (Thickness) [m] AC (Width) [m] 0.45 ~ 0.50 0.90 ~ 1.00 Crosswalk (Japan) A B 0.20 ~ 0.31 a Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Slide L Moving direction b The key idea of crosswalk detection Pedestrian Recognition by Single Camera for Driver Assistance D Crosswalk in the real world (3D) Assumption: The long sides of the crosswalk Range of cross ratio in Japan: are almost parallel to the vehicle moving direction [ ABCD ] = [abcd ] C c d ｌ Crosswalk in the camera image (2D) Flow Diagram for Crosswalk Detection Image sequence Binarization Edge extraction “AND” operation Two kinds of vertical edge lines extraction Calculation of cross ratio Verification of previous frames Crosswalk detection To reduce false positive Two kinds: Dark Bright Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 10 Bright Dark Validation of Crosswalk Detection Algorithm ▶Offline validation of the crosswalk detection in real-world traffic ▶There are 55 crosswalks on the specified driving route ▶The number of detected crosswalks, false detected crosswalks, and undetected crosswalks of trips under different weather conditions ▶Image resolution ▶480×372 pixels ▶PC ▶CPU: Celeron 3.06GHz ▶Memory: 1GB Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 18 Experimental Results of Crosswalk Detection Trip Trip Trip Trip Sum Weather #Detection Cloudy 54 Cloudy 54 Sunny 51 Rainy 54 213 #Undetection #False detection 1 13 Computational time: 30ms Examples of detected crosswalks Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 19 Example of undetected crosswalk Experimental validation of the pedestrian recognition algorithm ▶Pedestrian is walking on an artificial crosswalk perpendicular to the vehicle direction Pedestrian crosswalk ▶Image resolution ▶640×480 pixels ▶PC ▶CPU Celeron 3.06GHz ▶Memory 1GB The training results based on the Open Source Computer Vision Library* was used 20 km/h Vehicle *Reference : http://www.intel.com/technology/computing/opencv/index.htm Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 20 Experimental Results Computational time: 150ms Examples of the recognized pedestrian near the crosswalk ▶ The pedestrian detectable range is about 25 m ▶ Large number of false recognized pedestrian ▶ Training result based on database learning by the OpenCV contains many frontal images of pedestrians Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 21 Concluding Remarks This paper describes a pedestrian recognition method based on single visible camera image information ▶ The existence of crosswalk is detected based on the cross ratio of the real world crosswalk ▶ Pedestrian is recognized by using Haar-like features and the cascade of AdaBoost classifier ▶ The preliminary experiments show the effectiveness of the proposed algorithm Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 22 Future plans Crosswalk detection algorithm: ▶ Tuning parameters to enhance the robustness under various conditions ▶ Using information from other sensors (speed, steering angle, etc.) Pedestrian classification algorithm: ▶ Learning the database including the side images of pedestrians for more robust classification System improvement with other modules: ▶ Extraction of ROI (Region of Interests) to reduce the false detections ▶ Optical flow (moving object detection by camera image) ▶ Sensor fusion with millimeter wave radar (high accuracy of moving object detection) Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 23 An Example of Sensor Fusion ▶ Preliminary results of real-time pedestrian detection by the proposed method ▶ Detail will be presented in AAET2009 Braunschweig, Germany Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 24 Thank you for your kind attention Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 25 Appendix Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 26 Motivation ▶Statistics data about traffic accidents in Japan 2007 in Japan Pedestrian Pedestrians 35.1% 33.9% Traffic accident fatalities 5,732 9.8% 13.0% Bicyclists Small-scale motorcycle riders Motorcycle riders Car drivers and passengers 8.2% ・ Pedestrians were killed in traffic accidents, accounting for more than 30% of all traffic-related death 　 Technical Solution for pedestrian protection is required Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 27 Conventional method for crosswalk detection ▶Points at issue Undetection and false detection ▶Solutions ▶Shadow of the building under the sunny weather ▶Corruption of white lines forming a crosswalk Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 28 ▶Extraction of low intensity edge ▶Binarization ▶The periodicity of a crosswalk Experimental Results of Crosswalk Detection Weather Detection Undetection False detection number number number A B Trip Cloudy 54 54 Trip Cloudy 54 54 Trip Sunny 51 52 Trip Rain 54 54 Sum 213 214 A 1 B 1 A 13 Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 29 B 22 A: Binarization and edge detection B: Edge detection Computational cost: 30ms Different between Single Camera and Sensor Fusion ▶Single camera ▶Advantage ▶ Camera can verify what the interested objects are ▶ Camera is cheaper than the other sensors ▶Disadvantage ▶ The distance to the interested objects can’t be measured with accuracy ▶ Computational cost is high for an amount of information obtained single camera ▶Sensor fusion (Image sensor and millimeter wave radar) ▶Radar can directly measure the distance to the interested objects with high precision ▶Radar information can be used for determining ROI in image processing Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 30 Overview of Pedestrian Recognition Training Sets Features extraction Pattern Learning Images Pedestrian features Test sets ROI Features extraction Pedestrian Classifier Nonpedestrian Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance 　 Classification Slide 31 Method for Determining ROI (Region Of Interest) ▶Recognition accuracy and computational cost change by extraction accuracy of ROI ▶Single camera ▶Optical flow, Template matching – High computational cost Features extraction Pattern Learning Pedestrian features ROI Features extraction Classifier Pedestrian Nonpedestrian ▶Sensor fusion (Camera + Millimeter wave radar) ▶Kalman filter embedded in the millimeter wave radar provided by Continental corporation – Low computational cost Dr Pongsathorn Raksincharoensak Tokyo University of Agriculture and Technology Pedestrian Recognition by Single Camera for Driver Assistance Slide 32