{GplllGpGjGG pGpGOpukpuGYWW_P kjjSGkSGrGq GXZTX]SGYWW_ A Real-time Finite Line Detection System Based on FPGA Dongkyun Kim, Seung Hun Jin, Nguyen Tuong Thuy, Ki Hoon Kim, and Jae Wook Jeon School of Information and Communication Engineering, Sungkyunkwan University, Suwon, Korea {bluster, coredev, ntthuy}@ece.skku.ac.kr, dkkh1126@gmail.com, jwjeon@yurim.skku.ac.kr Abstract-Image processing and analysis are active research topics An intelligent vehicle and a service robot require these techniques In particular, there is a big demand for line detection because it has a wide range of applications The line features in an image are used for object identification, robot navigation, and intelligent vehicle control To detect the lines, a Hough transform is generally used The Hough transform has good detection results and it is robust to noise, but it takes a long time to execute and it requires a great deal of memory to store the parameter space This paper proposes a dedicated line detection hardware system To increase the processing speed, it has a parallel Hough transform unit, and it partitions the parameter space to decrease the memory requirements It can detect not only the line parameters, but also the exact start and end points of each line, and it sorts these lines by length It can display the detected line on a monitor via the DVI interface This system is designed with VHDL and implemented on an XC4VLX200 FPGA The device usage is about 15% and the maximum clock frequency is 67MHz It can detect up to 256 lines in one image frame and it can process up to 149 frames per second The simulation and real experimental results are given to verify the system performance I INTRODUCTION Research continues in image analysis to identify the contents and to understand an environment in an image In understanding the contents of image, line features are important information The line feature is widely used in industrial applications like image analysis [1], intelligent robots [2], intelligent vehicles [3], and 3D reconstruction [4] The Hough transform [5] is a standard method to find the line features in an image The Hough transform is robust to noise and changes in the illumination level, but it requires a great deal of memory and execution time It cannot satisfy the realtime constraint, so it is difficult to apply in products that require real-time performance Many studies have suggested methods of solving the problems associated with execution time and memory space when detecting lines Some studies tried to improve the line detection algorithm in a general PC environment and others concentrated on the development of dedicated hardware to detect lines The first group tried to improve the accuracy; the other group tried to improve the speed Of course, the first group also tried to improve the processing speed for line detection, but the improvement was poor Kiryati et al [6] suggested a probabilistic Hough transform to find the exact start and end points of a line However, this complex approach `^_TXT[Y[[TYX^XT_VW_VKY\UWWG YWW_Gplll still requires substantial execution time and memory, so it cannot be applied to a real product that needs a rapid response Many dedicated hardware systems have been proposed to achieve real-time line detection Tagzout et al [7] suggested an incremental Hough transform to reduce the computation cost It can replace multiplication by addition, but it needs more time to process and its accuracy is less than that of standard Hough transform Ming-Yang and Yi-Hsiang [8] presented a parallel Hough transform circuit It can process up to 166 frames per second of a 512x512 image, but it can find only line parameters Nagata and Maruyama [9] presented a line detection circuit to find the start and end points of a line However, it cannot identify multiple lines that have the same line parameters, so it detects only one line for each parameter and it is not accurate We present a finite line detection system to increase the speed and detection accuracy and to decrease the memory requirement It has a parallel Hough transform unit, a partitioned parameter space, and a parameter to edge image mapping mechanism To detect a line feature, a series of image processing techniques are used First, it extracts an edge image from a raw image Second, it extracts the line parameters theta and rho, which are based on the origin point in an image Third, it must find some parameters associated with features that are prominent and conspicuous Finally, it needs check up procedure to find the exact start and end point of each line feature It can detect multiple lines that have the same line parameters Moreover, it can sort the detected lines based on their lengths The system is designed using VHDL and implemented in an FPGA The system input consists of the image sequence from a progressive camera and the line detection results are displayed on a monitor The system has a line table that has aligned line information We validate this internal data using the ChipScope system Experiments are conducted in which the system analyzes various scenes and environments This paper is organized as follow Ch.2 explains the two major problems in line detection Ch.3 shows the details of the OLQH GHWHFWLRQ V\VWHP¶V FRQILJXUDWLRQ DQG explains how it solves the line detection problem Ch.4 presents the results of the simulation and the real environmental experiment Ch.5 is the conclusion ]\\ II THE PROBLEM OF LINE DETECTION B The Problem of Line Identification A Hough transform and peak detection can detect some line The line feature in an image is the oriented and connected parameters that are associated with features that are prominent pixel groups that are located at the boundary of two different and conspicuous For these lines, it can determine the theta and intensity regions This line information plays an important role rho values With this information, we can determine the length in most applications that use of image analysis, but the line and orientation of a line based on its image origin and the detection is not easy because of two major problems One is number of occupied edge points, but this solution is suitable the time and space complexity of the Hough transform, and the for various general applications Some applications need the other is how to find the exact start and end points of a line In exact start and end points location of line The line with a start this chapter, we explain these problems and end point location is a finite line and a line with only line A The Hough Transform and its Limitations parameter is an infinite line Fig.1 explains the difference The Hough transform has a high time and space complexity between the finite and infinite line cases In Fig.1(a), a yellow [10], and these two problems are tightly related The time line is considered to be one long red line, but in Fig.1(b), the complexity comes from the characteristic of requiring repeated line is considered to be three short red lines In the infinite line operations for a whole edge point Equation (1) is the line case, it cannot distinguish multiple lines that have same equation (in polar form) that is used for the Hough transform parameters However, in a finite line, it can distinguish each It needs two multiplications and one addition Moreover, this line The finite line case is more accurate and useful An equation is repeated for the whole theta resolution for each additional procedure is required to improve the interpretation edge point In general, a 9*$ UHVROXWLRQ LPDJH¶V YDOLG HGJH from the infinite to the finite line case A mapping operation point percentage is as high a 10%, and the rho and theta that used the line parameters and the edge image is needed resolutions are 800 and 315 In this case, the total number of This mapping operation requires a great deal of time and multiplications is 640*480*0.1(theta resolution)*2 and the memory It also needs memory space for storing one edge total number of additions is 640*480*0.1(theta resolution) image and searching operation that extracts the exact start and end points of a line from this edge image and the line U x cos T y sin T (1) parameter To solve this problem, Kiryati et al [6] suggested a probabilistic Hough transform algorithm It does not need an A more serious problem is the space complexity The additional mapping procedure; it combines a Hough transform memory size for storing the parameter space to vote is decided and edge-parameter mapping Unfortunately, this requires a by rho, theta, and the image resolution The rho and theta random operation and a large parameter space, so it is not resolution affects the height and width of the parameter space efficient to implement in hardware Nagata and Maruyama [9] The image resolution affects the width of the accumulated cells presented a line detection circuit that can find the start and end in the parameter space In a VGA resolution image, the points of a line However, this circuit cannot distinguish maximum line length is 1024, so the width of accumulated cell multiple lines that have the same line parameters, and it is size is 10bits The height and width of the parameter space is overly influenced by edge points that are not part of a located 315 and 800 respectively, so the total parameter size is line The result from detecting the start and end points of a line 315*800*10 bits This is the optimal size for line detection of a is not accurate We present a new finite line detection system It can identify VGA resolution image The problem comes not only from memory space, but also from the speed of memory operations multiple lines that have the same line parameters The system Indeed, even if the line equation can be processed faster, the has new mechanism for line parameter to edge image mapping memory operation speed cannot be faster The memory By using this method, the system can detect the exact start and operations for (1) consist of one read and one write Equation end points of a line The detailed explanation of this line (1) calculates the rho from the given x, y, and theta and the identification is given in the next chapter accumulation procedure is executed using the calculated rho value Some cell values in the parameter space are read and increased by one, so the next calculated value from (1) must wait until this accumulating operation is finished This is the bottleneck of the Hough transform, and the speed of a Hough transform depends on the speed of the memory operations We present a parallel Hough transform to reduce the execution time and we partition the parameter space to reduce the memory requirements The detailed description of the Hough transform hardware architecture appears in Ch3 (a) (b) Fig.1 The difference of infinite and finite line case ]\] edge calculator line equation calculator pixel position edge position list edge position data edge position estimator clock manager coordinate counter Hough Transform Unit voting mermory Line Identification Unit peak parameters LVDS transmitter raw image Camera Edge Extraction Unit edge image Camera Input Unit line identifier uses the camera clock, the vertical sync, the horizontal sync, and the data valid signal to count the pixel coordinate position If the data valid sLJQDO LV µ¶ then the horizontal position is increased by one at every rising edge of the clock, so the vertical position is increased by one at the horizontal sync The horizontal and vertical position is initialized to at the vertical sync Therefore, the top-OHIW SL[HO¶V SRVLWLRQ LV DQG WKH bottom-ULJKW SL[HO¶V SRVLWLRQ LV 7KLV SL[HO coordinate data is transferred to the edge extraction unit inverse line equation calculator peak detector line table peak table memory controller clock&sync signals D Edge Extraction Part The edge extraction unit is configured with three modules One is the ³edge calculator´, the second is the ³edge position estimator´, and the last is the ³edge position list´ Fig.2 The configuration of line detection system III HARDWARE ARCHITECTURE OF THE LINE DETECTION SYSTEM The hardware architecture of the line detection system is implemented in an FPGA and the design of the system is explained in this chapter The line detection system consists of an FPGA, a memory IC, and a signal interface IC Fig.2 shows the conceptual view of the line detection system This system is divided into four units These parts are the ³&DPHUD ,QSXW Unit´ the ³(GJH ([WUDFWLRQ Unit´ the ³+RXJK 7UDQVIRUP Unit´ DQG the ³/LQH ,GHQWLILFDWLRQ Unit´ :H SUHVHQW HDFK PRGXOH¶VLQSXWDQGRXWSXWVHTXHQFH and internal operation 1) Edge Calculator The edge calculator extracts an edge image from the raw image using real-time window processing [11] The raw image pixel and its position are transferred from the camera input unit The edge calculator can store the local region of a raw image using the window buffer, but the edge calculator does not use a frame buffer By using the canny algorithm, the edge calculator determines whether or not the center pixel of the window buffer is an edge point If this pixel is an edge, then the edge calculator outputs a àả ,I WKLV SL[HO LV QRW an edge point, the C Camera Input Part The Camera Input Unit is organized as the ³/9'6 edge calculator outputs a àả, so the edge calculator converts a 640*480*8 bit raw image to a 640*480*1 bit edge image, WUDQVPLWWHU´ WKH ³FORFN PDQDJHU´ and the ³coordinate which is binary image This edge image is transferred to the FRXQWHU´ edge position estimator and the memory controller of the line 1) LVDS Transmitter identification unit The camera outputs a 4bit LVDS data signal and a 1bit 2) Edge Position Estimator LVDS clock signal The LVDS transmitter converts the 4bit The edge position estimator makes an edge position list from LVDS to a 28bit TTL signal based on a 24.545MHz camera the edge image The edge image is not suitable for calculating clock In the 28bit TTL signal, the 24 bit signal represents the using line equation because the line equation needs the x and y red, green, and blue pixel data; each color pixel is allocated coordinates, but the edge image is just binary information The 8bits The others represent the vertical sync, the horizontal edge position estimator checks the result of the edge calculator sync, the data valid signal, and the reserved bit The color If this pixel is determined to be an edge by the edge calculator, information is not needed for line extraction A green pixel is then WKH HGJH SRVLWLRQ HVWLPDWRU VWRUHV WKLV SL[HO¶V SRVLWLRQ in used only for edge extraction The control and clock signals are the edge position list At the end of the edge image, the edge communicated to every unit, so each unit synchronizes on position estimator writes the ending data to the edge position these control and clock signals list to indicate the end of the list In the ending data, all of the 2) Clock Manager bits are àả, which is not valid a position, so the system can The clock manager increases the clock speed by factors of distinguish between the ending data and the valid positions and using the DCM core in the FPGA, so this line extraction 3) Edge Position List system has three clock signals; 24.545MHz, 49MHz, and The edge position list stores the position information of 98MHz The higher frequency clocks are used to increase the every edge point in one edge image The edge position list processing speed of the Hough transform unit and the line actually maintains two similar lists to prevent conflicts during identification unit because the Hough transform unit and the read and write operations It is impossible to perform read and line identification unit can operate independently of the camera write operations simultaneously One list is for reading; the operating speed The operation of the camera input unit and the other is for writing By using this double buffering method, the edge extraction unit depends on the camera operating speed, so reduction of the frame processing rate is prevented Moreover, these units use only the original camera clock the Hough transform unit can operate independently of the 3) Coordinate Counter camera clock, so the Hough transform can be processed at a The position counter tracks the position of the pixel that is higher speed by using a higher frequency clock input from the camera at the moment The coordinate counter ]\^ E Hough Transform Unit The Hough transform unit is configured as four modules One is the ³line equation calculator´ that calculates the rho value using the line equation The second is the ³voting memory´ that makes the partitioned parameter space using the rho value The third is the ³peak detector´ that finds the peak position in the partitioned parameter space The fourth is the ³peak table´ to store the theta and rho values of the detected peak The last is the ³Hough transform controller´ that controls these four modules The peak table transfers information to the line identification unit to identify the exact line position Fig.3 presents the Hough transform unit The Hough transform unit contains 15 line equation calculator and voting memory pairs, so it can operate 15 times faster The line equation calculator and voting memory operate in parallel to calculate 15 values of theta simultaneously, so this operation must be repeated 21 times to calculate the 315 theta values 1) Line Equation Calculator The line equation calculator operates according to (1) It has a sine table, a cosine table, two multipliers, and one adder The sine and cosine table are constructed using a read only memory (ROM) These whole sub-modules are configured as pipelined structure After defined pipeline latency, a final result consisting of the rho value is produced at every clock cycle The total pipeline latency that is involved in converting the (x, y, theta) information into the rho value is clock cycles 2) Voting Memory The voting memory is configured in one block memory of the FPGA The voting memory has two operation modes One is increase mode and the other is clear mode This voting memory accumulates the input rho values in increase mode First, it reads the value of a memory cell in the voting memory that is indicated by the rho value, aQGWKLVPHPRU\FHOO¶VYDOXH is increased and updated In clear mode, the contents of voting memory are transfer to the peak detector, and then whole data is cleared The address of the voting memory is sequentially increased in clear mode Peak detector voting memory 15 n max in columns max in rows 15to1 comparator R2 R15 Fig.4 The configuration of the peak detector 4) Peak Table The peak table stores the line parameters of the detected peak point in the partitioned parameter space The contents of the peak table are transferred to the line identification unit to identify the exact line position F Line Identification Unit The line identification part identifies the exact line position and their peak line parameters in an edge image The line identification unit is configured with four modules The first is the ³inverse line equation calculator´ to find the position of an image domain using the line parameters The second is the ³line identifier´ which checks the existence of a line in an edge image with the position value of the inverse line equation calculator The third is the ³line table´ to store the identified OLQH¶VVWDUWDQGHQGSRVLWLRQ0RUHRYHUWKLVOLQH¶VLQIRUPDWLon is sorted by line length The last is the ³memory controller´ which stores the edge image and transfers the edge pixels to the line identifier 1) Inverse Line Equation Calculator The inverse line equation calculator operates using (2) for i 1: 640 or 480 if 0.79 d T d 2.35 then U i cos T y sin T else U i sin T x cos T end if end for theta, x, y voting memory 15to1 comparator delay buffer Peak Detector The peak detector detects the peak point in the voting memory, which is partitioned in the parameter space The partitioned parameter space size is 15*800 Fig.4 shows the configuration of the peak detector The peak detector finds local maxima in 15*15 parameter space, and then compares this max value and a predefined threshold value ber 1to1 comparator with treshold R1 3) line equation calculator 15 num peak (2) peak detector umb er peak table Fig.3 The configuration of the Hough transform unit Using this equation, it is possible to know the line position by using the line parameters which are rho and theta The index of (2) is varies and it is a function of theta,IWKHWD¶VUHJLRQLV 0.79~2.35, the x coordinate is used as an index and (2) calculates the corresponding y position In this case, the range ]\_ 2) Line Identifier The line identifier identifies the line using the line parameters and the edge image The line identifier receives the position data of the line parameters from the inverse line equation calculator, and it UHTXHVWV WKLV SRVLWLRQ¶V HGJH GDWD from the memory controller The edge image in the memory controller is a 640*480*8 bit image If the pixel is an edge, then the value of the SL[HOLV³´otherwise the value of the SL[HO LV ³´ 7KH OLQH LGHQWLILHU PDNHs a connected list of line positions, as shown in Fig.5 In Fig.5, the connection list represents the connectivity of the edges that are based on the line parameters The connection list has 15 cells If the edge is located by using these line parameters (as in Fig.5(a)), then the connection list has an entry ³´ If no edge is located using the line parameters (as in Fig.5(b)), then the connection list has an entry ³´ 8VLQJ WKLV FRQQHFWLRQ OLVW LW FDQ determine the start and end points of line If the connection list is ³;´, then this is the start point of the line (as in Fig.5(c)) If the connection list is ³;´, then this is the end point of the line (as in Fig.5(d)) The position of the start and end points of the line is stored in the line table (a) 111111111111111 (b) 000000000000000 Start End Length Start rt n io ct so sele copy with End Length Fig.6 The line table 3) Line Table The line table stores the start and end positions of each identified line Moreover, the line table sorts the lines by length The line table is configured as two tables that have the same size, as shown in Fig.6 The position data is stored in the left list and this data is copied to the right table using a sorting operation The data of the left list is sorted according to its theta and rho A selection sorting is used to sort the line data of the line table It finds the maximum line length data, copies this data to the right list, and clears this data from the left list These operations are repeated until the left list is empty After sorting, the left list is empty, and the right list has line data that is sorted by length 4) Memory Controller The memory controller stores the edge image and transfers WKHHGJHSL[HOLQWKLVHGJHLPDJHZLWKWKHLGHQWLILHU¶VUHTXHVW The memory controller controls two FIFO memories and two SRAMs The FIFO is the frame buffer to separate the operating speed of the line identification unit from the camera clock speed By using the FIFO memory, the line identification unit becomes independent of the camera clock, so the identification unit can operate faster than the camera clock speed The SRAM stores the edge image The SRAM read and write simultaneously, so the memory controller has two SRAMs and switches between them when performing read and write operation Fig.7 shows the configuration of the memory controller IV EXPERIMENT An experiment using the proposed system is conducted Fig.8 shows the real experimental system The real-time finite line extraction system is implemented on a FPGA and designed VHDL This system receives a 24bit and VGA camera image at a rate of 60 fps The system can display processing result on edge image SRAM odd front FIFO back FIFO line image SRAM even memory controller (c) 0000000X1111111 256 items 256 items of the index value is from to 640 Alternatively, the y coordinate can be used an index of (2), in which cased the UDQJH RI WKH LQGH[ YDOXH¶V LV IURP WR The inverse line equation produces the (x,y) position of the line parameters This position data is transferred to the line identifier and memory controller to find the exact line position The line parameter comes from the peak table of the Hough transform unit If the index reaches the maximum value (either 480 or 640), the inverse line equation calculator reads the next line parameter in the peak table The inverse line equation calculator repeats these operations until no more parameters remain in the peak table Inverse line equation calculator (d) 1111111X0000000 Fig.5 The connected list of line position ]\` line position edge data data Fig.7 The memory controller line Identifier reconstruction, and intelligent vehicle Our system can also be used in these fields It can help in the creation of low-cost, lower-power, and high-speed applications Fig.8 The implemented system the monitor same at a rate of 60 fps Fig shows the experimental result of this system These images are a snapshot of operating DVI monitor Fig.9(a) is a raw image Fig.9(b) is a edge image Fig.9(c) is a infinite line image on the edge image Fig.9(d) is a infinite line image on the raw image Fig.9(e) is a finite line image on the edge image Fig.9(f) is a finite line image on the raw image Fig.10 is a line table and a constructed image using the contents of line table The system of the real-time finite line detection is designed using VHDL and is implemented in a Virtex-4 XC4VLX200 FPGA, with 200,448 Logic Cells(about 20M system gates), 6,048 Kbit Block RAM, 960 user I/O pins [12] This system takes an image from the camera and extracts an edge image from the raw image The system transforms from the edge image to a parameter space and finds peak line parameters The system finds an exact start and end position of line using the parameter to edge image mapping The system sorts line information by length and store internal block memory TABLE 1, shows the summary of the real-time finite line detection system design The clock frequency is 24.54MHz In this clock, our circuit can process at 60 fps The maximum frequency is 61MHz Then it can process at 149 fps Slice Flip Flops Input LUTs Occupied Slices Block RAMs DSP48s Global Clocks TABLE I THE DESIGN SUMMARY Used Available 17,713 178,176 13.517 178,176 13,793 89,088 246 336 60 96 32 Equivalent Gate Count Utilization 9% 7% 15% 73% 62% 21% 16,452,974 V CONCLUSION We present a real-time finite line detection system To increase line detecting speed, the system uses the parallel Hough transform and the partitioned parameter space To identify finite line, the system uses the line parameter to edge image mapping method This system can detect an exact start and end position of line The detected line information is sorted by length and stored in the line table This system can process images with VGA resolution at speeds of up to 149 fps The line information can be used in image analysis, 3D Fig.9 The experimental result of the finite line detection system REFERENCES Nagy G ³Twenty years of document image analysis in PAMI´ IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 22, pp 38-62, Jan 2000 [2] Kahn P, Kitchen L, Riseman E.M, ³$ IDVW OLQH ILQGHU IRU YLVLRQ-guided robot naviJDWLRQ´IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 12, pp 1098-1102, Nov 1990 [3] 4LQJ/1DQQLQJ=³6SULQJURERW $SURWRW\SHDXWRQRPRXVYHKLFOHDQG LWVDOJRULWKPVIRUODQHGHWHFWLRQ´IEEE Transactions on Pattern Analysis and Machine Intelligence, vol 5, pp 300-308, Dec 2004 [4] C Baillard, C Schmid, A Zisserman, A Fitzgibbon, ³$XWRPDWLF OLQH PDWFKLQJ DQG G UHFRQVWUXFWLRQ RI EXLOGLQJV IURP PXOWLSOH YLHZV´ Proceeding of ISPRS Conference on Automatic Extraction of GIS Objects from Digital Imagery, IAPRS vol 32, Sep 1999 [5] J Illingworth, J Kittler, ³A survey of the Hough transform´Journal of Computer Vision, Graphics, and Image Processing, vol 44, pp 87-116, 1988 [6] N Kiryati, Y Eldar, A M Bruckstein, ³A probabilistic hough transform´Journal of Pattern Recognition, vol 24, pp 303-316, 1991 [7] S Tagzout, K Achour, U Djekoune, ³Hough transform algorithm for FPGA implementation´Journal of Signal Processing, vol 81, pp 12951301, Jun 2001 [8] C Ming-Yang, L Yi-Hsiang, ³Desing and integration of parallel houghtransform chips for high-speed line detection´ Proceeding of the 2005 11th International Conference on Parallel and Distributed Systems, 2005 [9] N Nagata, T Maruyama, ³Real-time detection of line segments using the line hough (a) transform´ Proceeding of the 2004 (b)IEEE International Conference on Field-Programmable Technology, pp 89-96, 2004 [10] M G Albanesi, M Ferretti, D Rizzo, ³Benchimarking Hough transform architectures for real-time´ Journal of Real-Time Imaging, vol 6, pp 155-172, 2000 [11] C Torres-Huitzil, M Arias-Estrada, ³FPGA-based configurable systolic architecture of window-based image processing´ EURASIP Journal on Applied Signal Processing, Vol 2005, Issue 1, January 2005 [12] Xilinx Inc, ³Virtex-4 FPGA family data sheet´ available from www.xilinx.com [1] ]]W (c) (d) ... is a infinite line image on the edge image Fig.9(d) is a infinite line image on the raw image Fig.9(e) is a finite line image on the edge image Fig.9(f) is a finite line image on the raw image... information plays an important role rho values With this information, we can determine the length in most applications that use of image analysis, but the line and orientation of a line based on. .. with a start this chapter, we explain these problems and end point location is a finite line and a line with only line A The Hough Transform and its Limitations parameter is an infinite line