Implementation of sobel filter based on the zynq 7000 soc development board

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING CAPSTONE PROJECT ELECTRONICS AND TELECOMMUNICATIONS ENGINEERING TECHNOLOGY IMPLEMENTATION OF SOBEL FILTER BASED ON THE ZYNQ-7000 SOC DEVELOPMENT BOARD LECTURER: MSc TRUONG QUANG PHUC STUDENT: NGUYEN HOANG NHAT UYEN SKL 0 Ho Chi Minh City, 2022 HCMC UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH-QUALITY TRAINING 🙞🙡🕮🙣🙜 GRADUATION PROJECT IMPLEMENTATION OF SOBEL FILTER BASED ON THE ZYNQ-7000 SOC DEVELOPMENT BOARD Major: ELECTRONICS AND TELECOMMUNICATIONS ENGINEERING TECHNOLOGY Student Name: NGUYEN HOANG NHAT UYEN…18142241 Advisor: MSc TRUONG QUANG PHUC Ho Chi Minh City, July 2022 THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom – Happiness *** -Ho Chi Minh City, mm dd, 2022 GRADUATION PROJECT ASSIGNMENT Student name: Nguyen Hoang Nhat Uyen Student ID: 18142241 Major: Electronics and Telecommunications Engineering Technology Class: 18161CLCVT2A Advisor: Truong Quang Phuc, MSc Phone number: 0917731988 Date of assignment: 07/04/2022 Date of submission: 30/07/2022 Project title: Implementation of Sobel Filter based on the ZYNQ-7000 SoC Development Board Initial materials provided by the advisor: Sobel Edge-based Image Template Matching in FPGA, ZYNQ All Programmable SoC Sobel Filter Implementation Using the Vivado HLS Tool Contents of the project: - Research for image processing techniques, particularly edge detection filters such as Prewitt, and Sobel,… - Execute Sobel filter simulation and package IP core Sobel in Vivado HLS - Create Sobel filter block design in Vivado - Research how to connect and implement the Sobel filter on ZYNQ - Valuate result indicators - Write the final report Final product: a Sobel Filter model on the ZYNQ platform CHAIR OF THE PROGRAM (Sign with full name) ADVISOR (Sign with full name) i THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom – Happiness *** -Ho Chi Minh City, mm dd, 2022 ADVISOR’S EVALUATION SHEET Student name: Nguyen Hoang Nhat Uyen Student ID: 18142241 Major: Electronics and Telecommunications Engineering Technology Class: 18161CLCVT2A Project title: Implementation of Sobel Filter based on the ZYNQ-7000 SoC Development Board Advisor: Truong Quang Phuc, MSc EVALUATION Contents of the project: Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark: .(in words: .) Ho Chi Minh City, 30th July 2022 ADVISOR (Sign with full name) ii THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom – Happiness *** -Ho Chi Minh City, June 27, 2022 PRE-DEFENSE EVALUATION SHEET Student name: Nguyen Hoang Nhat Uyen Student ID: 18142241 Major: Electronics and Telecommunications Engineering Technology Class: 18161CLCVT2A Project title: Implementation of Sobel Filter based on the ZYNQ-7000 SoC Development Board Name of Reviewer: EVALUATION Contents and workloads of the project: Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark: (in words: .) Ho Chi Minh City, 30th July 2022 REVIEWER (Sign with full name) iii THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom – Happiness *** -Ho Chi Minh City, June 27, 2022 EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Nguyen Hoang Nhat Uyen Student ID: 18142241 Major: Electronics and Telecommunications Engineering Technology Class: 18161CLCVT2A Project title: Implementation of Sobel Filter based on the ZYNQ-7000 SoC Development Board Name of Defense Committee Member: EVALUATION Contents and workloads of the project: Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark: (in words: .) Ho Chi Minh City, 30th July 2022 COMMITTEE MEMBER (Sign with full name) iv DISCLAIMER Project Title: “IMPLEMENTATION OF SOBEL FILTER BASED ON THE ZYNQ7000 SOC DEVELOPMENT BOARD” Instructor: Truong Quang Phuc, MSc Student's first & last name: Nguyen Hoang Nhat Uyen Student ID: 18142241 Contact phone number: 0933065297 Email: 18142241@student.hcmute.edu.vn Promises: I hereby declare that the topic "IMPLEMENTATION OF SOBEL FILTER BASED ON THE ZYNQ-7000 SOC DEVELOPMENT BOARD" is my research work under the guidance of MSc Truong Quang Phuc The results published in the Graduation Project are honest and not copied from any other works The similarity rate is 11% Ho Chi Minh City, 30th July 2022 CHAIR OF THE PROGRAM (Sign with full name) v ACKNOWLEDGEMENTS First of all, I would like to express my appreciation to my advisor Truong Quang Phuc for his invaluable patience Throughout this project, he always helps and spends time giving me useful guidance I believe that I could not have undertaken everything without my advisor I am also deeply indebted to my teachers Nguyen Ngo Lam and Tran Tung Giang for their advice and encourage me to carry out this project Moreover, many thanks to my friends, who are willing to support me anytime CHAIR OF THE PROGRAM vi Table of Contents GRADUATION PROJECT ASSIGNMENT i ADVISOR’S EVALUATION SHEET ii PRE-DEFENSE EVALUATION SHEET iii EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER iv DISCLAIMER v ACKNOWLEDGEMENTS vi Table of Contents vii List of Acronyms x List of Tables xii List of Figures xiii Abstract xv Chapter 1: INTRODUCTION 1.1 Overview 1.2 Related work 1.3 Objective 1.4 Work content 1.5 Outline Chapter LITERATURE REVIEW 2.1 An overview of image 2.1.1 Image definition 2.1.2 Color space 2.2 Image processing 2.2.1 Introduction 2.2.2 Conversion of image format 2.2.3 Convolution operation 2.2.4 Binarization 2.2.5 Edge detection algorithm 10 2.2.5.1 Background 10 vii 2.2.5.2 Sobel filter 11 2.2.6 Comparison parameter 12 2.2.6.1 Mean Square Error 12 2.2.6.2 Peak Signal to Noise Ratio 13 2.3 Developing toolchain 13 2.3.1 FPGA introduction 13 2.3.2 IP Core 14 2.3.3 Vivado and High-Level Synthesis 15 2.3.3.1 An introduction to High-Level Synthesis 15 2.3.3.2 Vivado application tool 16 2.3.3.3 Software Development Kit 16 2.3.3.4 Xilinx HLS Video Library 16 2.3.3.5 Open source library for image processing 17 2.3.4 Dataflow technique 17 2.3.5 Python tool 18 2.4 ZYNQ-7000 platform 18 2.4.1 Introduction to ZYNQ board 18 2.4.2 Advanced Extensible Interface protocol 21 Chapter SYSTEM DESIGN 24 3.1 Implementation of Sobel Filter based on software application 24 3.2 Implementation of Sobel Filter based on co-design platform 24 3.2.1 System requirement specification 24 3.2.2 System design process 24 3.2.2.1 Flowchart of the edge detection algorithm 26 3.2.2.2 Overview of PL and PS parts cooperation 26 3.2.2.3 Flowchart over the PL part of IP core Sobel 28 3.2.2.4 Flowchart of the IP core Sobel in Test Bench 28 3.2.3 The PL part configuration 29 3.2.3.1 Pre-processing block design 30 3.2.3.2 Central processing block design 40 viii power of the user design, instantaneous and varies at each clock cycle It depends on voltage levels and logic and routing resources used This also includes static current from I/O terminations, clock managers, and other circuits that need power when used Total onchip power is the power consumed internally within the FPGA, equal to the sum of device static power and design power It is also known as thermal power (a) (b) Figure (a) Power summary (b) Power On-chip 4.2.2 Throughput value and Hardware utilization In Vivado HLS, the edge detection system uses a clock period of 13.5 ns Figure 4.3 shows a summary table with the estimated clock period of 12.49 ns It seems the timing is not really met the requirement with a margin of 1.01 ns (the required smallest margin is 1.69 ns), fortunately, the actual timing shown in Figure 4.5 is met The second table mentioned the latency and interval, we can see that the maximum throughput latency is 928503 clock cycles and it can start to process new input data after 928498 clock cycles This results in a throughput of ~80 frames/s (1/(928503 cycles/frame * 13.5 ns/cycle)) In Vivado HLS, we need to set up the target clock period for the IP core and it depends on the frequency of the system The period of 10 corresponds to the 100MHz value, the resolution of my design is 1280p so it needs a frequency and if we don't specify the uncertainly clock, it will be at 27% of the default clock period of 10ns so I calculate and add it in the target so the target is 13.5ns In addition, the latency is known as the number of cycles required for the system to accept next input For example if you have a design that can accept a new input on every clock cycle, but takes 10 cycles to propagate from input to output then we call latency as 10 HLS Analysis Tool knows when a design "starts" and "finishes" - it starts when we call the main function, and it finishes when the main function returns and the latency is the time between those two We can it manually in simulation: latency is defined (in HLS) as the time between two ap_ready signals All things we should is simulate the block, record when ap_ready goes low (because the 45 block has started), record when ap_ready goes high again (is ready to start on the next dataset0, and subtract one from the other, the result is latency Figure Measurements of timing and latency in the synthesis report In Figure 4.4, it can be found that the hardware utilization is meet the needed usage volume of the IP core Sobel, because the usage of BRAM, DSP48E, FF, and LUT is very low within the available For more detail, the below table shows how the hardware resources were divided into different functions in HLS The Sobel function obviously used the most resources Figure 4 The estimated hardware utilization of IP core Sobel 46 The report obtained from the RTL exportation gives the actual results of IP core Sobel implementation A larger timing margin was achieved of 3.53 ns, instead of 1.47 ns in the synthesis estimation In addition, the utilization of BRAM, FF, and LUT is significantly lower than what was estimated We can see that in Figure 4.5 below Figure The RTL exportation of IP core Sobel Figure 4.6 illustrates the ordering of performing functions, which can be observed by using the analysis tool in HLS Performance Figure The ordering of performing functions by HLS Performance 4.3 Output result and comparison The input and output image of the Sobel Filter implemented on the ZYNQ co-design platform is shown in Figure 4.7 Notice that this is the result from the HLS Test Bench Besides, Figure 4.8 shows the output when implementing the Sobel Filter on OpenCV – Python 47 Figure Result of edge detection algorithm on ZynQ-7000 platform Figure Result of edge detection algorithm on OpenCV – Python The result parameters that need to be compared will be shown in Table 4.1 below Platform OpenCV – Python ZynQ-7000 MSE 15234.953 2628.879 PSNR 6.30239 13.93310 Table Result parameters comparison As we can see, on the ZYNQ-7000 platform, the boundaries of the image are fetched more carefully and smoother than the OpenCV – Python, which saves a lot of capacity in the memory It interprets how the co-design has lower power consumption Additionally, the value of MSE is less than and PSNR is greater than OpenCV – Python, respectively 5.8 and 2.2 times It means the output image quality on the ZYNQ-7000 platform is better because we know that the lower the MSE and higher the PSNR are, the better the quality of the processed image is 4.4 An optimization of the throughput indicator Throughput in this thesis indicates the number of image frames passing through a process of the system To improve this indicator, it is found that we can increase or decrease the clock period value Two typical situations are carried out to check out the effect of the clock period on the throughput Firstly, the target clock is increased to 26 ns, as times the present, with this value the timing is also met The number of SLICEs is decreased by 5, the amount of LUTs is decreased by 6, along with the required FFs is less than 5.3% Despite that effective usage, the performance of the algorithm is worse than the 13.5 ns, 48 with a throughput of ~41 frames/s Secondly, the target clock is decreased to ns, the reason for this option indicates that ns is highly recommended by Xilinx’s designer for the minimum target value This solution resulted in an estimated clock period of 4.36 ns, a margin of 0.63 ns, and a clock period achieved post-implementation of 4.721 ns, which made the actual timing meet the requirement As well as that, it provides the most positive throughput of ~214 frames/s Unfortunately, this optimized choice does not save the hardware utilization as it expects more than 15.73% of SLICEs, 3.5 % of LUTs, and 30% of FFs All of the result indicators of the two solutions above are represented in detail in the Appendices section But in order to check out the overall performance of this solution, the graph in Figure 4.9 shows how the clock period changes the throughput value Figure Changing of throughput by the clock period 49 Chapter CONCLUSION AND FUTURE WORK 5.1 Conclusion In general, this thesis is regarded as providing quite coherent proof for the reason why implementing an image processing algorithm based on a co-design platform is better than a software platform According to all obtained result indicators of MSE and PSNR in the previous chapter, we can see that the quality of the output image on the Python software application is worse than on the ZYNQ co-design It is found that the implementation on co-design is memory saving, assume that this system is used as a subset of another larger system, there are several steps after detecting edge so the optimization in data storing is very important On the other side, the energy consumption of 1.936 W and the throughput of ~80 frames/s provide a power consumption of 0.024 J/frame This is a positive result and has the potential to gain maximum savings of hardware resources if some effective solutions are added to the design system Due to the complexity of the algorithm development, the adjustment of the target period clock is a considerable proposal for future work, as mentioned in section 5.2 However, it needs to take a step back, observe, and conclude the weaknesses of this thesis Firstly, the entire system has never been successfully implemented on the ZYNQ PS The Sobel design in Vivado and SDK occurs some issues with the VGA configuration and the result can not be displayed on the monitor This makes it impossible to check out the output image and the execution time Secondly, the Sobel design in Vivado is not really optimized since it has the Worst Pulse Width Slack of 0.185 ns (WPWS - a parameter that indicates the minimum width of highest and lowest pulses of a clock), fortunately, it seems to be within the required limitation The third drawback is the Sobel algorithm is not considered an innovation-driven operation for image processing, because there are other better algorithms such as Canny and LoG But the complexity of them is a hard challenge to the implementation 5.2 Future work In this section, we discuss different solutions to improve the edge detection system The first is the improvement of the IP core Sobel, as previously suggested in chapter 4, the adjustment of the target period clock is a considerable proposal Depending on the result indicators, we can see that increasing the clock period is not a good option, although it helps to save the utilization We know that when implementing an image processing algorithm based on a co-design system, the performance is the most concern Otherwise, the ns is wasting more resources, however, if the hardware of the co-design system is strong enough, this option supports achieving the significantly highest value of throughput The second solution is the integration of this system into a more advanced application For a simple example, we might consider combining this edge detection system with an 50 automatic insect detection model in [4], because it eliminates the drawbacks of only implementing the algorithm on Matlab and creates a real-time harmful insect retrieval system to improve agricultural yields In addition, the image edge detection system is also capable of becoming one of the main elements of traffic applications In particular, detecting traffic signs by combining edge detection and the template matching algorithms for the purpose of assisting people who are practicing driving or encountering an unfamiliar sign Furthermore, there are other applications in medicine, security, and the environment that need to use edge detection systems 51 REFERENCES [1] Nguyễn Thanh Hải, Giáo trình xử lý ảnh, Đại học Quốc Gia Thành phố Hồ Chí Minh, 2003 [2] Soma Prathap; Jatoth Ravi, "Hardware Implementation Issues on Image Processing Algorithms," National Institute of Technology Warangal, 2018 [3] Sagharichi Ha, Pooya; Shakeri, Mojtaba, "License Plate Automatic Recognition Based on Edge Detection," Faculty of Computer and IT Engineering, 2016 [4] Thenmozhi, K; Reddy U, Srinivasulu, "Image Processing Techniques for Insect Shape Detection in Field Crops," International Conference on Inventive Computing and Informatics, 2017 [5] Abbasi, Tanvir, "A Proposed FPGA Based Architecture for Sobel Edge Detection Operator," 2007 [6] Halder, Santanu; Hasnat, Abul; Khatun, Amina; Bhattacharjee, Debotosh; Nasipuri, Mita, "A Fast FPGA Based Architecture for Skin Region Detection," International Journal of Innovative Technology and Exploring Engineering, 2013 [7] Rafael C Gonzalez; Richard E Woods, Digital Image Processing, New York: Pearson, 2018 [8] Rang M H Nguyen; Michael S.Brown, "Why You Should Forget Luminance Conversion and Do Something Better," Computer Vision Foundation, IEEE Xplore, 2017 [9] Nguyễn Quang Hoan, Giáo trình xử lý ảnh, Học viện Cơng nghệ Bưu Viễn thông, 2006 [10] Sung Kim; Riley Casper, "Applications of Convolution in Image Processing with MATLAB," University of Washington, 2013 [11] Ramesh Jain; Rangachar Kasturi; Brian G Schunck, "Edge Detection," in MACHINE VISION, McGraw-Hill, 1995, pp 140-185 52 [12] Ansari, Mohd; Kurchaniya, Diksha; Dixit, Manish, "A Comprehensive Analysis of Image Edge Detection Techniques," International Journal of Multimedia and Ubiquitous Engineering, 2017 [13] Oskar Mencer; Dennis Allison; Elad Blatt; Mark Cummings; Michael J Flynn; Jerry Harris; Carl Hewitt; Quinn Jacobson; Maysam Lavasani; Mohsen Moazami; Hal Murray; Masoud Nikravesh; Andreas Nowatzyk; Mark Shand; Shahram Shirazi, "The History, Status, and Future of FPGAs: Hitting a nerve with fieldprogrammable gate arrays," 2020 [14] Ian Kuon; Russell Tessier; Jonathan Rose, "FPGA Architecture: Survey and Challenges," Foundations and Trends in Electronic Design Automation, 2008 [15] Xilinx, UltraScale Architecture DSP Slice, www.xilinx.com, 2021 [16] Rajewski, Justin, "How does an FPGA work?," Embedded Micro, 2015 [Online] Available: https://learn.sparkfun.com/tutorials/how-does-an-fpga-work/all [Accessed July 2022] [17] Eastland, Nate, "FPGA - Configurable Logic Block," Digilent Inc Blog, 2015 [Online] Available: https://digilent.com/blog/fpga-configurable-logic-block/ [Accessed July 2022] [18] M Rouse, "IP core (Intellectual Property core)," TechTarget Contributor, March 2011 [Online] Available: https://www.techtarget.com/whatis/definition/IP-coreintellectual-property-core [Accessed July 2022] [19] K Karras; J Hrica, "Designing protocol processing systems with vivado high level synthesis," 2014 [Online] Available: https://docs.xilinx.com/v/u/en-US/xapp1209designing-protocol-processing-systems-hls [Accessed July 2022] [20] Xilinx Inc., "Vivado HLS optimization methodology guide," 2017 [Online] Available: https://docs.xilinx.com/v/u/2017.4-English/ug1270-vivado-hls-optmethodology-guide [Accessed July 2022] [21] Xilinx Inc., "Accelerating OpenCV applications with ZYNQ-7000 all programmable SoC," 2015 [Online] Available: https://docs.xilinx.com/v/u/enUS/xapp1167 [Accessed July 2022] 53 [22] Arthur H Veen, "Dataflow Machine Architecture," Center for Mathematics and Computer Science, 1986 [23] Xilinx Inc., "Vivado Design Suite User Guide: High-Level Synthesis," 2017 [Online] Available: https://docs.xilinx.com/v/u/en-US/ug902-vivado-high-levelsynthesis [Accessed July 2022] [24] Digilent, ZYBO FPGA Board Reference Manual, 2017 [25] Xilinx Inc., "AXI Reference Guide," 2012 [Online] Available: https://docs.xilinx.com/v/u/en-US/ug761_axi_reference_guide [Accessed July 2022] [26] Digilent, "DVI to RGB (Sink) 2.0 IP Core User Guide," October 2019 [Online] Available: www.digilentinc.com [Accessed 12 July 2022] [27] Xilinx Inc., "Clocking Wizard v6.0 LogiCORE IP Product Guide," 20 April 2022 [Online] Available: www.xilinx.com [Accessed 12 July 2022] [28] Xilinx Inc., "LogiCORE IP Constant (v1.1)," April 2018 [Online] Available: www.xilinx.com [Accessed 12 July 2022] [29] Xilinx, "Video In to AXI4-Stream v4.0 LogiCore IP Product Guide," 18 November 2015 [Online] Available: www.xilinx.com [Accessed 23 June 2022] [30] Xilinx, "Video Timing Controller v6.2 LogiCORE IP Product Guide," 26 February 2021 [Online] Available: www.xilinx.com [Accessed 23 June 2022] [31] Xilinx Inc., "Processing System v5.5 Product Guide," 10 May 2017 [Online] Available: www.xilinx.com [Accessed 12 July 2022] 54 APPENDICES A Result indicators for IP core Sobel – a target clock period of 26 ns Synthesis Report for 'edge_detect' General Information Date: Wed Jul 20 21:27:22 2022 Version: 2016.4 (Build 1756540 on Mon Jan 23 19:31:01 MST 2017) Project: sobel_edge Solution: solution1 Product family: zynq Target device: xc7z010clg400-1 Performance Estimates • Timing (ns) Clock ap_clk Target Estimated Uncertainty 26.00 21.17 3.25 • Latency (clock cycles) Latency Interval max max 928502 928502 928498 928498 Type dataflow Utilization Estimates • Summary Name BRAM_18K DSP48E FF DSP Expression FIFO 80 Instance 933 Memory Multiplexer Register Total 1013 Available 120 80 35200 Utilization (%) LUT 334 1386 1721 17600 55 • Detail Instance Module BRAM_18K AXIvideo2Mat_U0 AXIvideo2Mat Block_proc_U0 Block_proc CvtColor_U0 CvtColor CvtColor_1_U0 CvtColor_1 Mat2AXIvideo_U0 Mat2AXIvideo Sobel_U0 Sobel Total DSP48E 0 0 FF 241 75 42 119 454 933 LUT 218 85 60 111 909 1386 Export Report for 'edge_detect' General Information Report date: Tue Jul 20 21:33:47 +0700 2021 Project: sobel_edge Solution: solution1 Device target: xc7z010clg400-1 Implementation tool: Xilinx Vivado v.2016.4 Resource Usage SLICE LUT FF DSP BRAM SRL Verilog 262 708 767 3 Final Timing CP required CP achieved post-synthesis CP achieved post-implementation Verilog 26.000 13.127 14.187 Timing met 56 B Result indicators for IP core Sobel – a target clock period of ns Synthesis Report for 'edge_detect' General Information Date: Thu Jul 21 20:05:25 2022 Version: 2016.4 (Build 1756540 on Mon Jan 23 19:31:01 MST 2017) Project: sobel_edge Solution: solution1 Product family: zynq Target device: xc7z010clg400-1 Performance Estimates • Timing (ns) Clock ap_clk Target Estimated 5.00 4.36 Uncertainty 0.63 • Latency (clock cycles) Latency Interval max max 932836 932836 932830 932836 Type max 932836 Utilization Estimates • Summary Name BRAM_18K DSP48E FF LUT DSP Expression FIFO 80 334 Instance 1589 1400 Memory Multiplexer Register Total 1669 1735 Available 120 80 35200 17600 Utilization (%) 57 • Detail Instance Module BRAM_18K AXIvideo2Mat_U0 AXIvideo2Mat Block_proc_U0 Block_proc CvtColor_U0 CvtColor CvtColor_1_U0 CvtColor_1 Mat2AXIvideo_U0 Mat2AXIvideo Sobel_U0 Sobel Total DSP48E 0 0 FF 241 200 42 119 985 1589 LUT 218 94 60 111 914 1400 Export Report for 'edge_detect' General Information Report date: Wed Jul 21 20:08:04 +0700 2021 Project: sobel_edge Solution: solution1 Device target: xc7z010clg400-1 Implementation tool: Xilinx Vivado v.2016.4 Resource Usage SLICE LUT FF DSP BRAM SRL Verilog 309 739 1053 3 Final Timing CP required CP achieved post-synthesis CP achieved post-implementation Verilog 5.000 4.707 4.721 Timing met 58 S K L 0