(Tiểu luận) midterm report digital image processing canny algorithm and code

VIETNAM GENERAL CONFEDERATION OF LABOR TON DUC THANG UNIVERSITY FACULTY OF INFORMATION TECHNOLOGY MIDTERM REPORT DIGITAL IMAGE PROCESSING Canny Algorithm and Code Instructor: PHD PHAM VAN HUY Authors: NGUYENMINHNHUT– 518H0545 TRAN NHAN TAI – 518H0560 Class: 18H50203 – 18H50204 Course: 22 HO CHI MINH CITY, 2021 0 Tieu luan VIETNAM GENERAL CONFEDERATION OF LABOUR TON DUC THANG UNIVERSITY FACULTY OF INFORMATION TECHNOLOGY MOBILE APP DEVELOPMENT DIGITAL IMAGE PROCESSING Canny Algorithm and Code Instructor: PHD PHAM VAN HUY Authors: NGUYENMINHNHUT– 518H0545 TRAN NHAN TAI – 518H0560 Class: 18H50203 – 18H50204 Course: 22 HO CHI MINH CITY, 2021 0 Tieu luan i Appreciation Letter Firstly, this should be an honor to send my regards to the Faculty of Information Technology, lecturers and staff f rom all departments of Ton Duc Thang University I would like to express my sincere thanks for the support and assistance during the implementation of the statistics and probability report I would like to express my gratitude to PHD Pham Van Huy - teacher who directly instructed and supervised me to complete this essay I sincerely thank my friends and classmates who are studying and working at Ton Duc Thang University and the family has encouraged, facilitated and helped me during the process Due to the fact that my actual ability is still weak, I ensure that I still have many shortcomings, so I hope my supervisor and the other professors will ignore it At the same time, I hope to receive many comments from many sources to help me accumulate more experience to complete the upcoming graduation report to achieve better results 0 Tieu luan ii THIS REPORT HAS BEEN CONDUCTED IN TON DUC THANG UNIVERSITY I assure that this is my own product and has been guided by PhD Pham Van Huy The research contents, results in this topic are all about honesty The data in the tables for analysis, comments and evaluation are collected by the me from various sources in the reference section In addition, comments and assessments as well as data from other authors or organizations are also used in the essay but with references and annotations If there is any fraud is detected, I ensure my complete responsibility for the contents of my work Ton Duc Thang University is not related to violations of authority and copyright caused by me during my work process (if any) Ho Chi Minh City, Saturday, 5th December, 2021 Authors (Sign and provide full name) Nguyen Minh Nhut Tran Nhan Tai 0 Tieu luan iii VERIFICATION AND EVALUATION FROM LECTURER Supervisor’s evaluation _ _ _ _ _ _ _ Ho Chi Minh city, date: (Sign and provide full name) Marking lecturer’s evaluation _ _ _ _ _ _ _ Ho Chi Minh city, date: (Sign and provide full name) 0 Tieu luan iv ABSTRACT Today life is blessed with the presence of technology and valuable inventions, one of them is camera, this one gadget is thing that you cannot go out without Then, images are things that you are able to modify and have good time with When images become extremely popular, the analysis of images has been a growing field of science and application for several decades Therefore, the ability to analyze images has been increasing One of the most well-known, classical techniques is edge detection There are various solutions for this topic such as Prewitt, Robert, Sobel, Marr Hildrith and Canny but this report we will focus on Canny edge detector and this algorithm is reacted as an efficient detector for edge extraction To make it easier to view but not try to rebuild the rest of the algorithm since “please not re-invent the wheel when it is already there for you”, we will go through all these headers below: PART 1: CANNY ALGORITHM THEORY Chapter 1: What is Edge detection? Chapter 2: Canny edge and Canny edge detection algorithm PART 2: CANNY ALGORITHM SCRIPTING Chapter 3: Canny algorithm scripting 0 Tieu luan TABLE OF CONTENTS MIDTERM REPORT .1 DIGITAL IMAGE PROCESSING Canny Algorithm and Code Instructor: PHD PHAM VAN HUY .1 Authors: NGUYENMINHNHUT– 518H0545 VIETNAM GENERAL CONFEDERATION OF LABOUR TON DUC THANG UNIVERSITY FACULTY OF INFORMATION TECHNOLOGY MOBILE APP DEVELOPMENT DIGITAL IMAGE PROCESSING Canny Algorithm and Code Instructor: PHD PHAM VAN HUY .2 Authors: NGUYENMINHNHUT– 518H0545 Appreciation Letter .i VERIFICATION AND EVALUATION FROM LECTURER iii Supervisor’s evaluation iii Marking lecturer’s evaluation iii ABSTRACT iv TABLE OF CONTENTS LIST OFTABLES AND ILLUSTRATIONS LIST OF ILLUSTRATION PART 1: CANNY ALGORITHM THEORY CHAPTER 1: WHAT IS EDGE DETECTION? 1.1 What is an edge? .4 Edges extraction from a sign Image intensity graph .5 1.1.1 Contour and Edge: Edge detection Contour detection .6 1.2 What is edge detection? Types of edge detections CHAPTER 2: Canny edge and Canny edge detection algorithm 2.1 What is gradient and gradient based operator? 0 Tieu luan 2.1.1 What is gradient? 2.1.2 What is image gradient .9 Image gradient properties 10 Gradient direction and edge 11 2.2 Who is Canny? 12 John F Canny portrait 12 2.3 Canny Edge Detection algorithm 12 2.2.1 Smoothing (remove noise): 13 2.3 Finding gradients: 15 2.4 Non-maximum suppression 16 2.5 Double thresholding 17 2.5 Edge tracking by Hysteresis 17 PART 2: CANNY ALGORITHM SCRIPTING .19 CHAPTER 3: Canny algorithm scripting 19 REFERENCES 25 Reading section 26 Watching section 26 0 Tieu luan LIST OFTABLES AND ILLUSTRATIONS LIST OF ILLUSTRATION Picture 1.1: Flutter’s log (source) Error! Bookmark not defined Picture 1.2: Flutter codebase example Error! Bookmark not defined Code snippet 1.1: Text sample Error! Bookmark not defined Picture 1.3: Text sample’ result Error! Bookmark not defined Picture 2.1: ToDoNhut’s starting screen Error! Bookmark not defined Picture 2.2: Step to create task Error! Bookmark not defined Picture 2.3: Step to create task Error! Bookmark not defined Picture 2.4: Finish creating task Error! Bookmark not defined Picture 2.5: Task infor screen Error! Bookmark not defined Picture 2.6: Edit Task screen Error! Bookmark not defined Picture 2.7: Mark as complete Error! Bookmark not defined Picture 2.8: Delete a task Error! Bookmark not defined Picture 2.9: Delete a task Error! Bookmark not defined Picture 2.10: Delete a task Error! Bookmark not defined Picture 3.1: ToDoNhut Github repo Error! Bookmark not defined 0 Tieu luan PART 1: CANNY ALGORITHM THEORY CHAPTER 1: WHAT IS EDGE DETECTION? Shapes within an image are usually defined by their boundaries, and these boundaries can be detected or enhanced with edge detection or enhancement algorithms Some texture measures are also reliant on the ability to detect edges Since Edge detection is a well-developed field on its own within image processing Edge detection is basically image segmentation technique, divides spatial domain, on which the image is defined, into meaningful parts or regions 1.1 What is an edge?  In low level image processing, we are about to “talk grayscale” and in grayscale images, edges are generally defined as large or abrupt changes in intensity along a line or curve  These changes would be evident as large values in the first derivative of a signal Since an image has at least two dimensions, the derivative has a slightly more complicated definition than in the one-dimensional case A hard edge exists when the change is very abrupt perhaps only or pixels wide A soft edge exists when the transition from bright to dark takes several pixels  Then, in simpler way to define edge is that “Edges are discontinuities in intensity” Edges extraction from a sign  As we can notice, in the sign, from red to white and reverse, those lines stand among them are called edge And here, imagine this sign as a diagram, you will see there is an area that the color red will span its weight to a point and then, it stops doing that and the color white kicks in so the breakpoint between them is an edge, but not just one 0 Tieu luan 12 2.2 Who is Canny? John Francis Canny (born in 1958) is an Australian computer scientist from MIT and Adelaide University, and Paul E Jacobs and Stacy Jacobs Distinguished Professor of Engineering in the Computer Science Department of the University of California, Berkeley He has made significant contributions in various areas of computer science and mathematics including artificial intelligence, robotics, computer graphics, human-computer interaction, computer security, computational algebra, and computational geometry John F Canny portrait (source) Of course, The Canny edge detector was developed by John F Canny in 1986 Canny also produced a computational theory of edge detection explaining why the technique works 2.3 Canny Edge Detection algorithm Canny’s edge approach is based on basic objectives - Low error rate All edges should be found, and there should be no spurious responses That is, the edges detected must be as close as possible to the true edges - Edge points should be well localized The edges located must be as close as possible to the true edges That is, the distance between a point marked as an edge by the detector and the center of the true edge should be minimum - Single edge point response The detector should return only one point for each true edge point That is, the number of local maxima around the true edge should be minimum This means that the detector should not identify multiple edge pixels where only a single edge point exists 0 Tieu luan 13 To get started, we need to discuss about steps that the algorithm follows After a period of time, I notice that there is no available standard writing resource that can describe steps of this algorithm concertedly until now but the Canny edge detection algorithm still will generally go through these steps below: Algorithm Flowchart 0 Tieu luan 14 2.3.1 Smoothing (remove noise) input image:  Since all images which taken from a camera cannot avoid to have some amount of noise Because of that, the noise must be reduced before implementing Canny edge detection algorithm  Then, you may ask: “If I not remove noises from my input image, can I still detect edges correctly?” The answer is relative but you should remove noises before applying Canny first order derivative edge detecting technic since the technic itself is very sensitive to noises, imagine that every detail on an image is represented by a pixel then you have to calculate derivative on that pixel and unfortunately, it is a noisy area, even you still get your extremum but that should be a wrong result after going through that area  Since Canny assumes that noises in image follow the rules of Gaussian Normal Distribution which can be depicted as:  x      f ( x)  e 2 2 Where f ( x ) is probability density function,  is standard deviation and  is mean  Gaussian filter is a technique which will apply smoothing to our images which will just obey the Normal Distribution rules  In 1-D, we can assume that the distribution has a mean of zero where  = then the equation has the form of:  f (x )  e 2 x2 2  As we have discussed above, image can be observed as 2-D union in two dimensions’ space such as images, the formula of Gaussian functions is: G ( x, y )  2  x y       2  e Where 𝑥 is the distance from the origin in the horizontal axis, similar 𝑦 is a distance in vertical axis and 𝜎 is a standard deviation of Gaussian filter 0 Tieu luan 15 Bonus, when we discuss about n-direction the equation can be like this: G( x,  )   2  n e  x   2       Then, we will not hesitate to convolve the G and f to create a smoothed image using Gaussian filter: fs ( x, y)  G( x, y)  f ( x, y ) 2.3.2 Finding gradients: As we have discussed above quite clearly, in image processing, gradient is a variation between pixels in the image which is in the grayscale intensity space The Canny edge detection algorithm based on finding where the gradient translation to the maxima By using the Sobel – operator, we can find the gradients at each pixel (first derivative) in the smoothed image We can apply the convolution operator in two dimensions’ space of image Two kernels are 𝐾𝐺𝑋 for horizontal axis and 𝐾𝐺𝑌for vertical axis −1 𝐾𝐺𝑋 = [−2 2] −1 1 𝐾𝐺𝑌 = [ −1 −2 0] −1 By using these two kernels with convolution operator, we can have 𝐺𝑋 = 𝐾𝐺𝑋 ∗ 𝐴 and 𝐺𝑌 = 𝐾𝐺𝑌 ∗ 𝐴 with 𝐺𝑋 and 𝐺𝑌 are the gradients in the x- and y-directions Then we can compute the gradient magnitudes (edge strengths) as Euclidean distance measure equation: |𝐺| = √𝐺𝑋2 + 𝐺𝑌2 Or apply the Manhattan distance measure: |𝐺| = |𝐺𝑋 | + |𝐺𝑌 | 0 Tieu luan 16 By implement that technics, we will have two-dimension matrix 𝐺 which shows the grayscale intensive transform in per pixel based on original image Similarly, we can have angle matrix with the same size as original image which will show the gradient direction in per pixel 𝐺𝑦 𝜃(𝑥, 𝑦) = arctan ( ) 𝐺𝑥 −𝜋 𝜋 Because the value domain of arctan is ( , ) , (−90°, 90 °) , so we can group the gradient direction (edge normal) into groups:  horizontal edge (0°)  vertical edge (asymptotic to −90°or 90°)  wing up right (45°)  wing up left (−45°) Then we can conclude the edge direction based on edge normal:  vertical edge ( 𝜃 ≈ 0°)  horizontal edge (𝜃 ≈ 90° 𝑜𝑟 𝜃 ≈ −90° )  wing up left ( 𝜃 ≈ 45°)  wing up right ( 𝜃 ≈ −45°) 2.3.4 Non-maximum suppression The purpose of this step is to convert the “blurred” edges into “sharp” edges in the image of the gradient magnitudes Basically, this process will keep all the maxima in the gradient image and deleting everything else by compare per pixel with their neighbors by edge strength value 0 Tieu luan 17    Round the gradient direction 𝜃 to nearest 45°, corresponding to the use of an 8connected neighborhood Compare the edge strength of the current pixel with the edge strength of the pixel in the positive and negative gradient direction Example: if the gradient direction is north (𝜃 ≈ 90° 𝑜𝑟 𝜃 ≈ −90° ) the we compare it with neighborhood pixels: above and below the current pixel Similarity with 3-other direction If the edge strength of the current pixel is largest, preserve the value else remove it 2.3.5 Double thresholding: The edge-pixels remaining after the non-maximum suppression step are marked with their strength Some of them may be caused by noise or color variations for instance due to rough surfaces Because of that, we can apply the simplest way to discern between noise and edges by using a threshold, which will only preserve the strong edges With two thresholds, we can use as:  Edge pixels which stronger than high threshold will be marked as strong  Edge pixels weaker than the low threshold will be remove  Edge pixels between two thresholds are marked as weak 2.3.6 Edge tracking by Hysteresis A result after step (Non-maximum suppression) still have strong and weak edges By using Hysteresis for edge tracking, we can reduce an amount of trivial edges The logic is that noise and other small variations are usually to result weak edges The strong edges almost be a true edge in the image The weak edges can be true edges or noise/ color variations For example, we have different edge units: A, B, C with A is a strong edge (edge strength higher than a high threshold which known as “maxVal”), B and C are weak edges (edge strengths between high threshold and low threshold) 0 Tieu luan 18 After apply hysteresis A will be preserved because it is a strong edge C is a weak edge but have a connection with a strong edge (A) so it will be preserved too B is a weak edge and standing alone so it need to be suppressed The edge tracking can be implement by using BLOB-analysis (Binary Large Object) Follow that technic, we divide 8-connected neighborhood with per BLOB unit BLOB unit which contain at least one strong edge pixel will be preserved, and suppressed it if not 0 Tieu luan 19 PART 2: CANNY ALGORITHM SCRIPTING CHAPTER 3: Canny algorithm scripting In OpenCV, there is a Canny function with structure as: void cv::Canny ( InputArray image, OutputArray edges, double threshold1, double threshold2, int apertureSize = 3, bool L2gradient = false ) With parameters:  image 8-bit input image  edges output edge map; single channels 8-bit image, which has the same size as image (binary image)  threshold1 first threshold for the hysteresis procedure  threshold2 second threshold for the hysteresis procedure  apertureSize aperture size for Sobel operator  L2gradient a flag, indicating weather a more accurate 𝐿2 = 𝑑𝐼 𝑑𝐼 √( ) + ( )should be used to calculate the image gradient magnitude 𝑑 𝑥 𝑑𝑦 (L2gradient = true), or whether the default 𝐿1 = |𝑑𝐼/𝑑𝑥 | + |𝑑𝐼/𝑑𝑦 | 𝑖𝑠 𝑒𝑛𝑜𝑢𝑔ℎ (L2gradient = false) In Canny function, we don’t have to attend where is the higher threshold parameters in procedure prepareThresh template inline void prepareThresh(f64 low_thresh, f64 high_thresh, s32 &low, s32 &high) { if (low_thresh > high_thresh) std::swap(low_thresh, high_thresh); #if defined GNUC low = (s32)low_thresh; high = (s32)high_thresh; low -= (low > low_thresh); high -= (high > high_thresh); #else low = internal::round(low_thresh); 0 Tieu luan 20 high = internal::round(high_thresh); f32 ldiff = (f32)(low_thresh - low); f32 hdiff = (f32)(high_thresh - high); low -= (ldiff < 0); high -= (hdiff < 0); #endif } The source code shows the different when change the L2gradient = true or false:  When L2gradient = true: for (; j < colscn; j++) _norm[j] = s32(_dx[j])*_dx[j] + s32(_dy[j])*_dy[j];  When L2gradient = false: for (; j < colscn; j++) _norm[j] = std::abs(s32(_dx[j])) + std::abs(s32(_dy[j])); Two loops above is apply in NormCanny procedure which have pointer as parameters (_dx, _dy and _norm) In source code, they apply the BLOB technic (consider 8-connected neighborhood pixels) when using Hyteresis: // now track the edges (hysteresis thresholding) while (stack_top > stack_bottom) { u8* m; if ((size_t)(stack_top - stack_bottom) + 8u > maxsize) { ptrdiff_t sz = (ptrdiff_t)(stack_top - stack_bottom); maxsize = maxsize * 3/2; stack.resize(maxsize); stack_bottom = &stack[0]; stack_top = stack_bottom + sz; } CANNY_POP(m); if if if if (!m[-1]) (!m[1]) (!m[-mapstep-1]) (!m[-mapstep]) CANNY_PUSH(m CANNY_PUSH(m CANNY_PUSH(m CANNY_PUSH(m Tieu luan + - 1); 1); mapstep - 1); mapstep); 21 if if if if (!m[-mapstep+1]) (!m[mapstep-1]) (!m[mapstep]) (!m[mapstep+1]) CANNY_PUSH(m CANNY_PUSH(m CANNY_PUSH(m CANNY_PUSH(m + + + mapstep + 1); mapstep - 1); mapstep); mapstep + 1); } An example for using canny in OpenCV: import cv2 image = cv2.imread('test2.jpeg', cv2.IMREAD_COLOR) blur = cv2.GaussianBlur(image, (5, 5), 0) canny = cv2.Canny(image = blur, threshold1 = 30, threshold2 = 100, apertureSize = 3, L2gradient = False) cv2.imshow('Result', image) cv2.imshow('Canny', canny) cv2.waitKey(0) cv2.destroyAllWindows() 0 Tieu luan 22 Here is the result: We can consider different between equations (Manhattan distance and Euclidean distance) 0 Tieu luan 23 0 Tieu luan 24 Using Euclidean Using Manhattan Tieu luan 25 Then, we can see the different between Canny and Sobel: (Sobel will record all the different include not an edges): Using Sobel Using Canny 0 Tieu luan 26 REFERENCES Reading section Slide Lecture 05 Edge detection Gradient of a straight line Gradient and generalized derivative of a multivariate function Differences between contour and edge from WikiDiff Contour detection and Edge detection with opencv John Canny John Canny Wikidata Gaussian smoothing Gaussian Filtering 10 The Gaussian kernel Watching section Image gradient Gradient to edges 0 Tieu luan ... below: PART 1: CANNY ALGORITHM THEORY Chapter 1: What is Edge detection? Chapter 2: Canny edge and Canny edge detection algorithm PART 2: CANNY ALGORITHM SCRIPTING Chapter 3: Canny algorithm scripting... Canny algorithm scripting 0 Tieu luan TABLE OF CONTENTS MIDTERM REPORT .1 DIGITAL IMAGE PROCESSING Canny Algorithm and Code Instructor: PHD PHAM VAN HUY ... THANG UNIVERSITY FACULTY OF INFORMATION TECHNOLOGY MOBILE APP DEVELOPMENT DIGITAL IMAGE PROCESSING Canny Algorithm and Code Instructor: PHD PHAM VAN HUY Authors: NGUYENMINHNHUT– 518H0545 TRAN