Neural networks

NEURAL NETWORKS Vedat Tavşanoğlu What Is a Neural Network?  Work on artificial neural networks, commonly referred to as "neural networks," has been motivated right from its inception by the recognition that the brain computes in an entirely different way from the conventional digital computer What Is a Neural Network?  The struggle to understand the brain owes much to the pioneering work of Ramon y Cajal (1911), who introduced the idea of neurons as structural constituents of the brain  Typically, neurons are five to six orders of magnitude slower than silicon logic gates; events in a silicon chip happen in the nanosecond (10-9 s) range, whereas neural events happen in the millisecond (10-3 s) range What Is a Neural Network?  However, the brain makes up for the relatively slow rate of operation of a neuron by having a truly staggering number of neurons (nerve cells) with massive interconnections between them What Is a Neural Network?  It is estimated that there must be on the order of 10 billion neurons in the human cortex, and 60 trillion synapses or connections (Shepherd and Koch, 1990) The net result is that the brain is an enormously efficient structure Specifically, the energetic efficiency of the brain is approximately 10-16 joules (J) per operation per second  The corresponding value for the best computers in use today is about 10-6 joules per operation per second (Faggin, 1991) What Is a Neural Network?  The brain is a highly complex, nonlinear, and parallel computer (informationprocessing system) It has the capability of organizing neurons so as to perform certain computations (e.g., pattern recognition, perception, and motor control) many times faster than the fastest digital computer in existence today What Is a Neural Network?  Consider, for example, human vision, which is an information-processing task (Churchland and Sejnowski, 1992; Levine, 1985; Marr, 1982)  It is the function of the visual system to provide a representation of the environment around us and, more important, to supply the information we need to interact with the environment What Is a Neural Network?  The brain routinely accomplishes perceptual recognition tasks (e.g., recognizing a familiar face embedded in an unfamiliar scene) in something of the order of 100-200 ms, whereas tasks of much lesser complexity will take hours on conventional computers What Is a Neural Network?  For another example, consider the sonar of a bat Sonar is an active echo-location system  In addition to providing information about how far away a target (e.g., a flying insect) is, a bat sonar conveys information about the relative velocity of the target, the size of the target, the size of various features of the target, and the azimuth and elevation of the target (Suga, 1990a, b) What Is a Neural Network?  The complex neural computations needed to extract all this information from the target echo occur within a brain the size of a plum Indeed, an echo-locating bat can pursue and capture its target with a facility and success rate that would be the envy of a radar or sonar engineer Using The Hopfield Network  a(  yi2 )  dy 1    diag  Wy  GΓ (y)  I  dt  2Ci    the state equations are obtained as   y 1  dy1  a  dt   2C1    dy2     dt     1  y1     ln   a y  g g y G        12 1   11           y22    g 21 g 22   y2   G2   1  y2   a    a ln  y   2C2      dy1   (1  y )(ag y  ag y  G ln  y1 )  11 12  dt   2C1  y1      dy2   (1  y )(ag y  ag y  G ln  y2 )  22 21  dt   2C2  y2  Discrete-Time Hopfield Networks Consider the state equation of the Gradient-Type Hopfield Network: dx( t ) C  Wy  Gx  I dt We can write dx( t ) -1 C  Wy  GΓ (y)  I dt Discrete-Time Hopfield Networks As the plot of the inverse bipolar activation function shows the second term in the above equation is zero for high gain neurons Hence: dx( t ) C  Wy  I dt Discrete-Time Hopfield Networks Now consider dxi df ( yi ) df ( yi ) dyi   dt dt dyi dt -1 Using the this plot we can conclude that df -1 ( yi ) 0 dyi for high gain neurons -1 Discrete-Time Hopfield Networks Hence dx( t ) 0 dt It follows that  Wy  I Now let us solve this equation using Jacobi’s algorithm To this end define: Discrete-Time Hopfield Networks where W'  W  D = L  U L,U and D = diag( wii ) Are the lower and upper triangular and diagonal matrices shown in the following decomposition of W W  w11 w  21     w N1 w12 w 22 w N2 w1N   w11 w 2N        w NN   0 w 22       w 21     w 31  w 32     w NN    w N1  w N2  w N,N1  0  w12  0        0  w1N   w 2N     w N1,N   Discrete-Time Hopfield Networks Now defining we obtain D  diag( wii ) -Dy = W'y + I -1 -1 y = -D W'y - D I Now define -D W' = W -1 -D I  I -1 Discrete-Time Hopfield Networks y = Wy  I Now replace the vector y on the right-hand side by an initial y(0) vector If the vector y on the lefthand side is obtained as y(0), then y(0) is the solution of the system If not then call the vector y obtained on the left-hand side y(1), i.e., Discrete-Time Hopfield Networks y(1) = W y(0)  I and in general we can write y(k + 1) = W y(k)  I Discrete-Time Hopfield Networks The method will always converge if the matrix W is strictly or irreducibly diagonally dominant Strict row diagonal dominance means that for each row, the absolute value of the diagonal term is greater than the sum of absolute values of other terms: wii   wij i j Discrete-Time Hopfield Networks The Jacobi method sometimes converges even if this condition is not satisfied It is necessary, however, that the diagonal terms in the matrix are greater (in magnitude) than the other terms Discrete-Time Hopfield Networks Solution by Gauss-Seidel Method In Jacobi’s method the updating of the unknowns is made after all N unknowns have been moved to the left side of the equation We will see in the following that this is not necessary, i.e., the updating can be made individually for each unknown and this updated value can be used in the next equation This is shown in the following equations: Discrete-Time Hopfield Networks x1 (n  1)  x2 (n  1)   a21x1 (n  1)  a23 x3 (n)   a2 N xN (n)  b2  a22 a22 x3 ( n  1)  and [ a12 x2 (n)  a13 x3 (n)   a1N xN (n)  b1 ] a11  a31x1 (n  1)  a32 x2 (n  1)  a34 x2 (n)   a3 N xN (n)  b3  a33 a33 xN (n  1)    aN x1 ( n  1)  aN x2 ( n  1)   aN , N 1 xN 1 ( n  1)  bN  aNN aNN In vector-matrix form, we can write: D x (n  1)  L x (n  1)  U x (n)  b ( D  L) x (n  1)  U x (n)  b x (n  1)  ( D  L) 1 (U x (n)  b) Discrete-Time Hopfield Networks This matrix expression is mainly used to analyze the method When implementing Gauss Seidel, an explicit entry-by-entry approach is used: xi (n  1)  aii   bi   aij x j (n  1)   aij x j (n)  j i j i   Discrete-Time Hopfield Networks Gauss-Seidel method is defined on matrices with non-zero diagonals, but convergence is only guaranteed if the matrix is either: diagonally dominant or symmetric and positive definite