Dynamics analysis and applications of neural networks

DYNAMICS ANALYSIS AND APPLICATIONS OF NEURAL NETWORKS TANG HUA-JIN NATIONAL UNIVERSITY OF SINGAPORE 2004 Founded 1905 DYNAMICS ANALYSIS AND APPLICATIONS OF NEURAL NETWORKS BY TANG HUA-JIN (M.Eng. Shanghai Jiao Tong Univ.) A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF ELECTRICAL AND COMPUTER ENGINEERING NATIONAL UNIVERSITY OF SINGAPORE 2004 Acknowledgments This work was done in the computational intelligence group led by Dr. Tan Kay Chen at the Department of Electrical and Computer Engineering, National University of Singapore and financially supported by the university. Firstly, I would like to express my sincere thanks to my supervisor Dr. Tan Kay Chen for his valuable guidance and supports throughout my research. He provided me with such an interesting interdisciplinary topic concerned with mathematics, computer science, and even biology. His enthusiasm, optimism and encouragement gave a strong impetus to my scientific work. Working with him proves to be a rewarding experience. I want to thank Dr. Chew Chee Meng and A/Prof. Ben M. Chen for their competent criticism and encouraging support. I am deeply indebted to Prof. Lawrence O. Hall (Univ. of South Florida) and Prof. Xin Yao (Univ.of Birmingham) for their encouragement and kind helps. I also want to express my deep gratitude to Prof. Zhang Weinian (Sichun Univ.) and Prof. Zhang Yi (Univ. of Electronic Science and Technology). I benefited much from many valuable discussions with them on the research topic. I am also grateful to all my colleagues in the Control & Simulation Lab which provides good research facilities. I highly appreciate the friendly atmosphere and all the nice time we spent together in the last three years. Special thanks go to my wife Yan Rui, for her support, encouragement and love. Particularly, she contributed a lot to this thesis with her uncountable constructive suggestions. Finally, I dedicate this work to my parents for their supports throughout my life. i Contents Summary vii List of Tables ix List of Figures x Introduction 1.1 Background and Motivations . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Feed-forward Neural Networks . . . . . . . . . . . . . . . . . 1.1.2 Recurrent Networks with Saturating Transfer Functions . . . 1.1.3 Recurrent Networks with Nonsaturating Transfer Functions . 1.2 Scope and Contributions . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Plan of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . New Dynamical Optimal Learning for Linear Multilayer FNN 11 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.3 The Dynamical Optimal Learning . . . . . . . . . . . . . . . . . . . 14 2.4 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.4.1 Function Mapping . . . . . . . . . . . . . . . . . . . . . . . 19 2.4.2 Pattern Recognition . . . . . . . . . . . . . . . . . . . . . . 21 2.5 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 ii Discrete-Time Recurrent Networks for Constrained Optimization 26 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.2 Preliminaries and Problem Formulation . . . . . . . . . . . . . . . . 28 3.3 The Discrete-Time RNN Model for Nonlinear Differentiable Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 29 GES Analysis for Strictly Convex Quadratic Optimization Over Bound Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.5 Discussions and Illustrative Examples . . . . . . . . . . . . . . . . . 39 3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 On Parameter Settings of Hopfield Networks Applied to Traveling Salesman Problems 44 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.2 TSP Mapping and CHN Model . . . . . . . . . . . . . . . . . . . . 46 4.3 The Enhanced Lyapunov Function for Mapping TSP . . . . . . . . 49 4.4 Stability Based Analysis for Network’s Activities . . . . . . . . . . . 51 4.5 Suppression of Spurious States . . . . . . . . . . . . . . . . . . . . . 52 4.6 Setting of Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . 58 4.7 Simulation Results and Discussions . . . . . . . . . . . . . . . . . . 59 4.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Competitive Model for Combinatorial Optimization Problems 65 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 5.2 Columnar Competitive Model . . . . . . . . . . . . . . . . . . . . . 66 5.3 Convergence of Competitive Model and Full Valid Solutions . . . . 69 5.4 Simulated Annealing Applied to Competitive Model . . . . . . . . . 73 5.5 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 5.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 iii Competitive Neural Network for Image Segmentation 80 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 6.2 Neural Networks Based Image Segmentation . . . . . . . . . . . . . 82 6.3 Competitive Model of Neural Networks . . . . . . . . . . . . . . . . 83 6.4 Dynamical Stability Analysis . . . . . . . . . . . . . . . . . . . . . 85 6.5 Simulated Annealing Applied to Competitive Model . . . . . . . . . 86 6.6 Local Minima Escape Algorithm Applied to Competitive Model . . 88 6.7 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 6.7.1 Error-Correcting . . . . . . . . . . . . . . . . . . . . . . . . 91 6.7.2 Image Segmentation . . . . . . . . . . . . . . . . . . . . . . 95 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 6.8 Qualitative Analysis for Neural Networks with LT Transfer Functions 100 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 7.2 Equilibria and Their Properties . . . . . . . . . . . . . . . . . . . . 102 7.3 Coexistence of Multiple Equilibria . . . . . . . . . . . . . . . . . . . 108 7.4 Boundedness and Global Attractivity . . . . . . . . . . . . . . . . . 112 7.5 Simulation Examples . . . . . . . . . . . . . . . . . . . . . . . . . . 116 7.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Analysis of Cyclic Dynamics for LT Networks 123 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 8.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 8.3 Geometrical Properties of Equilibria Revisited . . . . . . . . . . . . 126 8.4 Rotational Vector Fields . . . . . . . . . . . . . . . . . . . . . . . . 128 8.5 Existence and Boundary of Periodic Orbits . . . . . . . . . . . . . . 130 8.6 Winner-take-all Network . . . . . . . . . . . . . . . . . . . . . . . . 137 8.7 Examples and Discussions . . . . . . . . . . . . . . . . . . . . . . . 141 iv 8.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 LT Network Dynamics and Analog Associative Memory 145 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 9.2 Linear Threshold Neurons . . . . . . . . . . . . . . . . . . . . . . . 147 9.3 LT Network Dynamics (Revisited) . . . . . . . . . . . . . . . . . . . 149 9.4 Analog Associative Memory . . . . . . . . . . . . . . . . . . . . . . 156 9.5 9.6 9.7 9.4.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . 156 9.4.2 Design Method . . . . . . . . . . . . . . . . . . . . . . . . . 158 9.4.3 Strategies of Measures and Interpretation . . . . . . . . . . . 161 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 9.5.1 Small-Scale Example . . . . . . . . . . . . . . . . . . . . . . 163 9.5.2 Single Stored Images . . . . . . . . . . . . . . . . . . . . . . 165 9.5.3 Multiple Stored Images . . . . . . . . . . . . . . . . . . . . . 167 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 9.6.1 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . 168 9.6.2 Competition and Stability . . . . . . . . . . . . . . . . . . . 169 9.6.3 Sparsity and Nonlinear Dynamics . . . . . . . . . . . . . . . 170 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 10 Conclusions and Outlook 173 10.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 10.2 Suggestions for Future Research . . . . . . . . . . . . . . . . . . . . 175 Bibliography 177 A Relating the Derivative Bounds to Network Parameters 192 B Neuronal Trajectories in D1 and D2 195 B.1 Phase Analysis for Center Type Equilibrium in D1 v . . . . . . . . . 195 B.2 Phase Analysis in D2 . . . . . . . . . . . . . . . . . . . . . . . . . . 197 B.3 Neural States Computed in Temporal Domain . . . . . . . . . . . . 199 C Author’s Publications 200 vi Summary Neural networks have been studied for many years in hope of emulating some aspects of human brain functions, which have great potentials in areas that require a large amount of parallel computations, such as pattern recognition, optimization and sensory information processing, etc. Neural networks are proven to be a wealthy resource for developing various intelligent computation techniques which require multidisciplinary efforts in biology, mathematics, computer science and so forth. The goal of this thesis is to exploit new aspects of theories of neural networks, in an attempt to provide new methods for intelligent computation. This thesis comprises several parts with emphasizes on learning theory, dynamics analysis and applications of feed-forward neural networks, recurrent networks with saturating transfer functions and with nonsaturating transfer functions. Firstly, to overcome sensitivity and slow learning speed of conventional backpropagation algorithm, a new training algorithm for multilayer feed-forward neural networks is put forward by assuming that the transfer function always plays in its linear region. The new algorithm is able to determine the optimal learning rate dynamically along with the training procedure. A discrete-time recurrent network of Hopfield type is proposed which has its advantages in simple implementation for solving constrained quadratic optimization problems. Presented conditions on the global exponential stability extend the existing results on the discrete-time system. To improve the solution quality when solving combinatorial optimization problems, as another important application of Hopfield network, a new principle of parameter settings is set up based on the dynamical stability analysis of an enhanced energy function. With the new parameter settings, improved performance both with regard to less spurious solution and to shorter tour lengths can be obtained. vii The competitive model incorporating winner-take-all mechanism is presented which is capable of eliminating tedious process of parameter settings and increasing computation efficiency significantly. Different algorithms underlying the competitive model are developed with applications to combinatorial optimization and image segmentation, respectively. Such competitive networks deal with constraints in an intrinsical manner by the competitive updating rule. In the last part of this thesis, an important focus is placed on the linear threshold (LT) network, which is a prominent biologically motivated model that involves nonsaturating neural activities. Various dynamical properties are clarified in terms of geometrical properties of equilibria, boundedness and stability. New theoretical results will facilitate to develop the applications, such as associative memory and feature binding. Especially, the theory of cyclic dynamics of two-cell LT networks is established, and its implication to an important winner-take-all network of large scale is illustrated. Finally, the analog associative memory of LT networks is addressed, which shows the capability of storing and retrieving complicated gray-scale images. viii Bibliography Scherk, J. (2000). ALGEBRA: A Computational Introduction. Chapman & Hall/CRC. Seung, H.S. (1998). Continuous attractors and oculomotor control. Neural Networks 11, 1253–1258. Shah, J.V. and C.S. Poon (1999). Linear independence of internal representations in multilayer perceptrons. IEEE Trans. Neural Networks 10(1), 10–18. Smith, K.A. (1999). Neural networks for combinatorial optimization: A review of more than a decade of research. INFORMS Journal on Computing 11(1), 15– 34. Sum, J.P.F., C.S. Leung, P.K.S. Tam, G.H. Young, W.K. Kan and L.W. Chan (1999). Analysis for a class of winner-take-all model. IEEE Trans. Neural Networks 10(1), 64–70. Tachibana, Y., G.H. Lee, H. Ichihashi and T. Miyoshi (1995). A simple steepest descent method for minimizing hopfield energy to obtain optimal solution of the tsp with reasonable certainty. Proceedings Int. Conference on Neural Networks pp. 1871–1875. Talav´an, P.M. and J. Y´an ˜ez (2002). Parameter setting of the hopfield network applied to tsp. Neural Networks 15, 363–373. Tan, K. C., H. J. Tang and Z. Yi (2004). Global exponential stability of discretetime neural networks for constrained quadratic optimization. Neurocomputing 56, 399–406. Tang, H.J., K.C. Tan and Z. Yi (2002). Convergence analysis of discrete time recurrent neural networks for linear variational inequality problem. In: Proceedings of 2002 Int. Joint Conference on Neural Networks. Hawaii, USA. pp. 2470– 2475. 186 Bibliography Tank, D.W. and J.J. Hopfield (1986). Simple neural optimization networks: An a/d converter, signal decision circuit, and a linear programming circuit. IEEE Trans. Circuits Syst. 33(5), 533–541. Vemuri, V.R. (1995). Artificial Neural Networks: Concepts and Control Applications. IEEE Comput. Society Press. Vidyasagar, M. (1992). Nonlinear Systems Analysis. 2nd Edition. Prentice Hall. New Jersey. von der Malsburg, C. (1973). Self-organization of orientation sensitive cells in the striate cortex. Kybernetik 14, 85–100. Wang, C.H., H.L. Liu and C.T. Lin (2001). Dynamic optimal learning rates of a certain class of fuzzy neural networks and its applications with genetic algorithm. IEEE Trans. Systems, Man, and Cybernetics-Part B: Cybernetics 31(3), 467–475. Wang, D.L. (1999). Object selection based on oscillatory correlation. Neural Networks 12, 579–592. Wang, D.L. and D. Terman (1995). Locally excitatory globally inhibitory oscillator networks. IEEE Trans. Neural Networks 6(1), 283–286. Wang, D.L. and D. Terman (1997). Image segmentation based on oscillatory correlation. Neural Computation 9, 805–836. Wang, D.L. and X. Liu (2002). Scene analysis by integrating primitive segmentation and associative memory. IEEE Trans. Systems, Man and Cybernectis: B 32(3), 254–268. Wang, L. (1997a). Discrete-time convergence theory and updating rules for neural networks with energy functions. IEEE Trans. on Neural Networks 8(2), 445– 447. 187 Bibliography Wang, L. (1997b). On competitive learning. IEEE Trans. Neural Networks 8(5), 1214–1216. Wang, L. (1998). On the dynamics of discrete-time, continuous-state hopfield neural networks. IEEE Trans. on Circuits and Systems-II: Analog and Digital Signal Processing 45(6), 747–749. Wang, L. and F. Tian (2000). Noisy chaotic neural networks for solving combinatorial optimization problems. In: Proceedings of Int. Joint Conference on Neural Networks (IJCNN). Combo, Italy. Wang, L. and K. Smith (1998). On chaotic simulated annealing. IEEE Trans. on Neural Networks 9(4), 716–718. Weaver, S. and M. Polycarpou (2001). Using localizing learning to improve supervised learning algorithms. IEEE Trans. Neural Networks 12(5), 1037–1045. Wersing, H. and H. Ritter (1999). Feature binding and relaxation labeling with the competitive layer model. In: Proc. Eur. Symp. on Art. Neur. Netw. ESANN. pp. 295–300. Wersing, H., J.J. Steil and H. Ritter (1997). A layered recurrent neural network for feature grouping. In: Proceedings of Int. Conf. Art. Neur. Netw. ICANN. Lausanne. pp. 439–444. Wersing, H., J.J. Steil and H. Ritter (2001a). A competitive layer model for feature binding and sensory segmentation. Neural Computation 13(2), 357–387. Wersing, H., W.J. Beyn and H. Ritter (2001b). Dynamical stability conditions for recurrent neural networks with unsaturating piecewise linear transfer functions. Neural Computation 13(8), 1811–1825. 188 Bibliography Williamson, J.R. (1996). Neural network for dynamic binding with graph representation: form, linking and depth-from-occlusion. Neural Computation 8, 1203– 1225. Wilson, V. and G.S. Pawley (1988). On the stability of the tsp problem algorithm of hopfield and tank. Bio. Cybern. 58, 63–70. Wu, Y.Q., C.P. Yang and T.Z. Wang (2001). A new approach of color quantization of image based on neural network. International Joint Conference on Neural Networks pp. 973–977. Xia, Y. and J. Wang (2000). Global exponential stability of recurrent neural networks for solving optimization and related problems. IEEE Trans. Neural Networks 11(4), 1017–1022. Xie, X., R.H.R. Hahnloser and H.S. Seung (2002). Selectively grouping neurons in recurrent networks of lateral inhibition. Neural Computation 14(11), 2627– 2646. Yam, J.Y.F. and T.W.S. Chow (2001). Feedforward networks training speed enhancement by optimal initialization of the synaptic coefficients. IEEE Trans. Neural Networks 12(2), 430–434. Yao, X. (1999). Evolving artificial neural networks. Proceedings of IEEE 87(9), 1423–1447. Yao, X. and Y. Liu (1997). A new evolutionary system for evolving artificial neural networks. IEEE Trans. Neural Networks 8(3), 694–713. Yen, S.C. and L.H. Finkel (1998). Extraction of perceptually salient contours by striate cortical networks. Vision Research 38(5), 719–741. 189 Bibliography Yeung, D.S. and X. Sun (2002). Using function approximation to analyze the sensitivity of mlp with antisymmetrix squashing activation function. IEEE Trans. Neural Networks 13(1), 34–44. Yi, Z. and K.K. Tan (2002). Dynamic stability conditions for lotka-volterra recurrent neural networks with delays. Physical Review E 66, 011910. Yi, Z., K.K. Tan and T.H. Lee (2003). Multistability analysis for recurrent neural networks with unsaturating piecewise linear transfer functions. Neural Computation 15(3), 639–662. Yi, Z., P.A. Heng and A.W.C. Fu (1999). Estimate of exponential convergence rate and exponential stability for neural networks. IEEE Trans. Neural Networks 10(6), 1487–1493. Yi, Z., P.A. Heng and K.S. Leung (2001). Convergence analysis of cellular neural networks with unbounded delay. IEEE Trans. Circuits and Systems I 48(6), 680–687. Yi, Z., P.A. Heng and P. Vadakkepat (2002). Absolute periodicity and absolute stability of delayed neural networks. IEEE Trans. Circuits and Systems I 49(2), 256–261. Yi, Z., P.A. Heng and P.F. Fung (2000). Winner-take-all discrete recurrent neural networks. IEEE Trans. Circuits and Systems II 47(12), 1584–1589. Yi, Z., Y. Fu and H.J. Tang (2004). Neural networks based approach for computing eigenvectors and eigenvalues of symmetric matrix. Computer & Mathematics with Applications p. to appear. Yu, X., M.O. Efe and O. Kaynak (2002). A general backpropagation algorithm for feedforward neural networks learning. IEEE Trans. Neural Networks 13(1), 251–254. 190 Bibliography Zak, S.H., V. Upatising and S. Hui (1995). Solving linear programming problems with neural networks: A comparative study. IEEE Trans. Neural Networks 6(1), 94–104. Zhang, X.S. (2000). Neural networks in optimization. Kluwer Academic Publisher. Zhang, Z., T. Ding, W. Huang and Z. Dong (1992). Qualitative Theory of Differential Equations. Translated by W.K. Leung. American Mathematical Society. Providence, USA. Zucker, S.W., A. Dobbins and L. Iverson (1989). Two stages of curve detection suggest two styles of visual computation. Neural Computation 1, 68–81. Zurada, J.M. (1995). Introduction to Artificial Neural Systems. PWS Publishing Company. Boston. Zurada, J.M., I. Cloete and E. van der Poel (1996). Generalized hopfield networks for associative memories with multi-valued stable states. Neurocomputing 13, 135–149. 191 Appendix A Relating the Derivative Bounds to Network Parameters Proposition 4.1: Any invalid tour v ∈ HC − HT , S = vx,i = N − 1, the x i following bound is obtained max E (v) ≤ −A − B + S=N −1 C + 3DdU . Proof. For any point v ∈ HC , let I0 denote the index set of its columns whose elements are equal to 0, i.e., I0 = {i ∈ {1, 2, · · · , N} : Si = 0} and N0 = |I0| denotes the cardinal. Now consider S = N − 1, I0 = ∅ and N0 ≥ 1, then x and i exist such that vx ,i = and Sx = 0, e.g.,      v=   vx ,i  0 0      .  0   Therefore, from (4.18), Ex ,i (v) = −A − B + ≤ −A − B + C +D dx,y (vy,i −1 + vy,i +1 ) y=x C + DdU (Si −1 + Si +1 ). 192 Appendix A. Relating the Derivative Bounds to Network Parameters Taking minimum over all i ∈ I0, it is obtained that E (v) ≤ −A − B + C + DdU min(Si−1 + Si+1 ). i∈I0 Applying the technical result (Lemma A. 2, (Talav´an and Y´an ˜ez, 2002)), min(Si−1 + Si+1 ) ≤ − i∈I0 2(N − S) . N0 (A.1) and taking into account that Si−1 + Si+1 ∈ N , the following is obtained, min(Si−1 + Si+1 ) ≤ [4 − i∈I0 2(N − S) ] ≤ 3. N0 Hence it proves that E (v) ≤ −A − B + C + 3DdU . Proposition 4.2: Any invalid tour v ∈ HC − HT , S = vx,i = N , the x i following bound is obtained C E¯1 (v) ≥ {B, A + DdL , (N − 1)A} − . S=N Proof. To prove E (v) ≥ min{B, A + DdL , (N − 1)A} − S=N C , the following exhaustive cases for vx,i = are considered. Case : ∃vx ,i = : Sx ≥ 1, Si ≥ 2, e.g.,  0 0       0    v= .    0    0 From equation (4.18), it is derived Ex ,i = A(Sx − 1) + B(Si − 1) − ≥ B− C . C +D dx,y (vy,i −1 + vy,i +1 ) y=x (A.2) 193 Appendix A. Relating the Derivative Bounds to Network Parameters Case : ∃vx ,i = : Sx ≥ 2, Si = 1, e.g.,  0 0       0    v= .   1 0     0 Then Ex ,i = A(Sx − 1) − ≥ A− C + Ddx,y C + DdL . (A.3) Case : ∃vx ,i = : Sx = N, Si = 1, e.g.,  0 0       0 0    v= .    1 1    0 0 It follows that Ex ,i (v) ≥ A(N − 1) − C . (A.4) Considering (A.2)-(A.4), it holds E (v) ≥ min{B, A + DdL , A(N − 1)} − S=N This completes the proof. 194 C . Appendix B Neuronal Trajectories in D1 and D2 B.1 Phase Analysis for Center Type Equilibrium in D1 In D1 , the Jacobian matrix   w12   w11 − J= , w22 w22 − its eigenvalues are λ1,2 = ±ωi. By defining x1(t) = x1(t) − x∗1 , x2(t) = x2(t) − x∗2 ˙ Note that w12 w21 = 0. The for t ≥ 0, the linear system (8.1) becomes x˙ = J x. linear system has a pair of complex eigenvectors corresponding to λ1,2 ,         e1,2 =   ± i , −w11 +1 w12 or   e1,2 =  w11 −1 w21 ω w12     ± i ω w21   . Denote u = Re(e) and v = Im(e), according to the linear system theory (Perko, 2001), J can be reduced to the Jordan canonical form by defining a invertible 195 Appendix B. Neuronal Trajectories in D1 and D2 matrix P = (v, u). We choose the first pair of eigenvectors for our derivation. Adopting the second pair of eigenvectors will reach the same results. Therefore, we can define a linear transformation y(t) = P −1 x(t) for all t ≥ 0, where   P = ω w12 such that  −w11 +1 w12  ,    −ω  P −1 JP =  , ω then we have    −ω  y˙ =   y. ω It follows that  (B.1)   cos ωt − sin ωt  y(t) =   y(0). sin ωt cos ωt (B.2) y1(t)2 + y2(t)2 = y1 (0)2 + y2(0)2 . (B.3) Hence, Recall that y(t) = P −1 x(t), i.e.,    y1(t) = w11 −1 x1 (t) ω + w12 x2 (t), ω   y2(t) = x1(t), (B.4) substitute it back into (B.3), thus we obtain the trajectory equation in D1 , ( w12 w11 − w12 w11 − x1 (t) + x2 (t))2 + x1 (t)2 = ( x1(0) + x2(0))2 + x1 (0)2 , (B.5) ω ω ω ω that is, (1 + 2(w11 − 1)w12 w12 (w11 − 1)2 )x (t) + x (t)x (t) + x2(t)2 = x1(0)2 1 ω2 ω2 ω2 w11 − w12 x1 (0) + x2 (0))2 . +( ω ω 196 (B.6) Appendix B. Neuronal Trajectories in D1 and D2 B.2 Phase Analysis in D2 In D2 , the Jacobian matrix is given as   w12   −1 J= . w22 − By defining x1(t) = x1(t) − x∗1 , x2(t) = x2(t) − x∗2 for t ≥ 0, the linear system (8.1) ˙ Since J may have one or two distinct eigenvalues when w22 takes becomes x˙ = J x. different values, thus two different cases need to be considered. Case 1: w22 = 0. In this case the Jacobian matrix has two distinct eigenvalues, λ1 = −1, λ2 = w22 − 1. The corresponding linear-independent eigenvectors are given below,     w12    w  e1 =   , e2 =  22  . According to the linear system theory, J can be reduced to the Jordan canonical form by defining an invertible matrix P = (e1 , e2). Therefore, we can define a linear transformation y(t) = P −1 x(t) for all t ≥ 0, where   w12  w22  P = . Then   y = P −1 x =  It follows that 12 − ww22   w12 x w22   x1 − x =  x2    . (B.7)   −1  y˙ = P −1 JP y =   y. w22 − It is solved as    y1(t) = exp(−t)y1(0),   y2(t) = exp((w22 − 1)t)y2(0). (B.8) Thus we have y1(t)(w22 −1) y2 (t) = y1 (0)(w22 −1) y2 (0). 197 (B.9) Appendix B. Neuronal Trajectories in D1 and D2 Substitute (B.7) into (B.9), we get (x1 (t) − w12 w12 x2 (t))(w22 −1) x2(t) = (x1 (0) − x2(0))(w22 −1) x2 (0). w22 w22 (B.10) Case 2: w22 = 0. The eigenvalues are λ1,2 = −1. The Jacobian matrix has only one linear independent vector e1 = (1, 0)T . A generalized eigenvector e2 = (0, w112 )T is obtained such that Je2 = (−1)e2 + e1. Then we can define a similarity transformation y = P −1 x, where    P =  . (B.11) w12 Then we have      x1    y = P −1 x =  x =  , w12x2 w12 and  (B.12)   −1  y˙ = P −1 JP y =   y. −1 (B.13) Its solutions are given by    t  y(t) = e−t   y(0). Substitute (B.12) into (B.14), it is obtained that    x1 (t) = e−t (x1(0) + tw12x2 (0)),   x2 (t) = e−t x2(0), (B.14) (B.15) or equivalently, x1(t) x1 (0) = + tw12. x2(t) x2 (0) (B.16) From the above equation, it is derived that t= x1(t) x1(0) − ), ( w12 x2(t) x2(0) 198 (B.17) Appendix B. Neuronal Trajectories in D1 and D2 substitute it back into (B.15), we get ln x2 (t) + x1 (t) x1(0) = ln x2(0) + , w12 x2 (t) w12 x2(0) (B.18) for x2 (0) > 0. B.3 Neural States Computed in Temporal Domain The orbit of the system in D1 is computed by  w11 −w22 sin ωt + cos ωt τ  2ω e (x(t)−x ) = exp( t)  w21 sin ωt ω w12 ω w22 −w11 2ω sin ωt sin ωt + cos ωt   e  (x(0)−x ). (B.19) The orbit of the system in D2 is computed by   w12  exp(−t) w22 (exp((w22 − 1)t) − exp(−t))  e (x(t) − xe) =   (x(0) − x ), (w22 = 0) exp((w22 − 1)t) (B.20) or else   w12t exp(−t)   exp(−t) e (x(t)−xe) =   (x(0)−x ), (w22 = 0). (B.21) w22t exp(−t) + exp(−t) 199 Appendix C Author’s Publications The author has contributed to the following publications: Journal Publications 1. Huajin Tang, K. C. Tan and Weinian Zhang (2005). “Analysis of Cyclic Dynamics for Networks of Linear Threshold Neurons”. Neural Computation, vol. 17, no. 1, pp. 97–114. 2. Huajin Tang, K. C. Tan and Zhang Yi (2004). “A Columnar Competitive Model for Solving Combinatorial Optimization Problems”. IEEE Transactions on Neural Networks, vol. 15, no. 6, pp. 1568–1573. 3. K. C. Tan and Huajin Tang (2004). “New dynamical optimal learning for linear multilayer FNN”. IEEE Transactions on Neural Networks, vol. 15, no. 6, pp. 1562–1568. 4. Huajin Tang, K. C. Tan, and E. J. Teoh (2004). “Dynamics Analysis and Analog Associative Memory of Networks with LT Neurons”. IEEE Transactions on Neural Networks, accepted. 5. K. C. Tan and Huajin Tang and Zhang Yi (2004). “Global Exponential Stability of Discrete-Time Neural Networks for Constrained Quadratic Optimization”. NeuroComputing, vol. 56, pp. 399–406. 200 Appendix C. Author’s Publications 6. K. C. Tan, Huajin Tang and S. S. Ge (2004). “On Parameter Settings of Hopfield Networks Applied to Traveling Salesman Problems”. IEEE Transactions on Circuits and Systems - I: Fundamental Theory and Applications, accepted. 7. K. C. Tan, Huajin Tang and Weinian Zhang (2004). “Qualitative Analysis for Recurrent Neural Networks with Linear Threshold Transfer Functions”. IEEE Transactions on Circuits and Systems - I: Fundamental Theory and Applications, accepted. 8. Zhang Yi, Yan Fu and Huajin Tang (2004). “Neural Networks Based Approach for Computing Eigenvectors and Eigenvalues of Symmetric Matrix”. Computers and Mathematics with Application, vol. 47, pp. 1155–1164. Conference Publications 1. Huajin Tang, K. C. Tan and T. H. Lee (2003). “Stability analysis of Hopfield neural networks for solving TSP”. Proceedings of the Second International Conference on Computational Intelligence, Robotics and Autonomous Systems (CIRAS 2003), Singapore. 2. Huajin Tang, K. C. Tan and T. H. Lee (2003). “Competitive neural networks for solving combinatorial optimization problems”. Proceedings of the Second International Conference on Computational Intelligence, Robotics and Autonomous Systems (CIRAS 2003), Singapore. 3. Huajin Tang, K. C. Tan and Zhang Yi (2002). “Convergence analysis of discrete time recurrent neural networks for linear variational inequality”. Proceedings of IEEE International Joint Conference on Neural Networks 2002, pp. 2470-2475, Honolulu, Hawaii, USA. 4. Huajin Tang, K. C. Tan and T. H. Lee (2004). “Dynamical Optimal Learning for FNN and Its Applications”. FUZZ-IEEE, July 25–29, Budapest, Hungary. 201 [...]... in both theory and engineering application In this thesis, significant efforts are devoted to analyzing dynamic properties of neural networks and exploiting their applications in their dynamics regime 1.1 Background and Motivations Typically, the models of neural networks are divided into two categories in terms of signal transmission manner: feed-forward neural networks and recurrent neural 1 Chapter... extensive investigations of neural networks, both in dynamics analysis and applications 1.2 Scope and Contributions One focus of the thesis lies on the improvement of the training algorithm of feedforward neural networks by analyzing the mean-squared error function from the view point of dynamic stability The dynamical learning method is able to adaptively and optimally set the value of learning rate, hence... complexity and capability of modeled neural networks rely on our present understanding of biological neural systems On the other hand, neural networks provide efficient computation methods in making intelligent machines in multidisciplinary fields, e.g., computational intelligence, robotics and computer vision In the past two decades, the research on neural networks has witnessed a great deal of accomplishments... of promoting our understanding of the brain functions such as computation, perception and memory Secondly, the results in this thesis can provide meaningful techniques for developing real-world applications 1.3 Plan of the Thesis The first chapter motivates the issue of dynamics analysis as one crucial step to understand the collective computation property of neural systems and describes the scope and. .. hence the elimination of sensitivity of FNN networks with a fixed learning rate can be expected, as well as the reduction of convergence iterations and time Another emphasis is on the neurodynamics The dynamics of the recurrent networks with saturating and nonsaturating transfer functions are analyzed extensively New theoretical results on the nondivergence, stability and cyclic dynamics are established,... feed-forward neural networks, recurrent neural network (RNN) is described by a system of differential equations that define the exact evolution of the model as a function of time The system is characterized by a large number of coupling constants represented by the strengths of individual junctions, and it is believed that the computational power is the result of the collective dynamics of the system... segmentation (Wang and Terman, 1995; Wang and Terman, 1997; Wang, 1999; Chen and Wang, 2002) The CLM model, formulated by the LT network, realizes an energy-based approach to feature binding and texture segmentation and has been successfully applied to segmentation of real-world images (Ontrup and Ritter, 1998; Wersing et al., 1997; Wersing and Ritter, 1999) Dynamic binding in a neural network is of great interest... 171 xiii Chapter 1 Introduction Artificial neural networks, or simply called neural networks, refer to various mathematical models of human brain functions such as perception, computation and memory It is a fascinating scientific challenge of our time to understand how the human brain works Modeling neural networks facilitates us to investigate the information processing occurred... saturating transfer functions, Hopfield network and cellular neural network, have stimulated a great deal of research efforts over the past two decades because of their great potentials in applications in associative memory, optimization and intelligent computation (Hopfield, 1984; Hopfield and Tank, 1985; Tank and Hopfield, 1986; Bouzerdoum and Pattison, 1993; Maa and Shanblatt, 1992; Zak et al., 1995; Tan... the stability is of primary interest in the analysis and applications of recurrent networks, where the Lyapunov stability theory is a fundamental tool and widely used for analyzing nonlinear systems (Grossberg, 1988; Vidyasagar, 1992; Yi et al., 1999; Qiao et al., 2003) Based on the Lyapunov method, the conditions of global exponential stability of a continuoustime RNN were established and applied to . DYNAMICS ANALYSIS AND APPLICATIONS OF NEURAL NETWORKS TANG HUA-JIN NATIONAL UNIVERSITY OF SINGAPORE 2004 Founded 1905 DYNAMICS ANALYSIS AND APPLICATIONS OF NEURAL NETWORKS BY TANG. On this side, the complexity and capability of modeled neural networks rely on our present under- standing of biological neural systems. On the other hand, neural networks provide efficient computation. analyzing dynamic properties of neural networks and exploiting their applications in their dynamics regime. 1.1 Background and Motivations Typically, the models of neural networks are divided into