a hierarchical classification of first-order

A Hierarchical Classification of First-Order Recurrent Neural Networks J´er´emie Cabessa 1 and Alessandro E.P. Villa 1,2 1 GIN Inserm UMRS 836, University Joseph Fourier, FR-38041 Grenoble 2 Faculty of Business and Economics, University of Lausanne, CH-1015 Lausanne {jcabessa,avilla}@nhrg.org Abstract. We provide a refined hierarchical classification of first-order recurrent neural networks made up of McCulloch and Pitts cells. The classification is achieved by first proving the equivalence between the ex- pressive powers of such neural networks and Muller automata, and then translating the Wadge classification theory from the automata-theoretic to the neural network context. The obtained hierarchical classification of neural networks consists of a decidable pre-well ordering of width 2 and height ω ω , and a decidability procedure of this hierarchy is provided. Notably, this classification is shown to be intimately related to the at- tractive properties of the networks, and hence provides a new refined measurement of the computational power of these networks in terms of their attractive behaviours. 1 Introduct ion In neural computability, the issue of the computational power of neural networks has often been approached from the automata-theoretic perspective. In this con- text, McCulloch and Pitts, Kleene, and Minsky already early proved that the class of first-order recurrent neural networks discloses equivalent computational capabilities as classical finite state automata [5,7,8]. Later, Kremer extended this result to the class of Elman-style recurrent neural nets, and Sperduti discussed the computational power of different other architecturally constrained classes of networks [6,15]. Besides, the computational power of first-order recurrent neural networks was also proved to intimately depend on both the choice of the activation function of the neurons as well as the nature of the synaptic weights under consideration. In- deed, Siegelmann and Sontag showed that, assuming rational synaptic weights, but considering a saturated-linear sigmoidal instead of a hard-threshold acti- vation function drastically increases the computational power of the networks from finite state automata up to Turing capabilities [12,14]. In addition, Siegel- mann and Sontag also nicely proved that real-weighted networks provided with a saturated-linear sigmoidal activation function reveal computational capabilities beyond the Turing limits [10,11,13]. This paper concerns a more refined characterization of the computational power of neural nets. More precisely, we restrict our attention to the simple A H.Dediu,H.Fernau,andC.Mart´ın-Vide (Eds.): LATA 2010, LNCS 6031, pp. 142–153, 2010. c  Springer-Verlag Berlin Heidelberg 2010 A Hierarchical Classification of First-Order Recurrent Neural Networks 143 class of rational-weighted first-order recurrent neural networks made up of Mc- Culloch and Pitts cells, and provide a refined classification of the networks of this class. The classification is achieved by first proving the equivalence between the expressive powers of such neural networks and Muller automata, and then trans- lating the Wadge classification theory from the automata-theoretic to the neural network context [1,2,9,19]. The obtained hierarchical classification of neural net- works consists of a decidable pre-well ordering of width 2 and height ω ω ,anda decidability procedure of this hierarchy is provided. Notably, this classification is shown to be intimately related to the attractive properties of the considered networks, and hence provides a new refined measurement of the computational capabilities of these networks in terms of their attractive behaviours. 2 The Model In this work, we focus on synchronous discrete-time first-order recurrent neural networks made up of classical McCulloch and Pitts cells. Definition 1. A first-or der recurrent neural network consists of a tuple N = (X, U, a, b, c),whereX = {x i :1≤ i ≤ N} is a finite set of N activation cells, U = {u i :1≤ i ≤ M} is a finite set of M external input cells, and a ∈ Q N×N , b ∈ Q N×M ,andc ∈ Q N×1 are rational matrices describing the weights of the synaptic connections b etween cells as well as the incoming background activity. The activation value of cells x j and u j at time t, respectively denoted by x j (t) and u j (t), is a boolean value equal to 1 if the corresponding cell is firing at time t and to 0 otherwise. Given the activation values x j (t)andu j (t), the value x i (t + 1) is then updated by the following equation x i (t +1)=σ ⎛ ⎝ N  j=1 a i,j · x j (t)+ M  j=1 b i,j · u j (t)+c i ⎞ ⎠ ,i=1, ,N (1) where σ is the classical hard-threshold activation function defined by σ(α)=1 if α ≥ 1andσ(α)=0otherwise. Note that Equation (1) ensures that the whole dynamics of network N is described by the following governing equation x(t +1)= σ (a · x(t) + b · u(t) + c) , (2) where x(t) =(x 1 (t), ,x N (t)) and u(t) =(u 1 (t), ,u M (t)) are boolean vec- tors describing the spiking configuration of the activation and input cells, and σ denotes the classical hard threshold activation function applied component by component. An example of such a network is given below. Example 1. Consider the network N depicted in Figure 1. The dynamics of this network is then governed by the following equation:  x 1 (t+1) x 2 (t+1) x 3 (t+1)  = σ  000 1 2 00 1 2 00  ·  x 1 (t) x 2 (t) x 3 (t)  +  10 00 0 1 2  ·  u 1 (t) u 2 (t)  +  0 1 2 0  144 J. Cabessa and A.E.P. Villa 1/2 1/2 x 3 x 2 x 1 u 1 u 2 1 1/2 1/2 Fig. 1. A simple neural network 3 Attractors The dynamics of recurrent neural networks made of neurons with two states of activity can implement an associative memory that is rather biological in its de- tails [3]. In the Hopfield framework, stable equilibrium reached by the network that do not represent any valid configuration of the optimization problem are referred to as spurious attractors. According to Hopfield et al., spurious modes can disappear by “unlearning” [3], but Tsuda et al. have shown that rational successive memory recall can actually be implemented by triggering spurious modes [17]. Here, the notions of attractors, meaningful attractors, and spurious attractors are reformulated in our precise context. Networks will then be clas- sified according to their ability to switch between different types of attractive behaviours. For this purpose, the following definitions need to be introduced. As preliminary notations, for any k>0, we let the space of k-dimensional boolean vectors be denoted by B k , and we let the space of all infinite sequences of k-dimensional boolean vectors be denoted by [B k ] ω . Moreover, for any finite sequence of boolean vectors v, we let the expression v ω = vvvv ··· denote the infinite sequence obtained by infinitely many concatenations of v. Now, let N be some network with N activation cells and M input cells. For each time step t ≥ 0, the boolean vectors x (t) =(x 1 (t), ,x N (t)) ∈ B N and u(t) =(u 1 (t), ,u M (t)) ∈ B M describing the spiking configurations of both the activation and input cells of N at time t are respectively called the state of N at time t and the input submitted to N at time t.Anin- put stream of N is then defined as an infinite sequence of consecutive inputs s =(u(i)) i∈N = u(0)u(1)u(2) ··· ∈ [B M ] ω . Moreover, assuming the initial state of the network to be x(0) = 0, any input stream s =(u(i)) i∈N = u(0)u(1)u(2) ··· ∈ [B M ] ω induces via Equation (2) an infinite sequence of consecutive states e s =(x(i)) i∈N = x(0)x(1)x(2) ···∈[B N ] ω that is called the evolution of N induced by the input stream s. Along some evolution e s = x(0)x(1)x(2) ···, irrespective of the fact that this sequence is periodic or not, some state will repeat finitely often whereas other will repeat infinitely often. The (finite) set of states occurring infinitely often in the sequence e s is denoted by inf(e s ). It can be observed that, for any evolution e s , there exists a time step k after which the evolution e s will necessarily remain confined in the set of states inf(e s ), or in other words, there exists an index k A Hierarchical Classification of First-Order Recurrent Neural Networks 145 such that x(i) ∈ inf(e s ) for all i ≥ k. However, along evolution e s , the recurrent visiting of states in inf(e s ) after time step k does not necessarily occur in a periodic manner. Now, given some network N with N activation cells, a set A = {y 0 , ,y k }⊆ B N is called an attractor for N if there exists an input stream s such that the corresponding evolution e s satisfies inf(e s )=A. Intuitively, an attractor can be seen a trap of states into which some network’s evolution could become forever confined. We further assume that attractors can be of two distinct types, namely meaningful or optimal vs. spurious or non-optimal. In this study we do not extend the discussion about the attribution of the attractors to either type. From this point onwards, we assume any given network to be provided with the corresponding classification of its attractors into meaningful and spurious types. Now, let N be some network provided with an additional type specification of each of its attractors. The complementary network N  is then defined to be the same network as N but with an opposite type specification of its attractors. 1 In addition, an input stream s of N is called meaningful if inf(e s ) is a meaningful attractor, and it is called spurious if inf(e s ) is a spurious attractor. The set of all meaningful input streams of N is called the neural language of N and is denoted by L(N ). Note that the definition of the complementary network implies that L(N  )=L(N )  . Finally, an arbitrary set of input streams L ⊆ [B M ] ω is defined as recognizable by some neural network if there exists a network N such that L(N )=L. All preceding definitions are now illustrated in the next example. Example 2. Consider again the network N describedinExample1,andsuppose that an attractor is meaningful for N if and only if it contains the state (1, 1, 1) T (i.e. where the three activation cells simultaneously fire). The periodic input stream s =[( 1 1 )( 1 1 )( 1 1 )( 0 0 )] ω induces the corresponding periodic evolution e s =  0 0 0  1 0 0  1 1 1  1 1 1  0 1 0  1 0 0  ω . Hence, inf(e s )={(1, 1, 1) T , (0, 1, 0) T , (1, 0, 0) T }, and the evolution e s of N re- mains confined in a cyclic visiting of the states of inf(e s )alreadyfromtime step t = 2. Thence, the set {(1, 1, 1) T , (0, 1, 0) T , (1, 0, 0) T } is an attractor of N. Moreover, this attractor is meaningful since it contains the state (1, 1, 1) T . 4 Recurrent Neural Networks and Muller Automata In this section, we provide an extension of the classical result stating the equiv- alence of the computational capabilities of first-order recurrent neural networks and finite state machines [5,7,8]. More precisely, here, the issue of the expressive power of neural networks is approached from the point of view of the theory of automata on infinite words, and it is proved that first-order recurrent neural 1 More precisely, A is a meaningful attractor for N  if and only if A is a spurious attractor for N . 146 J. Cabessa and A.E.P. Villa networks actually disclose the very same expressive power as finite Muller au- tomata. Towards this purpose, the following definitions first need to be recalled. A finite Muller automaton is a 5-tuple A =(Q, A, i, δ, T ), where Q is a finite set called the set of states, A is a finite alphabet, i is an element of Q called the initial state, δ is a partial function from Q × A into Q called the transition function, and T⊆P(Q) is a set of set of states called the table of the automaton. A finite Muller automaton is generally represented by a directed labelled graph whose nodes and labelled edges respectively represent the states and transitions of the automaton. Given a finite Muller automaton A =(Q, A, i, δ, T ), every triple (q, a,q  )such that δ(q, a)=q  is called a transition of A.Apath in A is then a sequence of consecutive transitions ρ =((q 0 ,a 1 ,q 1 ), (q 1 ,a 2 ,q 2 ), (q 2 ,a 3 ,q 3 ), ), also denoted by ρ : q 0 a 1 −→ q 1 a 2 −→ q 2 a 3 −→ q 3 ···. The path ρ is said to successively visit the states q 0 ,q 1 , Thestateq 0 is called the origin of ρ,theworda 1 a 2 a 3 ··· is the label of ρ,andthepathρ is said to be initial if q 0 = i.Ifρ is an infinite path, the set of states visited infinitely often by ρ is denoted by inf(ρ). Besides, a cycle in A consists of a finite set of states c such that there exists a finite path in A with same origin and ending state that visits precisely all the sates of c. A cycle is called successful if it belongs to T ,andnon-succesful otherwise. Moreover, an infinite initial path ρ of A is called successful if inf(ρ) ∈T. An infinite word is then said to be recognized by A if it is the label of a successful infinite path in A,andtheω-language recognized by A, denoted by L(A), is defined as the set of all infinite words recognized by A.Theclassofallω-languages recognizable by some Muller automata is precisely the class of ω-rational languages. Now, for each ordinal α<ω ω , we introduce the concept of an α-alternating tree in a Muller automaton A, which consists of a tree-like disposition of the successful and non-successful cycles of A induced by the ordinal α (see Figure 2). We first recall that any ordinal 0 <α<ω ω can uniquely be written of the form α = ω n p ·m p +ω n p−1 ·m p−1 + +ω n 0 ·m 0 ,forsomep ≥ 0, n p >n p−1 > > n 0 ≥ 0, and m i > 0. Then, given some Muller automata A and some ordinal α = ω n p · m p + ω n p−1 · m p−1 + + ω n 0 · m 0 <ω ω ,anα-alternating tree (resp. α-co-alternating tree) is a sequence of cycles of A (C i,j k,l ) i≤p,j<2 i ,k<m i ,l≤n i such that: firstly, C 0,0 0,0 is successful (resp. not successful); secondly, C i,j k,l  C i,j k,l+1 ,and C i,j k,l+1 is successful iff C i,j k,l is not successful; thirdly, C i,j k+1,0 is strictly accessible from C i,j k,0 ,andC i,j k+1,0 is successful iff C i,j k,0 is not successful; fourthly, C i+1,2j 0,0 and C i+1,2j+1 0,0 are both strictly accessible from C i,j m i −1,0 ,andeachC i+1,2j 0,0 is successful whereas each C i+1,2j+1 0,0 is not successful. An α-alternating tree is said to be maximal in A if there is no β-alternating tree in A such that β>α. We now come up to the equivalence of the expressive power of recurrent neural networks and Muller automaton. First of all, we prove that any first- order recurrent neural network can be simulated by some Muller automaton. Proposition 1. Let N be a network provided with a type specification of its attr actors. Then there exists a Muller automaton A N such that L(N )=L(A N ). A Hierarchical Classification of First-Order Recurrent Neural Networks 147 C 0,0 0,n 0 C 0,0 1,n 0 C 0,0 m 0 −1,n 0 . . . . . . . . .    C 0,0 0,1 C 0,0 1,1 C 0,0 m 0 −1,1    C 0,0 0,0 −→ C 0,0 1,0 −→ · · · −→ C 0,0 m 0 −1,0 −→ −→ C 1,0 0,n 1 C 1,0 1,n 1 C 1,0 m 1 −1,n 1 . . . . . . . . .    C 1,0 0,1 C 1,0 1,1 C 1,0 m 1 −1,1    C 1,0 0,0 −→ C 1,0 1,0 −→ · · · −→ C 1,0 m 1 −1,0 ··· −→ −→ ··· C 1,1 0,n 1 C 1,1 1,n 1 C 1,1 m 1 −1,n 1 . . . . . . . . .    C 1,1 0,1 C 1,1 1,1 C 1,1 m 1 −1,1    C 1,1 0,0 −→ C 1,1 1,0 −→ · · · −→ C 1,1 m 1 −1,0 ··· −→ −→ ··· Fig. 2. The inclusion and accessibility relations between cycles in an α-alternating tree Proof. Let N be given by the tuple (X, U, a, b, c), with card(X)=N , card(U )= M, and let the meaningful attractors of N be given by A 1 , ,A K .Now,consider the Muller automaton A N =(Q, A, i, δ, T ), where Q = B N , A = B M , i is the N-dimensional zero vector, δ : Q × A → Q is defined by δ(x, u)=x  if and only if x  = σ (a · x + b · u + c), and T = {A 1 , ,A K }. According to this construction, any input stream s of N is meaningful for N if and only if s is recognized by A N .Inotherwords,s ∈ L(N ) if and only if s ∈ L(A N ), and therefore L(N )=L(A N ).  According to the construction given in the proof of Proposition 1, any evolution of the network N naturally induces a corresponding infinite initial path in the Muller automaton A N , and conversely, any infinite initial path in A N corre- sponds to some possible evolution of N. This observation ensures the existence of a biunivocal correspondence between the attractors of the network N and the cycles in the graph of the corresponding Muller automaton A N .Consequently, a procedure to compute all possible attractors of a given network N is simply obtained by first constructing the corresponding Muller automaton A N and then listing all cycles in the graph of A N . Conversely, we now prove that any Muller automaton can be simulated by some first-order recurrent neural network. For the sake of convenience, we choose to restrict our attention to Muller automata over the binary alphabet B 1 . Proposition 2. Let A be some Muller automaton over the alphabet B 1 .Then there exists a network N A such that L(A)=L(N A ). Proof. Let A be given by the tuple (Q, A, q 1 ,δ,T ), with Q = {q 1 , ,q N } and T⊆P(Q). Now, consider the network N A =(X, U, a, b, c) defined as follows: First of all, X = {x i :1≤ i ≤ 2N }∪{x  1 ,x  2 ,x  3 ,x  4 }, U = {u 1 },andeach state q i in the automaton A gives rise to a two cell layer {x i ,x N+i } in the network N A as illustrated in Figure 3. Moreover, the synaptic weights between 148 J. Cabessa and A.E.P. Villa u 1 and all activation cells, between all cells in {x  1 ,x  2 ,x  3 ,x  4 },aswellasthe background activity are precisely as depicted in Figure 3. Furthermore, for each 1 ≤ i ≤ N , both cells x i and x N+i receive a weighted connection of intensity 1 2 from cell x  4 (resp. x  2 ) if and only if δ(q 1 , (0)) = q i (resp. δ(q 1 , (1)) = q i ), as also shown in Figure 3. Farther, for each 1 ≤ i, j ≤ N , there exist two weighted connection of intensity 1 2 from cell x i (resp. from cell x N+i ) to both cell x j and x N+j if and only if δ(q i , (1)) = q j (resp. δ(q i , (0)) = q j ), as partially illustrated in Figure 3 only for the k-th layer. This description of the network N A ensures that, for any possible evolution of N A , the two cells x  1 and x  3 are firing at each time step t ≥ 1, and furthermore, one and only one cell of {x i :1≤ i ≤ 2N } are firing at each time step t ≥ 2. According to this observation, for any 1 ≤ j ≤ N ,let1 j ∈ B 2N+4 (resp. 1 N+j ∈ B 2N+4 ) denote the boolean vector describing the spiking configuration where only the cells x  1 , x  3 ,andx j (resp. x  1 , x  3 ,andx N+j ) are firing. Hence, any evolution x(0)x(1)x(2) ··· of N A satisfies x(t) ∈{1 k :1≤ k ≤ N }∪{1 N+l :1≤ l ≤ N } for all t ≥ 2, and thus any attractor A of N can necessarily be written of the form A = {1 k : k ∈ K}∪{1 N+l : l ∈ L},forsomeK, L ⊆{1, 2, ,N}. Now, any infinite sequence s = u(0)u(1)u(2) ··· ∈ [B 1 ] ω induces both a corresponding infinite path ρ s : q 1 u(0) −−−→ q j 1 u(1) −−−→ q j 2 u(2) −−−→ q j 3 ··· in A as well as a corresponding evolution e s = x(0)x(1)x(2) ··· in N A . The network N A is then related to the automaton A via the following important property: for each time step t ≥ 1, if u(t) =(1),thenx(t +1) = 1 j t ,andifu(t) =(0),thenx(t +1) = 1 N+j t . In other words, the infinite path ρ s and the evolution e s evolve in parallel and satisfy the property that the cell x j is spiking in N A if and only if the automaton A is in state q j and reads letter (1), and the cell x N+j is spiking in N A if and only if the automaton A is in state q j and reads letter (0). Finally, an attractor A = {1 k : k ∈ K}∪{1 N+l : l ∈ L} with K, L ⊆{1, 2, ,N} is set to be meaningful if and only if {q k : k ∈ K}∪{q l : l ∈ L}∈T.Consequently,forany infinite infinite sequence s ∈ [B 1 ] ω , the infinite path ρ s in A satisfies inf(ρ s ) ∈T u 1 x  1 x  2 x  3 x  4 −1/2 − 1 − 1 −1 1/2 1/2 +1 +1 +1 1/2 1/2 1/2 1/2 x 1 x i x N 1/2 1/2 x 2N x j x k x N+1 +1 1/2 1/2 x N+i x N+j x N+k Fig. 3. The network N A A Hierarchical Classification of First-Order Recurrent Neural Networks 149 if and only if the evolution e s in N A is such that inf(e s ) is a meaningful attractor. Therefore, L(A)=L(N A ).  Finally, the following example provides an illustration of the two translating procedures described in the proofs of propositions 1 and 2. Example 3. The translation from the network N described in Example 2 to its corresponding Muller automaton A N is illustrated in Figure 4. Proposition 1 ensures that L(N )=L(A N ). Conversely, the translation from some given Muller automaton A over the alphabet B 1 to its corresponding network N A is illustrated in Figure 5. Proposition 2 ensures that L(A)=L(N A ). 1/2 1/2 x 3 x 2 x 1 u 1 u 2 1 1 /2 1/2  0 0 0   0 1 0   1 1 0   1 1 1   0 1 1   1 0 0  ( 1 0 ) ( 0 0 ) ( 1 1 ) ( 0 0 ) ( 0 1 ) ( 1 0 ) ( 1 1 ) ( 0 0 ) ( 0 1 ) ( 1 0 ) ( 1 1 ) ( 0 0 ) ( 1 0 ) ( 1 1 ) ( 0 0 ) ( 0 1 ) ( 1 0 ) ( 1 1 ) ( 0 0 ) ( 0 1 ) ( 1 1 ) ( 0 1 ) ( 0 1 ) ( 1 0 ) A ⊆ B 3 is meaningful for N Table T = {A ∈ B 3 : A is meaningful for N} if and only if (1, 1, 1) T ∈ A Fig. 4. Translation from a given network N provided with a type specification of its attractors to a corresponding Muller automaton A N q 1 q 2 q 3 ( 1) ( 0) ( 1) ( 1) ( 0) ( 0) u 1 x 3 x 4 x 5 x  1 x  2 x  3 x  4 −1/2 −1 −1 −1 1 1/2 1/2 1/2 1/2 1/2 1/2 +1 +1 +1 1/2 1/2 1/2 1/2 x 6 x 1 x 2 Table T = {{q 2 }, {q 3 }} Meaningful attractors: A 1 = {1 5 } and A 2 = {1 3 }. Fig. 5. Translation from a given Muller automaton A to a corresponding network N A provided with a type specification of its attractors 150 J. Cabessa and A.E.P. Villa 5 The RNN Hierarchy In the theory of automata on infinite words, abstract machines are commonly classified according the topological complexity of their underlying ω-language, as for instance in [1,2,9,19]. Here, this approach is translated from the automata to the neural networks context, in order to obtain a refined classification of first- order recurrent neural networks. Notably, the obtained classification actually refers to the ability of the networks to switch between meaningful and spurious attractive behaviours. For this purpose, the following facts and definitions need to be introduced. To b egin with, for any k>0, the space [B k ] ω can naturally be equipped with the product topology of the discrete topology over B k . Thence, a function f : [B k ] ω → [B l ] ω is said to be continuous if and only if the inverse image by f of every open set of [B l ] ω is an open set of [B k ] ω . Now, given two first-order recurrent neural networks N 1 and N 2 with M 1 and M 2 input cells respectively, we say that N 1 Wadge reduces [18] (or continuously reduces or simply reduces)toN 2 , denoted by N 1 ≤ W N 2 , if any only if there exists a continuous function f :[B M 1 ] ω → [B M 2 ] ω such that any input stream s of N 1 satisfies s ∈ L(N 1 ) ⇔ f (s) ∈ L(N 2 ). The corresponding strict reduction, equivalence relation, and incomparability relation are then naturally defined by N 1 < W N 2 iff N 1 ≤ W N 2 ≤ W N 1 ,aswell as N 1 ≡ W N 2 iff N 1 ≤ W N 2 ≤ W N 1 ,andN 1 ⊥ W N 2 iff N 1 ≤ W N 2 ≤ W N 1 . Moreover, a network N is called self-dual if N≡ W N  ;itisnon-self-dual if N ≡ W N  ,whichcanbeprovedtobeequivalenttosayingthatN⊥ W N  . By extension, an ≡ W -equivalence class of networks is called self-dual if all its elements are self-dual, and non-self-dual if all its elements are non-self-dual. Now, the Wadge reduction over the class of neural networks naturally induces a hierarchical classification of networks. Formally, the collection of all first-order recurrent neural networks ordered by the Wadge reduction “≤ W ” is called the RNN hierarchy. Propositions 1 and 2 ensure that the RNN hierarchy and the Wagner hierarchy – the collection of all ω-rational languages ordered by the Wadge reduction [19] – coincide up to Wadge equivalence. Accordingly, a precise description of the RNN hierarchy can therefore be given as follows. First of all, the RNN hierarchy is well founded, i.e. there is no infinite strictly descending sequence of networks N 0 > W N 1 > W N 2 > W Moreover, the maximal strict chains in the RNN hierarchy have length ω ω , meaning that the RNN hierarchy has a height of ω ω . Furthermore, the maximal antichains of the RNN hierarchy have length 2, meaning that the RNN hierarchy has a width of 2. 2 More precisely, any two networks N 1 and N 2 satisfy the incomparability relation N 1 ⊥ W N 2 if and only if N 1 and N 2 are non-self-dual networks such that N 1 ≡ W N  2 .These properties imply that, up to Wadge equivalence and complementation, the RNN 2 A strict chain (resp. an antichain) in the RNN hierarchy is a sequence of neural networks (N k ) k∈α such that N i < W N j iff i<j (resp. such that N i ⊥ W N j for all i, j ∈ α with i = j). A strict chain (resp. an antichain) is said to be maximal if its length is at least as large as the length of every other strict chain (resp. antichain). A Hierarchical Classification of First-Order Recurrent Neural Networks 151 hierarchy is actually a well-ordering. In fact, the RNN hierarchy consists of an alternating succession of non-self-dual and self-dual classes with pairs of non-self- dual classes at each limit level, as illustrated in Figure 6, where circle represent the Wadge equivalence classes of networks and arrows between circles represent the strict Wadge reduction between all elements of the corresponding classes. For convenience reasons, the degree of a network N in the RNN hierarchy is now defined in order to make the non-self-dual (n.s.d.) networks and the self- dual ones located just one level above share the same degree, as illustrated in Figure 6: d(N )= ⎧ ⎪ ⎨ ⎪ ⎩ 1ifL(N )=∅ or ∅  , sup {d(M)+1:M n.s.d. and M < W N} if N is non-self-dual, sup {d(M):M n.s.d. and M < W N} if N is self-dual. Also, the equivalence between the Wagner and RNN hierarchies ensure that the RNN hierarchy is actually decidable, in the sense that there exists a algorithmic procedure computing the degree of any network in the RNN hierarchy. All the above properties of the RNN hierarchy are summarized in the following result. Theorem 1. The RNN hierarchy is a decidable pre-well-ordering of width 2 and height ω ω . Proof. The Wagner hierarchy consists of a decidable pre-well-ordering of width 2 and height ω ω [19]. Propositions 1 and 2 ensure that the RNN hierarchy and Wagner hierarchy coincide up to Wadge equivalence.  height ω ω degree 1 degree 2 degree 3 degree ω degree ω +1 degree ω · 2+1 degree ω · 2 Fig. 6. The RNN hierarchy The following result provides a detailed description of the decidability procedure of the RNN hierarchy. More precisely, it is shown that the degree of a network N in the RNN hierarchy corresponds precisely to the largest ordinal α such that there exists an α-alternating tree or an α-co-alternating tree in the Muller automaton A N . Theorem 2. Let N be a network provided with a type specification of its at- tractors, A N be the corr esponding Muller automaton o f N ,andα be an ordinal such that 0 <α<ω ω . • If there exists in A N a maximal α-alternating tree and no maximal α-co- alternating tree, then d(N )=α and N is non-self-dual. [...]...152 J Cabessa and A. E.P Villa • If there exists in AN a maximal α-co-alternating tree and no maximal αalternating tree, then d(N ) = α and N is non-self-dual • If there exist in AN both a maximal α-alternating tree as well as a maximal α-co-alternating tree, then d(N ) = α and N is self-dual Proof For any ω-rational language L, let dW (L) denote the degree of L in the Wagner hierarchy On the one hand,... proposes a new approach of neural computability from the point of view infinite word reading automata theory More precisely, the Wadge classification of infinite word languages is translated from the automata-theoretic to the neural network context, and a transfinite decidable hierarchical classification of first-order recurrent neural network is obtained This classification provides a better understanding of this... computational capabilities should also rather be approached from the point of view of finite word instead of infinite word reading automata, as for instance in [6,10,11,12,13,14,15] Unfortunately, as opposed to the case of infinite words, the classification theory of finite words reading machines is still a widely undeveloped, yet promising issue Finally, the study of hierarchical classifications of neural... path in AN could maximally alternate between successful and non-successful cycles along its evolution Therefore, according to the biunivocal correspondence between cycles in AN and attractors of N , as well as between infinite paths in AN and evolutions of the network N , it follows that the complexity of a network N in the RNN hierarchy actually refers to the capacity of this network to maximally alternate... corresponding Muller automaton AN (as described in Proposition 1), and then returning the ordinal α associated to the maximal α-(co)-alternating tree(s) in contained in AN (which can be achieved by some graph analysis of the automaton AN ) In other words, the complexity of a network N is directly related to the relative disposition of the successful and non-successful cycles in the Muller automaton AN , or in... hierarchies remain hard open problems Besides, this work is envisioned to be extended in several directions First of all, it could be of interest to study the same kind of hierarchical classification A Hierarchical Classification of First-Order Recurrent Neural Networks 153 applied to more biologically oriented models, like neural networks provided with some additional simple STDP rule In addition, neural networks’... one hand, propositions 1 and 2 ensure that d(N ) = dW (L(AN )) On the other hand, the decidability procedure of the Wagner hierarchy states that dW (L(AN )) corresponds precisely to the largest ordinal α such that there exists a maximal α-(co)-alternating tree in AN [19] The decidability procedure of the degree of a network N in the the RNN hierarchy thus consists in first translating the network N into... D.I., Palmer, R.G.: ‘unlearning’ has a stabilizing effect in collective memories Nature 304, 158–159 (1983) 4 Kaneko, K., Tsuda, I.: Chaotic itinerancy Chaos 13(3), 926–936 (2003) 5 Kleene, S.C.: Representation of events in nerve nets and finite automata In: Automata Studies Annals of Mathematics Studies, vol 34, pp 3–42 Princeton University Press, Princeton (1956) 6 Kremer, S.C.: On the computational power... computational power of elman-style recurrent networks IEEE Transactions on Neural Networks 6(4), 1000–1004 (1995) 7 McCulloch, W.S., Pitts, W.: A logical calculus of the ideas immanent in nervous activity Bulletin of Mathematical Biophysic 5, 115–133 (1943) 8 Minsky, M.L.: Computation: finite and infinite machines Prentice-Hall, Inc., Upper Saddle River (1967) 9 Selivanov, V.: Wadge degrees of ω-languages of deterministic... class of neural networks that could be relevant for implementation issues Moreover, the Wadge hierarchies of deterministic pushdown automata or deterministic Turing Machines both with Muller conditions [1,9] ensure that such Wadge-like classifications of strictly more powerful models of neural networks could also be described; however, in these cases, the decidability procedures of the obtained hierarchies . Grenoble 2 Faculty of Business and Economics, University of Lausanne, CH-1015 Lausanne {jcabessa,avilla}@nhrg.org Abstract. We provide a refined hierarchical classification of first-order recurrent neural. Siegelmann and Sontag showed that, assuming rational synaptic weights, but considering a saturated-linear sigmoidal instead of a hard-threshold acti- vation function drastically increases the. antichain) is said to be maximal if its length is at least as large as the length of every other strict chain (resp. antichain). A Hierarchical Classification of First-Order Recurrent Neural Networks