EURASIPJournalonAppliedSignalProcessing2003:7,620–628c 2003HindawiPublishing Corporation Analog VLSI Circuits for Short-Term Dynamic Synapses Shih-Chii Liu Institute of Neuroinformatics, University of Zurich and ETH Zurich, Winterthurerstrasse 190, CH-8057 Zurich, Switzerland Email: shih@ini.phys.ethz.ch Received 14 May 2002 and in revised form 25 September 2002 Short-term dynamical synapses increase the computational power of neuronal networks. These synapses act as additional fi lters to the inputs of a neuron before the subsequent integration of these signals at its cell body. In this work, we describe a model of depressing and facilitating synapses derived from a hardware circuit implementation. This model is equivalent to theoretical models of short-term synaptic dynamics in network simulations. These circuits have been added to a network of leaky integrate- and-fire neurons. A cortical model of direction-selectivity that uses short-term dynamic synapses has been implemented with this network. Keywords and phrases: shor t-term synaptic dynamics, depression, facilitation, silicon synapse, cortical models. 1. INTRODUCTION Cortical neurons show a wide variety of neuronal and synap- tic responses to their input signals. Networks with simplified models of spiking neurons and synapses and consisting of one or two time constants already exhibit a large number of possible operating regimes [1, 2]. Simulations of these spik- ing networks can take a long time on a serial computer. In most network simulations, synapses are assumed to be static. Recent physiological data, however, show that synapses frequently show activity-dependent plasticity which vary on a time scale of milliseconds to seconds. In particular, short- term dynamical synapses [3, 4, 5, 6, 7] with time constants of hundreds of milliseconds are seen in many parts of the visual cortex. When these synapses are stimulated with a train of input spikes, the amplitude of the membrane po- tential of the neuron or the excitatory postsynaptic potential (EPSP) decreases (depressing synapse) or increases (facilitat- ing synapse) with each subsequent spike. The recovery time of the maximum synaptic amplitude is in the order of hun- dreds of milliseconds. These synapses encode the history of their inputs and can be treated as time-invariant filters with fading memory [8]. These activit y-dependent synapses, when added to the network, allow for different forms of dynamical networks that can process time-varying patterns [9, 10]. Examples of how these synapses could contribute to visual cortical re- sponses include direction selectivity [11] and automatic gain control [12]. The simulation time of spiking networks with different types of activity-dependent synapses consisting of different time constants will increase significantly. This simu- lation time can be shortened by using a hardware implemen- tation of a network with spiking neurons and these activity- dependent synapses. Here, we describe a circuit model of short-term synaptic dynamics based on the silicon implementation of synaptic depression and facilitation in [13]. The dynamics of this cir- cuit model is qualitatively comparable to the dynamics of two theoretical models [14]: the phenomenological model from [6, 9, 15] and the model from [ 5, 12]. Measurements from these circuits on a fabricated chip show how these synapses filter the inputs to a leaky integrate-and-fire neuron under transient and steady-state conditions. The dynamics of short-term plastic synapses are depen- dent on the frequency of the presynaptic input. In the case of a neuron which is stimulated through a depressing synapse by a regular input spike train, the firing rate of the neuron decreases over time due to the decrease in synaptic input with each presynaptic spike. Interestingly, a class of neurons in the cortex also adapt their firing rate over time in response to a regular spike input through a normal synapse. This output adaptation mechanism is noninput spe cific whereas the first mechanism involves the filtering of specific inputs. The inclusion of these short-term synapses into networks of neurons allow processing of time-varying inputs. How- ever, the simulation time of such networks on a computer increases substantial ly as more different types of time con- stants are added to the circuits. The previous constructions of neuron circuits ranging from Hodgkin-Huxley models of neurons [16, 17] to integrate-and-fire neurons [18, 19, 20, 21, 22], together with long-time constant learning synapses [23, 24] and short-term dynamic synapses [13]canbeused to develop realistic, real-time, low-power, and spike-based networks. Analog VLSI Circuits for Short-Term Dynamic Synapses 621 2. SYNAPSES Synaptic circuits have been implemented using very few tran- sistors [13, 25]. However, their dynamics are usually differ- ent from the exponential dynamics of synaptic models used in simulations. To implement the exponential dynamics, we would have to use a linear resistor to obtain the exponential dynamics. A transistor can act as a linear resistor as long as the terminal voltages satisfy certain criteria. Additional cir- cuitry would be needed to satisfy these criteria, thus increas- ing the final size of the circuit. One alternative is to replace the linear-resistor dynamics with diode dynamics which is easily obtained with one diode-connected transistor. We will discuss the difference between the diode-connected transis- tor dynamics and the exponential dynamics for the different types of synapses. 2.1. Normal synapses In simulations, the synaptic current i(t) is either treated as a point current source at the time of the spike t sp : i(t) = I f δ t − t sp , (1) where I f is a fixed current, or as a current source with a finite decay time: i(t) = I f t τ g 1 − e −t/τ g , (2) where τ g is the time constant of the decay and t is measured right after a spike. The point current source can be implemented by two transistors (e.g., M 2 and M 3 in Figure 1a). If we need a synap- tic current with a finite decay time, we include the current- mirror circuit M 1 , M 4 ,andC. Unlike the dynamics in (4), the synaptic current I d has a 1/t decay dynamics [25] rather than exponential dynamics. The decay of I d is described by I d (t) = I d0 1+ AI d0 /Q T , (3) where Q T = CU T , A = e (κV dd −V gain )/U T , U T is the thermal volt- age, and I d0 is the value of I d at the time of the spike t = t sp . 2.2. Short-term synaptic dynamics Dynamical synapses can be depressing, facilitating, or a combination of both. In a depressing synapse, the synap- tic strength decreases after each spike and recovers towards its maximal value with a time constant τ d . In facilitating synapses, the strength increases after each spike and recov- ers towards its minimum value with a time constant τ f .Two prevalent models that are used in network simulations and also for fitting physiological data are the phenomenological model in [6, 9, 15] and the model from [5, 12]. We only con- sider the dynamics of the model from Abbott et al. [12]in this work. V gain M 1 M 4 I d I r C V d M 2 V pre M 3 (a) V a M 1 I r C V d M 2 V pre M 3 V x V gain M 6 M 7 I d C 2 M 5 I syn M 4 V pre (b) Figure 1: Current-mode circuits for a normal synapse (a) and a depressing synapse (b). 2.2.1 Simulation model of short-term dynamic synapses The dynamics of the depressing synapse is similar to the adaptation dynamics of the photoreceptor. Both elements code primarily changes in the input rather than the absolute level of the input. The photoreceptor amplifies the contrast of the visual signal and has a low gain to background illumi- nation. The output of the depressing synapse codes primarily changes in the presynaptic frequency. The synaptic strength adapts to a steady-state value that is approximately inversely dependent on the input frequency. Thus, the depressing synapse acts like a band-pass filter to spike rates, much like the photoreceptor has a band-pass re- sponse to illumination. The facilitating synapse, on the other hand, acts like a low-pass filter to changes in spike rates. A step increase in presynaptic firing rate leads to an increase in the synaptic strength. Both types of synapses can be treated as time-invariant fading memory filters [8]. In the theoretical model from Abbott et al. [12], the de- pression in the synaptic strength is defined by a variable D varying between 0 and 1. The synaptic strength is given by 622 EURASIPJournalonAppliedSignalProcessing gD(t)whereg is the maximum synaptic strength. The recov- ery dynamics of D is described by τ d dD dt = 1 − D, (4) where τ d is the recovery time constant of the depression, and the update dynamics is D t + sp = dD t − sp , (5) where d (d<1) is the amount by which D is decreased right after the spike. In the case of a regular spike train, the average steady-state value of D is D= 1 − e −1/(rτ d ) 1 − de −1/(rτ d ) . (6) In the facilitating case, the facilitation is defined by a vari- able F ≥ 1. The synaptic strength is g f F(t), where g f is the maximum synaptic strength. The recovery dynamics of F is τ f dF dt = 1 − F, (7) where τ f is the time constant in which F recovers exponen- tially back to 1. The update dynamics is now additive instead of subtrac- tive: F t + sp = F t − sp + f, (8) where f ( f<1) is the amount by which F is increased right after the spike. The variable F is updated additively because multiplicative facilitation can lead to increases of synaptic strength without bounds, especially at high frequencies, for the recovery dynamics in (7). 2.2.2 Circuit model of short-term dynamic synapses As before, we replace the exponential dynamics in (4)with the diode-connected tr ansistor dynamics. This replacement gives rise to the synaptic depressing circuit in Figure 1b which was proposed in [13]. The new circuit gives rise to the following recovery dynamics for the depressing variable D: dD dt = M 1 − D 1/κ , (9) where M is the equivalent of 1/τ d and κ is a transistor pa- rameter which is less than 1 in subthreshold operation. The update dynamics are similar to (5): D t + sp = dD t − sp . (10) 2.2.3 Depressing circuit The detailed analysis leading to (9)and(10)forD is de- scribed in [14]. The voltage V a determines the maximum synaptic strength g while the synaptic strength gD or I syn is exponential in the voltage V x . The subcircuit consisting of transistors M 1 , M 2 ,andM 3 controls the dynamics of I syn . The presynaptic input goes to the gate terminal of M 3 which acts like a sw itch. During a presynaptic spike, a quantit y of charge (determined by V d ) is removed from the node V x .In between spikes, V x recovers towards V a through the diode- connected transistor M 1 . Also during the presynaptic spike, transistor M 4 turns on and the synaptic current I syn flows into the membrane potential of the neuron. We can con- vert the I syn current source into an equivalent current I d with some gain and a “time constant” through the current-mirror circuit consisting of M 6 , M 7 , and the capacitor C 2 ,andby adjusting the voltage V gain . The synaptic strength is given by I syn (t) = I on e κV x /U T = gD(t), (11) where g = I on e κV a /U T ,and D(t) = (e κ(V dd −V a )/U T I op )/I rf (t), (12) where I rf := I op e κ(V dd −V x )/U T . T he recovery time constant (1/M)ofD is set by V a (M = (I op κ/Q T )e −(1−κ)(V dd −V a )/U T ). Becauseitisdifficult to compute a closed-form solution for (9) for any value of κ, we look at a simple case where κ = 0.5andsolveforD(t) after a spike has occurred at t = t 0 . 1 The actual value of κ changes for different operating condi- tions and also depends on fabrication parameters. The re- covery equation in (13) includes the current dynamics of the diode-connected transistor (M 1 in Figure 1b) in the region when D is close to the maximum value. The equation for D(t) is then dD dt = M 1 − D 2 =⇒ D(t) = D t 0 + D t 0 e 2Mt − 1+e 2Mt −D t 0 + D t 0 e 2Mt +1+e 2Mt = D t 0 cosh (Mt) + sinh (Mt) cosh (Mt)+D t 0 sinh (Mt) . (13) If D is not close to its maximum value of 1, we can ap- proximate the dynamics to dD/dt = M (regardless of κ)and solve for D(t): D(t) = Mt + D t 0 . (14) In this regime, D(t) fol lows a linear trajectory. Note that the same is true for (4) when t τ d . 2.2.4 Model of facilitating synapse The schematic for the facilitating synapse is shown in Figure 2.Thedifference in this circuit from the depress- ing synaptic circuit is that the node V x goes to the gate of a pFET instead of an nFET. The synaptic strength is now I syn (t) = I op e κ(V dd −V x )/U T and is directly proportional to the current variable I rf = I op e κ(V dd −V x )/U T ,so I syn (t) = g f F(t), (15) where g f = I op e κ(V dd −V a )/U T and F(t) = 1/D(t). 1 Note that if κ =1, then the equation reduces to (4). Analog VLSI Circuits for Short-Term Dynamic Synapses 623 V a M 1 I r C 1 V d M 2 V pre M 3 V x V preb M 4 I syn M 5 I d C 2 M 6 M 7 V b Figure 2: Synaptic facilitation circuit. The circuit on the left is the same as part of the circuit in Figure 1b. The voltage V x determines thesynapticstrengthandthecurrentI d goes to the neuron. This circuit would have to be inverted so that it can be combined with the neuron circuit in Figure 3. The update dynamics is multiplicative instead of additive as in Abbott’s model: F t + n = fF t − n , (16) where f = 1+a ≥ 1. The recovery dynamics is given by dF dt = MF 2 1 F 1/κ − 1 , (17) where M =(I op κ/Q T )e −(1−κ)(V dd −V a )/U T . In steady state, F =1. Using κ = 0.5, the equation for F(t)is dF dt = M 1 − F 2 =⇒ F( t) = F t 0 + F t 0 e 2Mt − 1+e 2Mt −F t 0 + F t 0 e 2Mt +1+e 2Mt = F t 0 cosh (Mt) + sinh (Mt) cosh (Mt)+F t 0 sinh (Mt) . (18) However , if F is far from its resting value of 1, we obtain the simpler dynamics dF/dt =−MF(t) 2 and solve for F(t): F(t) = F t 0 1+MtF t 0 . (19) The circuit model for facilitation is quite dissimilar to (7) and (8). Even though the update is multiplicative, the vari- able F will not increase without bounds because the recov- ery dynamics of the diode-connected transistor which is a negative-feedback element. In Section 5, we will see that the steady-state value of F is approximately linear in the presy- naptic rate r. 3. NEURON CIRCUIT The dynamics of the neuron circuit are similar to that of a leaky integrate-and-fire neuron with a constant leak (Figure 3). The circuit is described in detail in [26, 27]. It is a modified version of previous designs [18, 22] and also includes the circuitry which models firing-rate adaptation [21, 25]frequentlyseeninpyramidalcells.Theequationfor the depolarization of the soma is as follows: C m dV m (t) dt = i(t) − I leak − I ahp ,V m (t) <V thresh , (20) where i(t) is the synaptic current to the soma, I leak is the leak- age current, and I ahp is the after-hyperpolarization potassium (K) current which causes the adaptation in the firing rate of the cells. When V m (t) increases above V thresh at t = t s (t s is the time of spike), it increases by a step increment determined by the capacitive coupling C 1 and C m .TheoutputV o becomes ac- tive at this time and turns on the discharging current path through transistors M 5 and M 6 . The time during which V o remains high, T P , depends on the time taken for V m to dis- charge below V thresh . In this design, the pulse width T P is determined by the rate at which V m is discharged which in turn depends on the difference between the input current I d , the leak current I leak , and the current I pw . In other designs, V m is reset immediately below V thresh when V o becomes a c- tive because either the input current is blocked from charging the membrane or the current I pw is much larger than the in- put cur rent. T he refractory period T R is determined by V refr which keeps V o high so that I d cannot charge up the mem- brane. The spike output is taken from the node V o . The time taken for the neuron to charge up to threshold is T I = C m + C 1 V thresh i − I leak , (21) and, in the case of a constant input current I d , the spike rate is r = 1 T I + T P + T R . (22) Spike adaptation Transistors M 1 to M 4 and the capacitor C a in Figure 3 imple- ment the spike adaptation mechanism. The data in Figure 8b show the adaptation of the output spike rate when the neu- ron was driven by a 100 Hz regular input spike train through a nonplastic synapse. The amount of charge dumped onC a is determined by V ca . The dynamics of the current mirror cir- cuit (M 3 , M 4 ,andC a ) are used to set the dynamics of the I ahp current. The adapted spike rate is reduced from the initial rate by a factor γ = (1+A c q a /Q th )[21], where q a is the charge that is dumped onto the capacitor C a during each postsynap- tic spike (i.e., when V o is high), Q th is the amount of charge needed for V m to reach threshold, and A c = e κV t . 4. TRANSIENT RESPONSE The data in the figures in the remainder of the paper are obtained from a multineuron circuit with depressing and 624 EURASIPJournalonAppliedSignalProcessing V leak I leak I d C m I pw M 5 M 6 V pw V o V m V thresh C 1 V b V o V refr V o V m M 1 M 2 M 3 M 4 V o V ca V t C a Spike adaptation circuitry + − Figure 3: Schematic of the leaky integrate-and-fire neuron. The parameters, V refr sets the refractory period, V thresh sets the threshold voltage, V pw sets the pulse width of the spike, and V leak sets the leak current I leak . The circuit within the dashed-dotted inset implements the spike adaptation mechanism. The parameters V Ca and V t set the adaptation dynamics. facilitating synapses fabricated in a 0.8 µm CMOS process. To show the effect of synaptic depression, we measured V x over time as the input was dr iven by a regular spike train as shown in Figure 4. Remember that the synaptic strength gD is expo- nential in V x . When there are no spikes, V x is approximately equal to V a . During a spike, V x isdecreasedbyanamount dependent on V d . This node recovers in-between spikes at a rate that depends on the difference in voltage between V x and V a . The recovery rate is faster when V x is far from V a . The de- pendence of the recovery rate on this difference is due to the current-mirror circuit dynamics. The parameter V a controls both the synaptic strength and the recovery time constant. For a fixed V a , the dynamics and the steady-state value of D can be set by changing V d (or d) as shown in Figure 4b. The subsequent effect on the neuron is seen by measur- ing the EPSP response when a presynaptic spike occurs. The EPSPs recorded when the neuron was stimulated by a regular spiking input through these synapses are shown in Figure 5. The parameters of the synapse and the neuron have been tuned so that the EPSPs do not add up with each incom- ing spike. In Figure 5a, the EPSP amplitude decreases with each incoming spike, while in Figure 5b the amplitude in- creases instead. The EPSPs in response to the first few spikes in Figure 5b are not observable because the leak current is larger than the synaptic current. The amplitude reaches a steady-state value after a finite number of spikes. The num- ber of spikes needed to reach a steady state can be tuned by the parameters V a and V d .Different V d and V f values lead to different amounts of depression and facilitation as shown in Figure 6. The fits between the circuit model and the simu- lation model are described in [14]. 4.1. Depression and facilitation We can obtain a combination of facilitation and depres- sion dynamics in I d from the depressing synaptic circuit in Figure 1b by choosing cer tain circuit parameters. The out- put of the current-mirror synaptic circuit in Figure 1a can produce paired-pulse facilitation [25]. The equation for I d is the same as (15)forI syn in the facilitating synapse circuit I d = g fd F d (t), where g fd = I op e κ(V dd −V gain )/U T , F d (t) = I rfd (t) e κ(V dd −V gain )/U T I op , (23) and I rfd := I op e κ(V dd −V dx )/U T . This equation also applies to I d in Figure 1b.Thedifference between both circuits is that the factor f d that determines the change in F d right after a spike is constant in one circuit and varies for the other circuit ( f d = e κI rf δ t /Q T ). In the depressing synaptic circuit, I rf is not constant and depends on the input spike activity, whereas in the current-mirror synaptic circuit, I rf (= I r ) is constant. So, for certain parameter settings in the depressing circuit, the EPSPs show initial facilitation before depressing in response to a step input of a regular 100 Hz spike train as shown in Figure 7. 4.2. Depression or adaptation Both the synaptic depression and spike adaptation mecha- nisms lead to adaptation in the neuron’s firing rate to a step increase in the input rate as shown in Figure 8. In fact, the transient response to a step increase in the rate of a regular spiking input is almost indistinguishable using either mecha- nism. Although both mechanisms lead to gain control in the neuron, the individual mechanisms are sensitive to different signals. Synaptic depression gives rise to sensitivity in input rate changes whereas spike adaptation makes the neuron sen- sitive to changes in the neuron’s output rate. For example, if one of the inputs to the neuron is highly active, the spike adaptation mechanism of the neuron reduces its sensitivity to the continuous large input current regardless of the origin of the large input. On the other hand, the synaptic depressing mechanism only turns down the sensitiv ity of that particular active input so the neuron is still selective to all other inputs. The role of depression and facilitation in implementing gain control has been descr ibed in [12, 28, 29]. 5. STEADY-STATE RESPONSE The dependence of the steady-state values of D and F on the presynaptic frequency can be determined easily in the case of a regular spiking input. In the case of depression, we use (10) and (13) to compute the steady-state value of D: Analog VLSI Circuits for Short-Term Dynamic Synapses 625 0.04 0.06 0.08 0.1 0.12 0.14 0.16 Time (s) 0.1 0.15 0.2 0.25 0.3 0.35 V x (V) Update Slow recovery Fast recovery V d = 0.3V (a) 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 Time (s) 0.15 0.2 0.25 0.3 0.35 V x (V) V d = 0.24 V 0.26 V 0.28 V 0.3 V (b) Figure 4: Response of V x to a regular spiking input of 20 Hz with different values of V d . (a) Change of V x over time. It is decreased when an input spike arrives and it recovers back to the quiescent value at different rates dependent on its distance from the resting value of about 0.33 V. (b) The steady-state value and dynamics of V x can be tuned by changing V d . D ss = (−1+d) 1+e 2M/r + 4d −1+e 2M/r 2 +(1−d) 2 1+e 2M/r 2 2d −1+e 2M/r . (24) For the simpler dynamics of dD/dt = M, we use (14) instead of (13) and obtain a simpler expression for D ss : D ss = M (1 − d)r . (25) Thus the steady-state EPSP amplitude is inversely dependent on the presynaptic rate r as shown in Figure 9.Theformof the curve is similar to the results obtained in the work of Ab- bott et al. [12] where the data can be fitted with (6). 0.1 0.2 0.3 0.4 Time (s) 0 1 2 3 4 V m (V) (a) 0.15 0.2 0.25 0.3 0.35 0.4 Time (s) 0 0.5 1 1.5 2 2.5 V m (V) (b) Figure 5: Transient response of a neuron (by measuring its mem- brane potential, V m ) when stimulated by a regular spiking input through a depressing synapse (a) and a facilitating synapse (b). The leak current of the neuron has been adjusted so that the neuron does not reach threshold. (a) The EPSP decreases with each incom- ing input spike for a depressing synapse. (b) The EPSP increases with each incoming input spike for a facilitating synapse. The initial EPSPs are not seen because the leak current of the neuron is larger than the synaptic current. In the case of facilitation, we use (16)and(18)tocom- pute the steady-state value, F ss : F ss = (−1+ f ) 1+e 2M/r + 4 f 1−e 2M/r 2 +(1− f ) 2 1+e 2M/r 2 2 f −1+e 2M/r . (26) In the simpler case, where dF/dt =−MF(t) 2 , F ss = ( f − 1)r fM , (27) 626 EURASIPJournalonAppliedSignalProcessing 0.2 0.4 Time (s) 0 2 4 6 8 10 V m (V) V d = 0.2V V d = 0.3V V d = 0.35 V (a) 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 Time (s) 0 1 2 3 4 V m (V) V f = 0.35 V V f = 0.4V V f = 0.45 V (b) Figure 6: Transient response of a neuron to a regular spiking in- put for various values of V d and V f . (a) The amount by which each EPSP depresses for each subsequent pulse is set by V d .(b)The amount by which each EPSP facilitates is set by V f . which shows that the steady-state value of F is linear in the presynaptic rate and it does not increase without bounds as in the case of the exponential dynamics model for F. 6. DIRECTION SELECTIVITY USING SHORT-TERM SYNAPTIC DEPRESSION Depressing synapses have been implicated in the appear- ance of certain visual cortical cell responses, for example, direction-selectivity. Because these synapses act like a hig h- pass filter in the frequency domain, the response of the neu- ron shows a phase advance over its response if stimulated through a nonplastic synapse. This feature was exploited in a model that described the direction-selective responses of visual cortical neurons [11]. In this model, the neuron was driven by the outputs of a set of cells in the lateral geniculate nucleus (LGN) through depressing synapses and the outputs 051015 Spike number 0.05 0.1 0.15 0.2 0.25 0.3 ∆V m (V) (a) 02 4 6 81012 Spike number 0 0.2 0.4 0.6 0.8 1 1.2 1.4 ∆V m (V) (b) Figure 7: Change in the membrane potential for two different set- tings of V d and V a . (a) There is initial facilitation of the EPSPs be- fore depression. V d = 0.75 V and V a = 0.2 V. (b) Only depression is seen in the EPSPs. V d = 0.817 V and V a = 0.3 V. of a spatially shifted set of LGN cells through nondepress- ing synapses. We have attempted the same experiment by driving a “cortical neuron” on our chip with spikes recorded from an LGN cell in the cat visual cortex during stimulation with a drifting sinusoidal gra ting (courtesy of K. Martin) and a temporally shifted version of these spikes. An example of the direction-selective response is shown in Figure 10 [30]. The direction-selective results were qualitatively similar to the data in [11]. This chip has been used for exploring other spike-based cortical models, for example, orientation selec- tivity [27]. Analog VLSI Circuits for Short-Term Dynamic Synapses 627 0 0.2 0.4 0.6 0.8 1 Time (s) 0 1 2 3 4 5 6 Membrane potential (V) (a) 0 0.2 0.4 0.6 0.8 1 Time (s) 0 1 2 3 4 5 6 Membrane potential (V) (b) Figure 8: Mechanisms for spike frequency adaptation. (a) Adap- tation due to synaptic depression. Different adapted rates are ob- tained by using V d = 0.2 V (top curve) and V d = 0.1V (bottom curve). (b) Adaptation due to different after-hyperpolarization cur- rents (V ca = 4.3 V (top curve) and V ca = 4.5 V (bottom curve)). The sharp excursions of the membrane potential represent the out- put spikes of the neuron. 7. CONCLUSION The addition of short-term dynamical synapses to neuronal networks increases the computational power of such net- works, especially in processing time-varying inputs. Because of the similarity of the dynamics of the silicon models to the theoretical models, a silicon network of leaky integrate- and-fire neurons which incorporate these synapses can pro- vide an alternative to network simulations on the computer. 50 100 150 200 250 300 f in (Hz) 0 0.05 0.1 ∆V m (V) Figure 9: Average steady-state EPSP amplitude versus input fre- quency in the case of a depressing synapse. The curve shows an in- verse dependence of the amplitude on the frequency. 00.511.52 Time (s) 0 2 4 6 8 10 V m (V) Figure 10: Response to a drifting 1-Hz sinusoidal grating. The spikes from an LGN cell in the cat visual cortex in response to the drifting grating are depicted at the bottom of the curve. The spikes from a putative spatially shifted LGN cell were generated from these spikes by shifting them in time by 60 ms. The top curve shows the response of the silicon “cortical neuron” when the stimulus drifted in the preferred direction. The sharp excursions at the top of the po- tential are the output spikes of the neuron. The middle curve shows the response to the stimulus in the null direction. The membrane potential did not build up to threshold. Figure adapted from in [14, Figure 8] with permission. This type of spike-based network runs in real-time and the computational time does not scale with the size of the net- work. This chip is a basic module in a reconfigurable, rewire- able, and spike-based system that provides ease for prototyp- ing computational models. The system can also be useful for possible applications, for example, in interfacing with neural wetware. 628 EURASIPJournalonAppliedSignalProcessing ACKNOWLEDGMENTS I acknowledge Pascal Suter and Malte Boegershausen for some of the chip data and the simulation results in this pa- per. I also acknowledge Kevan Martin, Pamela Baker, and Ora Ohana for discussions on dynamic synapses. This work was supported in part by the Swiss National Foundation Research SPP grant. REFERENCES [1] C. van Vreeswijk and H. Sompolinsky, “Chaotic balanced state in a model of cortical circuits,” Neural Computation,vol. 10, no. 6, pp. 1321–1371, 1998. [2] N. Brunel, “Dynamics of sparsely connected networks of ex- citatory and inhibitory spiking neurons,” Journal of computa- tional neuroscience, vol. 8, no. 3, pp. 183–208, 2000. [3] K.Tarczy-Hornoch,K.A.Martin,J.J.Jack,andK.J.Stratford, “Synaptic interactions between smooth and spiny neurones in layer 4 of cat visual cortex in vitro,” Journal of Physiology, vol. 508, no. Pt 2, pp. 351–363, 1998. [4] K.Tarczy-Hornoch,K.A.Martin,K.J.Stratford,andJ.J.Jack, “Intracortical excitation of spiny neurons in layer 4 of cat stri- ate cortex in vitro,” Cerebral Cortex, vol. 9, no. 8, pp. 833–843, 1999. [5] J. A. Varela, K. Sen, J. Gibson, J. Fost, L. F. Abbott, and S. B. Nelson, “A quantitative description of short-term plasticity at excitatory synapses in layer 2/3 of rat primary visual cortex,” J. Neurosci., vol. 17, no. 20, pp. 7926–7940, 1997. [6] H. Markram, Y. Wang, and M. Tsodyks, “Differential signal- ing via the same axon of neocortical pyramidal neurons,” Proc. Natl. Acad. Sci. USA, vol. 95, no. 9, pp. 5323–5328, 1998. [7] A.Reyes,R.Lujan,A.Rozov,N.Burnashev,P.Somogyi,and B. Sakmann, “Target-cell-specific facilitation and depression in neocortical circuits,” Nat. Neurosci., vol. 1, no. 4, pp. 279– 285, 1998. [8] W. Maass and E. D. Sontag, “Neural systems as nonlinear fil- ters,” Neural Computation, vol. 12, no. 8, pp. 1743–1772, 2000. [9] M. Tsodyks, K. Pawelzik, and H. Markram, “Neural networks with dynamic synapses,” Neural Computation, vol. 10, no. 4, pp. 821–835, 1998. [10] W. Maass and A. Zador, “Computing and learning with dy- namic synapses,” in Pulsed Neural Networks,W.Maassand C. Bishop, Eds., chapter 6, pp. 321–336, MIT Press, Cam- bridge, Mass, USA, 1999. [11] F.S.Chance,S.B.Nelson,andL.F.Abbott,“Synapticdepres- sion and the temporal response characteristics of v1 cells,” The Journal of neuroscience, vol. 18, no. 12, pp. 4785–4799, 1998. [12] L. F. Abbott, J. A. Varela, K. Sen, and S. B. Nelson, “Synaptic depression and cortical gain control,” Science, vol. 275, no. 5297, pp. 220–224, 1997. [13] C. Rasche and R. Hahnloser, “Silicon synaptic depression,” Biological Cybernetics, vol. 84, no. 1, pp. 57–62, 2001. [14] M. Boegerhausen, P. Suter, and S C. Liu, “Modeling short- term synaptic depression in silicon,” Neural Computation, vol. 15, no. 2, pp. 331–348, 2003. [15] M. V. Tsodyks and H. Markram, “The neur al code between neocortical pyramidal neurons depends on neurotransmitter release probability,” Proc.Natl.Acad.Sci.USA,vol.94,no.2, pp. 719–723, 1997, erratum in Proc. Natl. Acad. Sci. USA, vol. 94, no. 10, pp. 54–95, May 13. [16] M. Mahowald and R. Douglas, “A silicon neuron,” Nature, vol. 354, no. 6354, pp. 515–518, 1991. [17] G. N. Patel and S. P. DeWeerth, “Analogue VLSI Morris-Lecar neuron,” Electronics letters, vol. 33, no. 12, pp. 997–998, 1997. [18] C. Mead, Analog VLSI and Neural Systems, Addison-Wesley, Reading, Mass, USA, 1989. [19] R. Sarpeshkar, L. Watts, and C. Mead, “Refractor y neuron circuits,” Tech. Rep. CNS-TR-92-08, California Institute of Technology, Pasadena, Calif, USA, 1992. [20] J. P. Lazzaro and J. Wawrzynek, “Low-power silicon axons, neurons, and synapses,” in Silicon Implementations of Pulse Coded Neural N etworks, M. E. Zaghloul, J. L. Meador, and R. W. Newcomb, Eds., pp. 153–164, Kluwer Academic Press, Norwell, Mass, USA, 1994. [21] K. A. Boahen, “The retinomorphic approach: Pixel-parallel adaptive amplification, filtering, and quantization,” Analog Integrated Circuits and Signal Processing,vol.13,no.1-2,pp. 53–68, 1997. [22] A. van Schaik, “Building blocks for electronic spiking neural networks,” Neural Networks, vol. 14, no. 6/7, pp. 617–628, 2001, Special Issue on Spiking Neurons in Neuroscience and Technology. [23] P. Hasler, C. Diorio, B. A. Minch, and C. Mead, “Single transistor learning synapses,” in Advances in Neural Informa- tion Processing Systems, vol. 7, pp. 817–824, MIT Press, Cam- bridge, Mass, USA, 1995. [24] P. H ¨ afliger and M. Mahowald, “Spike based normalizing Heb- bian learning in an analog VLSI artificial neuron,” in Learn- ing on Silicon, G. Cauwenberghs and M. A. Bayoumi, Eds., pp. 131–142, Kluwer Academic Press, Norwell, Mass, USA, 1999. [25] K. A. Boahen, Retinomorphic Vision Systems: Reverse Engi- neering the Vertebrate Retina, Ph.D. thesis, California Institute of Technology, Pasadena, Calif, USA, 1997. [26] G. Indiveri, “Modeling selective attention using a neuromor- phic analog VLSI device,” Neural Computation, vol. 12, no. 12, pp. 2857–2880, 2000. [27]S C.Liu,J.Kramer,G.Indiveri,T.Delbr ¨ uck, T. Burg, and R. Douglas, “Orientation-selective aVLSI spiking neurons,” Neural Networks, vol. 14, no. 6-7, pp. 629–643, 2001, Special Issue on Spiking Neurons in Neuroscience and Technology. [28] W. Senn, I. Segev, and M. Tsodyks, “Reading neuronal syn- chrony with depressing synapses,” Neural Computation,vol. 10, no. 4, pp. 815–819, 1998. [29] V. Matveev and X. Wang, “Differential short-term synaptic plasticity and transmission of complex spike trains: to depress or to facilitate?,” Cerebral Cortex, vol. 10, no. 11, pp. 1143– 1153, 2000. [30] S C. Liu, “Simple cortical modeling with aVLSI spiking neu- rons and dynamic synapses,” in ZNZ Sy mposium, University of Zurich, Zurich, 2001. Shih-Chii Liu received her B.S. degree in electrical engineering from Massachusetts Institute of Technology in 1983, and the M.S. degree in electrical engineering from University of California, Los Angeles in 1988. She received her Ph.D. degree in the Computation and Neural Systems Program from California Institute of Technology in 1997. She is currently an oberassistentin at the Institute of Neuroinformatics, Univer- sity of Zurich/ETH Zurich in Zurich, Switzerland. Dr . Liu was with Gould American Microsystems from 1983 to 1985, and with LSI Logic from 1985 to 1988. She worked at Rockwell International Research Labs from 1988 to 1997. Her research interests include neuromorphic neuronal modeling of vision and cortical process- ing, networks for behavior generation, and hybrid analog/digital signal processing. . EURASIP Journal on Applied Signal Processing 2003: 7, 620–628 c 2003 Hindawi Publishing Corporation Analog VLSI Circuits for Short-Term Dynamic Synapses Shih-Chii Liu Institute. synaptic dynamics, depression, facilitation, silicon synapse, cortical models. 1. INTRODUCTION Cortical neurons show a wide variety of neuronal and synap- tic responses to their input signals Nelson, “Synaptic depression and cortical gain control,” Science, vol. 275, no. 5297, pp. 220–224, 1997. [13] C. Rasche and R. Hahnloser, “Silicon synaptic depression,” Biological Cybernetics,