Evolution of Reinforcement Learning in Foraging Bees: A Simple Explanation for Risk Averse Behavior Yael Niv - Department of Psychology, Tel-Aviv University, Israel Email: yaeln@cns.tau.ac.il Daphna Joel - Department of Psychology, Tel-Aviv University, Israel Email: djoel@post.tau.ac.il Isaac Meilijson - School of Mathematical Sciences, Tel-Aviv University, Israel Email: isaco@math.tau.ac.il Eytan Ruppin - School of Computer Sciences & Department of Physiology and Pharmacology, Tel-Aviv University, Israel Email: ruppin@math.tau.ac.il Corresponding Author: Yael Niv Department of Psychology Tel-Aviv University Ramat-Aviv, Tel-Aviv 69978 Israel Tel +972-51-883178 Fax +972-3-6421889 Abstract Reinforcement learning is a fundamental process by which organisms learn to achieve a goal from interactions with the environment We use Evolutionary Computation techniques to derive (near-)optimal neuronal learning rules in a simple neural network model of decision-making in simulated bumblebees foraging for nectar The resulting bees exhibit efficient reinforcement learning The evolved synaptic plasticity dynamics give rise to varying exploration/exploitation levels and to well-documented animal foraging strategies These behaviors are shown to emerge directly from optimal reinforcement learning, which provides a biologically founded, parsimonious and novel explanation of risk-aversion Our results are corroborated by a rigorous mathematical analysis and by experiments in mobile robots Preferred mode of presentation: Oral presentation Introduction Reinforcement learning (RL) is a process by which organisms learn from their interactions with the environment to achieve a goal 1, and has been indicated as a fundamental means by which experience changes behavior in both vertebrates and invertebrates2 In RL, learning is contingent upon a scalar reinforcement signal that provides evaluative information about how good an action is, without providing an instructive cue to the most rewarding behavior RL has been demonstrated and studied excessively in the ethology of foraging bumble-bees Real 3,4 showed that when foraging for nectar in a field of blue and yellow artificial flowers, bees not only exhibit efficient RL, but also show a strong preference for landing on constant rewarding flowers, as opposed to variably rewarding flowers yielding the same mean reward Such risk-averse behavior5 has classically been accounted for by hypothesizing the existence of a subjective non-linear concave “utility function” for nectar6,7 RL has attracted ample attention in computational neuroscience, yet a fundamental question regarding the underlying mechanism has not been sufficiently addressed, namely, what are the optimal neuronal learning rules for maximizing reward in RL? And, given such rules, what foraging strategies emerge from optimal RL? In a previous neural network (NN) model, Montague et al simulated bee foraging in a 3D arena of flowers, based on a neurocontroller modelled after an identified interneuron in the honeybee suboesophogeal ganglion 9,10 While this model replicates Real’s foraging results and provides a basic and simple NN architecture to solve RL tasks, many aspects of the model, first and foremost the handcrafted synaptic learning rule, are arbitrarily specified and their optimality questionable Towards this end, we use a generalized and parameterized version of this model in order to determine the optimal synaptic learning rules for RL (with respect to maximizing nectar intake) using a genetic algorithm11 Previous attempts at evolving neuronal learning rules have used heavily constrained network dynamics and very limited sets of learning rules 12-15, or evolved only a subset of the learning rule parameters 16 We define a general framework for evolving learning rules, which encompasses all heterosynaptic Hebbian learning rules, along with other characteristics of the learning dynamics, such as learning dependencies between modules Preferred mode of presentation: Oral presentation (a) Regular Input Module B Y Different ial Input Module N B Reward Module Synapses N (b) W R (c) Y ij P Init ial Weight s m b Hebb Coefficient s Dependencies Figure 1: (a) The bee's neural network controller (b) The bee's action function (c) The genome sequence of the bee The Model A simulated bee flies above a 3D patch of 60x60 randomly scattered blue and yellow flowers In each trial the bee descends from height ~10, advancing in steps of unit in any downward heading direction The bee views the world through a cyclopean eye (10 cone view), and in each timestep decides whether to maintain the current heading direction or to change direction randomly, according to the visual inputs Upon landing, the bee consumes the nectar in the chosen flower and another trial begins A bee’s life consists of 100 trials The evolutionary goal (the fitness criterion) is to maximize nectar intake In the NN controlling the bee's flight (Fig 1a), three modules contribute their input via synaptic weights to a linear neuron P, whose continuous-valued output is P(t) = R(t) + (iregular)Wi(t)Xi(t) + (jdifferential)Wj(t)*[Xj(t) - Xj(t-1)] (1) The bee's action is determined according to the output P(t) using Montague et al's probabilistic action function6,8 (Fig 1b) p(change direction) = / [1 + exp(m*P(t) + b)] (2) During a bee's "lifetime" the synaptic weights are modified via a generalized heterosynaptic Hebb learning rule of the form Wi = [AXi(t)P(t) + BXi(t) + CP(t) + D], (3) where is a global learning rate parameter, X i(t) and P(t) are the presynaptic and the postsynaptic values respectively, Wi their connection weight, and A-D are real-valued evolvable parameters In addition, we allow for learning in one module to be dependent on the activity of another module via axo-axonical connections simulating gating of synaptic plasticity The simulated bee's “genome” (Fig 1c) consists of a string of 28 genes, each representing a parameter governing the network architecture or learning dynamics Using a genetic algorithm11, the bees were “evolved” and the optimal gene values determined In each generation one of the two flower types was randomly assigned as a constant-yielding flower (containing 0.7l nectar), and the other a variable-yielding flower (1l nectar in 1/5th of the flowers and zero otherwise) Reward contingencies were switched between the flower types in a randomly chosen trial during each bee's life Evolution of Reinforcement Learning About half of the evolutionary runs were successful runs, in which rewarddependent choice behavior was evolved These runs reveal that an interaction between eight genes governing the network structure and learning dependencies is essential for producing efficient learning in the bee's uncertain environment We find that in our framework only a network architecture similar to that used by Montague et al can produce above-random foraging behavior, supporting their choice as an optimal one However, our optimized networks utilize a heterosynaptic learning rule different from that used by Montague et al., which gives rise to several important behavioral strategies Examining the behavior of the evolved bees, we find that there are individual differences between the bees in their degree of exploitation of the high-rewarding flowers versus exploration of the other flowers (Figure 2(a,b)) This can be explained by differences in the Hebb rule coefficients of the evolved bees: In the common case when upon landing the bee sees only one color, the update rule for the corresponding differential synapse is Wi = [(A-C)*(-1)*(R(t)-Wi) + (D-B)] (4) leading to an effective monosynaptic coefficient of (A-C), and a general weight decay coefficient (D-B) For the rest of the differential synapses, the learning rule is Wj = [(C)(R(t)-Wi) + D] (5) (a) Preference for blue flow ers 0.8 0.6 0.4 0.2 0 25 50 75 100 (b) 0.8 0.6 0.4 0.2 0 25 50 75 Trials 100 Preference Preference for constant rew arding flow ers for blue flow ers Thus, positive C and D values result in spontaneous strengthening of competing synapses, leading to an exploration-inclined bee, while negative values result in exploitation-inclined behavior (c) 0.8 0.6 0.4 0.2 0 25 50 75 100 (d) 0.7 0.6 0.5 0.4 0.25 0.5 0.75 Learning rate Figure 2: (a,b) Preference for blue flowers for two different bees from the last generation of two successful runs, averaged over 40 test bouts, 100 trials each Blue is the initial constantrewarding flower, reward contingencies are switched at trial 50 (c) Previously evolved bees tested in a risk-aversion scenario show a marked preference for the constant-yielding flower (d) Risk aversion is ordered according to learning rate Emergence of Risk Aversion A prominent strategy exhibited by the evolved bees is risk-aversion Figure 2(c) shows that when tested in a new environment, where the mean rewards of the two kinds of flowers are identical, the previously evolved bees pronouncedly prefer the constantrewarding flower In contradistinction to the conventional explanations of risk aversion, our model does not include a non-linear utility function What hence brings about riskaverse behavior? Corroborating previous numerical results 17, we prove analytically that this is a direct consequence of Hebbian learning dynamics in an armed-bandit-like RL situation The bee's stochastic foraging decisions can be formally modeled as choices between a variable-rewarding(v) and a constant-rewarding(c) flower, based on memory (synaptic weights) We consider the bee's long-term choice dynamics as a sequence of N cycles, each choice of (v) beginning a cycle The frequency f v of visits to (v) can be determined (via Birkhoff's Ergodic theorem 18) by the expected number of visits to (c) in a cycle, an is fv = 1/E[1/pv(Wv,Wc)] (6) where pv(Wv,Wc) is the probability of choosing (v) in a trial in which the synaptic weights are Wv and Wc We show that if pv(Wv,Wc) is a positive increasing choice function such that [1/pv(Wv,Wc)] is convex, the risk order property of W v() always implies risk-averse behavior, i.e for every learning rate, the frequency of visits to the variable flower is less than 50%, further decreasing under higher learning rates Our simulations corroborate this analytical result (Figure 2d) (b) Yellow synapse Blue synapse (a) 0.5 0.5 1 0.5 0.5 1 BBBBBYYBBBBBBYYYYYYY 10 Trials 15 20 Figure 3: Synaptic weights of a mobile robot incorporating a NN controller of one of the previously evolved bees, performing 20 foraging trials (blue flowers - ½l nectar, yellow - 1l in half the flowers, contingencies switched at trial 10) Top – flower color chosen in each trial In order to assess the robustness of the evolved RL algorithm, we implemented it in a mobile Khepera mini-robot Moving with continuous speed and performing calculations in real-time in a noisy real-world environment, the foraging robot exhibited behavior fully analogous to that of the simulated bees (Figure 3) In summary, we present a novel model of evolved embodied reinforcement learning agents, which enables their concomitant study on both the macro-behavioral and micro-dynamic levels We further show that risk-averse behavior emerges directly from the evolved learning rule dynamics, making additional assumptions conventionally used to explain this behavior redundant References: Sutton R.S and Barto A.G., Reinforcement learning: An Introduction (MIT Press, 1998) Donahoe J.W and Packard-Dorsel V eds., Neural Network Models of Cognition: Biobehavioral Foundations (Elsevier Science, 1997) Real L.A., Animal Choice Behavior and the Evolution of Cognitive Architecture, Science 253, 980-986 (1991) Real L.A , Paradox, Performance and the Architecture of Decision Making in Animals, American Zoologist 36, 518-529 (1996) Kacelnik A and Bateson M., Risky Theories - The effects of Variance on Foraging Decisions, American Zoologist 36, 402-434 (1996) Harder L.D and Real L.A., Why are Bumble-bees Risk Averse? Ecology 68(4), 1104-1108 (1987) Smallwood P.D., An introduction to Risk Sensitivity: The use of Jensen's Inequality to clarify Evolutionary arguments of Adaptation and Constraint, American Zoologist 36, 392-401 (1996) Montague P.R., Dayan P., Person C and Sejnowski T.J., Bee foraging in Uncertain Environments using Predictive Hebbbian Learning, Nature 377, 725-728 (1995) Hammer M., An Identified Neuron mediates the Unconditioned Stimulus in Associative Olfactory Learning in Honeybees, Nature 366, 59-63 (1993) 10 Hammer M., The Neural Basis of Associative Reward Learning in Honeybees, Trends in Neuroscience 20(6), 245-252 (1997) 11 12 Mitchell T, Machine Learning (McGraw Hill, 1997) Chalmers D.J., The Evolution of Learning: An Experiment in Genetic Connectionism, in: Proceedings of the 1990 Connectionist Models Summer School, San-Mateo, CA (eds Touretzky D.S., Elman J.L., Sejnowski T.J and Hinton G.E.) 8190 (Morgan Kaufmann, 1990) 13 Fontanari J.F and Meir R., Evolving a Learning Algorithm for the Binary Perceptron, Network: Computation in Neural Systems 2(4), 353-359 (1991) 14 Baxter J., The Evolution of Learning Algorithms for Artificial Neural Networks, in: Complex Systems (eds Green D and Bossomaier T., IOS Press, 1992) 15 Floreano D and Mondada F., Evolutionary Neurocontrollers for Autonomous Mobile Robots, Neural Networks 11,1461-1478 (1998) 16 Unemi T., Nagayoshi M., Hirayama N., Nade T., Yano K and Masujima Y., Evolutionary Differentiation of Learning Abilities - a case study on optimizing parameter values in Q-learning by a genetic algorithm, in: Artificial Life IV (eds Brooks R.A., Maes P.) 331-336 (MIT Press, 1994) 17 March J G., Learning to be Risk Averse Psychological Review 103(2), 301-319 (1996) 18 Breiman L., Probability (Addison-Wesley, 1968) ... experience changes behavior in both vertebrates and invertebrates2 In RL, learning is contingent upon a scalar reinforcement signal that provides evaluative information about how good an action is,... providing an instructive cue to the most rewarding behavior RL has been demonstrated and studied excessively in the ethology of foraging bumble -bees Real 3,4 showed that when foraging for nectar in. .. only a subset of the learning rule parameters 16 We define a general framework for evolving learning rules, which encompasses all heterosynaptic Hebbian learning rules, along with other characteristics