Fundamentals of Computational Neuroscience 2e

Transcription

1 Fundamentals of Computational Neuroscience 2e Thomas Trappenberg March 21, 2009 Chapter 9: Modular networks, motor control, and reinforcement learning

2 Mixture of experts Expert 1 Input Expert 2 Integration network Output Expert n Gating network A. Absolute function f (x ) = abs (x ) B. Mixture of expert for absolute function X ΣΠ abs(x ) x

3 The what-and-where task B. Without bias towards short connections A. Model retina with sample image 5 4 Output node # Output node # Hidden node # C. With bias towards short connections Hidden node # Jacobs and Jordan (1992)

4 Coupled attractor networks A. Coupled attractor networks B. The left-right universe with letters Node group 1 Node group 2 Connections between groups

5 Limit on modularity Load capacity α : c m = 2 A. Load capacity B. Bounds on intermodular strength m = 4 m = g : Relative intermodular strength g : Relative intramodular strength m : Number of modules

6 Sequence learning A. Modular attractor model B. Time evolution of overlaps Input Pathway w AB w AA w w BB BA Overlap in A Overlap in B Module A Module B Time [τ] Lawrence, Trappenberg and Fine (2006); (Sommer and Wennekers (2005))

7 Working memory Prefrontal cortex Parietal motor cortex Hippocampus O Reilly, Braver, and Cohen 1999

8 Limit on working memory B. Two object C. Four object Time 20 Node number 120 Node number Node number A. One object Time Time 20

9 Motor learning and control Disturbance Desired state - Motor command generator Motor command Controlled object Actual state Sensory system Afferent Re-afferent

10 Forward model controller Disturbance Desired state - Motor command generator Motor command Controlled object Actual state Sensory system + Forward dynamic model Forward output model - - Afferent Re-afferent

11 Inverse model controller Inverse model Disturbance Desired state - Motor command generator - + Motor command Controlled object Actual state Sensory system Afferent Re-afferent

12 Cerebellum Stellate cell Parallel fibre Molecular layer Purkinje layer Granular layer { { { Climbing fibre Purkinje neuron Granule cell Basket cell Golgi cell Intracerebellar and vestibular nuclei Inferior olive Out Mossy fibre Excitatory synapse Inhibitory synapse Spinal cord External cuneate nucleus { Reticular nuclei Pontine nuclei

13 Reinforcement learning

14 Basal Ganglia A. Outline of basic BG anatomy Cerebral cortex C. Recordings of SNc neurons and simulations Caudate nucleus Putamen Thalamus Stimulus A No reward Globus pallidus Superior colliculus Subthalamic nucleus Substantia nigra pars compacta ( pars reticulata ) Stimulus B Stimulus A Reward rhat Pattern 4 Pattern 3 Pattern Episode

15 temporal difference learning A. Linear predictor node B. Temporal delta rule C. Temporal difference rule in in in r 1 (t) r 2 (t) r 3 (t) r 4 in (t) in in in r 1 (t) r 2 (t) r 3 (t) r 4 in (t) in in in r 1 (t) r 2 (t) r 3 (t) r 4 in (t) V(t) r j in (t-1) slow V(t) V(t 1) r (t) in (t-1) r j V(t) slow V(t 1) fast γ V(t) r (t) r (t) r (t)

16 Actor-critique and Q-learning (frontal) TH B. Actor-critic model of BG Cerebral cortex F C C C D. Q-learning model of BG state action Cerebral cortex state / action coding Matrix module ST (actor) SPm SPs PD DA Basal ganglia ST Striosomal module (critic) SNc Striatum reward prediction Pallidum action selection Thalamus Primary reinforcement Primary reinforcement

17 Actor-critique controller Critic Reinforcement signal Disturbance Desired state - Motor command generator (actor) Motor command Controlled object Actual state Sensory system Afferent Re-afferent

18 Further Readings Robert A. Jacobs, Michael I. Jordan, and Andrew G. Barto (1991), Task decomposition through competition in a modular connectionist architecture: the what and where tasks, in Cognitive Science 15: Geoffrey Hinton (1999), Products of experts, in Proceedings of the Ninth International Conference on Artificial Neural Networks, ICANN 99, 1:1 6. Yaneer Bar-Yam (1997), Dynamics of complex systems, Addison-Wesley. Edmund T. Rolls and Simon M. Stringer (1999), A model of the interaction between mood and memory, in Networks: Comptutation in neural systems 12: N. J. Nilsson (1965), Learning machines: foundations of trainable pattern-classifying systems, McGraw-Hill. O. G. Selfridge (1958), Pandemonium: a paradigm of learning, in the mechanization of thought processes, in Proceedings of a Symposium Held at the National Physical Laboratory, November 1958, , London HMSO. Marvin Minsky (1986), The society of mind, Simon & Schuster. Akira Miyake and Priti Shah (eds.) (1999), Models of working memory, Cambridge University Press. Daniel M. Wolpert, R. Chris Miall, and Mitsuo Kawato (1998), Internal models in the cerebellum, in Trends Cognitive Science 2: Edmund T. Rolls and Alessandro Treves (1998), Neural networks and brain function, Oxford University Press. James C. Houk, Joel L. Davis, and David G. Beiser (eds.) (1995), Models of information processing in the basal ganglia, MIT Press. Richard S. Sutton and Andrew G. Barto (1998), Reinforcement learning: an introduction, MIT Press. Peter Dayan and Laurence F. Abbott (2001), Theoretical Neuroscience, MIT Press.