The Cortex. Networks. Laminar Structure of Cortex. Chapter 3, O Reilly & Munakata.

Transcription

1 Networks The Cortex Chapter, O Relly & Munakata. Bology of networks: The cortex Exctaton: Undrectonal (transformatons) Local vs. dstrbuted representatons Bdrectonal (pattern completon, amplfcaton) Inhbton: Controllng bdrectonal exctaton. Constrant Satsfacton: Puttng t all together. Two separate populatons of neurons: Exctatory (glutamate): Pyramdal, Spny stellate. Inhbtory (GABA): Chandeler, Basket. / 0 / 0 Lamnar Structure of Cortex Lamnar Structure of Cortex (Layers,) (Layer ) Output (Layers,) Sensaton (Thalamus) Subcortex Motor/BG / 0 / 0

2 Area Structure of Cortex Area Structure of Cortex (Layers,) (Layer ) Output (Layers,) (Layers,) (Layers,) (Layer ) Output (Layers,) Output (Layers,) Sensaton (Thalamus) Subcortex BG Thalamus Subcortex Thalamus Subcortex Motor/BG / 0 / 0 Exctaton (Undrectonal): Transformatons Emphaszng Dstnctons Detectors work n parallel to transform nput actvty pattern to hdden actvty pattern. Emphaszes some dstnctons, collapses across others. Functon of what the detectors detect (and what they gnore). 0 9 Emphasze dstnctons: Dfferent dgts non-overlappng. Collapse dstnctons: Nosy dgts categorzed as same. / 0 / 0

3 Dstnctons: Cluster Plot Detectors are Dedcated, Content-Specfc a) NosyDgts Pattern: 0 b) _Acts Pattern: 0 a) Letters Pattern: 0 b) _Acts Pattern: A M Q W H D B V N IT Z L K E U J S R FP O CG Z W V U T R Q P O N M L K J I H G F E D C B A S / 0 0 / 0 Dstrbuted vs Localst Representatons Dgts Wth Dstrbuted Representatons Localst = unt actve at a tme (e.g., dgts). Dstrbuted = many unts actve, for multple nputs. / 0 / 0

4 energy Actvaton Advantages of Dstrbuted Representatons Networks: Bdrectonal Exctaton... Contnuous Dmenson Effcency: Fewer total unts requred. Smlarty: As a functon of overlap. Generalzaton: Can use novel combnatons. Robustness: Redundancy. Accuracy: By coarse-codng. Learnng: Bootstrappng of small changes. But: modularty has advantages, suggestng sparse representatons Top-down processng ( magery ). Pattern completon. Amplfcaton/bootstrappng. Attractor dynamcs. cf. Hopfeld networks... / 0 / 0 Attractor Dynamcs Networks: Inhbton attractor basn state y state x attractor state Bology: Feedforward and Feedback. Crtcal Parameters. KWTA Smplfcaton. Bdrectonal exctaton caused network to settle nto a partcular stable state over tme: the attractor. / 0 / 0

5 Inhbton: benefts Types of Inhbton a) b) Feedback Controls actvty (bdrectonal exctaton). Inhb Inhb Competton leads to selecton (Darwn!). Supports sparse dstrbuted representatons. Feed Forward Antcpates exctaton Reacts to exctaton / 0 / 0 Crtcal Parameters KWTA Approxmaton a) b) Feedback Inhb Feed Forward Inhb Inhb conductance nto hdden unts (g bar.hdden) Inhb conductance nto nhb unts (g bar.nhb) Strength of feedforward weghts to nhb (scale.ff) Strength of feedback weghts to nhb (scale.fb) Approxmate nhbton wth max of k unts actve at any tme. Approxmates set pont behavor of negatve feedback systems: thermostat settng. Implemented by computng g for entre layer, such that k unts would stll get actve, but the rest wll be too nhbted. 9 / 0 0 / 0

6 KWTA Approxmaton: Smple a) b) c) g g g 0 k k+ n 0 k k+ n 0 k k+ n Compute g such that k unts have above-threshold equlbrum membrane potental, apply to all unts n the layer. Frst sort the unts by g e (.e., most actve). Recall: V m = g eḡee e + g ḡ E + g l ḡ l E l g e ḡ e + g ḡ + g l ḡ l () Substtutng threshold potental Θ for V m : g Θ = g eḡe(e e Θ) + g l ḡ l (E l Θ) Θ E () Select g between unt k and unt k + n the rankng. KWTA Approxmaton: Average-Based a) b) c) < > k < >k g < >n k < > k g < > n k g Actual Actual < >n k actvty actvty 0 k k+ n 0 k k+ n 0 k k+ n The same, but usng the average of the top k unts and the average of the bottom n k unts. g Θ k = k k g Θ () () = g Θ n k = n k n g Θ () () =k g = g Θ n k + q( g Θ k g Θ n k ) () g = g Θ (k + ) + q(g Θ (k) g Θ (k + )) () / 0 / 0 Other Smplfcatons Constrant Satsfacton Compettve Learnng (WTA). Soft Compettve Learnng (Mxture of Gaussans) a j = l j P k l k Kohonen Networks (Neghborhood Kernels). Drect Inhbton (IAC). Now we can return to Hopfeld networks but better. Energy Functon Nose. Inhbton. / 0 / 0

7 energy The Energy Functon Energy, Harmony attractor basn state y state x attractor state Fallng thngs are mnmzng ther potental energy: nature always seeks to mnmze the energy of a system. If we can wrte an expresson for the energy of a network, wll the nature of the network mnmze t? es! Energy: E = x w j y j () j Harmony: H = x w j y j () j (Symmetrc weghts) Actvatons are constraned to be consstent wth the weghts. / 0 / 0 Updatng Global Satsfacton Locally The Role of Nose Just updatng actvatons ncreases global constrant satsfacton! y j = x w j (9) H = x w j y j (0) H y j = j x w j () Local Mnmum Global Mnmum In PDP++ mplement wth spke actvaton functon, or add nose to the rate-coded functon / 0 / 0

8 Unt Unt The Role of Inhbton a) No KWTA b) KWTA Unt Unt For KWTA: average actvty s constant so search s restrcted but mples sparse dstrbuted representatons 9 / 0