Networks of Neurons (Chapter 7)

Transcription

1 CSE/NEUBEH 58 Networks of Neurons (Chapter 7) Drawng by Ramón y Cajal Today s Agenda F Computaton n Networks of Neurons Feedforward Networks: What can they do? Recurrent Networks: What more can they do?

2 Flashback Frng-Rate-Based Network Model Output frng rate changes lke ths: Input current changes lke ths: r d di s s F( I ( t)) I s s w u F s the actaton functon What happens when:? s? r Statc nput? r s 3 What f there are multple output neurons? Scalar Vector w Vector u Vector Matrx W Vector u Sngle Output Input Vector Output Vector j Input Vector w j d F ( w u) (Assumng relately fast synapses, w u at each t) I s d F( Wu) : N ector; u : K ector W : N K matrx; F : pontwse functon 4

3 5 General Equaton for Modelng Networks ) M W ( u F d For feedforward networks, M = matrx of zeros Output Decay Input Feedback Feedforward Recurrent Image Source: Dayan & Abbott textbook 6 Example: Lnear Feedforward Network u W d Dynamcs: Steady State (set d/ to ): u W ss W u What s ss?

4 7 Lnear Feedforward Network Wu ss What s the network dong? Network s performng Lnear Flterng for Edge Detecton Output Input (and shfted ersons n W) Flter u Input u Output

5 Example of Edge Detecton n a D Image Input u Output 9 Image from Edge detectors n the bran Retna Lateral Genculate Nucleus (LGN) Prmary Vsual Cortex (V) Examples of recepte felds n prmary sual cortex (V)

6 The Bran can do Calculus! V neurons are bascally computng derates! One row of W df f ( x h) f ( x) lm dx h h Dscrete approxmaton f ( x ) f ( x) Wu u( x ) u( x) d f lm dx h Dsc. approx. f ( x h) f ( x) h f ( x ) f ( x) f ( x) f ( x ) f ( x ) f ( x) f ( x ) Lnear flterng wth Wu s fne but what about Wu-sng more than layers of neurons? Wu

7 Lnear Multlayer Feedforward Network? W 4 W 3 W W u 3 Deep (Nonlnear) Feedforward Networks u F F F F W W W 3 W 4 F( W F(W3 F(W F(W 4 u )))) How do get the W s? Answer: Stay tuned Rch Fgure adapted from 4

8 Recurrent Neural Networks d F( Wu M) Output Decay Input Feedback 5 Image Source: Dayan & Abbott textbook What can a Lnear Recurrent Network do? d Wu M h u : K ector W : N K matrx, h : N ectors M : N N matrx Want to fnd out how (t) behaes for dfferent M How? 6 Image Source: Dayan & Abbott textbook

9 Egenectors to the rescue! d h M F Idea: Use egenectors of M to sole dfferental equaton for F Suppose N N matrx M s symmetrc F M has N orthogonal egenectors e and N egenalues whch satsfy: Me e h 7 Usng Egenectors to Sole for Network Output (t) F We can represent output ector (t) usng egenectors of M: N ( t) c ( t) e d F Substtutng aboe n the dff. equaton for : h M usng Me e and orthonormalty of e, we can sole for c (and therefore (t)!): h e t( ) t( ) c ( t) exp( ) c () exp( ) (For full deraton, see Lecture Notes on course webste) 8

10 Egenalues determne Network Stablty! N ( t) c ( t) e h e t( ) t( ) c ( t) exp( ) c () exp( ) If any (e.g., ), ( t) explodes network s unstable! If all, network sstable and ( t) conerges ss h e e to steady state alue : 9 Amplfcaton of Inputs n a Recurrent Network ss h e e If all and one (say ) h e ss e s close to wth others much smaller : Amplfcaton of nput projecton by a factor of E.g.,.9,

11 Example of a Lnear Recurrent Network h Each output neuron codes for an angle between -8 to +8 degrees Recurrent connectons M = cosne functon of relate angle M (, ') cos( ') Exctaton nearby, Inhbton further away Is M symmetrc? M(, )= M(, )? ( - ) Amplfcaton n the Lnear Recurrent Network M (, ') cos( '), all egenalues = except =.9 ( h e ) e Amplfcaton ( h e ) ss e - Nosy Input Output Preferred angle of neuron (From secton 7.4 n Dayan & Abbott textbook)

12 Memory n Lnear Recurrent Networks d h M N ( t) c ( t) e dc Suppose and all other. Then, If nput h s ( t) c ( t) e h e turned onand then off, can show that een after t e ce h( t' ) e ' Sustaned actty wthout any nput! Networks keeps a memory of ntegral of past nput h : (For full deraton, see Lecture Notes on course webste) 3 The Bran can do Calculus (Part II: Integraton)* Input: Bursts of spkes from bran stem oculomotor neurons Output: Memory of eye poston n medal estbular nucleus *For Part I: Dfferentaton, see earler slde 4 (Image: Dayan & Abbott based on (Seung et al., ))

13 Nonlnear Recurrent Networks Input ector h Output ector d F( h M) Output Decay Example: Rectfcaton nonlnearty: F(x) = [x] + = x f x > and o.w. Input Recurrent Feedback 5 Nonlnear Recurrent Network performs Amplfcaton Input Output As before, recurrent connectons All egenalues but.9 M (, ') cos( ') (yet stable due to rectfcaton) 6 Image Source: Dayan & Abbott textbook

14 Same Nonlnear Network performs Selecte Attenton Input Output Network performs Wnner-Takes-All nput selecton 7 Image Source: Dayan & Abbott textbook Gan Modulaton n the Nonlnear Network Inputs Outputs Addng a constant amount to the nput h multples the output 8 Image Source: Dayan & Abbott textbook

15 Memory n the Nonlnear Network Local Input + Background Output Network mantans a memory of preous actty when nput s turned off. Turn off nput Output Memory R. Rao, 58: Lecture s mantaned 9 by recurrent actty Smlar to shortterm memory or workng memory n prefrontal cortex 9 Image Source: Dayan & Abbott textbook What about Non-Symmetrc Recurrent Networks? F Example: Network of Exctatory (E) and Inhbtory (I) Neurons Connectons can t be symmetrc: Why? + + E I ms de E di I Parameter we wll ary to study the network I - E M M II EE I E M M IE EI E I How do we analyze the dynamc behaor of such a network? I E 3

16 Stablty Analyss de E M EEE M EII E Take derates of rght E hand sde wth respect to di I M III M IEE I both E and I I Stablty Matrx (aka the Jacoban Matrx): ( M EE ) E J M IE I.5 ms M EI E ( M II ) I (For all the gory detals of ths stablty analyss, see Lecture Notes on course webste) - Egenalues of J can hae real and magnary parts These egenalues determne dynamcs of the nonlnear network near a fxed pont 3 Damped Oscllatons n the Network Choose I = 3 ms (makes real part of egenalues negate) Stable Fxed Pont 3 Image Source: Dayan & Abbott textbook

17 Unstable Behaor and Lmt Cycle Choose I = 5 ms (makes real part of egenalues poste) Lmt cycle 33 Image Source: Dayan & Abbott textbook Oscllatory Actty n Real Networks Snff Snff Snff Actty n rabbt (or wabbt) olfactory R. Rao, 58: Lecture bulb 9 durng 3 snffs (see Chapter 7 n textbook for detals) 34

18 F Thngs to do: Start readng Chapter 8 n D & A Homework #3 due Sunday Feb 9 Fnalze a fnal project topc and partner(s) Emal Raj, Adrenne and Rch your topc and partners, or ask to be assgned to a team That s all folks! Next Class: Guest lecture by Prof. Erc Shea-Brown 35