15 Grossberg Network 1

Transcription

1 Grossberg Network

2 Biological Motivation: Vision Bipolar Cell Amacrine Cell Ganglion Cell Optic Nerve Cone Light Lens Rod Horizontal Cell Retina Optic Nerve Fiber Eyeball and Retina

3 Layers of Retina The retina is a part of the brain that covers the back inner wall of the eye and consists of three layers of neurons: Outer Layer: Photoreceptors - convert light into electrical signals Rods - allow us to see in dim light Cones - fine detail and color Middle Layer Bipolar Cells - link photoreceptors to third layer Horizontal Cells - link receptors with bipolar cells Amacrine Cells - link bipolar cells with ganglion cells Final Layer Ganglion Cells - link retina to brain through optic nerve 3

4 Visual Pathway Primary Visual Cortex Lateral Geniculate Nucleus Retina 4

5 Photograph of the Retina Blind Spot (Optic Disk) Vein Fovea 5

6 Imperfections in Retinal Uptake Edge Vein Retina Blind Spot Stabilized Images Fade 6

7 Compensatory Processing Emergent Segmentation: Complete missing boundaries. Featural Filling-In: Fill in color and brightness. Before Processing After Processing Emergent Segmentation Featural Filling-in 7

8 Visual Illusions Illusions demostrate the compensatory processing of the visual system. Here we see a bright white triangle and a circle which do not actually exist in the figures. 8

9 Vision Normalization Variable Illumination Separate Constant Illumination The vision systems normalize scenes so that we are only aware of relative differences in brightness, not absolute brightness. 9

10 Brightness Contrast If you look at a point between the two circles, the small inner circle on the left will appear lighter than the small inner circle on the right, although they have the same brightness. It is relatively lighter than its surroundings. The visual system normalizes the scene. We see relative intensities. 0

11 Leaky Integrator (Building block for basic nonlinear model.) ε dn () t = nt () pt () dt Leaky Integrator p - /ε ṅ n ε dn/dt = - n p

12 Leaky Integrator Response nt () = e t ε n() 0 t -- e ( t τ) ε pt ( τ) dτ ε 0 For a constant input and zero initial conditions: nt () = p( e t ε )

13 Shunting Model Input Basic Shunting Model p Excitatory Input /ε ṅ b - n b p n(t) -b - Inhibitory Input b - ε dn/dt = -n (b - n) p - (n b - ) p - Gain Control (Sets upper limit) Gain Control (Sets lower limit) 3

14 Shunting Model Response ε dn () t = nt () ( b nt ())p ( nt () b - )p - dt b = b - = 0 ε = p - = 0 Upper limit will be, and lower limit will be p = p =

15 Grossberg Network Input Layer Layer (Retina) (Visual Cortex) STM LTM (Adaptive Weights) Normalization Contrast Enhancement LTM - Long Term Memory (Network Weights) STM - Short Term Memory (Network Outputs) 5

16 Layer Input Layer b S p W S x S n /ε S x - - W S x S -. n - b - a S εdn /dt = - n ( b - n ) [ W ] p - (n - b ) [ - W ] p 6

17 Operation of Layer ε dn () t = n () t ( b dt n () t )[ W ]p ( n () t - b )[ - W ]p Excitatory Input [ W ]p Inhibitory Input - [ W ]p W - W = = On-Center/ Off-Surround Connection Pattern b bi - - = = 0 b Normalizes the input while maintaining relative intensities. 7

18 Analysis of Normalization Neuron i response: At steady state: ε dn i () t = n dt i () t ( b n i () t )pi n i () t p j n i 0 = n i ( b )p i n i Define relative intensity: p i Steady state neuron activity: p i j i p j = ---- P p P where = j S j = j i n i = b p i S p j j = n i b P = pi P Total activity: n j S j = S b P = P p j = j = b P P b 8

19 Layer Example t ( 0.) dn () = n dt () t ( n() t )p n ()p t t ( 0.) dn () = n dt () t ( n() t )p n ()p t n n p = p 0 = n 0.5 n t t 9

20 Characteristics of Layer The network is sensitive to relative intensities of the input pattern, rather than absolute intensities. The output of Layer is a normalized version of the input pattern. The on-center/off-surround connection pattern and the nonlinear gain control of the shunting model produce the normalization effect. The operation of Layer explains the brightness constancy and brightness contrast characteristics of the human visual system. 0

21 Layer Layer S x S W On-Center b W S x S a S - - /ε. n - n f a Off-Surround - b - W S x S S εdn /dt = - n ( b - n ) {[ W ] f (n ) W a } - (n - b ) [ - W ] f (n )

22 Layer Operation ε dn () t dt = n () t ( b n () t ){[ W ]f ( n () t ) W a } ( n () t - b ) W Excitatory Input: {[ W ]f ( n () t ) W a } - [ ]f ( n () t ) W W = W (On-center connections) (Adaptive weights) Inhibitory Input: - W [ ]f ( n () t ) - W - = W (Off-surround connections)

23 Layer Example ε = 0. b = b = W ( w ) T f 0 () n ( n ) = = = 0 ( n) ( w ) T Correlation between prototype and input. t ( 0.) dn () n dt () t ( n() t ) f ( n() t ) ( w ) T a n t = ()f ( n() t ) Correlation between prototype and input. t ( 0.) dn () n dt () t ( n() t ) f ( n() t ) ( w ) T a n t = ()f ( n() t ). 3

24 Layer Response ( w ) T a n () t 0.75 a 0. = ( w ) T a n () t Contrast Enhancement and Storage Input to neuron : Input to neuron : t ( w ) T a = = ( w ) T a = =

25 Characteristics of Layer As in the Hamming and Kohonen networks, the inputs to Layer are the inner products between the prototype patterns (rows of the weight matrix W ) and the output of Layer (normalized input pattern). The nonlinear feedback enables the network to store the output pattern (pattern remains after input is removed). The on-center/off-surround connection pattern causes contrast enhancement (large inputs are maintained, while small inputs are attenuated). 5

26 Oriented Receptive Field When an oriented receptive field is used, instead of an on-center/off-surround receptive field, the emergent segmentation problem can be understood. Active Active Inactive 6

27 Choice of Transfer Function f (n) Linear Stored Pattern n ( ) Comments Perfect storage of any pattern, but amplifies noise. n i(0) Slower than Linear Amplifies noise, reduces contrast. i Faster than Linear Sigmoid Winner-take-all, suppresses noise, quantizes total activity. Supresses noise, contrast enhances, not quantized. 7

28 Adaptive Weights dw i, j Hebb Rule with Decay () t = α { w dt i, j () t n i ()n t j() t } dw i, j Instar Rule (Gated Learning) () t = αn dt i t () wi, j { () t n j() t } Learn when n i (t) is active. Vector Instar Rule d[ w i () t ] = αn dt i () t { [ i w () t ] n () t } 8

29 Example dw, () t = n dt t () w, { () t n () t } dw, () t = n dt t () w, { () t n () t } dw, () t = n dt t () w, { () t n () t } dw, () t = n dt t () w, { () t n () t } 9

30 Response of Adaptive Weights Two different input patterns are alternately presented to the network for periods of 0. seconds at a time. For Pattern : n = 0.9 n = w, () t w, () t For Pattern : 0.5 w, () t n = 0.45 n = w, () t The first row of the weight matrix is updated when n (t) is active, and the second row of the weight matrix is updated when n (t) is active. 30

31 Relation to Kohonen Law Grossberg Learning (Continuous-Time) d[ w i () t ] = αn dt i () t { [ i w () t ] n () t } Euler Approximation for the Derivative d[ w i () t ] dt w i ( t t) w i () t t Discrete-Time Approximation to Grossberg Learning w i ( t t) w = i () t α t ( )n i () t { i w () t n () t } 3

32 Relation to Kohonen Law Rearrange Terms w i ( t t) = { α t ( )n i () t }w i () t α t ( )n i () t { n () t } Assume Winner-Take-All Competition w ( t t) = { α' } w () t { α}'n () t where α' α t ( )n i i = i () t Compare to Kohonen Rule w i ( q) = ( α) i w( q ) αp( q) 3