CS623: Introduction to Computing with Neural Nets (lecture-10) Pushpak Bhattacharyya Computer Science and Engineering Department IIT Bombay

Transcription

1 CS6: Iroducio o Compuig ih Neural Nes lecure- Pushpak Bhaacharyya Compuer Sciece ad Egieerig Deparme IIT Bombay

2 Tilig Algorihm repea A kid of divide ad coquer sraegy Give he classes i he daa, ru he percepro raiig algorihm If liearly separable, covergece ihou ay hidde layer If o, do as ell as you ca pocke algorihm This ill produce classes ih misclassified pois

3 Tilig Algorihm cod Take he class ih misclassified pois ad break io subclasses hich coai o ouliers Ru PTA agai afer recruiig he required umber of percepros Do his uil homogeous classes are obaied Apply he same procedure for he firs hidde layer o obai he secod hidde layer ad so o

4 Illusraio XOR problem Classes are,,,,

5 As bes a classificaio as possible, ve, -ve, -ve, ve

6 Classes ih error,,,, oulier

7 Ho o achieve his classificaio Give he labels as sho: eqv o a OR problem -,,,, oulier

8 The parially developed / Ge he firs euro i he hidde layer, hich compues OR h.5..

9 Break he icorrec class,,,,, oulier,, - Do care: Make

10 Solve classificaio for h,,,, - This is

11 Ne sage of he / Compues Compues h h

12 Geig he oupu layer Solve a ilig algo problem for he hidde layer h h y,,, -, AND problem

13 Fial / AND / y.5 Compues.. Compues Compues h h

14 Lab eercise Impleme he ilig algorihm ad ru i for. XOR. Majoriy. IRIS daa

15 Hopfield e Ispired by associaive memory hich meas memory rerieval is o by address, bu by par of he daa. Cosiss of N euros fully coeced ih symmeric eigh sregh ij = ji No self coecio. So he eigh mari is - diagoal ad symmeric. Each compuig eleme or euro is a liear hreshold eleme ih hreshold =.

16 Compuaio Y Neuro i θ = W W W σ σ... σ Figure: A euro i he Hopfield Ne.

17 Eample = = 5 = = = = A ime = s = s = - s = Usable sae: Neuro ill flip. A sable paer is called a aracor for he e. 5 Figure: A eample Hopfield Ne

18 Sabiliy Asychroous mode of operaio: a ay isa a radomly seleced euro compares he e ipu ih he hreshold. I he sychroous mode of operaio all euros updae hemselves simulaeously a ay isa of ime. Sice here are feedback coecios i he Hopfield Ne he quesio of sabiliy arises. A every ime isa he eork evolves ad fially seles io a sable sae. Ho does he Hopfield Ne fucio as associaive memory? Oe eeds o sore or sabilize a vecor hich is he memory eleme.

19 Eergy cosideraio Sable paers correspod o miimum eergy saes. Eergy a sae <,,,, > E = -/ j j<>i ji i j Chage i eergy alays comes ou o be egaive i he asychroous mode of operaio. Eergy alays decreases. Sabiliy esured.

20 Hopfield Ne is a fully coeced eork i h euro is coeced o - euros

21 Cocep of Eergy Eergy a sae s is give by he equaio: E s = [ K K M ]

22 Coecio mari of he eork, -diagoal ad symmeric j... k i... ij k diagoal

23 Sae Vecor Biary valued vecor: value is eiher or - X = < > e.g. Various aribues of a sude ca be represeed by a sae vecor heigh address 5 roll umber hair color Ram = ~Ram = - 4

24 Relaio beee eigh vecor W ad sae vecor X W T X Weigh vecor Traspose of sae vecor W = T X 5 5 = W = T X For eample, i figure, A ime =, sae of he eural eork is: s = <, -, > ad correspodig vecors are as sho. Fig.

25 W.X T gives he ipus o he euros a he e ime isa 5 5 W = T X = 7 =. sg T X W This shos ha he / ill chage sae

26 Eergy Cosideraio A ime =, sae of he eural eork is: s = <, -, > 5 - E = -[5**-** *-*] = 4 - The sae of he eural eork uder sabiliy is <-, -, -> Esable sae = - -[5*-*-*-* -*-*-] = -

27 Sae Chage s = <, -, > s = compue by comparig ad summig ==sg[σ j= j j ] = if Σ j= j j > = - oherise

28 Theorem I he asychroous mode of operaio, he eergy of he Hopfield e alays decreases. Proof: M [ E = K K ]

29 Proof Le euro chage sae by summig ad comparig We ge folloig equaio for eergy [ E = K K ] M

30 Proof: oe ha oly euro chages sae [ ]} K [ ] = = j j j j {[ ] = K E E E = [ ][ ] = = j j j Sice oly euro chages sae, j = j, j=,, 4,, ad hece

31 = Proof coiued [ ][ ] j j j= S D Observaios: Whe he sae chages from - o, S has o be ve ad D is ve; so E becomes egaive. Whe he sae chages from o -, S has o be -ve ad D is ve; so E becomes egaive. Therefore, Eergy for ay sae chage alays decreases.

32 The Hopfield e has o coverge i he asychroous mode of operaio As he eergy E goes o decreasig, i has o hi he boom, sice he eigh ad he sae vecor have fiie values. Tha is, he Hopfield Ne has o coverge o a eergy miimum. Hece he Hopfield Ne reaches sabiliy.