Introduction to Neural Networks Computing. CMSC491N/691N, Spring 2001

Transcription

1 Iroduco o Neurl Neorks Compug CMSC49N/69N, Sprg 00

2 us: cvo/oupu: f eghs: X, Y j X Noos, j s pu u, for oher us, j pu sgl here f. s he cvo fuco for j from u o u j oher books use Y f _ j j j Y j X j Y j

3 bs: b j hreshold: θ egh mr: W{ } j cos pu for us h sep/hreshold cvo fuco j : ro de; j: colum de

4 ro vecors vecors of eghs: j L j, j, 3 j L,, colum vecors eghs come o u j eghs go ou of u

5 smll usull of scle he specfes re lerg : compuo vecorfor pu..., vecor pu rgeou or rg..., vecor pu rg... } { rg lerg/,, j m j j j j s s s s W old e α α

6 Reve of Mr Operos Vecor: sequece of elemes he order s mpor e.g.,, deoes vecor legh sqr*+* oreo gle,,,, dmesol vecor po o dmesol spce colum vecor: 5 8 ro vecor 5 8 T T T X, rspose

7 orms of vecor: mgude vecor operos: orm L orm L orm L m / r r r r r T T T Σ M M...,..., dmeso sme of vecors colum re, produc er do vecor : colum : scler,,...,

8 Cross produc: defes oher vecor orhogol o he pl formed b d.

9 he eleme o he h ro d jh colum dgol eleme egh egh mr W ech ro or colum s vecor jh colum vecor h ro vecor m j m m m m A } { M m m j A M... : : : : : j j

10 colum vecor of dmeso m s mr of m rspose: A m T m m squre mr: de mr: jh colum becomes jh ro I A j 0 f j oherse

11 smmerc mr: m A A or or j T,, mr operos: ra r,... r r j j j T A m T... m,...,... T The resul s ro vecor, ech eleme of hch s T er produc of d colum vecor j

12 produc of o mrces: vecor ouer produc: j j p m p m b C here C B A m m A I A m m m m T,...,,...,... M M M

13 & Clculus d Dfferel Equos, he dervve of, h respec o me Ssem of dfferel equos & f M & f soluo:, L dffcul o solve uless f re smple

14 Mul-vrble clculus: prl dervve: gves he dreco d speed of chge of, h respec o,..., f cos s e e e e

15 he ol dervve: Grde of f : Ch-rule: s fuco of, s fuco of T f f f d df & & & & & +..., f f f,..., f

16 dmc ssem: & & M f f,...,... chge of m poell ffec oher ll coue o chge he ssem evolves reches equlbrum he 0 sbl/rco: specl equlbrum po mml eerg se per of,... sble se ofe represes soluo &

17 Chper : Smple Neurl Neorks for Per Clssfco Geerl dscusso Ler seprbl Hebb es Percepro Adle

18 Per recogo Geerl dscusso Pers: mges, persol records, drvg hbs, ec. Represeed s vecor of feures ecoded s egers or rel umbers NN Per clssfco: Clssf per o oe of he gve clsses Form per clsses Per ssocve recll Usg per o recll reled per Per compleo: usg prl per o recll he hole per Per recover: dels h ose, dsoro, mssg formo

19 Geerl rchecure Sgle ler b e pu o Y: e b + Y bs b s reed s he egh from specl u h cos oupu. hreshold oupu θ reled o Y f e - f f e e < θ θ clssf,... o oe of he o clsses

20 Decso rego/boudr, b! 0, θ 0 b or s le, clled decso boudr, hch pros he ple o o decso regos If po/per Oherse, b + + clss o b + + clss oe b, 0 s he posve rego, he, d he oupu s oe belogs o <, oupu belogs o, b 0, θ! 0 ould resul smlr pro 0 + -

21 If 3 hree pu us, he he decso boudr s o dmesol ple hree dmesol spce I geerl, decso boudr b + 0 s - dmesol hper-ple dmesol spce, hch pro he spce o o decso regos Ths smple eork hus c clssf gve per o oe of he o clsses, provded oe of hese o clsses s erel oe decso rego oe sde of he decso boudr d he oher clss s oher rego. The decso boudr s deermed compleel b he eghs W d he bs b or hreshold θ.

22 Ler Seprbl Problem If o clsses of pers c be sepred b decso boudr, represeed b he ler equo b + 0 he he re sd o be lerl seprble. The smple eork c correcl clssf pers. Decso boudr.e., W, b or θ of lerl seprble clsses c be deermed eher b some lerg procedures or b solvg ler equo ssems bsed o represeve pers of ech clsses If such decso boudr does o es, he he o clsses re sd o be lerl seprble. Lerl seprble problems co be solved b he smple eork, more sophsced rchecure s eeded.

23 Emples of lerl seprble clsses - Logcl AND fuco pers bpolr decso boudr b θ Logcl OR fuco pers bpolr decso boudr b - θ o o o : clss I o: clss II - o : clss I o: clss II -

24 Emples of lerl seprble clsses - Logcl XOR eclusve OR fuco pers bpolr decso boudr No le c sepre hese o clsses, s c be see from he fc h he follog ler equl ssem hs o soluo b b b b < < becuse e hve b < 0 from + 4, d b > 0 from + 3, hch s cordco o o : clss I o: clss II -

25 XOR c be solved b more comple eork h hdde us - - θ z z θ 0 -, - -, - - -, -,, -, -,, - Y

26 Hebb Nes Hebb, hs fluel book The orgzo of Behvor 949, clmed Behvor chges re prmrl due o he chges of spc sreghs beee euros I d j j j creses ol he boh I d j re o : he Hebb lerg l j I ANN, Hebb l c be sed: creses ol f he oupus of boh us d hve he j sme sg. I our smple eork oe oupu d pu us or, e old j j j j j e j old α

27 Hebb e supervsed lerg lgorhm p.49 Sep 0. Ilzo: b 0, 0, o Sep. For ech of he rg smple s: do seps -4 /* s s he pu per, he rge oupu of he smple */ Sep. : s, I o /* se s o pu us */ Sep 3. : /* se o he rge */ Sep 4. : + *, o /* upde egh */ b : b + * /* upde bs */ Noes: α, ech rg smple s used ol oce. Emples: AND fuco Br us, 0,, b,,, 0, 0 0,, 0 0, 0, 0 bs u A correc boudr: Is lered fer usg ech smple oce

28 Bpolr us, -,, b,,, -, ,, - - -, -, - - A correc boudr s successfull lered I ll fl o ler ^ ^ 3, eve hough he fuco s lerl seprble. Sroger lerg mehods re eeded. Error drve: for ech smple s:, compue from s bsed o curre W d b, he compre d Use rg smples repeedl, d ech me ol chge eghs slghl α << Lerg mehods of Percepro d Adle re good emples

29 B Rosebl 96 Percepros For modelg vsul percepo re Three lers of us: Sesor, Assoco, d Respose Lerg occurs ol o eghs from A us o R us eghs from S us o A us re fed. A sgle R u receves pus from A us sme rchecure s our smple eork For gve rg smple s:, chge eghs ol f he compued oupu s dffere from he rge oupu hus error drve

30 Percepro lerg lgorhm p.6 Sep 0. Ilzo: b 0, 0, o Sep. Whle sop codo s flse do seps -5 Sep. For ech of he rg smple s: do seps 3-5 Sep 3. : s, o Sep 4. Sep 5. compue If! : + α *, o b : b + α * Noes: - Lerg occurs ol he smple hs! - To loops, compleo of he er loop ech smple s used oce s clled epoch Sop codo - Whe o egh s chged he curre epoch, or - Whe pre-deermed umber of epochs s reched

31 Iforml jusfco: Cosder d - To move ord, should reduce e_ If, * < 0, eed o reduce * s reduced If -, * >0 eed o crese * s reduced See book pp for emple of eecuo Percepro lerg rule covergece heorem Iforml: problem h c be represeed b percepro c be lered b he lerg rule Theorem: If here s W such h f p W p for ll P rg smple pers { p, p}, he for 0 sr egh vecor W, he percepro lerg rule ll coverge o egh vecor * f p W sme. p such h * for ll p. W d W m o be he Proof: redg for grd sudes pp W *

32 Adle B Wdro d Hoff 960 Adpve Ler Neuro for sgl processg The sme rchecure of our smple eork Lerg mehod: del rule oher of error drve, lso clled Wdro-Hoff lerg rule The del: _ NOT becuse f _ s o dffereble Lerg lgorhm: sme s Percepro lerg ecep Sep 5: b : b + α _ : + α * _

33 Dervo of he del rule Error for ll P smples: me squre error E s fuco of W {,... } Lerg kes grde desce pproch o reduce E b modf W he grde of E: There for P p p p P E _..., E E E E P p P p p p P p p p p P E ] _ [ _ ] _ [ E P p p P ] _ [

34 Ho o ppl he del rule Mehod sequel mode: chge fer ech rg per b α p _ p Mehod bch mode: chge he ed of ech epoch. Wh epoch, cumule α p _ p for ever per p, p Mehod s sloer bu m provde slghl beer resuls becuse Mehod m be sesve o he smple orderg Noes: E mooocll decreses ul he ssem reches se h locl mmum E smll chge of ll cuse E o crese. A locl mmum E se, E /, bu E s o 0 gureed o be zero

35 Summr of hese smple eorks Sgle ler es hve lmed represeo poer ler seprbl problem Error drve seems good o r e Mul-ler es or es h o-ler hdde us m overcome ler seprbl problem, lerg mehods for such es re eeded Threshold/sep oupu fucos hders he effor o develop lerg mehods for mul-lered es

36 Wh hdde us mus be o-ler? Mul-ler e h ler hdde lers s equvle o sgle ler e v v v v Becuse z d z re ler u z * *v + *v + b z * *v + *v + b _ z* + z* z z *u + *u + b+b here u *v+ *v, u *v + *v* _ s sll ler combo of d. θ 0 Y