The Perceptron. Nuno Vasconcelos ECE Department, UCSD

Size: px

Start display at page:

Download "The Perceptron. Nuno Vasconcelos ECE Department, UCSD"

Jonathan Fletcher
5 years ago
Views:

1 he Perceprn Nun Vscncels ECE Deprmen, UCSD

2 Clssfcn clssfcn prblem hs pes f vrbles e.g. X - vecr f bservns feures n he rld Y - se clss f he rld X R = fever, bld pressure Y = {dsese, n dsese} X, Y reled b unknn funcn f. = f gl: desgn clssfer h: X Y such hh h h = f

3 Lner dscrmnn he clssfer mplemens he lner decsn rule f g > 0 h = = sgn[ g ] f g < 0 hs he prperes dvdes X n hlf-spces bundr s he plne h: nrml dsnce he rgn b/ g/ s he dsnce frm pn he bundr g = 0 fr pns n he plne h g g > 0 n he sde pns psve sde g < 0 n he negve sde = b b g 3

4 Lner dscrmnn he clssfer mplemens he lner decsn rule f g > 0 h = = sgn[ g ] f g < 0 h gven lnerl seprble rnng se D = {,,..., n, n } n errrs f nd nl f, = nd g > 0 r = - nd g < 0.e..g > 0 g =- = b = b g hs lls ver cncse epressn fr he sun f n rnng errr r zer emprcl rsk 4

5 Lernng s pmzn necessr nd suffcen cndn fr zer emprcl rsk b > 0, hs s neresng becuse lls he frmuln f he lernng prblem s ne f funcn pmzn srng frm rndm guess fr he prmeers nd b e mmze he rerd funcn n = b r, equvlenl, mnmze he cs funcn J n, b b = = 5

6 he grden e hve seen h he grden f funcn f z s f z f = 0 f z, L, n z herem: he grden pns n he drecn f mmum grh grden s he drecn f grees ncrese f f z, nrml he s-cnurs f f. f f, 0 f f, 0 f, 6

7 Crcl pn cndns le f be cnnuusl dfferenble s lcl mnmum f f f nd nl f s lcl mnmum f f f nd nl f f hs zer grden 0 = f nd he Hessn f f s psve defne 0 = f n d d f d R 0 here d d f d R 0, f f = 0 0 f f f n M L 7 0 f f n n L

8 Grden descen hs sugges smple mnmzn echnque pck nl esme 0 f fll he negve grden = η f n n n η f hs s grden descen n η s he lernng re nd needs be crefull chsen f η lrge, descen m dverge f mn eensns re pssble mn pn: nce frmed s pmzn, e cn n generl slve n η f n n 8

9 he perceprn hs s he mn nsgh f Rsenbl, hch led he Perceprn he bsc de s d grden descen n ur cs J e kn h: n, b b = = f he rnng se s lnerl seprble here s les pr,b such h J,b < 0 n mnmum h s equl r beer hn hs ll d Q: cn e fnd ne such mnmum? 9

10 Perceprn lernng he grden s srghfrrd cmpue f = nd grden descen s rvl f = b here s, hever, ne prblem: J,b s n bunded bel f J,b < 0, cn mke J b mulplng nd b b λ > 0 he mnmum s ls hch s que bd, numercll hs s rell jus he nrmlzn prblem h e lred lked bu 0

11 Rsenbl s de resrc enn he pns ncrrecl clssfed ech ern defne se f errrs { < 0} E = b nd mke he cs J p ne h, b = b E J p cnn be negve snce, n E, ll b re negve f e ge zer, e kn e hve he bes pssble slun E emp

12 Perceprn lernng s rvl, jus d grden descen n J p,b b n n = = b n n η η E E hs urns u n be ver effecve f he D s lrge lp ver he enre rnng se ke smll sep he end ne lernve h frequenl s beer s schsc grden descen ke he sep mmedel fer ech pn n gurnee hs s descen sep bu, n verge, u fll he sme drecn fer prcessng enre D ver ppulr n lernng, here D s usull lrge

13 Perceprn lernng he lgrhm s s flls: se k = 0, k = 0, b k = 0 se R = m d { fr = :n { f b k < 0 hen { k = k η b k = b k η R k=k k e ll lk bu R shrl! } } } unl b k 0, n errrs 3

14 Perceprn lernng des hs mke sense? cnsder he emple bel se k = 0, k = 0, b k = 0 se R = m d { = fr = :n { f b k < 0 hen { k = k η b k = b k η R k=k k } } } unl b k 0, n errrs b k k =- 4

15 Perceprn lernng des hs mke sense? cnsder he emple bel se k = 0, k = 0, b k = 0 se R = m d { = fr = :n { f b k < 0 hen { k = k η b k = b k η R k=k k } } } unl b k 0, n errrs b k k =- 5

16 Perceprn lernng des hs mke sense? cnsder he emple bel se k = 0, k = 0, b k = 0 se R = m d { = fr = :n { f b k < 0 hen { k = k η b k = b k η R k=k k } } } unl b k 0, n errrs k η b k k =- 6

17 Perceprn lernng des hs mke sense? cnsder he emple bel se k = 0, k = 0, b k = 0 se R = m d { = fr = :n { f b k < 0 hen { k = k η b k = b k η R k=k k } } } unl b k 0, n errrs k b k =- 7

18 Perceprn lernng des hs mke sense? cnsder he emple bel se k = 0, k = 0, b k = 0 se R = m d { = fr = :n { f b k < 0 hen { k = k η b k = b k η R k=k k } } } unl b k 0, n errrs k b k b k η R =- 8

19 Perceprn lernng des hs mke sense? cnsder he emple bel se k = 0, k = 0, b k = 0 se R = m d { = fr = :n { f b k < 0 hen { k = k η b k = b k η R k=k k } } } unl b k 0, n errrs k b k =- 9

20 Perceprn lernng OK, mkes nuve sense h d e kn ll n ge suck n lcl mnmum? hs s Rsenbl s semnl cnrbun herem: Le D = {,,..., n, n } nd R = m If here s,b such h = nd b >, γ hen he Perceprn ll fnd n errr free hper-plne n ms R erns γ 0

21 Prf n h hrd dene ern b, ssume pn prcessed ern - s, fr smplc, use hmgeneus crdnes. Defnng z = R lls he cmpc nn = b / R b = z = snce nl msclssfed pns re prcessed, e hve z < 0

22 Prf h des evlve? d h l l b b/r z R R b R b η η = = = / / denng he pml slun b =, b/r b z = = η η nd, frm, b = η > ηγ slvng he recursn > ηγ ηγ ηγ ηγ > > > >...

23 Prf hs mens cnvergence f e cn bund he mgnude f. Wh s hs mgnude? Snce g g e hve z η = nd, z z η η = = R z η η = < frm frm def f z slvng he recursn R η < frm 3 R η <

24 Prf cmbnng he R nd R η ηγ. < < < b = < b R R b R R γ γ frm def f snce b/ = b s he dsnce he = R b R γ frm = snce b / = b s he dsnce he rgn, e hve b < R = m nd b g R 4 < γ R

25 Ne hs s n he sndrd prf e.g. Dud, Hr, Srk sndrd prf: regulr lgrhm n R n upde equns gher bund < R/γ hs ppers beer, bu requres chsng η = R /γ hch requres knledge f γ, h e dn hve unl e fnd.e. he prf s nn-cnsrucve, cnn desgn lgrhm h he lgrhm bve jus rks! hence, I lke hs prf beer despe lser bund. 5

26 Perceprn lernng herem: Le D = {,,..., n, n } nd R = m If here s,b such h = nd b > γ, hen he Perceprn ll fnd n errr free hper-plne n ms R erns γ hs resul s he sr f lernng her fr he frs me here s prf h lernng mchne culd cull lern smehng! 6

27 he mrgn ne h b γ, ll hld f nd nl f = b γ = mn = mn b =- hch s h e defned he mrgn hs ss h he bund n me cnvergence s nversel prprnl he mrgn even n hs erl resul, he mrgn ppers s mesure f he dffcul f he lernng prblem R γ 7

28 he rle f R sclng he spce shuld n mke dfference s heher he prblem s slvble R ccuns fr hs f he re re-scled bh R nd γ re re-scled nd he bund R γ remns he sme nce gn, jus quesn f nrmlzn γ R = =- llusres he fc h he nrmlzn = s usull n suffcen 8

29 Sme hsr Rsenbl s resul genered lf ecemen bu lernng n he 50s ler, Mnsk nd Pper denfed serus prblems h he Perceprn here re ver smpl lgc prblems h cnn slve mre n hs n he hmerk hs klled ff he enhussm unl n ld resul b Klmgrv sved he d herem: n cnnuus funcn g defned n [0,] n cn be represened n he frm n d g = Γ j Ψ j j = = 9

30 Sme hsr nng h he Perceprn cn be ren s n h l k lk h P l [ ] = = = n b h 0 sgn sgn hs lks lke hvng Perceprn lers ler : J hper-plnes j n ler : hperplne v J j, h n j j j,..., sgn 0 = = = = = J J J j j j j v h v u 0 sgn 30 = = = J j j J j j j j v v 0 0 sgn sgn

31 Sme hsr hch cn be ren s [ ] J sgn u = sgn g h g = v j nd resembles suggesed he de h g = J j = j = hle ne Perceprn s n gd enugh nn d Γ j Ψ j j = = mbe mul-lered Perceprn MLP ll rk j j 0 v lf rk n MLPs ensued under he nme f neurl nerks evenull, s shn h ms funcns cn be pprmed b MLPs j 0 3

32 Grphcl represenn he Perceprn s usull represened s npu uns: crdnes f eghs: crdnes f hmgeneus crdnes: =, bs erm h = sgn 0 = sgn 3

33 Sgmds he sgn[.] funcn s prblemc n s: n dervve 0 f nn-smh pprmns cn be pprmed n vrus s fr emple b he hperblc ngen f f = nh σ = e e e e σ σ σ σ σ cnrls he pprmn errr, bu hs dervve everhere smh f neurl nerks re mplemened h hese funcns 33

34 Neurl nerk he MLP s funcn pprmn 34

35 35

Chapter 2 Linear Mo on

Chapter 2 Linear Mo on Chper Lner M n .1 Aerge Velcy The erge elcy prcle s dened s The erge elcy depends nly n he nl nd he nl psns he prcle. Ths mens h prcle srs rm pn nd reurn bck he sme pn, s dsplcemen, nd s s erge elcy s