Radial Basis Function Networks

Transcription

1 Radal Bass Fucto Netorks

2 Radal Bass Fucto Netorks A specal types of ANN that have three layers Iput layer Hdde layer Output layer Mappg from put to hdde layer s olear Mappg from hdde to output layer s lear PR ANN & ML

3 Comparso Mult-layer perceptro Multple hdde layers Nolear mappg W: er product lobal mappg Warp classfers Stochastc appromato RBF Netorks Sgle hdde layer Nolear + lear W: dstace Local mappg Warp data Curve fttg PR ANN & ML 3

4 Aother Ve: Curve Fttg We try to estmate a mappg from patters to classes fpatters->classes f f->d Patters are represeted as feature vector Classes are decsos d rag samples: f ->d =... Iterpolato of the f based o samples d PR ANN & ML 4

5 Yet Aother Ve: Warpg Data If the problem s ot learly separable MLP ll use multple euros to defe complcated decso boudares arp classfers Aother alteratve s to arp data to hgher dmesoal space that they are much more lkely to be learly separable sgle perceptro ll do hs s very smlar to the dea of Support Vector Mache PR ANN & ML 5

6 Eample OR Warpped OR y e [00] t y e t [] ] PR ANN & ML 6

7 More Eample PR ANN & ML 7

8 A Pure Iterpolato Approach ve: d = Desred: f = d Soluto: f th f = d Radal bass fucto soluto geeral form s shft ad rotato varat Shft varat requres - Rotato varat requres - Eample Multquadrcs Iserve Multquadrcs aussa f r r c r r c r r e PR ANN & ML 8

9 raphcal Iterpretato Each euro respods based o the dstace to the ceter of ts receptve feld he bottom level s a olear mappg he top level s a lear eghted sum f PR ANN & ML m 9

10 Other Alteratves: lobal Lagrage polyomals k y k L f y 0 k k k k k k o k k k o k k k k L y f y PR ANN & ML 0

11 Other Alteratves: Local Bezer Bass B-sple bass PR ANN & ML

12 B-Sple Iterpolato A bg subect mathematcs Used may dscples Appromato Patter recogto Computer graphcs As far as patter recogto s cocered Determe order of sple DOFs Kot vectors partto to tervals Fttg each terval PR ANN & ML

13 Iterpolato Soluto d f d d d D Φ W D ΦW s symmetrcal s vertable f all s are dstct PR ANN & ML 3 s vertable f all s are dstct

14 Practcal Issue: Accuracy cot. he fucto represets the ree s fucto for a certa dfferetal operator Whe t s shft ad rotatoal varat e ca rte as - aga aussa Kerel s a popular choce here PR ANN & ML 4

15 Accuracy Practcal Issues Ho about data are osy? Speed Ho about there are may sample pots? rag What s the trag procedure? PR ANN & ML 5

16 Practcal Issue: Accuracy Whe data are osy pure terpolato represets a form of overfttg Need a stablzg or smoothg regularzato term he soluto should acheve to thgs ood fttg Smoothess PR ANN & ML 6

17 Practcal Issue: Accuracy cot. Df d f m Df d * D W D W o o he soluto s rooted the regularzato theory hch s ay beyod the scope of ths course read hch s ay beyod the scope of ths course read the papers o the class Web stes for more detals ry to mmze error as a eghted sum of to PR ANN & ML 7 y g terms hch mpose the fttg ad the smoothess costrats

18 Sdebar I It ca be prove that MAP estmator Baysa rule gves the same results as regularzed RBF soluto U-regularzed fttg soluto assumes the same pror Df d f P f P f P f P c f P f P f P P log log log PR ANN & ML 8

19 Sdebar II Regularzato s also smlar to or call rdge regresso statstcs he problem here s to ft a model to data thout t overfttg I lear case e have rdge arg m y o rdge arg m y o subectto s PR ANN & ML 9

20 Ituto Whe varables are hghly correlated ther coeffcets become poorly determed th hgh varace E.g. dely large postve coeffcet o oe ca be caceled by a smlarly large egatve coeffcet o ts correlated cous Sze costrat s helpful Caveat: costrat s problem depedet PR ANN & ML 0

21 Soluto to Rdge Regresso Smlar to regularzato o rdge y arg m W W Y W Y W W W Y W Y W rdge d m arg W Y W W W Y W Y W d d 0 0 Y I Y W I W Y W rdge λ 0 PR ANN & ML Y I rdge λ

22 Ugly Math Y I rdge λ Y U VΣ I UΣ U VΣ UΣ Y I Y rdge λ V V λ Y U Σ V I UΣ U VΣ UΣ Y U VΣ I UΣ U VΣ UΣ λ V V λ V V Y U Σ I Σ Σ UΣ Y U Σ I V UΣ U VΣ V UΣ λ V λ V V Y u u λ PR ANN & ML λ

23 Physcal Iterpretato Sgular values of represets the spread of data alog dfferet body-fttg dmesos o estmate Y= rdge regularzato mmzes the cotrbuto from less spread-out dmesos Less spread-out dmesos usually have much larger varace hgh dmeso ege modes race +I - s called effectve degrees of freedom PR ANN & ML 3

24 More Detals race +I - s called effectve degrees of freedom Cotrols ho may ege modes are actually used or actve Dfferet methods are possble Shrkg smoother: cotrbutos t are scaled Proecto smoother: cotrbutos are used or ot used 0 PR ANN & ML 4

25 Practcal Issue: Speed Whe there are may trag samples ad matrces are of sze by Ivertg such a matr s of O 3 Reducg the umber of bases used PR ANN & ML 5

26 Practcal Issue: Speed cot. m * m< m Df d f * * m o D W m m m m m o m PR ANN & ML 6 m m m

27 Practcal Issue: rag Ho ca the ceter of radal bass fuctos for the reduced bass set be determed? Chose radomly rag volves fdg usg SVD PR ANN & ML 7

28 rag th K-mea Usg usupervsed clusterg Fd here data are clustered that t s here the radal bass fuctos should be placed Wth k-mea PR ANN & ML 8

29 K-Meas Algorthm fed # of clusters Arbtrarly pck N cluster ceters assg samples to earest ceter Compute sample mea of each cluster Reassg samples to clusters th the earest mea for all samples Repeat f there are chages otherse stop PR ANN & ML 9

30 PR ANN & ML 30

31 PR ANN & ML 3

32 PR ANN & ML 3

33 rag th radet Decet Error Epresso m d Free varables the error epresso are Weght Ceter locato Bass spread PR ANN & ML 33

34 Effect of Weghts m d m d d PR ANN & ML 34

35 Effect of Ceter Postos m d m d ' Σ PR ANN & ML 35

36 Effect of Bass Spread m d m d ' Q Q Σ Σ Σ Σ Σ PR ANN & ML 36 Σ

37 Detals A lot of theoretcal developmet results are omtted here E.g. relato to kerel regresso ad SVM A lot of tug cosderatos are ot covered here E.g. ho to determe? hs s a actve research area PR ANN & ML 37

38 Eamples PR ANN & ML data pots ceters Accuracy of.40-6 for all data pots

39 Problem Defto ve a pot cloud of fdata From laser rage scaer or C MR etc. Fd a sgle aalytcal surface appromato Or a sde-outsde fucto Rage data are s=0 Outsde s s>0 Isde s s<0 Just sample data s=0 s ot eough s ca be a trval zero fucto Need off-surface data geerato PR ANN & ML 39

40 Procedures. Off surface data geerato. Choose a subset from the terpolato ode ad ft a RBF oly to these 3. Evaluate the resdeual e = f s 4. If mae <accuracy the stop 5. Else apped e ceters here e s large 6. Re-ft RBF ad go back to step PR ANN & ML 40

41 More Results Less smoothg More smoothg PR ANN & ML 4