Reproducing kernel Hilbert. Nuno Vasconcelos ECE Department, UCSD

Transcription

1 Reprducng erne Hbert spaces and reguarzatn Nun Vascnces ECE Department, UCSD

2 Casscatn a casscatn prbem has tw types varabes X - vectr bservatns eatures n the wrd Y - state cass the wrd Perceptrn: casser mpements the near decsn rue h sgn[ g ] wth T g w + b w apprprate when the casses are neary separabe t dea wth nn-near separabty we ntrduce a erne b w g w

3 Types ernes these three are equvaent dt prduct erne pstve dente erne Mercer erne 3

4 Dt-prduct ernes Dentn: a mappng : X X R,y,y s a dt-prduct erne and ny,y <Φ,Φy> where Φ: X H, H s a vectr space, <.,.> s a dt-prduct n H X 1 Φ n 1 3 H 4

5 pstve dente and Mercer ernes Dentn:,y s a pstve dente erne n X X and { 1,..., }, X, the Gram matr { 1 } s pstve dente. [ K ], Dentn: a symmetrc mappng : X X R such that, y y ddy 0, s a Mercer erne st s.t. d < 5

6 Tw derent pctures derent dentns ead t derent nterpretatns what the erne des Reprducng erne map Mercer erne map..., 1 m H K, m m ' ' 1 1, g β H M, g d 1 d T g < Φ :., Φ : φ, φ, L T : 1 1 φ where,φ are the egenvaues and egenunctns,y > 0 6

7 The dt-prduct pcture when we use the Gaussan erne K, e the pnt X s mapped nt the Gaussan G,,σI σ H s the space a unctns that are near cmbnatns Gaussans the erne s a dt prduct n H, and a nn-near smarty n X reprducng prperty n H: anagy t near systems X Φ 1 n 1 3 H K 7

8 The Mercer pcture ths s a mappng rm R n t R d X 1 Φ d 1 3 Φ d much mre e t a mut-ayer Perceptrn than bere the ernezed Perceptrn as a neura net Φ 1. Φ. Φ d. 8

9 The reprducng prperty wth ths dentn H K and <.,.> H K, <.,,.> ths s caed the reprducng prperty an anagy s t thn near tme-nvarant systems the dt prduct as a cnvutn., as the Drac deta. as a system nput the equatn abve s the bass a near tme nvarant systems thery eads t reprducng Kerne Hbert Spaces 9

10 reprducng erne Hbert spaces Dentn: a Hbert space s a cmpete dt-prduct space vectr space + dt prduct + mt pnts a Cauchy sequences Dentn: Let H be a Hbert space unctns : X R. H s a RKHS endwed wth dt-prduct <.,.> : X X R such that 1. spans H,.e., { }, { }, such that span.,..., H { }.<.,.,>, H, 10

11 Mercer ernes hw derent are the spaces H K and H M? Therem: Let : X X R be a Mercer erne. Then, there ests an rthnrma set unctns φ φ d δ and a set 0, such that 1, y, y ddy 1 φ φ y 11

12 RK vs Mercer maps nte that r H M we are wrtng r Φ φ e + L + φ φ1 but, snce the φ. are rthnrma, there s a 1-1 map Γ d : r e span { φ.} φ. r e e d and we can wrte Γ Φ φ φ. + L + φ φ , d d d rm hence., maps nt M span{φ.} 1

13 The Mercer pcture we have d Φ Φ X e 1 e d ΤοΦ 1 e 3 e M Γ ΤοΦ., φ 1 φ d Γ 13 φ 3 φ φ d

14 RK vs Mercer maps dene the dt prduct n M s that 1 1 φ, φ φ φ d δ then {φ.} } s a bass, M s a vectr space, any unctn n M can be wrtten as φ and.,.,.,. φ φ φ φ φ, φ φ δ.e., s a reprducng erne n M 14

15 RK vs Mercer maps urthermre, snce., M, any unctns the rm are n M and., g.. β.,, g.,, β β β β m., φ. φ, m m m φ φm φ., φm. φ φ m φ. φ m β, nte that,g H K and ths s the dt prduct we had n H K 15

16 The Mercer pcture urthermre, nte that r n H M φ and snce φ d δ φ φ φ φ 1 1, the dt prduct n H M s g φ φ β g β φ φ β,, 16

17 The Mercer pcture we have d Φ Φ X e 1 e d ΤοΦ 1 e 3 e M H K Γ ΤοΦ.,,, g β φ 1 φ d Γ 17 φ 3 φ φ d g β,

18 RK vs Mercer maps H K M and <.,.> n H K s the same as <.,.> n M Questn: s M H K? need t shw that any H K rm, and, r any sequence { 1,..., d }, φ., φ1 φ1. + L+ φ φ. 1 d d d., 1 1φ 1 1 d φd 1 φ1. M M L M.,. d 1φ 1 d d φd d φd there s an nvertbe P, then φ., H K and M H K P 18

19 RK vs Mercer maps snce > 0 φ1 1 φ d P M L M φ 1 d φ d d 0 d Π s nvertbe when Π s. I Π s nt nvertbe, then 0 0 s.t. φ there s n sequence r whch Π s nvertbe then 0 0 s.t. φ the φ cannt be rthnrma. Hence there must be nvertbe P and M H K. 19

20 The Mercer pcture we have H M d Φ Φ Μ X H M Μ e 1 e d ΤοΦ 1 e 3 e MH K Γ ΤοΦ.,,, g β φ 1 φ d Γ 0 φ 3 φ φ d g β,

21 In summary H K M and <.,.> n H K s the same as <.,.> n M the reprducng erne map and the Mercer erne map ead t the same RKHS, Mercer gves us an.n. bass Reprducng erne map H m K..., 1 m m ' 1 1 ', g β r :.,, Mercer erne map H M, g d 1 d T g < Φ Φ φ,, L T Γ ΦM Φ r M : 1 1 φ d : { φ.} ths turns ut t be true r any RKHS btaned rm,y 1-1 reatnshp Γ r e span φ. 1

22 Reguarzatn Q: RKHS: why d we care? A: reguarzatn we want t d we utsde the tranng set mnmzng emprca rs s nt enugh need t penaze cmpety eampe: regressn gve me n pnts, I w gve yu a mde zer errr pynma rder n-1 ncredby wggy pr generazatn

23 Reguarzatn we need t reguarze the rs R reg [ ] R [ ] + Ω [ ] emp s the unctn we are tryng t earn > 0 s the reguarzatn parameter Ω s a cmpety penazng unctna arger when s mre wggy the reguarzed rs can be usted n varus ways cnstraned mnmzatn Bayesan nerence structura rs mnmzatn... 3

24 Cnstraned mnmzatn reguarzed rs as the sutn t the prbem [ ] subect t [ ] t arg mn R Ω emp we w ta a t mre abut ths sutn s the sutn the Lagrangan prbem arg mn { R [ ] + Ω[ ]} emp s a Lagrange mutper changng s equvaent t changng t the cnstrant n the cmpety the ptma sutn 4

25 Bayesan nerence n Bayesan nerence we have ehd unctn P X ehd unctn P X prr P and search r the mamum a psterr MAP estmate arg ma P P P X { } arg ma arg ma P P P P P X X X { } [ ] { } g g arg ma arg ma P P P P X X + 5

26 Bayesan nerence P X e -Remp[] and P e -Ω[] the MAP estmate s the mnmum the reguarzed rs [ ] + Ω[ ] { R } arg mn emp the prr P assgns w prbabty t unctn wth arge Ω[] ths reects ur prr bee that t s uney that the sutn w be very cmpe eampe: Ω[] - 0,theprr s a Gaussan centered at 0 wth varance 1/ 1/ we beeve that the mst ey sutn s 0 the arger the, the mre we penaze sutns whch h are derent rm ths 6

27 Structura rs mnmzatn start rm a nested cectn ames unctns S L S 1 where S {h,, r a } r each S, nd the unctn set parameters that mnmzes the emprca rs R emp 1 mn n 1 seect the unctn cass such that R mn n L[ y, h, ] h { R + Φ } emp Φh s a unctn VC dmensn cmpety the amy S VC have shwn that ths s equvaent t mnmzng a bund n the rs, and prvdes generazatn guarantees reguarzatn wth the rght reguarzer! 7

28 The reguarzer what s a gd reguarzer? ntutn: wgger unctns have a arger nrm than smther unctns r eampe, n H K we have, c φ φ φ φ φ 8

29 The reguarzer and wth c hence, δ cc φ, φ cc φ grws wth the number c derent than zer ths s the case n whch s mre cmpe, snce t becmes a sum mre bass unctn φ dentca t what happens n Furer type decmpstns mre cecents means mre hgh-requences r ess smthness c 9

30 Reguarzatn OK, reguarzatn s a gd dea rm mutpe pnts vew prbem: mnmzng the reguarzed rs R reg [ ] R [ ] + Ω[ ] emp ver the set a unctns seems e a nghtmare t turns ut that t s nt, under sme reasnabe cndtns n the reguarzer Ω the ey s the representer therem 30

31 Representer therem Therem: Let Ω:[0, R be a strcty mntncay ncreasng unctn, H the RKHS asscated wth a erne,y L[y,] a ss unctn then, n arg mn L 1 [ ] y, + Ω [ ] admts a representatn the rm n 1.,.e. H 31

32 Pr decmpse any nt the part cntaned n the span the., and the part n the rthgna cmpement where 0 w 0 and w wth., 1 0 n + + where 0 w 0 and w wth { } { } { } 0 0 0,,., w g g w span w then { } 0, g, g [ ] + Ω Ω n [ ] Ω + Ω + Ω Ω 1., n n Ω mn. ncreasng 3 Ω + Ω 1 1.,.,

33 Pr ths shws that the secnd term n n arg mn L y, + Ω 1 s mnmzed by a unctn the stated rm. [ ] the rst, usng the reprducng prperty ,.,.,.,.,., + n 1, s aways a unctn the stated rm. hence, the mnmum must be ths rm as we 48 33

34 The pcture due t reprducng prperty: H s the sutn n H > s a mre cmpe unctn whch des nt reduce the ss hence, s the ptma sutn H e 1 e d e 3 34

35 Reevance the remarabe cnsequence the therem s that: we can reduce the mnmzatn ver the nnte dmensna space unctns t a mnmzatn n a nte dmensna space! n.,, t see ths nte that, because we have 1 T K.,,,.,, K, 35

36 Reguarzatn ths prves the wng therem Therem: Ω:[0, R s a strcty mntncay ncreasng unctn, then r any dt-prduct erne,y and ss unctn L[y,] the sutn n arg mn L 1 n [ ] y, + Ω n [ ] T K s wth., arg mn LY, K + Ω 1 1 K the Gram matr [, ] and Y...,y,... 36

37 Nte the resut that H hds r any nrm, nt ust L hwever, the eact rm the prbem w n nger be n arg mn LY T K 1 [, K ] + Ω K the argument the secnd term w have a derent rm 37

38 38

39 Prects at ths tme yu shud have a reasnabe dea what yu are gng g t d ur questns what are yu gng t d? why shud yu d t? hw are yu gng t d t? what resuts d yu epect t get? net thursday: n cass, mnute meetngs t dscuss the ur questns send me ema sayng yu are avaabe at 3:00 be prepared, n tme t spare 39

40 Pnters besdes the bs n syabus Prceedngs Neura Inrmatn Systems Prceedngs Internatna Cnerence n Machne Learnng Neura Cmputatn IEEE Transactns n Pattern Anayss and Machne Integence Jurna Machne Learnng Research IEEE Cnerence n Cmputer Vsn and Pattern Recgntn Internatna Cnerence n Cmputer Vsn IEEE Int. Cnerence e Acustcs, cs, Speech, Sgna Prcessng IEEE Internatna Cnerence Image Prcessng Internatna Jurna Cmputer Vsn As the INSPEC database and Gge 40

41 Prect tpcs msty undergrad casscatn graphcs vs natura mages n gge vsuazng data mands cmparsn LDA-based ace recgntn methds 41