Reprucng ernel Hlbert spaces Nun Vascncels ECE Department UCSD
Classfcatn a classfcatn prblem has tw tpes f varables X -vectr f bservatns features n the wrl Y - state class f the wrl Perceptrn: classfer mplements the lnear ecsn rule h sgn[ g ] wth T g w + b w apprprate when the classes are lnearl separable t eal wth nn-lnear separablt we ntruce a ernel b w g w
Kernel summar. D nt lnearl separable n X appl feature transfrmatn Φ:X Z such that mz >> mx. cmputng Φ t epensve: wrte ur learnng algrthm n t-pruct frm nstea f Φ we nl nee Φ T Φ 3. nstea f cmputng Φ T Φ efne the t-pruct ernel K z Φ T Φ z an cmpute K rectl nte: the matr M K LK L M s calle the ernel r Gram matr 4. frget abut Φ an use Kz frm the start! 3
4 Plnmal ernels ths maes a sgnfcant fference when Kz s easer t cmpute that Φ T Φz e.g. we have seen that whle Kz has cmplet O Φ T Φz s O fr Kz T z we g frm O t O T T T z z z K L L L M R R Φ Φ Φ : wth
Questn n practce: pc a ernel frm a lbrar f nwn ernels we tale abut the lnear ernel Kz T z the Gaussan faml the plnmal faml K z e z σ what f ths s nt g enugh? T + z { L} K z hw I nw f a functn s a t-pruct ernel? 5
Dt-pruct ernels let s start b the efntn Defntn: a mappng : X X R s a t-pruct ernel f an nl f <ΦΦ> where Φ: X H H s a vectr space <..> s a t-pruct n H nte that bth H an <..> can be abstract nt necessarl R n 3 Φ X H 6
Dt pruct vs pstve efnte ernels that s prett abstract. hw I turn t nt smethng I can cmpute? Therem: X s a t-pruct ernel f an nl f t s a pstve efnte ernel ths can be chece let s start b the efntn Defntn: s a pstve efnte ernel n X X f l an {... l } X the Gram matr s pstve efnte. [ K ] 7
Pstve efnte matrces recall that e.g. Lnear Algebra an Applcatns Strang Defntn: each f the fllwng s a necessar an suffcent cntn fr a real smmetrc matr A t be pstve efnte: T A 0 0 all egenvalues f A satsf λ 0 all upper-left submatrces A have nn-negatve etermnant v there s a matr R wth nepenent rws such that A R T R upper left submatrces: A a A a a a a A 3 a a a 3 a a a 3 a a a 3 3 33 L 8
9 Dt pruct vs pstve efnte ernels equvalence between t pruct an pstve efnte autmatcall prves sme smple prpertes t-pruct ernels are nn-negatve functns t-pruct ernels are smmetrc Cauch-Schwarz nequalt fr t-pruct ernels nte that ths s ust X 0 X X cs - 44 4 3 4 4 θ
Dt pruct vs pstve efnte ernels the prf actuall gves nsght n what the ernel es we efne H as the space spanne b lnear cmbnatns f. H m f. f. α. m X e.g. the the space f all lnear cmbnatns f Gaussans when the we apt the Gaussan ernel. σ K. e nte: ths s a functn f the frst argument s fe 0
Dt pruct vs pstve efnte ernels f f. an g. H wth f. m α we efne the t-pruct <..> * n H as fr the Gaussan ernel ths s m '. g. β. ' m m ' α ' * f g β f m m ' ' σ g α β e * a t pruct n H a nn-lnear measure f smlart n X smewhat relate t lelhs
The t-pruct <..> * t s nt har t shw that ths means that.. * wth Φ Φ * Φ: X H. the feature transfrmatn asscate wth the ernel sens the pnts t the functns. furthermre <..> * s tself a t-pruct ernel n H H
The reprucng prpert wth ths efntn f H an <..> * f H <.f.> * f ths s calle the reprucng prpert an analg s t thn f lnear tme-nvarant sstems the t pruct as a cnvlutn. as the Drac elta f. as a sstem nput the equatn abve s the bass f all lnear tme nvarant sstems ther we wll see that t als plas a funamental rle n the ther an use f reprucng Kernel Hlbert Spaces 3
The bg pcture when we use the Gaussan ernel the pnt X s mappe nt the Gaussan G σi H s the space f all functns that are lnear cmbnatns f Gaussans the ernel s a t pruct n H an a nn-lnear smlart n X reprucng prpert n H: analg t lnear sstems X Φ n K e 3 * * σ * H 4
Hlbert spaces pla an mprtant rle n functnal analss we wll ver nfrmal revew H s a vectr space wth a t pruct: ths s nwn as a t-pruct space r a pre-hlbert space the fference t a Hlbert space s mstl techncal Defntn: a Hlbert space s a cmplete t-pruct space what we mean b cmpleteness? Defntn: S s cmplete f all Cauch sequences n S cnverge ths s useful mstl t prve cnvergence results 5
Cauch sequences ust fr cmpleteness n pun ntene Defntn: a sequence {...} n a nrme space s a Cauch sequence f ε n N s.t. n ' n" > n n ε 0 ' n " u can pcture ths as ε n wh mght ths nt cnverge? well the lmt pnt cul be utse the space wh s ths mprtant? cmplete cnvergent Cauch 6
reprucng ernel Hlbert spaces t turn ur pre-hlbert space H nt a Hlbert space we have t cmplete t wth respect t the nrm f f f * * ths s ust ang t H the lmt pnts f all ts Cauch sequences we represent the cmpletn f H b H mre than an Hlbert space H becmes a reprucng ernel Hlbert space RKHS 7
reprucng ernel Hlbert spaces Defntn: Let H be a Hlbert space f functns f: X R. H s a RKHS enwe wth t-pruct <..> * f : X X R such that. spans H.e. { } {α } such that H.<f..> * f f H n summar span {. } f. f. α. the H we bult can easl be transfrme nt a RKHS b ang t t the lmt pnts functns f all Cauch seqs Questn: s there a ne-t-ne mappng between ernels an RKHSs? 8
ernels vs RKHSs answer: RKHS es nee specf ernel unquel prf: assume that ernels an span H usng reprucng prpert <.. > * <..> * hence frm smmetr f t pruct <..> * we must have an frm smmetr f ernel t fllws that snce ths hls fr all an the ernels are the same 9
ernels vs RKHSs hwever t s nt clear that a ernel unquel specfes the RKHS there mght be multple feature transfrms an t-pructs that are cnsstent wth a ernel t stu ths we nee t ntruce Mercer ernels Defntn: a smmetrc mappng : X X R such that f f 0 s a Mercer ernel f s.t. f < * 0
Mercer ernels wh we care abut them? Tw reasns Therem: a ernel s pstve efnte an t-pruct f an nl f t s a Mercer ernel prf that Mercer mples PD: cnser an sequence {... n } X. Let snce f < f s Mercer t fllws frm * that 0 n f s. t. w < f wδ f n w w δ δ
Mercer ernels 0 w w δ δ w w w T Kw snce ths hls fr an w s PD
3 Mercer ernels prf that PD mples Mercer: suppse there s a g s.t. cnser a parttn f X X fne enugh that hence wth u g... g n T K s nt PD an nt a PD ernel the Therem fllws b cntractn < 0 ε g g ε g g g g 0 0 < < Ku u g g T
Mercer ernels the have the fllwng prpert wthut prf Therem: Let : X X R be a Mercer ernel. Then there ests an rthnrmal set f functns an a set f λ 0 such that φ φ δ λ λ φ φ ntutn: thn f ths as a D Furer seres +Parseval ** 4
5 Egenfunctns nte: f we efne the peratr then the functns φ are the egenfunctns f Tf agan a cnnectn t lnear sstems LTI f h- f Tf T λ φ φ φ φ λ φ φ φ λ φ φ δ 4 43 4 4
Mercer ernels the egenfunctn ecmpstn gves us anther wa t esgn the feature transfrmatn Φ : X l λ φ λ φ L T where l s the space f vectrs s.t. a < an the number f nn-zer egenvalues λ clearl Φ T Φ.e. there s a vectr space l ther than H s.t. s a t pruct n that space 6
The pcture ths s a mappng frm R n t R X Φ l Φ 3 much mre le t a mult-laer Perceptrn than befre the ernelze Perceptrn as a neural net Φ. Φ. Φ. 7
In summar Reprucng ernel map Mercer ernel map f. f. α. m H K m m ' α ' * f g β Φ :. H M f g l * f T g < λ φ λ L T Φ : φ where λ φ are the egenvalues an egenfunctns f tw ver fferent pctures f what the ernel es are the tw spaces reall that fferent? 8
RK vs Mercer maps nte that fr H M we are wrtng r Φ λ φ e + L+ λφ r but snce the φ. are rthnrmal there s a - map Γ : l r e span λ { φ.} φ. e an we can wrte Γ Φ λ φ φ.. + L+ λ φ φ. frm ** *** hence. maps nt M span{φ.} 9
RK vs Mercer maps efne the t pruct n M s that then {φ.} s a bass M s a vectr space an functn n M can be wrtten as an φ φ φ φ ** λ f α φ f.. **.e. s a reprucng ernel n M α φ. λ α λφ φ φ 3 δ λ φ. φ δ λ ** ** α φ f 30
RK vs Mercer maps furthermre snce. M an functns f the frm are n M an. g.. α f β. f g ** α. α α α β β β lm l l λ φ. φ l φ φ φ. φ. λ φ φ l l λ λ m l l β. l nte that fg H an ths s the t pruct we ha n H l l m m ** λ m l m φ. φ m m ** ** α β 3
In summar H M an <..> * n H s the same as <..> ** n M Questn: s M H? nee t shw that an f H frm *** an fr an sequence {... } α φ. λ φ φ. + L+ λ f there s an nvertble P then φ α. H an M H. λφ λ φ φ. M M L M.. λφ λ φ φ 4444 44444 3 P φ φ. 3
In summar snce λ > 0 φ φ λ P M L M φ φ 0 444 4444 3 Π 0 λ s nvertble when Π s. If Π s nt nvertble then f there s n sequence fr whch Π s nvertble then the φ cannt be rthnrmal. Hence there must be nvertble P an M H. λ 0 α 0 s.t. αφ 0 α 0 s.t. α φ 33
In summar H M an <..> * n H s the same as <..> ** n M the reprucng ernel map an the Mercer ernel map lea t the same RKHS Reprucng ernel map m H K f. f. α. m m ' α * ' f g β Φ r :. Mercer ernel map H M Φ l f g * f T g < λ φ λ L T M : φ Γ ΦM Φ r Γ : l span{ φ.} r e λ φ. 34
35