Other Topics in Kernel Method Statistical Inference with Reproducing Kernel Hilbert Space

Transcription

1 Oher Topcs Kerel Mehod Sascal Iferece wh Reproducg Kerel Hlber Space Kej Fukumzu Isue of Sascal Mahemacs, ROIS Deparme of Sascal Scece, Graduae Uversy for Advaced Sudes Sepember 6, 008 / Sascal Learg Theory II

2 Oule. Relao o fucoal daa aalyss. Sple smoohg 3. Relao o radom process

3 Oule. Relao o fucoal daa aalyss. Sple smoohg 3. Relao o radom process 3

4 Fucoal daa aalyss For fucoal daa aalyss, see Ramsay & Slverma (005) Wha are fucoal daa? Daa: f (), f (),, f () -- fucos o a erval [a, b] Example: Berkeley Growh Sudy See hp:// 4

5 Coverg raw daa o fucoal form Daa are ofe gve by a se of {( j, y ( j )) =,,, j =,, m }. Coverg daa by smoohg For each, f a curve φ () o dvdual daa {( j, y ( j )) j =,, m } by smoohg (e.g. B-sple) The covered daa are of he form φ ) = c θ ( ) + L+ c θ ( ), θ ( ), K, θ ( ) : bass fucos Aalyss o fucoal daa Apply lear mehods o he covered daa a fuco space (ypcally L ). Examples: ( l l Fucoal PCA Fucoal CCA Fucoal lear modelg, ec. l 5

6 Fucoal PCA Fucoal daa: φ (),, φ () (already covered) Fd a fuco o maxmze Varace of he projecos o he dreco of f If bass fucos θ ( ), K, θ ( ) are used, l Solve: where The egral R s compued umercally, or by he propery of he bass 6

7 Kerel mehod v.s. fucoal daa Smlary aalyss Boh he mehods exeds lear mehods o fucoal daa. Dfferece I kerel mehods, he daa coverso s gve by a posve defe kerel, whle FDA he daa are assumed o be fucoal. Kerel mehods use RKHS as a fuco space, whle he FDA uses L space prcple. Roughess pealy FDA I FDA, smoohess s somemes mposed o he soluo. Ths s esseally he Sobolev orm (RKHS). Wh roughess pealy, FDA s more smlar o kerel mehods. 7

8 Oule. Relao o fucoal daa aalyss. Sple smoohg 3. Relao o radom process 8

9 Sple smoohg (X,Y ),..., (X, Y ) : X R, Y R P: dffereal operaor o R Sple smoohg: = ( ) + λ m Y f( X ) Pf( x) dx f Roughess pealy 9

10 Laplaca ad Gree fuco Laplaca Self-adjo: f f f Δ f = + + L+ x x x f f (x), g (x) 0 Δ f ( x) g( xdx ) = f ( x) Δg( xdx ) Gree fuco for Laplaca Δ Gx (, ξ ) = δ( x ξ).e. Gree fuco solves a dffereal equao: ) f ( x) = G( x, y) ϕ( y) dy Δ f = ϕ gve ϕ. 0

11 Smoohg pealy Regularzao erm Cosder fucos o R for smplcy (o boudary) m! α Jm( f) = D f α! α! Lα! = α + L+ α = m m m! f α α α!!! α + L+ α = mα α Lα x x L x L L orm of m-h dervave dx example ( = m = ) f f f J ( f) = + + dx x x x x

12 Smoohg ( ) + λ m m f = m= 0 m Y f( X ) a J ( f) ( a m 0) Expresso by Laplaca Paral egral shows m m ( ) ( m ) J ( f) = f, Δ f The smoohg problem s expressed by = ( Y f X ) + λ ( f Af ) m ( ), f L L where A = ( ) m a Δ m m= 0 m

13 Case Two cases a0 0 The Gree fuco s a posve defe kerel. The pealy erm s equal o he squared RKHS orm. Case a = 0 0 Sple smoohg The Gree fucos s codoally posve defe. The fucoal space s RKHS + polyomal of some order The pealy erm s equal o he squared RKHS orm of he projeco of f oo he RKHS. 3

14 a 0 0 : RKHS regularzao Soluo = Varaoal calculus ( Y f X ) + λ ( f Af ) m ( ), f = ( ) Y f( x) δ( x X ) + λaf = 0 Af = Y f( x) δ ( x X ) λ = = = ( ) If we have he Gree fuco G for A.e. f ( ξ ) = ( Y f ( x)) δ( x X ) G( x, ξ) dx λ = ( Y f( X )) G( ξ, X ) λ oe: f(x ) ukow L 4

15 The soluo s o have he form: f = cg(, X ) = Plug o he orgal problem: c R = ( ) j j j = j + λ, j= j m Y c G( X, X ) cc G( X, X ) Q) By dffereao, c= ( G+ λi) Y where G j = G( X, X ) j Y = ( Y, K, Y ) T The soluo: T Y where g ( ) (, x = G x X ) f( x) = ( G+λI) g( x) 5

16 Gree fuco Theorem If a0 0, a j 0( j ), he Gree fuco of A s a posve defe kerel. Proof. Sce Α s shf vara, so s G. Thus, m= 0 By Fourer rasform m m ( ) a Δ G( z) = δ ( z) m If a0 0, a j 0( j ), he Fourer verso s possble. Use Bocher s heorem. 6

17 Regularzao by RKHS orm Assume a0 0, a 0 G: Gree fuco of A. H G : RKHS w.r.. G. ( ) + λ m m f = m= 0 m Y f( X ) a J ( f) The soluo s gve by The pealy erm s, he, f = = cg (, X ) The above regularzao s equvale o he kerel rdge regresso = ( ) Y f X + λ f m ( ) f H G 7

18 a 0 = 0: Sple smoohg Th-plae sple = ( ) m m Y f( X ) + λj ( f) f m! α Jm( f ) = D f α + L+ α = α! α! Lα! m L The Gree fuco of s o ecessarly posve defe, bu codoally posve defe The fuco space for f s ad m α B : D f L ( R ) ( α = m) m J ( f) = 0 f P m P m- : Polyomals of degree a mos m - 8

19 B m = P H Le be decomposo by drec sum. m * Theorem (Megue 979) If m > /,he subspace H * s a RKHS wh er produc m! α = m α! Lα! I parcular, he orm s gve by H f = J ( f) * ( α α ) ( m m ) f, g = D f, D g = ( ) Δ f, g H * L L m = ( ) m m Y f( X ) + λj ( f) f g H* p Pm, = ( ) Y g X + p X + λ g H* m ( ( ) ( )) 9

20 Soluo of sple smoohg By he represeer heorem, he soluo s o be of he form: By pluggg, f ( x) = ck( x X ) + bφ ( x) M l l = l= The soluo: 0

21 Codoally posve defe Defo. K(x,y) : Ω x Ω R s sad o be codoally posve defe of order m f. K(x,y) = K(y,x). If pos x,, x Ω ad real umbers c,, c sasfy = cpx ( ) = 0 for ay polyomal (geeralzed creme of order m), he j, = cc jk( x, xj) 0 A posve defe kerel s codoally posve defe of order 0. A egave defe kerel s egao of a codoally posve defe kerel of order. Iuo: he above c,, c s a geeralzao of he m-h order dfferece. Thus, he defo uvely says ha he m-h dervave of K s posve defe.

22 s order dff.: Coeff. of f ( + ) f( ) + f( ) c c c 3 c c c L+ = coeffces of s order dfferece d order dff.: Coeff. of ( ) f f( + ) f( + ) f( + ) f( ) c+ c + L+ c = 0 c + c + L + c = c c c 3 c coeffces of d order dfferece

23 Oule. Relao o fucoal daa aalyss. Sple smoohg 3. Relao o radom process 3

24 Gaussa process A Gaussa process s a radom process (radom varables wh dex Ω) such ha for ay fe subse {,..., } of Ω, he radom vecor ( X, K, ) s a Gaussa radom vecor. Mea fuco Covarace fuco X A Gaussa process s uquely deermed by he mea ad covarace fuco. X = ( X,..., X ) R (, ) R (, ) L R (, ) R (, ) R (, ) L R (, ) μx = ( μ( ), K, μ( )), Σ X = M M O M R (, ) R (, ) L R (, ) 4

25 Examples σ = σ = 0.3 mea zero covarace fuco Rs (,) = exp ( s ) σ Geeraed by Malab gpml oolbox (Rasmusse ad Wllams) 5

26 Radom process ad posve defe kerel Covarace fuco s a posve defe kerel Theorem The covarace fuco R(s, ) of a radom process s a posve defe kerel. Q) For smplcy, mea = 0., j= cc jr(, j ) =, j= cc je[ X, X ] j E cx j cjx E = = = cx =, = ( ) 0 j A radom process o Ω deermes a RKHS o Ω. 6

27 Posve defe kerel defes Gaussa process k(s,): posve defe kerel o Ω. For ay fe subse = (,, ) of Ω, he Gram marx Σ = ( k(, j ) ) s always posve semdefe. By Kolmogorov exeso heorem, here s a Gaussa process wh dex se Ω such ha X = ( X,..., X ) The covarace fuco = k(s,). 7

28 RKHS by radom process { X } Ω :radom process o Ω wh mea zero ad fe d momes. X : Ξ R radom varable defed by a probably space. X L ( Ξ, B, P) closed subspace of L ( Ξ) Hlber space geeraed by { X } Ier produc ( UV, ) EUV [ ] L( Ξ ) = (er produc of L (Ξ)) Ω UV, L( X) 8

29 RKHS ad radom process Theorem k : posve defe kerel o a se Ω { X } :radom process wh mea 0 ad covarace fuco k Ω L( X ) Hk (somorphc as Hlber space) X k(, ) ( UV, ), L ( Ξ ) = f g, U f V g ( X, X ) Ξ = EXX [ ] = ks (, ) = k (, ), k (, s ) 注 ) L( ) (er produc) (cov) (reproducg) s s H k 9

30 Saoary process ad shfvara kerel Saoary case : radom process o R m saoary process m EX [ + X + ] = EXX [ ] ( sh,, R ) h s h s covarace fuco s gve by Rs (, ) R ( s) Posve defe kerel for a saoary process s gve by Ks (, ) = K ( s) Bocher s heorem Weer-Khche s heorem (covarace fuco of a saoary process o R m s he verse Fourer rasform of he power specral.) 30

31 Iferece wh radom process Esmao of radom process Modelg by a radom process X : radom process o Ω wh mea zero ad fe d momes Y = X + ε ε : ose dep. wh X E[ ε ] = 0, Cov[ ε, ε ] = σ δ( s) s R(,s) = Cov[X, X s ] : kow. σ : kow Esmao X 0 Esmae for 0 gve he observao Y,, Y K 3

32 Mmzg mea square error Lear esmaor for radom process Lear esmaor Xˆ Mea square error 0 = Leas square error esmaor = j α Y m E X Xˆ = m E X Y j j j= α j α j Xˆ = ˆT α Y = r T ( K + σ I ) Y 0 ) T m α ( K + σ I ) α r α α ˆ α = ( K + σ I ) r T 3

33 Bayesa esmao of Gaussa Jo probably process where ) K = ( R(, )) R r = ( R(, 0)) R j = + σ δ EY [, Xs] = R (, s) EY [, Y] R (, s) ( s), s Bayesa esmao = LSE esmao 0 T EX [ Y] = r ( K+ σ I) Y 33

34 Gaussa process ad regularzao LSE esmao of a process = Regularzao wh RKHS Lear LSE esmaor of a process (Bayesa esmaor of Gaussa process) ˆ m E X X = m E X α Y Sol j= j α ˆ T X = r ( K + σ I ) Y 0 j Rdge regresso o RKHS = m ( ) f H ( Y f ) + λ f H decal σ λ Sol. () () T f = r ( K + λi ) Y 34

35 Correspodece bewee RKHS ad radom process RKHS radom process Pos. def. kerel K(,s) Covarace fu. K(,s) = E[X,X s ] ck(, ) c X lm ck(, ) (compleo) lmc X (closure) Regularzao (smoohg) m ( ) f H = ( Y f ) + λ f () () T f = r ( K + λi ) Y Shf-vara kerel K(,s) = K(-s) Bocher s heorem H Lear esmao m E X α Y α 0 0 j= j ˆ T X = r ( K + σ I ) Y Cov. fu. of a saary process K(,s) = K(-s) Weer-Khche s heorem j 35

36 m-irf Ierave radom fucos A radom process s sad o be a m-erave radom fucos (m-irf) f for ay fe subse = (,, ) of Ω ad ay geeralzed creme c,, c of order m, he process s secod-order saoary. A saoary process s called (-)-IRF coveo. Modelg by o-saoary process Krgg s a modelg by 0-IRF. The geeralzed covarace fuco G(-s) s used sead of covarace fuco K(, s) for he modelg. 36

37 Geeralzed covarace Theorem (Mahero 973) : couous m-irf. There s a couous fuco G K such ha for ay fe subse = (,, ) of Ω ad ay wo geeralzed cremes (c,, c ) ad (d,, d ) of order m, The fuco G K s called geeralzed covarace. The geeralzed covarace s codoally posve defe of order m (obvous by defo ad above heorem). Mahero (973) proves he coverse, also. There s a correspodece bewee m-irf ad codoally posve defe fucos of order m. (Geeralzao of he correspodece bewee he saoary processes ad posve defe fucos.) 37

38 Refereces Ramsay, J. ad B. W. Slverma. Fucoal Daa Aalyss (d ed.). Sprger, 005. Gree, P.J. ad Slverma, B.W. oparamerc Regresso ad Geeralzed Lear Models. A Roughess Pealy Approach. Chapma & Hall Wahba, G. Sple Models for Observaoal Daa. CBMS-SF Regoal Coferece Seres Appled Mahemacs 59. SIAM Megue, J. (979) Mulvarae Ierpolao a Arbrary Pos Made Smple. J. Appled Mahemacs ad Physcs (ZAMP) 30, Mahero, G. (973) The rsc radom fucos ad her applcaos. Adv. Appl. Prob., 5, pp Berle, A. ad C. Thomas-Aga. Reproducg Kerel Hlber Spaces Probably ad Sascs. Kluwer Academc Publshers, 003. Gel fad, I.M. ad Vlek,.Ya. Geeralzed Fucos Vol.4: Applcaos of Harmoc Aalyss. Academc Press