Scatterplot Smoothing

Transcription

1 1 Sttstcs 540, Smoothng Sctterplot Smoothng Overvew Problem... The usul settng for sctterplot smoothng s the delzed regresson model y = f(x )+σɛ, ɛ N(0, 1), =1,...,n, where the observtons (x,y ) re ndependent. Independence s crucl, wheres the ssumpton of normlty s more of convenence thn necessty s n lest squres regresson. The gol s to estmte the underlyng expectton functon f from the n observtons (x,y ). For the moment, we ll ssume tht x s sclr. Assumptons... We need to mtch the estmtor ˆf to the propertes tht we ssume hold for f. For exmple, f we ssume tht f s perodc wth perod d, then we ought to hve our smoother ˆf shre ths property. A more resonble ssumpton s tht f s contnuous or perhps dfferentble functon. From the dt lone, we cnnot determne n upper bound on the roughness of f (see Donoho 1988), though we cn obtn lower bounds. In the relted context of densty estmton, for exmple, we cn fnd one-sded ntervl for the number of modes (ndctng tht we need t lest certn number), but not two-sded ntervl. How could the dt ndcte tht the true densty ws not multmodl, wth mode t ech observton? Issues... One needs to keep vrety of ssues n mnd when smoothng, s trde-offs need to be mde. Key ttrbutes of ll smoothers re 1. Smoothness propertes of estmtor, supportng rtonle. 2. Locl senstvty to dt. 3. Bs/vrnce trde-off. 4. Estmtors re blend lner nd nonlner functons. A common crteron tht mkes some of these ssues more concrete s to defne the estmtor mplctly, s the soluton of ˆf = rg mn (y f(x )) 2 + λ f (t) 2 dt. (1) f Some further ssues tht re often forgotten untl too lte re

2 Sttstcs 540, Smoothng 2 1. Behvour t endponts. 2. Effect of mssng dt nd the ssumpton of equl spcng. 3. Robustness to outlyng vlues. 4. Computtonl speed versus generlty. Approches... The most common estmton methods re Sldng regressons lnk seres of lner/polynoml fts computed from overlppng subsets of some wdth. The lowess smoother n LspStt s the best exmple of ths group. Kernel methods weght the dt by movng smoothng kernel K of some wdth (typcl kernels resemble the Gussn densty). The estmtor of f(x) hs the form of weghted verge of the dt, y K( ˆf(x x x w )= K( ) x x ) (2) w The choce of kernel functon K s much less mportnt n pplctons thn the choce of the smoothng wdth w. Smoothng splnes jon contnuous low-order polynomls tht stsfy some externl smoothness ssumpton. These re the mn topc for tody s clss. Wvelets nd thresholdng whch together comprse loclzed orthogonl decomposton of the dt wth selected coeffcents shrunken towrd zero. We wll study these seprtely lter s tme permts. Ech of these cn be mde more robust (e, tolernt of outlers) by dptng the estmton method pproprtely. For exmple, lowess uses robust regresson rther thn lest squres regresson nd one cn replce the weghted verge (2) of the kernel smoother by robust estmte of locton. Cubc Splnes Knots... Let the ponts = x 1 < <x n =bdefne prtton of the ntervl [, b], nd ssume tht we hve observtons (x,y ), =1,...,n. The ponts x re known s the knots. Splnes... A splne s pecewse polynoml functon. The smplest splne s pecewse constnt functon, s 0 (x) =y j, x j x<x j+1.

3 Sttstcs 540, Smoothng 3 Ths splne hs no contnuty t the knot loctons. The lner splne s 1 (x) =y j +(y j+1 y j ) x x j x j+1 x j, x j x<x j+1. s contnuous, but ts frst dervtve ( zero order splne) s step functon. Note tht we defne s 0 from one knot, s 1 from two (n ntervl). The qudrtc splne s 2 requres two ntervls (3 knots) nd the cubc splne s(x) =s 3 (x) requres three ntervls (4 knots). Cubc splnes occupy specl plce n the theory of smoothers, nd we ll focus on these. Defnton... The functon s(x) s cubc splne on [, b] ft 1. Interpoltes: s(x )=y,s 2. Smooth: s(x),s (x),s (x) re contnuous, nd s 3. Cubc polynoml on ech ntervl [x,x +1 ]. Extreml property... Cubc splnes hve n mportnt extreml property. Among ll nterpoltng, dfferentble functons, the so-clled nturl cubc splnes mnmze the squred ntegrted second dervtve: s (t) 2 dt g (t) 2 dt, wth s () =s (b) =0, (3) Proof Begn by expndng the squre, wth the terms rerrnged n useful mnner: 0 (s (x) g (x)) 2 dx = g (x) 2 s (x) 2 2s (x)(g (x) s (x))dx We re done f we cn show tht the lst term s zero. Strt by formultng t s n ntegrton by prts, then use the fct tht s (x) s pecewse constnt on the prtton: s (x)(g (x) s (x))dx = s (x)d(g (x) s (x)) = s (x)(g (x) s (x)) b s (x)(g (x) s (x))dx = s (b)(g (b) s (b)) s ()(g () s ()) = s (b)(g (b) s (b)) s ()(g () s ()) = 0 x+1 x s (x)(g (x) s (x) k (g(x) s(x)) x +1 x f we lso ssume the stndrd condton tht s () =s (b) = 0. Wth these ddtonl boundry condtons (tht produce lner extrpolton), s(x) s known s nturl cubc splne.

4 Sttstcs 540, Smoothng 4 Unque clculton soluton... Gven n prs (x,y ) wth dstnct x s, there s but one cubc nterpoltng splne. The ssocted n 1 cubc polynomls hve 4(n 1) coeffcents tht we must be ble to determne unquely. The nterpolton condton mples 2(n 1) lner constrnts on the coeffcents. The smoothness of ech dervtve mples further n 2 constrnts (one for ech nteror knot. Combnng ll of these leds to huge lner system of equtons wth the 4(n 1) unknown coeffcents nd 4(n 1) 2 lner equtons. Addng the two ddtonl nturl boundry condtons gves complete system (whch s nerly dgonl). Regresson splnes... So fr, nothng hs been sd bout smoothng wth splnes. As defned, splnes smply smoothly nterpolte dt wth smoothly joned pecewse polynomls. Splnes were desgned s low-order nterpoltng polynomls, not s smoothers. They were needed to vod the end-vlue problems tht one runs nto wth hgh-order nterpoltng polynomls. There re two brod wys tht one cn smooth wth pecewse polynomls. The frst, loosely clled regresson splnes, s qute smple nd underles Fredmn s MARS method nd the Turbo-Smoother of Fredmn nd Slvermn. Gven observtons (x 1,y 1 ),...,(x n,y n ), use only few observtons s knots nd compute polynoml on ech ntervl by lest squres. Suppose n = 100 nd we use x 50 = 0 s the sngle knot. Ft cubc on ech ntervl. The ntl polynoml s, sy, s(x) =A(x)= x+ 2 x x 3 for x 0. Let B(x) = 3 j=0 b j x j denote the polynoml for postve x. When fttng from dt, one does not smply ft two seprte cubcs v lest squres these two would not stsfy the smoothness condtons. In fct, ddng the second polynoml dds one degree of freedom to the ft. Fx j nd see wht you cn tell bout B(x). The nterpoltng condton A(0) = B(0) mples b 0 = 0. Contnuty of s (x) nd s (x) t zero mples tht b 1 = 1 nd b 2 = 2. All tht s left s to fnd the new cubc term b 3. Non-zero knots mke the lgebr more complex, but ech new cubc dds but one degree of freedom ( new coeffcent) to the ft (four coeffcents mnus 3 lner constrnts equls one new coeffcent). Smoothng splnes... Smoothng splnes trdtonlly men somethng dfferent. Here s the de. Pck ny smoother tht you lke, nd let ŷ denote ts ftted vlues t the gven x. No mtter wht smoother hs been used to determne the ŷ, the extreml property (3) mples you cn do better n terms of the crteron (1) by nterpoltng these ŷ wth cubc splne. The trck to clcultons, however, s to ft the mnmzer of (1) rther thn mprovng nother estmtor.

5 Sttstcs 540, Smoothng 5 Computng Smoothng Splnes Methods of clculton... You hve severl wys to pproch the computton of smoothng splnes. We ll consder two. The frst s to return to the regresson splnes, nd thnk of fttng these wth knot t every pont. The second s more drect nd reles upon set of bss functons known s B-splnes for the spce of functons spnned by the cubc splnes. V regresson splnes... To fnd cubc nterpoltng splne v regresson, consder the representton s(x) =β 0 +β 1 x+β 2 x 2 +β 3 x 3 + n 1 j=2 θ j (x x j ) 3 + (4) where x + s the postve prt of x. Superfclly, ths expresson hs n+2 unknowns, but the boundry condtons dd two more constrnts tht nl thngs down. You cn check tht the resultng functon stsfes our condtons for cubc splne. (Numerclly, one vods ths representton snce t ntroduces very lrge vlues (the cubcs) nd leds to ner sngulr desgn mtrces even wth fewer thn n 1 ntervls. It does help one see wht s hppenng, however.) The hrd prt of ths representton s to decern how to ncorporte the penlty term f (x) 2 dx nto the estmton of the coeffcents of the regresson splne (4). If we substtute s(x) from (4) nto ths ntegrl for f, we obtn for the th ntervl the sum x+1 x s (x) 2 dx = x+1 x (2β 2 +6β 3 x+6 θ j (x x j )) 2 dx j The ccumultng nture of the bss functons (they spn ll ntervls to the rght) mke ths expresson pretty unweldy. The remedy s to use dfferent, equvlent set of regressors. B-splnes... An lterntve to the truncted power bss used mplctly n (4) s to use polynomls whch re zero outsde of smll rnge known s B-splnes. For exmple, n the lner cse, the regresson splne formulton s (new θ s) s 1 (x) =β 0 +β 1 x+ n 1 j=2 θ j (x x j ) +. The ssocted regresson desgn mtrx hs trngulr shpe. Alterntvely, we cn prmeterze s 1 usng trngulr functons tht spn just two ntervls, x x j x B j,2 (x) = j+1 x j, x j x<x j+1 1 x x j+1 x j+2 x j+1, x j+1 x<x j+1

6 Sttstcs 540, Smoothng 6 Then wrte the lner splne s s 1 (x) = j γ j B j,2 (x), so tht the regressors re more nerly orthogonl. Consequently, the clcultons re more stble, nd snce B-splnes re nerly orthogonl mkng X X s lmost dgonl, the clcultons re lso qute fst. Cubc B-splnes behve smlrly nd re computed n essence by ntegrtng up from the lner B-splnes. It s mportnt to note tht the B-splnes re polynomls defned by the grd of x s nd do not depend on the y. B-splnes re n generl defned by the recurrence expresson B j,k (x) = x x j x j+k 1 x j B j,k 1 (x)+ x j+k x x j+k x j B j+1,k 1 (x), nd one gets consderble smplfctons when the grd of x s s eqully spced. The recurson s strted wth the ndctor functons B j,1 = 1 for x j x<x j+1 nd zero elsewhere (see deboor 1978, eqns 4,5 of Chpter 10). Smoothng, t lst... Wrte the cubc splne n vector form s lner combnton of cubc B-splnes, s(x) = j γ j B j (x) (f(x 1 ),...,f(x n )) = Bγ where B s the mtrx wth element B j = B j (x ). Then substtute ths expresson nto the smoothng expresson (1) to obtn where the mtrx M hs elements (Y Bγ) (Y Bγ)+λγ Mγ (5) M j = B (x)b j (x)dx. The expresson (6) s now n the form more suted to be recognzed s penlzed lest squres estmtor, wth soluton (B B + λm)ˆγ = B Y. (6) Ths type of estmtor hs long hstory, ncludng Mrqurt s method for nonlner optmzton (keepng the Hessn postve defnte) nd rdge regresson (Hoerl nd Kennrd, where t provdes bsed estmtor to overcome collnerty).

7 Sttstcs 540, Smoothng 7 Byesn nterpretton... There s lso Byesn rgument tht leds to penlzed lest squres soluton. Assume tht the dt re condtonlly norml followng the usul regresson model, Y = Xβ + ɛ, ɛ N(0,σ 2 I n ). However, dd pror dstrbuton on the slopes, mkng them norml s well nd centered on zero wth vrnce v 2, β N(0,v 2 I k ). Thus, Y β N(Xβ,σ 2 I n ) nd Cov(Y,β) =XVr(β) =v 2 X, nd the jont dstrbuton of Y nd β s ( Y β ) ( ( v 2 XX + σ 2 I = N 0, n v 2 )) X v 2 X v 2 I k The posteror men for β s then (usng the usul regresson expressons nd lettng r = σ 2 /v 2 ) Eβ Y = v 2 X (v 2 XX + σ 2 I n ) 1 Y = (1/r)(I k X X(X X + ri k ) 1 )X Y = (X X +ri k ) 1 X Y. The frst step comes s specl cse of the formul for the nverse of prttoned mtrx. In prtculr, for squre mtrx prttoned s M j (, j =1,2), we hve (M 11 M 12 M 1 22 M 21 ) 1 = M M 1 11 M 12 (M 22 M 21 M 1 11 M 12 ) 1 M 21 M (7) You cn derve ths expresson by dgonlzng the mtrx M n blocks usng the regresson expressons. You cn reproduce the lst step by rerrngng the correspondng sclr expresson s 1 x2 x 2 + r = r x 2 + r =(x2 /r +1) 1. Choosng the Smoothng Prmeter Pckng λ... So how does one choose λ n the crteron (1)? A populr choce s bsed on crossvldton, mechnsm for ssessng the out-of-smple performnce of sttstcl estmtor. Judgng regresson model... So how ought one pck the vrbles n regresson model? One pproch s to try to

8 Sttstcs 540, Smoothng 8 pck the model tht you beleve wll predct best when ppled to new set of dt (from the sme populton s the one used to construct the model). Suppose tht we observe n observtons whose men s some vector η whch s unknown to us, though fxed: Y = η + σɛ, ɛ N(0,I n ), For model buldng, ssume tht we re gong to pproxmte η by projectng t nto subspce ssocted wth collecton of k predctors, collected nto the n k mtrx X. Let ˆη = X ˆβ be the lest squres estmte of η bsed on ths projecton of η nto the spn of X, Xβ = Hη for H = X(X X) 1 X. For convenence of notton, let y 2 = Ey y. The expected predcton error sum of squres for predctng n ndependent vector Y = η + σɛ wth the sme men η usng the ft to the orgnl n observtons s Y ˆη 2 = η Hη 2 + σɛ + Hη ˆη 2 = η Hη 2 + σɛ 2 + Hη HY 2 = η Hη 2 + nσ 2 + Hɛ 2 = η Hη 2 } {{ } bs 2 2 +(n+k)σ }{{} vrnce. (8) The bs shrnks s we ncrese the sze of the subspce for projecton (e, dd more vrbles to the model, ncresng k), wheres the vrnce term gets lrger s the number of predctors k ncreses. We see the clssc trde-off of bs versus vrnce. Notce tht f you consder the MSE of ˆη, you wll fnd MSE(ˆη)= η ˆη 2 = η Hη 2 + kσ 2, droppng the term nσ 2 whch does not depend on the ftted model nd thus does not ffect whch model we would choose. Tht s, the model tht mnmzes Y ˆη 2 lso mnmzes the MSE. Now, f we hope to fnd the model tht mnmzes ths sort of out-of-smple predcton error, we need some wy of computng (8) from the vlble dt. For exmple, the expected resdul sum of squres (expected n-smple predcton error) s Y ˆη 2 = η Hη 2 + σɛ + Hη ˆη 2 = η Hη 2 + σɛ + Hη HY 2 = η Hη 2 + σ 2 ɛ Hɛ 2 = η Hη 2 +(n k)σ 2. (9) Tht s, the resdul sum of squres s the sum of bs plus nother term whch now lso shrnks s k ncreses. Ths s not the needed behvour nd leds to the problem of overfttng (usng too mny predctors). Among the ptches for ths problem re Mllow s (1973) C p sttstc s well s...

9 Sttstcs 540, Smoothng 9 Cross-vldton... In the sprt of out-of-smple predcton, cross-vldton seeks the regresson model (relly, the set of predctors tht mke up X) whch mnmzes (y x ˆβ ( ) ) 2 (10) where ˆβ ( ) denotes the slope estmtes bsed on ll of the dt except for the th observton. Rther thn settng prt some frcton of the dt for vldton, ech observton s left out, n ths cse one t tme (better lterntves leve out 2 or more), nd predcted from model ft to the rest of the dt. For smoothng, the correspondng expresson s (y ˆf ( ) ) 2, (11) wth ˆf ( ) denotng the smooth ft wthout (x,y ). Expressons for regresson clcultons... For regresson, there s very useful specl cse of the prttoned nverse expresson (7): (M b ) 1 = M 1 + M 1 b M 1 (12) 1+ M 1 b where M s squre mtrx nd nd b re conformble vectors. (Proof de: Look t the geometrc expnson 1/(1 x) =1+x+x 2 +. Wrte (M b) 1 = M 1 (I b M 1 ) 1 nd try to expnd smlrly.) In regresson, the vlue of (12) s to notce tht (X ( )X ( ) ) 1 =(X X x x ) 1 =(X X) 1 + (X X) 1 x x (X X) 1 1 h, where h = x (X X) 1 x s the so-clled leverge for the th observton (the dgonl of the projecton mtrx H = X(X X) 1 X ). Ths expresson then gves ˆβ ( ) = (X( )X ( ) ) 1 X ( )Y ( ( ) = (X X) 1 + (X X) 1 x x (X X) 1 ) (X Y x y ) 1 h = ˆβ (X e X)x, 1 h where e s the usul resdul e = y x ˆβ. Thus ts qute esy to compute the summnds n (10): y x ˆβ( ) = y x ( ˆβ (X X) 1 x e 1 h ) = e + h e 1 h

10 Sttstcs 540, Smoothng 10 e =, 1 h so tht (10) becomes (y x ˆβ ( ) ) 2 = e 2 (1 h ). (13) 2 Thus, the cross-vldton sum of squres (CVSS) s smply weghted sum squred resduls, nd very esy to compute. Generlzed cross-vldton (GCV)... Generlzed cross vldton goes one step further. Notce frst tht h =trh=trx(x X) 1 X =k, the number of regressors (ncludng the constnt). Rther thn compute CVSS n (13) drectly, replce h n the denomntor wth ts verge, h = k/n, obtnng the pproxmton whch hs from (9) expected vlue (y x ˆβ ( ) ) 2 e 2 (1 h) 2 = e 2 (1 k/n) 2 ( ) n 2 = e 2 n k bs 2 + n2 n k σ2 bs 2 +(n+k)σ 2, s motvted by out-of-smple predcton. Another wy to look t ths lst expresson s to notce tht the MSE of ˆη s η Hη 2 + kσ 2, so tht mnmzng the CVSS s ttemptng to mnmze the MSE of the ftted model. Applctons n smoothng... In generl, thngs re not so smple when delng wth smoothng, but one gets long wy by tretng most smoothers n mnner resemblng regresson. One cn lwys resort to brute force to obtn ˆf ( ), but better methods re esly obtned nd generlzed cross vldton smplfes thngs further. Smoother mtrx... The key step n levergng the regresson clcultons s to express smoother n lner form resemblng the lest squres projecton Ŷ = HY. For exmple, the movng verge smoother of length w cn be wrtten s ˆf m = 1 w WY

11 Sttstcs 540, Smoothng 11 where W s the n n mtrx whose row contns the weghts (mostly 1 s) used to compute the th smoothed vlue. Smlrly, usng the B-spne representton for the ftted model (6) we cn wrte ˆf = Bˆγ = B(B B + λm) 1 B Y = S λ Y. (14) We re not lkely compute ˆf ths wy, but t mkes the problem more esy to thnk bout. Note tht S λ depends only upon the x s nd choce of λ. By comprson to the true projecton mtrx H from regresson, S λ for smoothng splnes 1. Is not projecton mtrx snce S λ Sλ, 2 2. Is symmetrc (look t (14)), nd 3. Hs trce whch s often used s the degrees of freedom for the smoother. (Others, such s tr Sλ 2 re lso possble snce S λ s not projecton mtrx.) MSE of smoother... Wrtng ˆf = S λ Y, we get nce expresson for the MSE of the smoother, f S λ Y 2 = f S λ f σs λ ɛ = f S λ f 2 + σ 2 S λ ɛ 2 = f (I S λ ) 2 f + σ 2 trs 2 λ. Compre ths to the prevous regresson expressons nd you ll see tht the dfference s relly just the second term, whch for regresson would be kσ 2. Cross-vldton wth smoothers... Note frst tht for smoothng splnes tht S λ hs two egenvectors wth egenvlue 1, S λ 1 = 1, S λ x = x where x denotes lner vector wth x j = jx 1. A method tht leds to smple expressons for CVSS s to defne (suppressng λ from S λ ) j S j y j ˆf ( ) =, (15) 1 S tht s, set the weght on y to zero nd renormlze the others so tht they sum to one (How do you know they ought to sum to one?). From (15) we obtn (1 S ) ˆf ( ) = ˆf S y = ˆf S ( ˆf + e ) = (1 S ) ˆf S e so tht we cn express the leve-one-out ft s ˆf ( ) = ˆf S 1 S e. (16)

12 Sttstcs 540, Smoothng 12 Thus the CVSS for smoothng becomes (y ˆf ( ) ) 2 = = = (y ˆf + S 1 S e ) 2 (e + S e ) 2 1 S e 2 (1 S ), 2 whch corresponds to the expresson (13) for regresson. One then chooses the vlue of λ tht mnmzes ths expresson. Alterntvely, usng GCV, one cn replce S by the verge trs λ /n s ws done wth regresson. Further redng... The book Generlzed Addtve Models by Hste nd Tbshrn dscusses ll ths nd more.