Smoothing, penalized least squares and splines

Transcription

1 Smthing, penalized least squares and splines Duglas Nychka, Lcally weighted averages Penalized least squares smthers Prperties f smthers Splines and Reprducing Kernels The interplatin prf CV and the smthing parameter Supprted by the Natinal Science Fundatin L Ecle D Ete, Ovrnnaz Sep 2005

2 Estimating a curve r surface. The additive statistical mdel: Given n pairs f bservatins (x i, y i ), i = 1,..., n y i = g(x i ) + ɛ i ɛ i s are randm errrs and g is an unknwn smth functin. The gal is t estimate a functin g based n the bservatins

3 A 2-d example Predict surface zne where it is nt mnitred Ambient daily zne in PPB June 16, 1987, US Midwestern Regin

4 Lcal linear surface estimates Use lcal infrmatin t predict unbserved values Use a linear regressin based n clse by bservatins. find ˆβ by least squares. y i = β 1 + ln i β 2 + lat i β 3 + ɛ i The predictin at lcatin ( 88, 41) is just a weighted average f the bservatins. ĝ( 88, 41) = ˆβ ˆβ ˆβ 3 = i=1,n w i y i

5 A kernel estimatr Determine the weights based n the distance t the predictin pints w i (1/h)K((x x i )/h) (and nrmalize s that the weights sum t ne.) Kernel: K K is bump shaped e.g. a nrmal Bandwidth: h h cntrls the spread f K as h gets large the estimate is just the average.

6 Sme kernel estimates fr zne

7 Linear smthers Let ĝ = g(x 1 ),..., g(x n ) be the predictin vectr at the bserved pints. A smther matrix satisfies ĝ = Ay where A is an n n matrix eigenvalues f A are in the range [0,1]. Nte: Ay y Usually values in between the data are filled in by interplating the predictins at the bservatins.

8 Prblems with lcal regressin and kernels Hw large shuld the neighbrhd/bandwidth be? What is the uncertainty f the predictin? Predicting in between bservatins is ad hc and can get weird when the errr is small.

9 Prblems with lcal regressin and kernels Hw large shuld the neighbrhd/bandwidth be? What is the uncertainty f the predictin? Predicting in between bservatins is ad hc and can get weird when the errr is small. But the theretical prperties f kernel estimatrs are well understd... E [g(x) ĝ(x)] 2 = h 4 K 2 /4 + σ2 nh K 0 MSE= Bias 2 + Variance

10 Penalized least squares Ridge regressin Start with yur favrite n basis functins {ψ k } n k=1 estimate has the frm f(x) = n l=1 θ k ψ k (x) where θ = (θ 1,..., θ n ) are the cefficients. The Let W i,k = ψ k (x i ) s f = W θ

11 Estimate the cefficients by a penalized least squares. Sum f squares(θ) + penalty n θ min θ n i=1 (y [W θ] i ) 2 + λθ T Bθ with λ > 0 a hyperparameter and B a nnnegative definite matrix.

12 Estimate the cefficients by a penalized least squares. Sum f squares(θ) + penalty n θ min θ n i=1 (y [W θ] i ) 2 + λθ T Bθ with λ > 0 a hyperparameter and B a nnnegative definite matrix. r in general, - lg likelihd + λ penalty n θ In any case nce we have the parameter estimates these can be used t evaluate ĝ at any pint.

13 The frm f the smther matrix Just calculus... Take derivatives f the penalized likelihd w/r t θ, set equal t zer, slve fr θ

14 The frm f the smther matrix Just calculus... Take derivatives f the penalized likelihd w/r t θ, set equal t zer, slve fr θ The mnster... ˆθ = (W T W + λb) 1 W T y ĝ = W ˆθ = W (W T W + λb) 1 W T y = A(λ)y

15 Why is this a smthing matrix? If W is symmetric A(λ) = W (W T W + λb) 1 W T = (I + λw 1 BW 1 ) 1

16 Effective degrees f freedm in the smther Fr linear regressin trace A(λ) gives us the number f parameters. (Because it is a prjectin matrix) By analgy, tra(λ) is measure f the effective number f degrees f freedm attributed t the smth surface

17 Effective degrees f freedm in the smther Fr linear regressin trace A(λ) gives us the number f parameters. (Because it is a prjectin matrix) By analgy, tra(λ) is measure f the effective number f degrees f freedm attributed t the smth surface A useful decmpsitin Recall: A(λ) = (I + λw 1 BW 1 ) 1 One can always find an rthgnal matrix, U s that U T U = I and (W 1 BW 1 ) = UΓU T where Γ is diagnal. A(λ) = (I + λuγu T ) 1 = U(I + λγ) 1 U T

18 A simple frmula fr the trace S tra(λ) = tru(i + λγ) 1 U T = tr(i + λγ) 1 U T U = tr(i + λγ) 1 = n i= λγ ii The Gamma s are all nnnegative s this must be increasing as λ decreases. The relatinship is ne-t-ne with λ and independent f the data s we can always talk abut λ in terms f the effective degrees f freedm.

19 Splines One btains a spline estimate using a specific basis and a specific penalty matrix. Splines are cnfusing because the basis is a bit mysterius. The classic cubic smthing spline: Fr curve smthing in ne dimensin, min f n i=1 (y i f(x i )) 2 + λ (f (x)) 2 dx The secnd derivative measures the rughness f the fitted curve.

20 Splines One btains a spline estimate using a specific basis and a specific penalty matrix. Splines are cnfusing because the basis is a bit mysterius. The classic cubic smthing spline: Fr curve smthing in ne dimensin, min f n i=1 (y i f(x i )) 2 + λ (f (x)) 2 dx The secnd derivative measures the rughness f the fitted curve. The slutin, is cntinuus up t its secnd derivative and is a piecewise cubic plynmial in between the bservatin pints. Where des this cme frm?

21 Climate fr Clrad CO met statin lcatins elevatin July average min/max temp

22 Cubic splines with different λ s July average min/max temp elevatin

23 Sme abstractin Reprducing kernels Think f these as cvariance functins... k(x, x ) T get basis functins we will hld ne argument fixed at sme value f x and let the ther vary. A Space f functins Let H be the smallest Hilbert space with inner prduct, such that k(x, ) H fr all x and k reprduces itself! k(x, ), k(x, ) = k(x, x )

24 Sme abstractin Reprducing kernels Think f these as cvariance functins... k(x, x ) T get basis functins we will hld ne argument fixed at sme value f x and let the ther vary. A Space f functins Let H be the smallest Hilbert space with inner prduct, such that k(x, ) H fr all x and k reprduces itself! k(x, ), k(x, ) = k(x, x ) Fr f in H, k(x, ), f = f(x)!

25 A variatinal prblem With H and its inner prduct, min f H n i=1 (y i f(x i )) 2 + λ f, f

26 A variatinal prblem With H and its inner prduct, min f H n i=1 (y i f(x i )) 2 + λ f, f The slutin Chse basis functins k(., x i ) fr 1 i n ˆf(x) = n i=1 ˆθ i k(x, x i ) Chse rughness penalty matrix B = W

27 A variatinal prblem With H and its inner prduct, min f H n i=1 (y i f(x i )) 2 + λ f, f The slutin Chse basis functins k(., x i ) fr 1 i n ˆf(x) = n i=1 ˆθ i k(x, x i ) Chse rughness penalty matrix B = W this gives ˆθ = (W W T + λw ) 1 W T y = (W + λi) 1 y ĝ = W ˆθ = W (W + λi) 1 y = (I + λw 1 ) 1 y

28 Cvariances Here W ij = k(x i, x j ) this lks like a cvariance matrix desn t it. The Prf The main part is t use the minimum interplatin prperties f splines. It is easiest t prve this in mre generality fr any reprducing kernel Hilbert space. A simple prf is t guess at the minimizing slutin and use the ptimal interplatin results t shw that there can nt be anther slutin. mre n this later.

29 A 1-d cubic smthing spline The key step is a decmpsitin: f(x) = β 1 + β 2 x + h(x) becmes min β,h min f n i=1 n i=1 (y i f(x i )) 2 + λ (y i β 1 + β 2 x i + h(x i )) 2 + λ H: All functins with 2 derivatives Inner prduct: related t the integral Usual (Wahba style) Reprducing Kernel: k(x, x ) = x x 3 + linear terms (f (x)) 2 dx (h (x)) 2 dx

30 A less scary reprducing kernel fr the same cubic spline prblem On the interval [0, 1] Let k(u, v) = u 2 v/2 u 3 /6 fr u < v and k(u, v) = v 2 u/2 v 3 /6 fr u v k(., v) fr fixed v, is a functin that is a cubic plynmial frm 0 t v and is then a linear functin fr u > v. It is twice differentiable. Let H be a Hilbert space f functins n [0, 1] where f(0) = 0 and f (0) = 0 and with inner prduct f, g = 1 0 f (u)g (u)du k is a reprducing kernel fr this space! Shw this just using integratin by parts.

31 Mre n cubic splines T be explicit the cubic smthing spline slutin has the frm: ˆf(x) = ˆβ 1 + ˆβ 2 x + n i=1 ˆθ i k(x, x i ) Besides verifying that the reprducing n the previus page wrks with the integrated secnd derivative inner prduct ne can als check that it is the cvariance functin fr integrated Brwnian mtin,b(t): with B(0)=0. i.e. X(u) = u 0 B(t)dt, E(X(u)X(v)) = k(u, v) S as we will see later, the Bayesian prir assciated with the cubic smthing spline is an integrated Brwnian mtin.

32 The last details... Given the reprducing kernel, the prblem is min β,θ y Xβ W θ 2 + λθ T W θ X is the regressin matrix with clumns 1 and {x i } First minimize ver θ with β fixed. Plug in slutin and then minimize ver β. Ω = (W + λi) ˆβ = (X T Ω 1 X) 1 X T Ω 1 y ˆθ = (W + λi) 1 (y X ˆβ)

33 A 2-d thin plate smthing spline min f n i=1 (y i f i ) 2 + λ R 2 ( 2 ) 2 ( f 2 f u u v ) 2 + ( 2 ) 2 f dudv 2 v Cllectin f secnd partials is invariant t a rtatin. Again, separate ff the linear part f f. f(x) = β 1 + β 2 x 1 + β 3 x 2 + h(x) Reprducing Kernel: k(x, x ) = x x 2 lg( x x ) + linear terms leading t basis functins that are bumps at the bservatin lcatins.

34 Sme thin plate splines fr the zne data

35 The interplatin prblem: a editrial Splines arse in numerical analysis with the interplatin prblem: Find a curve g that minimizes (g (x)) 2 dx, subject t g(x i ) = y i

36 The interplatin prblem: a editrial Splines arse in numerical analysis with the interplatin prblem: Find a curve g that minimizes (g (x)) 2 dx, subject t g(x i ) = y i We already knw hw t d this by letting λ 0 Statisticians are, f curse, suspicius f fitting data exactly 1-d splines have sme very fast cmputatinal prperties that d nt extend t higher dimensins! Assuming a reprducing kernel is equivalent t a mdel fr the unknwn functin. What is that mdel?

37 The general interplatin prblem Given {x i, y i } and a reprducing kernel Hilbert space. Find a functin g in the space s that minimizes g, g. subject t the cnstraints g(x i ) = y i, 1 i n The slutin g = n i=1 θ i k(., x i ) Where θ is determined by slving a system f n linear equatins t guarentee the interplatin cnstraints.

38 Interplatin Prf Suppse g = n i=1 θ i k(., x i ) interplates {x i, y i } and s des sme ther functin h. We will shw that h, h > g, g. Let δ = h g and s δ, δ > 0 h, h = g + δ, g + δ = g, g + 2 g, δ + δ, δ The cl part, using the reprducing prperty g, δ = n i=1 θ i k(., x i ), δ = n i=1 θ i k(., x i ), δ = because δ(x i ) = h(x i ) g(x i ) = y i y i = 0. S h, h = g, g + δ, δ > g, g Dne! n i=1 θ i δ(x i ) = 0

39 A prf f the smthing spline slutin The prf is by cntradictin and uses the interplatin result. Let ĝ be the smthing spline btained as a linear cmbinatin f the kernel basis functins and pssibly a linear r lw rder plynmial. This is fund as a penalized smther by plugging this frm int the penalized least squares criterin and minimizing by rdinary calculus. Suppse there is an h that has a smaller penalized least squares than ĝ. Cnstruct a g that interplates h at the x i and uses the same basis functins as ĝ. By the interplatin result we knw that h, h > g, g and since bth h and g have the same residuals sums f squares the penalized least squares criterin will be smaller fr g. Thus, h can nt be a minimizer. Because ĝ hwever, has the same frm as the interplant and minimizes the criterin. Thus it must in fact be the minimizer ver all functins in the Hilbert space.

40 Chsing λ by Crss-validatin Sequentially leave each bservatin ut and predict it using the rest f the data. Find the λ that gives the best ut f sample predictins. Refitting the spline when each data pint is mitted, and fr a grid f λ values is cmputatinally demanding. Frtunately there is a shrtcut. The magic frmula residual fr g(x i ) having mitted y i (y i ĝ i ) = (y i ĝ i )/(1 A(λ)) i,i This has a simple frm because adding a data pair (x i, ĝ 1 ) t the data des nt change the estimate.

41 CV and Generalized CV criterin CV (λ) (1/n) n (y i ĝ i ) 2 = (1/n) n i=1 i=1 (y i ĝ i ) 2 (1 A(λ)) i,i ) 2 GCV (λ) ni=1 (y i ĝ i ) 2 (1/n) (1 tra(λ)/n) 2 Minimize CV r GCV ver λ t determine a gd value

42 GCV fr the zne data GCV( eff. degrees f freedm), the estimated surface GCV functin GCV pints, slid GCV mdel, dashed GCV ne Eff. number f parameters

43 GCV fr the climate data GCV( eff. degrees f freedm), the estimated surface GCV July average min/max temp eff.df elevatin

44 Summary We have frmulated the curve/surface fitting prblem as penalized least squares. Splines treat estimating the entire curve but als have a finite basis related t a cvariance functin (reprducing kernel). One can use CV r GCV t find the smthing parameter.

45 Thank yu!