Alternatives. The D Operator

Transcription

1 Using Smoothness Alternatives Text: Chapter 5 Some disadvantages of basis expansions Discrete choice of number of basis functions additional variability. Non-hierarchical bases (eg B-splines) make life more complicated. Not necessarily easy to tailor to problems at hand Kernel Methods: One "bump" function K i (t) at each data point calculate a weighted average: x(t) = y i K i (t/h)/ K i (t/h) Use the bandwidth, h, to regulate smoothness. Define smoothness and regulate it explicitly. What do we mean by smoothness? Some things are fairly clearly smooth: constants straight lines What we really want to do is eliminate small wiggles in the data. The D Operator We use the notation that for a function x(t), Dx(t) = d dt x(t) We can also define further derivatives in terms of powers of D: D 2 x(t) = d 2 dt 2 x(t),...,dk x(t) = d k dt k x(t),... Dx(t) is the instantaneous slope of x(t); D 2 x(t) is its curvature. We measure the size of the curvature for all of f by J 2 (x) = [D 2 x(t) ] 2 dt

2 Curvature of Vancouver Precipitation Curvature of Vancouver Precipitation 3 Fourier bases: 13 Fourier bases: Curvature of Vancouver Precipitation Curvature of Vancouver Precipitation 25 Fourier bases: 53 Fourier bases:

3 The Roughness of Derivatives Penalized Squared Error Sometimes we may want to examine a derivative of x(t), say D 2 x(t). We then should consider the roughness of that derivative This means we measure D 2 [ D 2 x(t) ] = D 4 x(t) J 4 [x] = [D 4 x(t) ] 2 dt And the roughness of the mth derivative is J m+2 [x] = [D m+2 x(t) ] 2 dt We now have two competing desires: fit to data and smoothness. We will explicitly trade them off by minimizing penalized squared error: PENSSE λ (x) =[y x(t)] T [y x(t)] + λj 2 [x] λ is a smoothing parameter measuring compromise between fit and smoothness. As λ increases, roughness is increasingly penalized and x(t) will become linear. As λ decreases, the penalty is reduced and allows x(t) to fit the data better. log λ = 1 log λ = 3

4 log λ = 7 log λ = 11 log λ = 15 log λ = 19

5 The Smoothing Spline Theorem Computing the smoothing spline Consider the usual penalized squared error: PENSSE λ (x) =[y x(t)] T [y x(t)] + λ [D 2 x(t) ] 2 dt A remarkable theorem tells us that the function x(t) that minimizes PENSSE λ (x) is a spline function of order 4 (piecewise cubic) with a knot at each sample point t j this is often referred to simply as cubic spline smoothing. The theorem tells us that x(t) is of the form x(t) =φ(t) T c where φ(t) is a vector of B-spline basis functions. The number of basis functions is (n 2)+4 = n + 2wheren is the number of sampling points How do we calculate c? Calculating the Penalized Fit When x(t) =Φ(t)c, wehavethat [D m x(t)] 2 dt = c T D m Φ(t)D m Φ(t) T c = c T Rc R is known as the penalty matrix. The penalized least squares estimate for c is now ĉ = This is still a linear smoother: [ Φ T W Φ + λr] 1 Φ T W y [ 1 ŷ = Φ Φ T W Φ + λr] Φ T W y Linear Smooths and Influence A procedure is a linear smooth if for some S. Eg ŷ = Sy [ 1 S λ = Φ Φ T W Φ + λr] Φ T W Computing the smoothing spline is also a linear operator [ 1 ˆx(t) =Φ(t) Φ T W Φ + λr] Φ T W y and we can calculate the influence of each y i on the curve.

6 Influence in Vancouver Precipitation Linear Smooths and Degrees of Freedom In least squares fitting, the degrees of freedom used to smooth the data is exactly K, the number of basis functions In penalized smoothing, we can have K > n. The smoothing penalty reduces the flexibility of the smooth (ie, we say we know something). The degrees of freedom are controlled by λ. A natural measure turns out to be df (λ) =traces λ Vancouver precipitation above was fit with 365 basis functions, λ = 10 4 resulting in df = Alternative Definitions of Roughness Harmonic Acceleration in Vancouver Precipitation D 2 x(t) is only one way to measure the roughness of x. If we were interested in D 2 x(t), we might think of penalizing D 4 x(t) cubic polynomials are not rough. What about the weather data? we know it s periodic, and not very different from a sinusoid. The Harmonic acceleration of x is and L cos(ωt) =0 = L sin(ωt). Lx = ω 2 Dx + D 3 x We can measure departures from a sinusoid by J L (x) = [Lx(t)] 2 dt

7 A Very General Notion We can be even more general and allow roughness penalties to use any linear differential operator Lx(t) = m α m (t)d m x(t) k=1 Then x is smooth if Lx(t) =0. Wewillseelateronthatwecanevenaskthedatatotelluswhat should be smooth. However, we will rarely need to use anything so sophisticated. Basis Expansions and Roughness There is always a function x minimizing PENSSE L (x) =(y x(t)) T (y x(t)) + λ [Lx(t)] 2 dt However, it can be annoying (and expensive) to calculate explicitly. Instead, we simply use a basis expansion with R = LΦ(t)LΦ(t) T dt This is given by formulae for simple cases, but may need to be evaluated numerically. Although the handwriting data potentially needs 1403 cubic B-splines, 50 does a very nice job. There are mathematical results to back this up. Choosing the Smoothing Parameter Generalized Cross Validation There are a number of data-driven methods for choosing smoothing parameters. Ordinary Cross Validation: leave one point out and see how well you can predict it. For any linear smooth OCV(λ) = 1 (yi x(t i )) 2 n (1 S ii ) 2 where S is the smoothing matrix. Generalized cross validation (see next slide) Various information criteria (not used here) ReML estimation GCV(λ) = n 1 (y i x(t i )) 2 [n 1 trace(i S)] 2 Doesn t actually generalize anything We can re-write this as ( GCV(λ) = n n df (λ) )( SSE ) n df (λ) SSE is discounted for degrees of freedom like usual, but we then also make a discount for minimizing over λ. GCV smooths more than OCV; even then, it may need to be tweaked a little to produce pleasing results.

8 Various Statistics on Vancouver Precipitation Confidence Intervals for Linear Probes Optimal λ = Most of the quantities that we are interested in are linear functionals of x. N t [x] =x(t), N Dt [x] =Dx(t), N ρ [x] = ρ(t)x(t)dt All possess the property that That is the are linear. This means that N[x] =N[Φ]c. N[ax 1 + bx 2 ]=an[x 1 ]+bn[x 2 ] Confidence Intervals for Linear Probes Recall c = Cy, y (x(t), Σ) Confidence Intervals for a Derivative We have Dx(t) =DΦ(t)Cy Var[Dx(t)] = DΦ(t)CVar[y]C T DΦ(t) T Then Var(c) =CΣC T and Var(N[x]) = N[Φ]CΣC T N[Φ] T So that a confidence interval for N[x] can be given by N[ˆx] ± 2 N[Φ]CΣC T N[Φ] T

9 A Probe for the Summer Dip Summary p = 0.5(x(190)+x(230)) x(210) =[0.5(Φ(190)+Φ(230)) Φ(210)] Cy Test the "dip" in summer by contrast between day 210 and average of days 190 and Defined smoothness in terms of derivatives of x. 2 We can use smoothness as a way to regularize the estimate of x. 3 Use of λ to trade-off smoothness and fit. 4 GCVasameansofchoosingλ 5 Linear smooths and linear probes gives a means of providing confidence for features of x. p ± 2se ( p) = ±