The Hodrick-Prescott Filter

Transcription

1 The Hodrick-Prescott Filter A Special Case of Penalized Spline Smoothing Alex Trindade Dept. of Mathematics & Statistics, Texas Tech University Joint work with Rob Paige, Missouri University of Science and Technology Funded in part by the National Security Agency Grants: H (Paige) & H (Trindade) June 2012 SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University JuneJoint 2012work with 1 / 38 Rob

2 Outline 1 Motivation Examples: the need for a semiparametric smoothing approach... 2 Semiparametric Models Partially linear models & penalized spline models Penalized splines as linear mixed models (LMMs) 3 Result #2: Estimation of Smoothing Parameter Estimators as roots of quadratic estimating equations (QEEs) Saddlepoint-based bootstrap (SPBB) inference for QEEs Exact ML & REML Inference in LMMs 4 Result #1: The Hodrick-Prescott Filter (HPF) HPF solution coincides with linear penalized spline 5 Examples and Simulations U.S. Gross National Product (GNP) Returns of GNP SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University JuneJoint 2012work with 2 / 38 Rob

3 The LIDAR Data (Ruppert, Wand, & Carroll, 2003) Model: y = µ(x) + error. Goal: estimate mean function µ(x), i.e. smooth data. log ratio range SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University JuneJoint 2012work with 3 / 38 Rob

4 Standard Parametric Approaches Fail Polynomial regression: µ(x) = β 0 + β 1 x + + β p x p p = 3, 4: don t capture sharp drop (x 600) p = 10: µ(x) too wiggly; µ (x) too variable (and changes sign!) Nonlinear regression, e.g. µ(x) = β 0 e β 1x, requires thinking about correct (nonlinear) relationship... Can often transform x and/or y such that get linear relationship, e.g. take logs above: log µ(x) = log(β 0 ) + β 1 x No transform here seems to help in linearizing (and stabilizing variance of) relationship. SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University JuneJoint 2012work with 4 / 38 Rob

5 The Fossil Data (Ruppert, Wand, & Carroll, 2003) strontium ratio dip here? obvious dip! age (millions of years) SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University JuneJoint 2012work with 5 / 38 Rob

6 U.S.A. Detrended Log(GNP) Quarterly Data (Hodrick & Prescott, 1997) A cycle of long duration? Time SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University JuneJoint 2012work with 6 / 38 Rob

7 A Simple Spline Model (LIDAR Data) Linear spline model with knots at κ 1 = 575 and κ 2 = 600 is promising µ(x) = β 0 + β 1 x + u 1 (x 575) + + u 2 (x 600) + Piecewise linear function, called spline (thin strip of flexible wood). A spline model is a flexible model. log ratio range SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University JuneJoint 2012work with 7 / 38 Rob

8 Polynomial Spline Models In general, have polynomial spline model of degree p with K knots µ(x) = β 0 + β 1 x + + β p x p + K k=1 u k (x κ k ) p + Given pairs of obs (x i, y i ), i = 1,..., n, can write mean in matrix form µ = X β + Zu Bθ Can also add vector of covariates w entering mean linearly µ = W γ }{{} + }{{} Bθ linear part nonlinear part Model can allow for autocorrelation, R, in residuals (e.g. time series): y = µ + ε, ε (0, σ 2 ε R) SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University JuneJoint 2012work with 8 / 38 Rob

9 Penalized Spline Models Polynomial spline model Estimate θ by minimizing µ = X β + Zu Bθ ˆθ PS = arg min θ { (y Bθ) R 1 (y Bθ) + αθ Dθ } D: a non-negative definite penalty matrix (usually θ Dθ = u u) α is a smoothing parameter controlling balance between: LIDAR: linear spline fits with max and min smoothing (24 knots) fidelity to data (α = 0) smoothness of fit (α = ) log ratio α = 0 α = range SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University JuneJoint 2012work with 9 / 38 Rob

10 Linear Mixed Model (LMM) Formulation With assumptions: [ ] u E = ε [ ] 0, Cov 0 [ ] [ ] u σ 2 = u I K 0 ε 0 σε 2 R Penalized spline can be recast as LMM with one variance component (Brumback, Ruppert, & Wand, 1999) y = X β }{{} + Zu }{{ + ε } fixed effects random effects BLUP of y in this context is ỹ = B θ, where θ = arg min θ { (y Bθ) R 1 (y Bθ) + σ2 ε σu 2 u u }. SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work10 with / 38 Rob

11 Smoothing Parameter is BLUP in LMM Formulation Define: α = σε 2 /σu 2 Let D be zero with I K as lower diagonal block: [ ] 0 0 D = J 0 I p+1,k = θ Dθ = u u K We obtain that: θ = ˆθ PS Fitted penalized spline for mean of y coincides with BLUP of y in LMM defined above. Implies BLUP-optimal value for α is: α = σ 2 ε /σ 2 u SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work11 with / 38 Rob

12 Estimation of Smoothing Parameter Since α is ratio of variance components in LMM many methods available in this context; typically require normal errors. Call these parametric. (SAS: proc mixed; R: lme.) Methods that do not require LMM representation and specification of error distribution: nonparametric. Examples (Parametric) Maximum Likelihood (ML) REstricted Maximum Likelihood (REML) Examples (Nonparametric) Akaike s Information Criterion (AIC) Generalized Cross-Validation (GCV) SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work12 with / 38 Rob

13 A Unified View of Smoothing Parameter Estimators (New) Above estimators can be viewed as roots of a quadratic estimating equation (QEE) in normal random variables Q(α) = y A α y The n n matrix A α has a (complicated, but) closed-form expression in each case... Example (A α for GCV estimator) Define: S α = B ( B R 1 B + αd ) 1 B R 1, and Ṡ α = S α / α. Then: A α = (I n S α )Ṡ α {1 Tr(S α )/n} (I n S α ) 2 Tr(S α )/n SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work13 with / 38 Rob

14 Example of QEE Based Inference: REML Example (A α for REML estimator) Assume: y u N(X β + Zu, σε 2 R), u N(0, σui 2 K ) Differentiation of restricted log-likelihood in β and σε 2 leads to profile REML(α, R) criterion, and A α =... too long to fit here... Krivobokova & Kauermann (2007): REML less sensitive to misspecification of residual correlation than AIC or GCV. Inspect ACF/PACF plots for ideas on what R should be... SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work14 with / 38 Rob

15 Saddlepoint-Based Bootstrap (SPBB) Inference for QEEs Technique pioneered by Paige, Trindade, & Fernando (2009): Assume y N(µ, Σ); want confidence interval (CI) for scalar α from Q(α). Estimator ˆα is (appropriate) root of Q(α). Q(α) usually obtained by differentiation of (profile) log likelihood (nuisance parameters replaced by conditional MLEs). Can express MGF of Q(α) in closed-form. Use MGF to saddlepoint approximate CDF of Q(α). Under monotonicity of Q(α) in α, relate saddlepoint approximations: CDF of Q(α) to CDF of ˆα. CI (α L, α U ) obtained by pivoting CDF of ˆα (inverting 2-sided test). SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work15 with / 38 Rob

16 SPBB Details: Slide 1 Approx (1 ζ)100% CI obtained by solving: Fˆα (ˆαobs α L, ˆθ(α L ), ˆσ 2 ε (α L ) ) = 1 ζ/2 Fˆα (ˆαobs α U, ˆθ(α U ), ˆσ 2 ε (α U ) ) = ζ/2 2nd order accurate: coverage prob is (1 ζ) + O(n 1 ) (DiCiccio & Romano, 1995). Fˆα ( ) is difficult to compute..., but if Q(α) is monotone (e.g. decreasing) in α: (Daniels, 1983). Fˆα (α) = P(ˆα α) = P(Q(α) 0) = F Q (0) F Q(α) ( ) also difficult to compute..., but have closed-form expression for its MGF (linear combo of indep noncentral χ 2 r.v. s). SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work16 with / 38 Rob

17 SPBB Details: Slide 2 Saddlepoint approximate CDF of Q(α): F Q(α) ( ) ˆF Q(α) ( ) (Lugannani & Rice, 1980). Saddlepoint approximations extremely accurate (3rd order accurate over sets of bounded central tendency), e.g. Butler (2007). (1 ζ)100% CI obtained by solving (numerically over a grid): ( ˆF Q(ˆαobs ) 0 αl, ˆθ(α L ), ˆσ ε 2 (α L ) ) = 1 ζ/2 ( ˆF Q(ˆαobs ) 0 αu, ˆθ(α U ), ˆσ ε 2 (α U ) ) = ζ/2 Still retain 2nd order accuracy in coverage prob (Paige, Trindade, & Fernando, 2009). SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work17 with / 38 Rob

18 SPBB Summary Method is similar to a parametric bootstrap: Use plug-in estimator ˆα; Usually sample from bootstrap distribution Fˆα ( ) via Monte Carlo simulation; BUT, replace simulation with saddlepoint approximation of Fˆα ( ). ( Bootstrap is in the name, but there s no re-sampling...) What is a saddlepoint approximation? Accurate method to approx CDF & PDF of X from its MGF, M X ( ). Alternative to Fourier inversion of characteristic function (FFT). Most computationally expensive step: for each x X, solve for ŝ log M X (s) s = x s=ŝ Original paper: Daniels (1954). Recent book: Butler (2007). SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work18 with / 38 Rob

19 Graphical Overview of SPBB (Result #2) Intractable! (Except via parametric bootstrap... slow.) Fˆα (ˆα obs ) (α L, α U ) ˆα solves pivot Q(α) = 0 ˆF Q(ˆαobs )(0) Q(α) monotone Fˆα (ˆα obs ) = F Q(ˆαobs )(0) saddlepoint approx via MGF of Q(α) SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work19 with / 38 Rob

20 Exact ML & REML Inference for α Exact finite sample inference for α = σ 2 ε /σ 2 u in LMMs with one variance component (Crainiceanu, Ruppert, Claeskens, & Wand, 2005): Note: asymptotic χ 2 dist is poor approx in finite samples due to substantial point mass at 0 (Crainiceanu & Ruppert, 2004). Assume y N(µ, Σ). Invert (restricted) likelihood ratio test corresponding to (REML-based) ML-based inference. Grid search needed to locate endpoints of CI (α L, α U ): At each grid value of α, use fast algorithm to draw large sample from H 0 distribution of RLRT/LRT statistic in order to (empirically) calculate p-value(α) p-value(α L ) = 0.95 and p-value(α U ) = 0.95 SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work20 with / 38 Rob

21 Comparison of SPBB and Exact Method Exact y normal. α: ratio of vars in LMM. Dataset-specific (needs manual tuning). LIDAR: 4 hours/ci. Performance: exact. Estimators: ML & REML only. SPBB y normal. α: root of QEE. QEE-specific (needs derivatives of QEE). LIDAR: 10 mins/ci. Performance: near-exact. Estimators: many (ML, REML, AIC, GCV, etc.) Fact SPBB is able to furnish a CI when no other computationally feasible method, exact or otherwise, exists, e.g. AIC & GCV. SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work21 with / 38 Rob

22 The Hodrick-Prescott Filter (Hodrick & Prescott, 1997) Smoothing technique for series y t : viewed as sum of trend µ t and residual component ε t y t = µ t + ε t, t = 1,..., n For choice of smoothing parameter α, trend fitted by minimizing weighted sum of squares ŷ HP = arg min µ R n n t=1 balance (y t µ t ) 2 {}}{ + α }{{} fidelity to data n t=1 (µ t 2µ t 1 + µ t 2 ) 2 }{{} smoothness of fit = (I n + α 2 2 ) 1 y, 2 = function({0, 1, 2}) SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work22 with / 38 Rob

23 HPF is Linear Penalized Spline (Result #1) Theorem Consider penalized spline model of degree p = 1, explanatory variable time x t = t, K = n 2 equi-spaced knots at 2,..., n 1, and uncorrelated residuals R = I n : n 1 y t = β 0 + β 1 t + u k (t k) + + ε t, t = 1,..., n. k=2 Then HPF solution coincides with penalized spline solution: ŷ PS = B PS (B PS B PS + αj 2,n 2 ) 1 B PS y = (I n + α 2 2 ) 1 y = ŷ HP SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work23 with / 38 Rob

24 HPF Coincides With Linear Penalized Spline: Proof Proof. HPF can be formulated as LMM (Schlicht, 2005): y = X HP β + Z HP u + ε B HP θ + ε where Z HP = 2 ( 2 2 ) 1, and X HP is any n 2 matrix satisfying: 2 X HP = 0, and X HP X HP = I 2 LMM formulation implies HPF solution is BLUP of y: ŷ HP = B HP (B HP B HP + αj 2,n 2 ) 1 B HP y SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work24 with / 38 Rob

25 Proof (continued...) Defining L = B 1 PS B HP, transform basis functions (but preserve degree and knot locations): ŷ HP = B HP [B HP B HP + αj 2,n 2 ] 1 B HP [ ( y = B PS L L B PS B PS + αl T J 2,n 2 L 1) 1 L] L B PS y = B PS [B PS B PS + αl T J 2,n 2 L 1] 1 B PS y =... = B PS (B PS B PS + αj 2,n 2 ) 1 B PS y = ŷ PS Equivalence follows if we can show in step above that L T J 2,n 2 L 1 = J 2,n 2 Decomposing L 1 into block structure, result follows...

26 Remarks Schlicht (2005): calls LMM representation of HPF a stochastic model, unaware that it is in fact a LMM; proposes maximum (Gaussian) likelihood estimator (MLE) for α; proposes also a method of moments estimate. Successive econometrics papers have built on this; propose new estimators e.g. Dermoune, Djehiche, and Rahmania (2008). But, there is already a vast existing body of work on the estimation of fixed and random components in LMMs... e.g. Christensen (1996). Fact LMM formulation of penalized spline endows it, and therefore HPF, with rich variety of existing estimation methods, e.g. REML (a parametric choice) and GCV (a nonparametric choice). SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work26 with / 38 Rob

27 Example 1: U.S. Gross National Product (GNP) Quarterly (seasonally adjusted) GNP for period to (billions of dollars). Have n = 189 pairs of (t, y t ) values: y t = log GNP, t = time (quarter since ). Strong linearly increasing trend, remove it! Hodrick & Prescott (1997): smooth via HPF with choice α = 1, 600 advocated for quarterly data (the HPF smooth). Theorem: HPF equivalent to 187-knot (K = 187) linear (p = 1) penalized spline model with uncorrelated residuals (R = I n )... Alternatives to HPF α = 1, 600 choice: REML and GCV. SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work27 with / 38 Rob

28 Example 2: Log Returns of GNP Analyze also first differences, {y t y t 1 } of log GNP (log returns), corresponds to a growth rate. Have n = 188 pairs of (t, y t ) values. Again compare smooths: HPF, REML, GCV. Theorem: HPF equivalent to 186-knot (K = 186) linear (p = 1) penalized spline modell with uncorrelated residuals (R = I n ). SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work28 with / 38 Rob

29 Log GNP and Log Returns of GNP Data Detrended Log GNP Log Returns of GNP Time Time ACF of Detrended Log GNP ACF of Log Returns of GNP Lag Lag SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work29 with / 38 Rob

30 Results: Fitted Models Log GNP: ACF/PACF of REML-IID & GCV-IID (R = I n ) residuals suggests AR(3) autocorrelation structure... so fit REML-AR(3). Log Returns GNP: REML-IID & GCV-IID (R = I n ) residuals white. Dataset Method α 95% Estimate (lo, pt, hi) Log GNP Log Returns GNP HPF (?, 1600,?) Exact-REML-IID (0.07, 0.16, 0.32) SPBB-GCV-IID (0.04, 0.15, 0.59) SPBB-REML-AR(3) (97.6, 337.3, 871.4) HPF (?, 1600,?) Exact-REML-IID (8, 595, 65, 302, 330, 436) SPBB-REML-IID (5, 572, 65, 302, 463, 185) SPBB-GCV-IID (6, 022, 79, 164, ) SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work30 with / 38 Rob

31 Smoothed Log GNP: REML & GCV with R = I n underfit... Detrended Log GNP HPF REML IID & GCV IID Quarter since 1947 SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work31 with / 38 Rob

32 Smoothed Log GNP: REML-AR(3) coincides with HPF! Detrended Log GNP HPF REML AR(3) Quarter since 1947 SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work32 with / 38 Rob

33 Smoothed Returns of Log GNP: HPF underfits? Log Returns of GNP REML IID & GCV IID HPF Quarter since 1947 SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work33 with / 38 Rob

34 Simulations: Performance of Exact vs. SPBB CI s for α 1,000 reps from REML-fitted model to Log Returns of GNP. Performance of SPBB is near-exact! SPBB: only method to get CI in case of GCV or REML (non-iid). Estimation Median Underage Coverage Overage Method CI Length Prob. Prob. Prob. Exact-REML 374, SPBB-REML 466, SPBB-GCV SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work34 with / 38 Rob

35 Summary Avoid blind usage of HPF: equivalent to linear penalized spline with equi-spaced knots and uncorrelated residuals. Advocate structured penalized spline approach: pay attention to degree of polynomial (p), number of knots (K), residual autocorrelaton (R). Use data-driven method for estimation: REML is less sensitive to misspecification of R than GCV or AIC. Use SPBB for α CI s: if desired; gives range of plausible smooths. SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work35 with / 38 Rob

36 Application: The Fossil Data strontium ratio Point and 95% Interval Smoothing Estimates REML & GCV point estimate (solid) Exact REML lower bound (dash) Exact REML upper bound (dot) SPBB GCV lower bound (dot dash) SPBB GCV upper bound (long dash) age (millions of years) SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work36 with / 38 Rob

37 Acknowledgements Reviewers: valuable suggestions. Ciprian Crainiceanu: exact ML & REML code. High Performance Computing Center, Texas Tech University: supercomputing. National Security Agency: not having to teach in summer. SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work37 with / 38 Rob

38 Key References Crainiceanu, C., Ruppert, D., Claeskens, G., and Wand, M. (2005), Exact likelihood ratio tests for penalized splines, Biometrika, 92, Hodrick, R.J., and Prescott, E.C. (1997), Postwar U.S. business cycles: an empirical investigation, J. Money, Credit, and Banking, 29, Krivobokova, T., and Kauermann, G. (2007), A Note on Penalized Spline Smoothing with Correlated Errors, J. Amer. Statist. Assoc., 102, Paige, R.L., Trindade, A.A. and Fernando, P.H. (2009), Saddlepoint-based bootstrap inference for quadratic estimating equations, Scand. J. Stat., 36, Paige, R.L. and Trindade, A.A. (2010), The Hodrick-Prescott Filter: A Special Case of Penalized Spline Smoothing, Electronic J. Statist., 4, Paige, R.L., and Trindade, A.A. (2012), Fast and Accurate Inference for the Smoothing Parameter in Semiparametric Models, Austral. N. Zeal. J. Statist., (to appear). Ruppert, D., Wand, M.P. and Carroll, R.J. (2003), Semiparametric Regression, London: Cambridge. Schlicht, E. (2005), Estimating the smoothing parameter in the so-called Hodrick-Prescott filter, J. Japan Statist. Soc., 35, SPBB (Dept. Inference of Mathematics for HPF & Statistics, Texas Tech University June 2012 Joint work38 with / 38 Rob