Smoothing and variable selection using P-splines

Transcription

1 Smoothing and variable selection using P-splines Irène Gijbels Department of Mathematics & Leuven Statistics Research Center Katholieke Universiteit Leuven, Belgium Joint work with Anestis Antoniadis, Anneleen Verhasselt, Sophie Lambert-Lacroix

2 Anestis Antoniadis XIVth International Biometrics Conference in Namur, Belgium, in July 1988 first reading: Nonparametric penalized maximum likelihood estimation of the intensity of a counting process, AISM, 1989 joint work on model selection using wavelet decomposition (with G. Grégoire) change point detection change point detection in hazard function (with B. MacGibbon) unfolding sphere size distributions using wavelets (with J. Fan) Workshop in honor of Anestis, Villard de Lans, March 2011 p.1

3 penalized wavelet monotone regression (with J. Bigot) smoothing non equispaced heavy noisy data with wavelets (with Jean-Michel Poggi) penalized likelihood regression for generalized linear models with nonquadratic penalties (with Mila Nikolova) variable selection in additive models and in varying coefficient models using P-splines (with Anneleen Verhasselt and Sophie Lambert-Lacroix) Workshop in honor of Anestis, Villard de Lans, March 2011 p.2

4 Seminal contributions to many areas wavelets and their applications intensity function estimation survival analysis and point processes inverse problems (in particular Poisson inverse problems) constraint estimation analysis of functional data development of statistical methods for microarray data excellent and very dynamic researcher extremely broad knowledge Workshop in honor of Anestis, Villard de Lans, March 2011 p.3

5 Some characteristics many services to the profession associate editor of several international journals serving in scientific evaluation boards for many years very supporting to young (moderate and old) researchers always helping with scientific advise a remarkable honesty and modesty an admirable personality; an example for many... Workshop in honor of Anestis, Villard de Lans, March 2011 p.4

6 THANK YOU!

7 Y : response variable Additive models: introduction (X 1,...,X d ) vector of d explanatory variables additive model Y = f 0 + d f j (X j ) + ε E (f j (X j )) = 0 j=1 ε random noise term; mean 0 and variance σ 2 f j unknown univariate functions often only a few components f j are different from 0 aim: to select and estimate the non-zero f j components Workshop in honor of Anestis, Villard de Lans, March 2011 p.6

8 Nonnegative garrote method Original nonnegative garrote method proposed by Breiman (1995) in a multiple linear regression model data (Y i, X i1,...,x id ) from Y i = β 0 + d β j X ij + ε i j=1 i = 1,...,n β OLS j ordinary least squares estimator for β j Workshop in honor of Anestis, Villard de Lans, March 2011 p.7

9 basic idea: the nng method shrinks the least squares estimators β OLS j shrinkage done via: c j βols j with c j 0 and a bound on d j=1 c j task : how to find the shrinkage factors c j? Workshop in honor of Anestis, Villard de Lans, March 2011 p.8

10 the nonnegative garrote shrinkage factors ĉ j are found by solving 1 n ( d ) 2 OLS (ĉ 1,...,ĉ d ) = argmin c1,...,c d Y i β 0 c 2 βols j j X ij i=1 j=1 d s.t. 0 c j (j = 1,...,d), c j s j=1 for given s, or equivalently (ĉ 1,...,ĉ d ) = argmin c1,...,c d s.t. 0 c j (j = 1,...,d) 1 2 n i=1 ( Y i β OLS 0 d j=1 ) 2 d c βols j j X ij + θ j=1 c j for given θ > 0 s > 0 and θ > 0; regularization parameters (see e.g. Xiong (2010)) Workshop in honor of Anestis, Villard de Lans, March 2011 p.9

11 the nonnegative garrote estimator of the regression coefficient β j is β NNG j = ĉ j βols j special case: orthogonal design, i.e. X X = I n ĉ j = ( 1 θ ( β OLS j ) 2 ) + z + = max(z, 0) the larger θ, the stronger the shrinkage effect Workshop in honor of Anestis, Villard de Lans, March 2011 p.10

12 β NNG θ=0 θ=0.01 θ=0.05 θ= β OLS Figure 1: Shrinkage effect of the nonnegative garrote for different θ s Workshop in honor of Anestis, Villard de Lans, March 2011 p.11

13 Relation with other estimation methods LASSO Ridge : ( β Lasso 1,..., s.t. d j=1 β j s ( β Ridge 1,..., s.t. d j=1 β2 j s β Lasso d ) = argmin β n i=1 ( Y i β 0 d j=1 β jx ij ) 2 Ridge ( β d ) = argmin n β i=1 Y i β 0 ) 2 d j=1 β jx ij identity nonnegative garrote lasso ridge Least Squares Estimate Workshop in honor of Anestis, Villard de Lans, March 2011 p.12

14 More relations with (other) thresholding rules (see, e.g., literature on wavelet methods, work by Anestis Antoniadis) ) hard-thresholding rule δλ H 0 if β j λ ( βj = β j if β j > λ ) 0 if β j λ soft-thresholding rule δλ S ( βj = β j λ if βj > λ β j + λ if βj < λ hard-thresholding (a discontinous function): keep or kill rule soft-thresholding (a continuous function): shrink or kill rule Bruce & Gao (1996) and Marron, Adak, Johnstone, Newmann & Patil (1998),..., Gao (2008),... Workshop in honor of Anestis, Villard de Lans, March 2011 p.13

15 another thresholding rule Antoniadis & Fan (2001) suggested the SCAD (Smoothed Clipped Absolute Deviation) thresholding rule ) ( ) sign ( βj max 0, β j λ if β j 2λ ) ) δλ SCAD (a 1) β ( βj = j aλsign ( βj if 2λ < β j aλ a 2 β j if β j > aλ where a > 2 this is also a shrink or kill rule (a piecewise linear function) this rule does not over-penalize large values of β j and hence does not create excessive bias when the regression coefficients are large Antoniadis & Fan (2001), based on a Bayesian argument, have recommended to use the value a = 3.7 Workshop in honor of Anestis, Villard de Lans, March 2011 p.14

16 thresholding functions δ λ Hard-thresholding Soft-thresholding x NNG-thresholding x SCAD x x Workshop in honor of Anestis, Villard de Lans, March 2011 p.15

17 Functional nonnegative garrote method extension of the nng to additive models: see Cantoni, Fleming & Ronchetti (2011) and Yuan (2007) start with an initial estimator f init j (X j ) of f j (X j ) the nonnegative garrote shrinkage factors are then found via ( {1 min n c1,...,c d 2 i=1 Y i s.t. 0 c j (j = 1,...,d) f init 0 d j=1 c j ) 2 init f j (X ij ) + θ d j=1 c j } the nonnegative garrote estimate of f j is NNG init f j ( ) = ĉ j f j ( ) Cantoni, Fleming & Ronchetti (2011): use smoothing splines for the initial estimator using P-splines... Workshop in honor of Anestis, Villard de Lans, March 2011 p.16

18 univariate P-spline estimation P-splines, introduced by Eilers & Marx (1996), in the univariate nonparametric smoothing context Y i = f(x i ) + ε i for i = 1,...,n P-splines are an extension of regression splines with a penalty on the coefficients of adjacent B-splines (X i, Y i ), for i = 1,...,n, with X i [0, 1] IR regression spline model: approximate f(x) with m α j B j (x; q) where {B j ( ; q) : j = 1,...,K + q = m} is the q-th degree B-spline basis, using normalized B-splines such that j B j(x; q) = 1, with K + 1 equidistant knot points t 0 = 0, t 1 = 1 K,...,t K = 1 in [0, 1] α = (α 1,...,α m ) : unknown column vector of regression coefficients j=1 Workshop in honor of Anestis, Villard de Lans, March 2011 p.17

19 penalized least squares estimator α is the minimizer of S(α) = n i=1 ( Y i m j=1 λ > 0 : smoothing parameter ) 2 m α j B j (X i ; q) + λ j=k+1 ( k α j ) 2 the differencing operator: k α j = k t=0 ( 1)t( k t) αj t, with k IN examples: k = 1 : 1 α j = α j α j 1 k = 2 : 2 α j = α j 2α j 1 + α j 2 rewriting in matrix-notation: S(α) = (Y Bα) (Y Bα) + λα D kd k α the elements B ij of B ( IR n m ) are B j (X i ; q) D k ( IR (m k) m ) : matrix representation of the kth order differencing operator k Workshop in honor of Anestis, Villard de Lans, March 2011 p.18

20 Additive model: Nonnegative garrote with P-splines initial estimator additive model Y = f 0 + d f j (X j ) + ε E (f j (X j )) = 0 j=1 key ingredients to prove the consistency of the nonnegative garrote with P-splines (i) consistency result for (univariate) P-splines (Claeskens, Krivobokova & Opsomer (2009)) (ii) extension of a univariate smoothing estimator to additive models via backfitting (rely on results from Horowitz, Klemelä & Mammen (2006)) (iii) on a consistency result for the functional nonnegative garrote (e.g. Yuan (2007)) Workshop in honor of Anestis, Villard de Lans, March 2011 p.19

21 Consistency of nonnegative garrote with P-splines notation : f j = (f j (X 1j ),, f j (X nj )) Theorem: Under some assumptions, and if θ n tends to 0 such that κ n = n (q+1) 2q+3 = o( θ n ), then (given X ij = x ij ) (1). P( f j NNG = 0) 1 for any j such that f j = 0 1 (2). sup j n f (( NNG j f j 2 2 = O θ ) 2 ) P n in other words: the nonnegative garrote method with P-splines is variable selection consistent (1) estimation consistent (2) Workshop in honor of Anestis, Villard de Lans, March 2011 p.20

22 Other selection methods COSSO (Component Selection and Smoothing Operator, Lin & Zhang (2006)) ACOSSO (Adaptive Component Selection and Smoothing Operator, Storlie, Bondell, Reich & Zhang (2010)) APSO (Adaptive P-splines Selection Operator, Antoniadis et al. (2011)) Workshop in honor of Anestis, Villard de Lans, March 2011 p.21

23 Varying coefficient models: introduction (Fan and Zhang (2000),...) Y (t) = β 0 (t) + d X (p) (t)β p (t) + ε(t) = X(t) β(t) + ε(t) p=1 Y (t) is the response at time t (t T = [0, T]) X(t) = (X (0) (t),...,x (d) (t)) covariate vector at time t with X (0) (t) 1 β(t) = (β 0 (t),...,β d (t)) the vector of coefficients at time t β 0 (t) is the baseline effect ε(t) a mean zero stochastic process at time t Workshop in honor of Anestis, Villard de Lans, March 2011 p.22

24 longitudinal data, i.e. samples with n independent subjects or individuals each measured repeatedly over a time period the j-th measurement for subject i of (t, Y (t),x(t)) is denoted by (t ij, Y ij,x ij ) 1 i n, 1 j N i N i is the number of repeated measurements of subject i t ij is the measurement time, Y ij is the observed response at time t ij and X ij = (X (0) ij,...,x(d) ij ) N = n N i is the total number of observations i=1 Workshop in honor of Anestis, Villard de Lans, March 2011 p.23

25 P-spline estimation in varying coefficient models (see Lu, Zhang & Zhu (2008), Wang & Huang (2008),...) suppose: each unknown function β p (t), p = 0,...,d, can be approximated by a B-spline basis expansion β p (t) = m p l=1 B pl (t; q p )α pl where {B pl ( ; q p ) : l = 1,...,K p + q p = m p } is the q p -th degree B-spline basis with K p + 1 equidistant knots for the p-th component Workshop in honor of Anestis, Villard de Lans, March 2011 p.24

26 the P-spline estimates of the regression coefficients α pl are obtained by minimizing S(α) with respect to α = (α 0,...,α d ) IR dim 1, where α p = (α p1,...,α pmp ) and dim = p m p: S(α) = n i=1 1 N i N i j=1 ( Y ij d p=0 m p l=1 X (p) ij B pl(t ij ; q p )α pl ) 2 + d p=0 λ p α pd k p D kp α p k p is the differencing order for the p-th component λ p are the smoothing parameters Workshop in honor of Anestis, Villard de Lans, March 2011 p.25

27 S(α) = = n i=1 1 N i N i j=1 ( Y ij d p=0 m p l=1 X (p) ij B pl(t ij ; q p )α pl ) 2 + d n (Y i U i α) W i (Y i U i α) + αq λ α i=1 p=0 λ p α pd k p D kp α p Y i = (Y i1,..., Y ini ) B 01 (t; q 0 )... B 0m0 (t; q 0 ) B(t) = B d1 (t; q d )... B dmd (t, q d ) U ij = X ij B(t ij) IR 1 dim U i = (U i1,...,u in i ) IR N i dim ( ) W i = diag N 1 i,..., N 1 i IR N i N i N 1 i on the diagonal) (a diagonal matrix with N i times Q λ = diag(λ 0 D k 0 D k0,..., λ d D k d D kd ) IR dim dim (a block diagonal matrix with the matrices λ p D k p D kp on the diagonal) Workshop in honor of Anestis, Villard de Lans, March 2011 p.26

28 S(α) = n (Y i U i α) W i (Y i U i α) + αq λ α i=1 introducing further matrix notations: Ỹ Ũα αq λ α Y = (Y 1,...Y n) IR N 1 W = diag(w i ) i=1,...,n IR N N Ỹ = W 1/2 Y U = [U 0,...,U d ] Ũ = W 1/2 U Workshop in honor of Anestis, Villard de Lans, March 2011 p.27

29 consistency results are proved for the case that the number of knots K p + 1 (and thus m p = K p + m p ) grows with n β p ( ) is not a spline function itself, but can be approximated by a spline function theoretical results consistency result ( ( ) ) q/(2q+1) 1 β p (t) β p (t) L2 = O n P n 2 i=1 1 N i asymptotic normality Workshop in honor of Anestis, Villard de Lans, March 2011 p.28

30 Nonnegative garrote selection method the nonnegative garrote shrinkage factors ĉ = (ĉ 0,...,ĉ d ) are obtained from the optimization problem min c0,...,c d n i=1 1 N i d j=1 s.t. 0 c p (p = 0,...,d) ( Y ij ) 2 d p=0 X(p) ij c p β p init d (t ij ) + θ p=0 c p β init p ( ) : initial estimator for the regression coefficient function β p ( ) θ > 0 is a regularization parameter we use the P-spline estimator as an initial estimator Workshop in honor of Anestis, Villard de Lans, March 2011 p.29

31 some more matrix notations: min c Ỹ Zc θ d p=0 c p s.t. 0 c p (p = 0,...,d) where Y = (Y 1,...Y n) IR N 1 W = Ỹ = diag(w i ) i=1,...,n IR N N W 1/2 Y z (p) i = (X (p) i1,...,x(p) in i ) diag( β init p (t ij )) j=1,...,ni IR 1 N i Z p = (z (p) 1,...,z(p) n ) IR N 1 Z = [Z 0,...,Z d ] Z = W 1/2 Z c = (c 0,...,c d ) Workshop in honor of Anestis, Villard de Lans, March 2011 p.30

32 the P-spline estimator f p (t) = X (p) (t) β p (t) for the p-th component f p (t) = X (p) (t)β p (t) is consistent it can be shown that the nonnegative garrote estimator with the P-spline estimator as initial estimator for β p (t) f p NNG (t) = ĉ p fp (t) is estimation consistent variable selection consistent other selection methods: Adaptive P-spline Selection Operator (APSO)... Workshop in honor of Anestis, Villard de Lans, March 2011 p.31

33 example: CD4 data example the data are a subset from the Multicenter AIDS Cohort Study (Kaslow et al. (1987)) contain repeated measurements of physical examinations, laboratory results, CD4 cell counts and CD4 percentages of 283 homosexual men who became HIV-positive between 1984 and 1991 unequal numbers of repeated measurements and different measurement times for each individual aim: try to evaluate the effects of cigarette smoking, pre-hiv infection CD4 cell percentage and age at HIV infection on the mean CD4 percentage after infection the number of repeated measurements ranged from 1 to 14, with a median of 6 and mean of 6.57 Workshop in honor of Anestis, Villard de Lans, March 2011 p.32

34 the number of distinct time points was 59 covariates: X (1) i the smoking status of the i-th individual (1 or 0 if the individual ever or never smoked cigarettes) X (2) i X (3) i the centered age at HIV infection for the i-th individual the centered pre-infection CD4 percentage Workshop in honor of Anestis, Villard de Lans, March 2011 p.33

35 varying coefficient model for Y ij Y ij = β 0 (t ij ) + 3 β p (t ij ) + ε ij p=1 X(p) i β 0 (t) is the baseline CD4 percentage, represents the mean CD4 percentage t years after the HIV infection for a nonsmoker with average pre-infection CD4 percentage and average age at infection Table 1: Aids data. Summary parameters. method NS RSS ngp APSO Workshop in honor of Anestis, Villard de Lans, March 2011 p.34

36 36 34 ngp APSO ngp APSO baseline CD time since infection coefficient of smoking status time since infection (a) (b) ngp APSO ngp APSO coefficient of age coefficient of pre infection CD time since infection time since infection (c) (d) Figure 2: Aids data. Fitted (a) baseline effect; (b) coefficient of smoking status; (c) coefficient of age at HIV infection; (d) coefficient of pre-infection CD4. Workshop in honor of Anestis, Villard de Lans, March 2011 p.35

37 70 65 pre infection CD4= pre infection CD4=15 pre infection CD4=69 60 CD4 percentage time since infection Figure 3: Aids data. Fitted CD4 percentage for 3 pre-infection CD4 percentages. Workshop in honor of Anestis, Villard de Lans, March 2011 p.36

38 Grouped regularization methods Y (t) = β 0 (t) + d X (p) (t)β p (t) + ε(t) = X(t) β(t) + ε(t) p=1 approximation in terms of a basis of smooth functions β p (t) L p l=1 B (p) l (t)γ pl (L p large) as before, introducing the appropriate matrix notations n i=1 1 N i N i j=1 ( Y i (t ij ) p L k k=1 l=1 γ k,l X (k) i (t ij )B (k) l (t ij ) ) 2 Ỹ Zγ 2 Z : dimension N ( d p=0 L p) γ : dimension ( d p=0 L p) 1 2 Workshop in honor of Anestis, Villard de Lans, March 2011 p.37

39 grouped Lasso regularization minimize 1 2n Ỹ Zγ λ d w p γ p 2 p=0 w p = L p with respect to the vector of parameters γ = (γ 0,...,γ d ) defining p λ (v) = λv, for v 0, this can be written as minimize 1 2n Ỹ Zγ d ( ) p λ wp γ p 2 p=0 w p = L p Liu and Zhang (2008) Workshop in honor of Anestis, Villard de Lans, March 2011 p.38

40 grouped SCAD regularization the SCAD penalty p λ (v), for v 0, is λv if 0 v λ p λ (v) = v2 2aλv + λ 2 if λ < v < aλ 2(a 1) (a + 1)λ 2 if v aλ 2 grouped SCAD procedure: minimize 1 2n Ỹ Zγ d p λ (ω p γ p 2 ) p=0 w p = L p with respect to the vector of parameters γ Workshop in honor of Anestis, Villard de Lans, March 2011 p.39

41 possible approach a Taylor expansion of p λ (v) for v around v 0 gives p λ (v) p λ (v 0 ) p λ (v 0) v 0 (v 2 v 2 0) (see Fan and Li (2001)) this leads to solving a Ridge-regression type of problem (restricted to d n < n) Workshop in honor of Anestis, Villard de Lans, March 2011 p.40

42 first grouped SCAD regularization procedure minimize 1 2n Ỹ Zγ d p λ (ω p γ p 2 ) p=0 w p = L p with respect to the vector of parameters γ second grouped SCAD regularization procedure minimize with respect to γ 1 2n Ỹ Zγ d p λ ( γ p 1 ) p=0 algorithm for solving this optimization problem: in Breheny & Huang (2009) Workshop in honor of Anestis, Villard de Lans, March 2011 p.41

43 grouped Bridge regularization penalty function, for v > 0, p λ (v) = λ v q with0 < q < 1 grouped Bridge approach: minimizing the objective function 1 2n Ỹ Zγ d p λ (ω p γ p 2 ) p=0 an algorithm for solving this optimization problem: from Breheny & Huang (2009) Workshop in honor of Anestis, Villard de Lans, March 2011 p.42

44 Simulation studies comparison of 5 methods of grouped regularization methods gbridge: grouped Bridge (q = 1/2); local coordinate descent algorithm from Breheny & Huang (2009) gscad: grouped SCAD; idem gmcp: grouped MCP; idem glasso 1: grouped Lasso; idem glasso 2: grouped Lasso; implemented by Meier, van de Geer & Bühlman (2008) Workshop in honor of Anestis, Villard de Lans, March 2011 p.43

45 tuning parameter λ chosen by a BIC-type of criterion: ( RRSλ log n i=1 N i ) + n log( N i ) i=1 n i=1 N i df λ where RSS λ is the residual sum of squares df λ is the number of nonzero coefficients of γ Workshop in honor of Anestis, Villard de Lans, March 2011 p.44

46 simulation model (from Huang, Wu & Zhou (2002) and Wang, Li & Huang (2008)) 23 Y i (t ij ) = β 0 (t ij )+ p=1 β p (t ij )X (p) i (t ij )+ε i (t ij ), i = 1,...,n j = 1,...,Ñ intercept term and the three true relevant variables: ( ) ( ) πt π(t 25) β 0 (t) = sin β 1 (t) = 2 3 cos β 2 (t) = 6 0.2t β 3 (t) = 4 + (20 t) t [1, 30] remaining coefficients: β p (t) = 0, p = 4,...,23 time points t ij given by 1, 2,...,30 (Ñ = 30) and n = 100 Workshop in honor of Anestis, Villard de Lans, March 2011 p.45

47 simulation of three relevant variables X (k) i (t), k = 1,...,3: at any point t, the variable X (1) i (t) is sampled uniformly from [t/10, 2 + t/10] conditioning on X (1) i (t), the variable X (2) i (t) is centered Gaussian with variance given by (1 + X (1) i (t))/(2 + X (1) i (t)) the variable X (3) i (t) is independent of X (1) i and X (2) i and is a Bernoulli random variable with success rate equal to 0.6 the irrelevant variables X (k) i, k = 4,...,23 are paths of centered Gaussian process with covariance function Cov(X (k) i (t), X (k) i (s)) = 4 exp( t s ) the irrelevant variables are independent between them and of the others three first variables Workshop in honor of Anestis, Villard de Lans, March 2011 p.46

48 three levels of noise for the random error: σ = 1, 1.25 and 2 corresponds to signal-to-noise ratio (SNR) : 6.39, 5.11 and 3.19 SNR is defined by γ T Z T Zγ/N for each simulated data set: use cubic splines with five equidistant internal knots number of simulations: 500 Workshop in honor of Anestis, Villard de Lans, March 2011 p.47

49 reported criteria: mean value of the tuning parameter λ the average number of variables selected the average number of truly zero variables that where selected (false positives) the average number of truly nonzero variables that where not selected (false negatives) the mean and standard deviation of the model error: ( γ γ) T Z T Z( γ γ)/n Workshop in honor of Anestis, Villard de Lans, March 2011 p.48

50 Table 2: Selection model ability λ S FP FN ME σ = 1 gbridge (0.0033) gscad (0.0142) gmcp (0.0124) glasso (0.0066) glasso (0.0497) σ = 2 gbridge (0.0137) gscad (0.0316) gmcp (0.0317) glasso (0.0407) glasso (0.1209) Workshop in honor of Anestis, Villard de Lans, March 2011 p.49

51 remarks from this simulation study: for all signal-to-noise ratios, gbridge and glasso 1 lead to the best result for the selection ability and for the model error compared to the other methods the gbridge method is better in model error while glasso 1 method is better in selection ability the gscad and gmcp procedures are not very good in selection ability: the number of false positives is rather high the implementation of group lasso (glasso 2) gives relatively correct result in selection model but leads to very bad result in term of model error Workshop in honor of Anestis, Villard de Lans, March 2011 p.50

52 typical performance of the estimators of the four first coefficients for a signal-to-noise ratio = 1.25 Model error, SNR = gbridge gscad gmcp glasso 1 Model error, SNR = gbridge gscad gmcp glasso 1 Model error, SNR = gbridge gscad gmcp glasso 1 Figure 4: Workshop in honor of Anestis, Villard de Lans, March 2011 p.51

53 THANK YOU!