Envelopes: Methods for Efficient Estimation in Multivariate Statistics

Transcription

1 Envelopes: Methods for Efficient Estimation in Multivariate Statistics Dennis Cook School of Statistics University of Minnesota Collaborating at times with Bing Li, Francesca Chiaromonte, Zhihua Su, Inge Helland & Henry Zhang

2 Response envelopes in multivariate linear regression, (Cook, et al. 2010) Y i = α + βx i + ε i, i = 1,..., n Y R r : multivariate response X R p : non-stochastic predictors centered at 0 ε R r : normal errors, mean 0 and covariance Σ > 0 α R r : unknown intercept β R r p : unknown coefficients Goal: estimate β, prediction. MLE β Std of β is obtained by doing r univariate linear regressions, one for each response.

3 Rationale for envelopes Envelopes arise by asking if there are linear combinations of Y whose distribution is invariant under changes in X. Parameterize the MLM in terms of the smallest subspace E R r so that (P E = projection onto E, Q E = I P E ) Q E Y (X = x 1 ) Q E Y (X = x 2 ), (x 1, x 2 ) P E Y Q E Y X This implies that the impact of X on Y is concentrated in P E Y. We refer to P E Y and Q E Y informality as the material and immaterial parts of Y. (Li & Zhang, 2015: Generalized sparsity principle)

4 The conditions Q E Y X Q E Y and P E Y only if B := span(β) E, so E envelops B Q E Y X hold if and Σ = P E ΣP E + Q E ΣQ E, so E is a reducing subspace of Σ Formally, the intersection of all subspaces E with these properties is called the Σ-envelope of B and represented as E Σ (B) with u = dim(e Σ (B)).

5 Let the columns of the semi-orthogonal matrices Γ R r u and Γ 0 R r (r u) be bases for E Σ (B) and E Σ (B). Envelope Model: Y = α + ΓηX + ε, Σ = ΓΩΓ + Γ 0 Ω 0 Γ T 0. 0 u r. u = 0 means Y does not depend on X. u = r means all linear comb of Y depend on X and the model reduces to the standard model. Estimation via maximum likelihood with u determined by AIC, BIC,..., likelihood ratio testing, cross validation or a holdout sample. We are still interested in β and Σ, which depend on the envelope E Σ (B), but not on the particular basis Γ selected to represent it.

6 How do envelopes work? Multivariate regression with two responses, Y 1 and Y 2, and a single predictor, X = 0 or 1, to indicate two populations. Y = ( Y1 Y 2 ) = ( α1 α 2 ) ( β1 + β 2 ) ( ε1 X + ε 2 α 1 = E(Y 1 X = 0), β 1 = E(Y 1 X = 1) E(Y 1 X = 0), α 2 = E(Y 2 X = 0), β 2 = E(Y 2 X = 1) E(Y 2 X = 0). Standard estimators are obtained by substituting sample moments. )

7 Possible configurations for uncomplicated study Y 1 Y Y 2 0 Y 2

8 Inference for β 2 = E(Y 2 X = 1) E(Y 2 X = 0)

9 Inference for β 2 = E(Y 2 X = 1) E(Y 2 X = 0)

10 Inference for β 2 = E(Y 2 X = 1) E(Y 2 X = 0)

11 Schematic representation of an envelope analysis E Σ (B) EΣ(B)

12 Schematic representation of an envelope analysis E Σ (B) EΣ(B)

13 Schematic representation of an envelope analysis E Σ (B) EΣ(B)

14 Wheat protein data (Y 1, Y 2 ) are spectral intensities at two wave lengths for high (red) and low protein wheat. n = 50. SE s for 2 elements in β: Standard estimator: 8.6, 9.5 Envelope estimator: 0.4, 0.6 n 20, 000

15 Wheat protein data Y2 (2nd response) A Y2 (2nd response) EΣ(B) B2 B1 E Σ (B) Y1 (1st response) Standard analysis Y1 (1st response) Envelope analysis

16 Maximum Likelihood Estimation The estimated envelope ÊΣ(B) can be represented as Ê Σ (B) = arg min S (log P S S Y X P S 0 + log Q S S Y Q S 0 ), where 0 means the product of the non-zero eigenvalues and S is a u-dim subspace of R r. Let Γ be a basis for ÊΣ(B). Estimators of other parameters: β = P Γ βstd, Σ = P Γ ΣStd P Γ + Q Γ ΣStd Q Γ avar{ nvec[ β]} avar( nvec[ β Std ]) massive gains when Γ T ΣΓ Γ T 0 Σ 0Γ 0.

17 Cattle data Experiment: Two treatments, each assigned randomly to 30 cows. Weight measured at weeks 2, 4, 6,...,16, 18, 19. Do the treatment have a differential effect; if so, about when it is first apparent?

18 Ï Ø ½ ¼ ¾½¼ ¾ ¼ ¾ ¼ ¾ ¼ ¾ ¼ ½¼ ¼ ¼ ¼ Profile plot of cattle data ¼ ¾ ½¼ ½¾ ½ ½ ½ ¾¼ Ï Y i = α + βx i + ε i, X = 0, 1 β Std = Ȳ trt1 Ȳ trt2

19 Mean profile plot of cattle data max i ( β Std ) i /SE{( β Std ) i } 1.3. LRT stat. for β = 0 is about 27 on 10 df.

20 Fitted profile plots, after inferring that u = 5. From envelope fit, β i /SE( β i ) > 4.1 for i 10

21 Cattle weight, week 12 vs week Envelope Γ T Y 340 Weight on week Γ 0 T Y E S E Weight on week 12

22 Brain Volumes for Alzheimer s patients Y holds log volumes of 93 different regions of the brains of n = 309 female Alzheimer s patients of varying age X 1 and log intracerebroventricular volumes X 2, (Zhu, et al. 2015) We inferred that u = 1 so the envelope model becomes Y = α + Γ{η 1 X 1 + η 2 X 2 } + ε, Since û = 1, we can easily plot Γ T Y vs the two predictors.

23 Γ T Y vs. Age T Y Age

24 Other applications in linear models Partial response envelopes Partial response envelopes for prediction Predictor envelopes Response-predictor envelopes Envelope version of reduced rank regression Tensor envelopes Bayesian envelopes

25 Beyond linear models, (Cook & Zhang, 2015) Suppose we have an an asymptotically normal estimator θ of θ R p, n( θ θ) N(0, V(θ)). The estimator can often be improved by projecting it onto a root-n consistent estimator of the V(θ)-envelope of span(θ). 1 Reproduces known envelope methods, and applicable to GLMs. 2 Links envelopes to a pre-specified estimator, MLE, robust estimator, OLS,... Likelihood not required. 3 V(θ) can now depend on the parameter being estimated, plus perhaps nuisance parameters. 4 Envelopes and Aster models (Eck, et al. poster session)

26 Computing for linear model applications: MatLab toolbox: Papers: dennis/. Thank you!

27 General Context

28 Other envelope application in MLMs Partial response envelopes for part of β = (β 1, β 2 ). (Su and Cook, Biometrika, 2011) Y = α + β 1 X 1 + β 2 X 2 + ε = α + ΓηX 1 + β 2 X 2 + ε Σ = ΓΩΓ T + Γ 0 Ω 0 Γ T 0 Simultaneous envelopes for reducing X and Y (Cook and Zhang, Technometrics, to appear) Y = α + βx + ε = α + ΓηΦ T X + ε Σ = ΓΩΓ T + Γ 0 Ω 0 Γ T 0 Σ X = Φ Φ T + Φ 0 0 Φ T 0

29 Scaled predictor envelopes, when predictors are in different scales. (Su and Cook, submitted) Y = α + η T Φ T Λ 1 X + ε, Σ X = ΛΦ Φ T Λ + ΛΦ 0 0 Φ T 0 Λ, Λ = diag(1, λ 2,..., λ p ) Scaled response envelopes, when responses are in different scales. (Su and Cook, Biometrika, 2013) Inner envelopes, when envelopes don t offer improvement. (Su and Cook, Biometrika, 2012) based on the largest reducing subspace of Σ that is contained within span(β). Heteroscedastic envelopes for comparing multivariate means in populations with different covariance matrices. (Su and Cook, Statistica Sinica, 2013).

30 Egyptian Skulls 4 measurements Y in cm on 30 male skulls in each of 5 epochs, 4000, 3300, 1850, 200 BC & 150 AD, included as indicators X. (Thomson, A. and Randall-Maciver, R., 1905, Ancient Races of the Thebaid, Oxford University Press.) Y = α + β 3300 X 1 + β 1850 X 2 + β 200 X 3 + β 150 X 4 + ε We inferred that u = 1 so the envelope model becomes Y = α + Γ{η 3300 X 1 + η 1850 X 2 + η 200 X 3 + η 150 X 4 } + ε, Since û = 1, we can easily plot Γ T Y vs epoch.

31 ½¼ ¾¼ ¼ ¼ ¼ Skull Boxplots vs Epoch Ì ¹ ¼¼¼ ¹ ¼¼ ¹¾¼¼¼ ¹ ¼¼ ½¼¼¼ Ê

32 Table: Estimated coefficients from cattle data. Week B B/se(B) β β/se( β) se(b)/se( β) We would need n 1500 for OLS to match the envelope results.

33 The standard MLE is β Std = (5.5, 4.8) T with bootstrap standard errors (4.2, 4.4) T, while the envelope estimate is β = (5.4, 5.1) T with bootstrap standard errors (1.12, 1.07) T. About 1500 observations would be needed for a likelihood analysis to yield the standard errors from an envelope analysis with 60 observations.

34 Intro. MLR 2Pop. MLE Cattle BrVol Other Heights of Boys and Girls Finis Extras Skulls MSE

35 Heights of Boys and Girls Ages 13 and EΣ(B) E Σ (B) Age Age13 Ω = 1.57 and Ω 0 = SE ratios for sm/em: 8.49 and 8.61.

36 Heights of Boys and Girls Ages 17 and EΣ(B) 190 Age E Σ (B) Age17 Ω = and Ω 0 = SE ratios for sm/em: 1.01 and 0.99

37 Heights of Boys and Girls: Bootstrap SEs Table: Bootstrap and estimated asymptotic standard errors of the two elements in β under the standard model (SM) and envelope model (EM). Response SM BSM EM BEM SM/EM BSM/BEM Age Age Age Age

38 SIMPLS algorithm for Φ (de Jong, 1993) Set w 0 = 0 and let φ k = (w 0,..., w k ) R p k. Then given φ k, the next vector w k+1 is constructed as S k = span(s X Φk ) w k+1 = l max (Q Sk S XY S T XY Q S k ) Φ k+1 = (w 0,..., w k, w k+1 ) for k = 1,..., m 1. m, the number of components, is chosen by cross-validation or a hold-out sample. Then Φ = Φ m. Envelope connection: With known m, span( φ m ) is a n-consistent estimator of the ΣX -envelope of span(β T ), E ΣX (B ), where B = span(β T ) and m = dim(e ΣX (B )).

39 Alternatively, we can use an envelope estimator for the same tasks: Y = α + η T {φ T X} + ε Σ X = φ φ T + φ 0 0 φ T 0 Σ = Σ β = β Std P Ṱ φ(s X ) where φ = arg min S {log P S S X Y P S 0 + log Q S S X Q S 0 } and S is an m-dim subspace of R p.

40 Estimation Let (B, B 0 ) denote an orthogonal basis of R p, where B R p q, B 0 R p (p q) and span(b) E M (B). Then v E B T 0 MB 0 (B T 0 B) implies that B 0 v E M (B).

41 1 Set initial value g 0 = G 0 = 0. 2 For k = 0,..., u 1, g k+1 R q is obtained direction in the envelope E V(θ) (span(θ)) as follows, 1 Let G k = (g 1,..., g k ) if k 1 and let (G k, G 0k ) be an orthogonal basis for R q. 2 Define the stepwise objective function L k (w) = log(w T A k w) + log(w T B 1 k w), (1) where A k = G T 0k V(θ)G 0k, B k = G T 0k V(θ)G 0k + G T 0k θ θ T G 0k and w R q k. 3 Solve w k+1 = arg min w L k (w) subject to a length constraint w T w = 1. 4 Define g k+1 = G 0k w k+1 to be the unit length (k + 1)-th stepwise direction.

42 Envelopes and MSE Y 1 y 0 0 Y 2

43 Y Y 2

44 E Σ (B) EΣ(B)