Partial factor modeling: predictor-dependent shrinkage for linear regression

Transcription

1 modeling: predictor-dependent shrinkage for linear Richard Hahn, Carlos Carvalho and Sayan Mukherjee JASA 2013 Review by Esther Salazar Duke University December, 2013

2

3 Factor framework The factor framework may be written in two parts: Linear for a scalar response Y i : (Y i X i, β, σ 2 ) N (Xi t β, σ 2 ) Marginal model for a p-dimensional vector of predictor variables X i X i = Bf i + ν i, ν i N (0, Ψ) f i N (0, I k ) where B R p k and Ψ is a diagonal matrix

4 Factor framework The factor framework may be written in two parts: Linear for a scalar response Y i : (Y i X i, β, σ 2 ) N (Xi t β, σ 2 ) Marginal model for a p-dimensional vector of predictor variables X i X i = Bf i + ν i, ν i N (0, Ψ) f i N (0, I k ) where B R p k and Ψ is a diagonal matrix This work considers modification to a Gaussian factor model, suited for and variable. It differs from previous work on Bayesian variable in that it explicitly accounts for predictor correlation structure

5 Gaussian factor model Linear for a scalar response Y i : (Y i X i, β, σ 2 ) N (Xi t β, σ 2 ) Marginal model for a p-dimensional vector of predictor variables X i X i = Bf i + ν i, ν i N (0, Ψ) f i N (0, I k ) This paper asks the question: how should the prior of β depend on B and Ψ?

6 Two extreme answers: (1) Pure linear which ignores the marginal distribution the predictors π(x) π(β B, Ψ) = π(β) (2) Pure factor model where Y i depends linearly on the same k latent factors that captures the covariation in X i π(y i X i, f i, θ) = π(y i θ, f i ) and E(Y i θ, f i) = θf i Intuition: Y i N (θf i, σ 2 ), X i N (Bf i, Ψ), f i N (0, I k ) ( ) ( [ Xi 0 N Y i 0 ] [, BB T + Ψ Bθ T θb T θθ T + σ 2 ]) E(Y i X i ) = θb T (BB T + Ψ) 1 X i This entails that β is a deterministic function β T = θb T (BB T + Ψ) 1 Also, f i = B T (BB T + Ψ) 1 X i, projection of X i onto a k-dimensional subspace

7 Bayesian linear factor model Factor model for the predictors: X i N (Bf i, Ψ), f i N (0, I k ) Integrating over f i : cov(x i ) Σ X = BB T + Ψ Assuming that the p predictors influence Y i only through the k-dimensional vector f i

8 Effects of misspecifying k

9 Effects of misspecifying k By the likelihood criterion the two models are nearly identical. In terms of predicting X 10 the two-factor model is nearly always the best

10 Idea: Relax the assumption that the latent factors capturing the predictor covariance Σ X are sufficient for predicting the response Y i This is achieved by using the covariance structure where V = (v 1,..., v p ), the 1 p row vector, is not exactly equal to θb T Novelty: Prior for V, conditional on θ, B and Ψ v j N ( {θb T } j, ω 2 wj 2 ψj 2 ) where ω 2 is a global variance, wj 2 is a predictor-specific variance (Carvalho, Polson, Scott 2010), ψj 2 is the diagonal element of Ψ

11 Hierarchical specification Let Λ = (V θb T )Ψ 1/2. Considering the prior v j N ( {θb T } j, ω 2 w 2 j ψ 2 j ) and the reparameterization given by Λ, we have that λ j N (0, ω 2 w 2 j )

12 Hierarchical specification Priors for τ, ω, and the individual elements of w, q, t: half-cauchy This corresponds to the horseshoe priors ( Carvalho et al, 2010) over the elements of B, θ and Λ Posterior inference: the model can be fit using a Gibbs sampling approach

13 Out-of-sample prediction applied example They compare partial factor to five other methods: (1) ridge, (2) partial least square, (3) lasso, (4) principal component and (5) Bayesian factor model using the model prior of Bhattacharya and Dunson (2011)

14 Problem: With the assumption that the predictors X and the data Y come from a joint normal distribution, the variable problem is related to infer exactly zero entries in the precision matrix Σ 1 X,Y From the partial factor model We can use a spike-and-slab prior over Λ such that Analogous priors are placed on the elements of B and θ

15 : beyond the linear model It is straightforward to extend the method to a binary or categorical response variable Z i by treating the continuous response Y i as an additional latent variable For instance: if Z i is binary Z i = 1(Y i < 0) where Y i follows the partial factor model called partial factor probit model