Simple Linear Regression

Transcription

1 Simple Linear Regression Reading: Hoff Chapter 9 November 4, 2009

2 Problem Data: Observe pairs (Y i,x i ),i = 1,... n Response or dependent variable Y Predictor or independent variable X GOALS: Exploring p(y x) as a function of x Understanding the mean (variability) in Y as a function of x Special cases: linear regression (normal Y) logistic regression (binomial Y success probability depends on x)

3 Models Model with additive error: for i = 1,...,n, Y i = f (X i ) + ǫ i Regression function E(Y x) = f (x) Taylors series expansion of f (x i ) = f (x 0 ) + f (x 0 )(x i x 0 ) + Remainder leads to locally linear approximation Y i = α + βx i + ε i ε i : independent errors (sampling, measurement, lack of fit)

4 BIG PICTURE: Y i = α + βx i + ε i, ε i iid N(0,σ 2 ) Estimate parameters (α,β,σ 2 ) interpretation of parameters: β, α assess model fit adequate? good? if inadequate, how? predict new ( future ) responses at new x n+1,... how much variability does x explain?

5 Example: Body Fat data For a group of male subjects, various body measurements were obtained An accurate measurement of the percentage of body fat is recorded for each Goal is to use the other body measurements as a proxy for predicting body fat Focus on simple linear regression model for predicting body fat as a function of abdomen circumference

6 Data Percent Bodyfat Circumference of Abdomin (cm)

7 Sample Summary Statistics Sample means: x, ȳ Sample variances: s 2 y = S yy /(n 1) s 2 x = S xx /(n 1) sample covariance s xy = S xy /(n 1) where the Sums of Squares are: Syy = n i=1 (y i ȳ) 2 : Total Variation in response Sxx = n i=1 (x i x) 2 Sxy = n i=1 (x i x)(y i ȳ)

8 Correlation Sample correlation is covariance in a standardized scale (unit-less) r = s xy s x s y measure of dependence 1 r 1

9 Ordinary Least Squares (OLS) For any chosen α,β, Q(α,β) = n ε 2 i = i=1 n (y i α βx i ) 2 i=1 measures fit of chosen line α + βx to response data OLS estimator: Choose ˆα, ˆβ to minimize Q(α,β) Ad-hoc principal of least squares estimation Under normal error assumption OLS is equivalent to MLE

10 MLEs FACTS: or ˆβ = s xy sx 2, ˆα = ȳ ˆβ x ( ) sy ˆβ = r s x ˆβ is correlation coefficient r, corrected for relative scales of y : x so that the units of the fitted values ˆα + ˆβx are on scale of Y For use in theoretical derivations ˆβ = Sxy S xx

11 R 2 measure of model fit: Simplest model: β = ˆβ = 0 so Y i are a normal random sample with mean α ˆα = ȳ, Q(ȳ,0) = S yy = Total Sum of Squares = TSS Any other model fit: SSE = Sum of Squares Error Q(ˆα, ˆβ) R 2 = 1 Q(ˆα, ˆβ)/Q(ȳ,0) Sum Squares Error = 1 Total SS TSS = SS due to Regression on X + SSE

12 Facts R 2 = r 2 Higher % variation explained is better: Higher correlation Measures linear correlation only not general dependence not causation Can be used to compare other simple linear regression models with transformations of X Cannot be used to compare models with transformed Y Does NOT provide a measure of model adequacy

13 Summarizing Model Fit Fitted values Ŷ i = ˆα + ˆβx i Residuals ˆε i = Y i Ŷi estimates of ε i Residual sum of squares = SSE = Q(ˆα, ˆβ) = n i=1 ˆε2 i measures remaining/residual variation in response data s 2 Y X is a point estimate of σ2 from fitted model s 2 Y X SSE n = MSE = n 2 = ˆε 2 i n 2 i=1 note: n 2 degrees of freedom, not n 1 lose 2 degrees of freedom for estimation of α, β

14 Some R commands bodyfat.lm = lm(bodyfat abdomin, data = bodyfat) summary(bodyfat.lm) (regression output) plot(bodyfat.lm) (residual plots) anova(bodyfat.lm) (Analysis of Variance) regr1.plot(bodyfat.lm) in library(hh)

15 Model Assessment Residual analysis: Graphical exploration of fitted residuals ˆε i Standardize: r i = ˆε i / var(ˆε i ) var(ˆε i ) = ˆσ 2 (1 h ii ) h ii leverage measure of potential influence Check normality assumption, constant variance, outliers, influence Treat ˆε i as new data look at structure, other predictors Other predictors? Transformations? Revise model before making interpretations...

16 Residuals Residuals Residuals vs Fitted Standardized residuals Normal Q Q Fitted values Theoretical Quantiles Standardized residuals Scale Location Standardized residuals Residuals vs Leverage Cook s distance Fitted values Leverage

17 Diagnostics Residuals versus fitted values (or versus x) Normal quantile plot of residuals (check distributional assumptions of the errors) Scale-location plot: ε i versus Ŷi. Detect if the spread of the residuals is constant over the range of fitted values. Cook s distance plot: shows if any data points have a large influence on the predicted values of the response variable. Values greater than 1 are considered influential. Case 39 appears influential...

18 Residuals Without Case 39 lm(bodyfat ~ Abdomen, subset=c(-39)) Residuals Residuals vs Fitted Standardized residuals Normal Q Q Fitted values Theoretical Quantiles Standardized residuals Scale Location Standardized residuals Residuals vs Leverage Cook s distance Fitted values Leverage

19 Conjugate Priors for Regression Model: Y i ind N(α + βx i,σ 2 ) Normal-Gamma distribution is conjugate for α and β and precision φ 1 σ 2 (α,β) φ N((α 0,β 0 ),φ 1 Σ) variances and covariance φ Gamma(ν 0 /2,ν 0 σ 2 0 /2) Σ is a 2 2 matrix of A Reference Prior for Regression: Limiting case of conjugate prior as the prior variances goes to infinity (information goes to zero) p(α,β,φ) 1/φ

20 Theory for Inference β ˆβ t n 2 (0,1) 1 S xx s 2 Y X ˆβ is OLS (MLE) estimate of β, s 2 Y X = ˆσ 2 is the MSE (marginal) posterior for β is a Student t distribution with n 2 df. Sampling distribution of ˆβ given β is Student t distribution with n 2 df Used for classical and Bayesian (Reference) analysis

21 Distributions Continued (marginal) posterior for α is a Student t distribution with n 2 df. Sampling distribution of ˆα is Student t distribution with n 2 df α ˆα t n 2 (0,1) sy 2 X (1 n + x2 S xx )

22 Significance of Regression Measuring the explanatory power of predictor X Credible Intervals (HPD or equal-tailed): ˆβ ± t α/2 s β where t α/2 is 100 α/2% quantile of a standard t n 2 and s β = sy 2 X 1/S xx Confidence Intervals: and SE(ˆβ) = s 2 Y X 1/S xx ˆβ ± t α/2 SE(ˆβ)

23 Body Fat Example: summary(lmobj) > summary( lm(bodyfat ~ Abdomen, data=bodyfat, subset=c(-39))) Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** Abdomen <2e-16 *** --- Residual standard error: on 249 df Multiple R-squared: , F-statistic: on 1 and 249 DF, p-value: < 2.2e-16 Roughly 68% of the variation in bodyfat may be explained by abdominal circumference

24 HPD intervals Estimate Std. Error t value Pr(> t ) (Intercept) Abdomen % HPD interval ±qt(.025,249) = (0.61,0.73) For every additional cm of abdominal circumference, percent bodyfat increases by 0.61 to 0.73 percent with probability For every additional inch of abdominal circumference, percent bodyfat increases by 1.6 to 1.9 percent with probability 0.95.

25 Predictions The (posterior) predictive distribution for a new case, y n+1 = α + βx n+1 + ε n+1 is also a Student t distribution with n 2 df. y n+1 y 1,... y n t n 2 (ŷ,s 2 y n+1 ) ŷ = ˆα + ˆβx n+1 s 2 y n+1 = s 2 Y X (1 + 1 n + (x n+1 x) 2 S xx ) posterior uncertainty about α + βx n+1 depends on x n+1 spread is higher for x n+1 far from x additional variability +s 2 Y X due to ε n+1

26 Intervals: ci.plot(bodyfat.lm) 95% confidence and prediction intervals for bodyfat.lm 60 xbar Bodyfat 30 observed fit conf int pred int xbar Abdomen

27 Intervals: without case 39 95% confidence and prediction intervals for bodyfat.lm2 xbar Bodyfat observed fit conf int pred int 20 0 xbar Abdomen Percent Body fat for 34 inch abdomin = 14.9%

28 Significance of the Regression Question: How probable is β = 0 under the posterior? Informal Answer: Compute posterior probability on β values with lower posterior density than β = 0 Measures probability of β less likely than β = 0 Informal test : Probability in tails = significance level = (Bayesian) p-value p-value = P( t > ˆβ/s β ) = P( t > ˆβ/SE(β) ) Classical testing terminology: The regression on x is significant at the 5% level (or 1%, etc) if the p-value is smaller than 0.05 (or 0.01, etc)

29 Lindley s Method Lindley suggested rejecting the hypothesis that β = 0 at the α100% level of significance if the (1 α)100% HPD region does not include 0. 0 / (ˆβ t 1 α/2 s β, ˆβ + t 1 α/2 s β ) Equivalent to comparing the p-value to α and concluding that the regression is significant if the p-value is less than α. Alternative approach is to compute a Bayes Factor.

30 Bayes Factors Testing H o : β = 0 versus H a : β 0 Assign prior probabilities to H o and H a Find P(H i Y ) via Bayes Theorem Bayes Factor for comparing evidence in favor of H o BF[H o : H a ] = p(h o Y )/p(h o ) p(h a Y )/p(h a ) Often difficult to calculate, instead use lower bound based on p-values (Berger, Selke and Bayarri ) BF[H o : H a ] = ep log(p)

31 Bodyfat Example P( t > 22.11) = The regression of bodyfat on abdominal circumference is highly significant (p-value = ). Lower bound on Bayes Factor BF[H 0 : H a ] = exp(1)log( ) = Approximate posterior probability of H 0 = P(H 0 Y ) = BF[H 0 : H a ]O[H 0 : H a ] 1 + BF[H 0 : H a ]O[H 0 : H a ]

32 Jeffreys Scale of Evidence Bayes Factor Interpretation B 1 H 0 supported 1 > B minimal evidence against H > B 10 1 substantial evidence against H > B 10 2 strong evidence against H > B decisive evidence against H 0 B = BF[H o : B a ] Decisive evidence against hypothesis that bodyfat is not associated with abdominal circumference