Chapter 1: Linear Regression with One Predictor Variable also known as: Simple Linear Regression Bivariate Linear Regression

Transcription

1 BSTT523: Kutner et al., Chapter 1 1 Chapter 1: Linear Regression with One Predictor Variable also known as: Simple Linear Regression Bivariate Linear Regression Introduction: Functional relation between two variables: Y = f(x) Value of X Value of Y Example: F = 32 + (9/5) C is a deterministic relationship 1 value of X 1 unique value of Y Statistical relation between two variables: 1 value of X a distribution of values of Y Y = Dependent / Response / Outcome Variable X = Independent / Explanatory / Predictor Variable

2 BSTT523: Kutner et al., Chapter 1 2 Linear Equation: General equation for a straight line Y = b 0 + b 1 X b 0 : Intercept = value of Y when X=0 b 1 : Slope = change in Y per unit change in X b 1 = change in Y value = "rise" change in X value "run" What if X increases by 1 unit? Y = b 0 + b 1 (X + 1) = {b 0 + b 1 X} + b 1 Y increases by b 1 units { Other than linear: e.g. curvilinear Y = b 0 + b 1 X + b 2 X 2 } Regression of Y on X Observe data points {(X 1, Y 1 ),..., (X n, Y n )} At each point X i there is a distribution of Y i s

3 Y = Head Circumference (cm) BSTT523: Kutner et al., Chapter 1 3 Example: Y = head circumference (cm), X = gestational age (wks) in a sample of 100 low birth weight infants X = Gestational Age (weeks) Qs: Does average head circumference change with gestational age? What is the form of the relationship? (linear? curvilinear?) How to estimate the relationship, given the data? How to make predictions for new observations?

4 BSTT523: Kutner et al., Chapter 1 4 Descriptive Data: Scatterplot Correlation Coefficients Some examples:

5 BSTT523: Kutner et al., Chapter 1 5 Population Correlation Coefficient: Random Variables X and Y with parameters μ X, μ Y, σ X 2, σ Y 2 ρ = Cov(X,Y) σ X σ Y = E[(X μ X )(Y μ Y )] σ X σ Y, 1 ρ +1 ρ measures the direction and strength of linear association between X and Y Maximum likelihood estimator of ρ is the Pearson Correlation Coefficient: r = (X i X)(Y i Y) = (X i X)(Y i Y) (X i X) 2 (Y i Y) 2 (n 1)s X s Y Inference on ρ: If X and Y are both Normal, H 0 : ρ = 0 vs. H a : ρ 0 T = r n 2 1 r 2 ~ t (n 2) under H 0 : ρ = 0 critical value = ±t (n 2,α 2)

6 BSTT523: Kutner et al., Chapter 1 6 Spearman Rank Correlation Coefficient: For X or Y non-normal R Xi = rank of X i R Yi = rank of Y i R X = R Y = n+1 2 means of ranks R Xi or R Yi Spearman rank correlation coefficient is r s = (R X i R X )(R Yi R Y ) (R Xi R X ) 2 (R Yi R Y ) 2, 1 r s +1 H 0 : There is no association between X and Y H a : There is association between X and Y T = r s n 2 1 r s 2 ~ t (n 2) under H 0 critical value = ±t (n 2,α 2)

7 BSTT523: Kutner et al., Chapter 1 7 The Simple Linear Regression Model Y i = β 0 + β 1 X i + ε i, i = 1,...,n observations Y i value of response for i th observation X i value of predictor for i th observation Population parameters (unknown): β 0 Population intercept β 1 Population regression coefficient ε i is i th random error term Mean: E(ε i ) = 0 Variance: Var(ε i ) = σ 2 Independence: ε i and ε j are uncorrelated for i j Normality: ε i ~N(0, σ 2 ) i. i. d. for all i Y i = β 0 + β 1 X i Constant + ε i Random,i.i.d.N(0,σ 2 ) E(Y i ) = E(β 0 + β 1 X i + ε i ) = β 0 + β 1 X i Var(Y i ) = Var(β 0 + β 1 X i + ε i ) = Var(ε i ) = σ 2 Y i ~ N(μ Y, σ 2 ) i. i. d. where μ Y = β 0 + β 1 X i

8 BSTT523: Kutner et al., Chapter 1 8 How to obtain β 0 and β 1, estimates for β 0 and β 1? Least Squares Estimators (LSE): LSEs minimize the sum of squared deviations of Y i from E(Y i ) Least Squares Criterion: Q = n i=1 [Y i E(Y i )] 2 = n i=1 [Y i (β 0 + β 1 X i )] 2 Minimize Q: set first derivatives w.r.t. each parameter = 0 First derivatives are: Q β 0 = 2 (Y i β 0 β 1 X i ) (1) Q β 1 = 2 X i (Y i β 0 β 1 X i ) (2) Normal Equations: set (1)=0 and (2)=0; call solutions β 0 and β 1 2 (Y i β 0 β 1X i ) = 0 2 X i (Y i β 0 β 1X i ) = 0 Y i = nβ 0 + β 1 X i 2 X i Y i = β 0 X i + β 1 X i

9 BSTT523: Kutner et al., Chapter 1 9 Solution to Normal Equations: Least Squares Estimators (LSE): β 1 = (X i X)(Y i Y) (X i X) 2 β 0 = Y β 1X Properties of LSE: Note: Unbiased estimators (accuracy) E(β 0) = β 0, E(β 1) = β 1 Minimum variance (precision) Robust against Normality assumption functions are called estimators calculated values from data are called estimates Interpretation: Intercept (β 0) Value of Y when X=0 (not always meaningful!) Slope (β 1) Average change in Y per unit increase in X Effect of X on Y; regression coefficient

10 Y = Head Circumference (cm) BSTT523: Kutner et al., Chapter 1 10 Example: X = gestational age (wks), Y = head circumference (cms) X = Gestational Age (weeks) Formula for least squares regression line is: Y = X Intercept: not meaningful! (extrapolation to X = 0 weeks) Slope: For every increase of one week gestational age, there is an increase of about 0.78 cm head circumference.

11 BSTT523: Kutner et al., Chapter 1 11 Another approach: Method of Maximum Likelihood The MLE maximizes the likelihood function (the likelihood of the observed data, given the model parameters) Q. Under which parameter values is the sample data most likely to occur? [see explanation of MLE on p.27-29] For simple linear regression: ε i = Y i β 0 β 1 X i ~ N(0, σ 2 ) f(ε i ) = 1 1 exp { (y 2πσ 2σ 2 i β 0 β 1 x i ) 2 } likelihood = L = n i=1 f(ε i ) log e L = ln { 1 (2πσ 2 ) n 2 1 exp [ (y 2σ 2 i β 0 β 1 x i ) 2 n i=1 ]} = n 2 ln(2πσ2 ) 1 n (y 2σ 2 i=1 i β 0 β 1 x i ) 2 log e L β 0 = 0, log e L β 1 = 0, log e L σ = 0 same solution as LSE (please prove for yourself!) same nice properties

12 BSTT523: Kutner et al., Chapter 1 12 After calculating the fitted regression line: Fitted value Y i Y i = β 0 + β 1X i On the fitted line for the value X i Fitted Y- values are estimates of the Mean Response Function Y i is an unbiased estimator of the mean response at X i The fitted line is an unbiased estimator of the mean response function Note: the point (X, Y) is ALWAYS on the fitted regression line, i.e, Y = β 0 + β 1X

13 BSTT523: Kutner et al., Chapter 1 13 The i th residual e i : e i = Y i Y i = Y i (β 0 + β 1X i ) it is the vertical distance between (X i, Y i ) and (X i, Y i) it is the estimate of the i th error term, e i = ε i n i=1 e i = 0 Proof: e i = [Y i (β 0 + β 1X i )] = Y i nβ 0 β 1 X i = 0 (by normal equation 1)

14 BSTT523: Kutner et al., Chapter 1 14 Error Sum of Squares: SSE = n (Y i Y i) 2 i=1 = n 2 i=1 e i minimum when residuals are from LSE or MLE. associated degrees of freedom df = n 2 (generally df = n p where p = # of parameters in the model) Mean Squared Error: unbiased estimator of σ 2 MSE = SSE df = SSE n 2 E(MSE) = σ 2

15 Y = Head Circumference (cm) BSTT523: Kutner et al., Chapter 1 15 Example: X = gestational age and Y = head circumference 100 observations scatterplot, fitted line, fitted values X = Gestational Age (weeks)

16 BSTT523: Kutner et al., Chapter 1 16 EXCEL: SUMMARY OUTPUT Regression Statistics Multiple R R Square Adjusted R Square Standard Error Observations 100 ANOVA df SS MS F Significance F Regression E-21 Residual Total Coefficients Standard Error t Stat P-value Intercept X Variable E-21

17 BSTT523: Kutner et al., Chapter 1 17 SAS output: The REG Procedure Model: MODEL1 Dependent Variable: headcirc Number of Observations Read 100 Number of Observations Used 100 Analysis of Variance Sum of Mean Source DF Squares Square F Value Pr > F Model <.0001 Error Corrected Total Root MSE R-Square Dependent Mean Adj R-Sq Coeff Var Parameter Estimates Parameter Standard Variable DF Estimate Error t Value Pr > t Intercept gestage <.0001