Lecture 2. Simple linear regression

Transcription

1 Lecture 2. Simple linear regression Jesper Rydén Department of Mathematics, Uppsala University Regression and Analysis of Variance autumn 2014

2 Overview of lecture Introduction, short historical remarks Model formulation. Estimation Example: dummy variables Validation, outliers Alternatives to Least Squares estimation

3 Historical remarks: Least Squares method The method of least squares is the automobile of modern statistical analysis: despite its limitations, occasional accidents, and incidental pollution, it and its numerous variations, extensions, and related conveyances carry the bulk of statistical analyses, and are known and valued by nearly all. But there has been some dispute, historically, as who was the Henry Ford of statistics. SM Stigler (1981)

4 A scientific dispute Adrien-Marie Legendre ( ) Carl Friedrich Gauss ( ) Publication in Publication in 1809.

5 F Galton (1886): Regression towards mediocrity

6 Simple linear regression Observations (x i, y i ), i = 1,..., n. Model: Y i = β 0 + β 1 x i + ɛ i, i = 1,..., n, where ɛ i N(0, σ 2 ) are independent. Important expression in the view of linear models: µ i = µ(x i ) = β 0 + β 1 x i, i = 1, 2,..., n

7 Linear regression: Estimation of parameters LS estimates: ˆβ 1 = n i=1 y ix i n xȳ n n i=1 x i 2 n x 2 = i=1 (x i x)(y i ȳ) n i=1 (x i x) 2 = S xy S xx and ˆβ 0 = ȳ ˆβ 1 x where ȳ = n 1 y i and x = n 1 x i. Estimated model: with residuals ŷ i = ˆβ 0 + ˆβ 1 x i e i = y i ŷ i Blackboard

8 Idea: Minimize a sum of squares

9 Sum of squares identity By algebraic computations, the following identity can be shown to hold: n n n (y i ȳ) 2 = (ŷ i ȳ) 2 + (y i ŷ i ) 2 i=1 i=1 i=1 Important interpretation: Total variation = Explained variation + Unexplained variation Example of notation: SS T = SS R + SS E

10 Inference It can be shown that under the assumptions, and ˆβ 1 N(β 1, σ 2 S xx ) ˆβ 0 N(β 0, σ 2 ( 1 n + x 2 S xx ) Hypothesis testing of H 0 : β 1 = β1 0. Test quantity T = ˆβ 1 β 0 1 d[ ˆβ 1 ] t(n 2) under H 0, where d[ ˆβ 1 ] is the estimated standard error of ˆβ 1.

11 Predicted values It can be shown that where Moreover, ŷ i = n h ij y j j=1 h ij = 1 n + (x i x)(x j x) S xx V[e i ] = σ 2 (1 h ii ). Blackboard

12 Example: Dummy variable regression Example. A large food-processing centre needs to be able to swith from one type of package to another quickly, to react to changes in order patterns. A new method has been developed for changing the production line. New method, sample: 48 change-over times (min). Existing method, sample: 72 change-over times (min). Consider a simple linear regression model Y = β 0 + β 1 x + ɛ where Y is change-over time, x is a dummy variable (x = 1 for new method, x = 0 for existing method).

13 Example cont. Typical test of hypothesis in regression analysis: H 0 : β 1 = 0 against H 1 : β 1 < 0 Test statistic: when H 0 is true. T = β 1 0 d[ β 1 ] t(n 2)

14 Example, cont. Some figures

15 Regression output from R ANOVA table from R: Call: lm(formula = Changeover ~ New) Residuals: Min 1Q Median 3Q Max Coefficients: Estimate Std. Error t value Pr(> t ) (Intercept) <2e-16 *** New * --- Signif. codes: 0 *** ** 0.01 * Residual standard error: on 118 degrees of freedom Multiple R-squared: ,Adjusted R-squared: F-statistic: on 1 and 118 DF, p-value:

16 Danger of extrapolation :-)

17 Centering observations Some authors center the explanatory variable and hence study y i = β 0 + β 1 (x i x) + ɛ i, i = 1, 2,..., n instead of y i = β 0 + β 1 x i + ɛ i, i = 1, 2,..., n. Simpler calculations when calculating by hand, advantage in multiple regression when dealing with the problem of multicollinearity. The estimates of β 0 and β 1 can be shown to be independent.

18 What is an Outlier? Discordant observations may be defined as those which present the appearance of differing in respect of their law of frequency from other observations with which they are combined. Edgeworth (1887) An outlying observation, or outlier, is one that appears to deviate markedly from the other member of the sample in which it occurs. Grubbs (1969)

19 An non-regression example 10 observations, intended to be independent realizations from a common normal population: (Example from Johnson and Hunt (1979)) Which observations could be labelled outliers? Answer: 3 obs. were generated from N(3,1) (1.04, 2.04, 4.99); the remaining from N(0,1).

20 Possible definitions Discordant obs Any obs. that appears surprising or discrepant to the investigator. Contaminant obs Any obs. that is not a realization from the target distribution. Outlier A collective to refer to either a contaminant or a discordant obs. (Beckman and Cook (1983))

21 Outliers, Leverage, and Influence Outlier (common x value), low leverage and little influence.

22 Outliers, Leverage, and Influence Outlier (unusal x value), high leverage and substantial influence.

23 Outliers, Leverage, and Influence Outlier, high leverage and little influence.

24 J.D. Forbes experiment (1857) Altitude could be determined from atmospheric pressure. Middle of 19th century, barometers were fragile. Could a simpler measurement of the boiling point of water substitute for a direct reading of barometric pressure? Barometric pressure (inches Hg) Boiling point (F)

25 Residuals (Forbes data) Residual value Residual number

26 Residuals (Forbes data) 5 Normal probability plot 4 3 Quantile Data

27 Simple regression: Measure of fit Sample coefficient of determination: n R 2 i=1 = 1 e2 i n i=1 (y i ȳ) 2 This number lies between 0 and 1. The closer to 1, the better the fit. For simple linear regression, R 2 = r 2 (the estimated correlation coefficient).

28 Simple linear regression: Prediction of mean values Given x = x 0, the value of the response variable y is predicted by Statistical properties: ŷ 0 = ˆβ 0 + ˆβ 1 x 0 E[ŷ 0 ] = β 0 + β 1 x 0 [ 1 V[ŷ 0 ] = σ 2 n + (x 0 x) 2 ] n i=1 (x i x) 2

29 Simple linear regression: Prediction interval A future observation y 0 related to x 0 ; hence the difference y 0 ŷ 0 is of interest. E[y 0 ŷ 0 ] = 0 V[y 0 ŷ 0 ] = V[y 0 ] + V[ŷ 0 ] [ = σ n + (x 0 x) 2 ] n i=1 (x i x) 2

30 Forbes data 32 Barometric pressure (inches Hg) Boiling point (F)

31 Other approaches Rao et al. (2008): Linear models and generalizations. Least squares and alternatives.

32 Other approaches Consider a vector x = (x 1,..., x n ). Definition. For a real number p 1, the p-norm or L p -norm of x is defined by Euclidean distance: 2-norm. Manhattan distance: 1-norm. x p = ( x 1 p + x 2 p + + x n p ) 1 p The L -norm or maximum norm (or uniform norm) is the limit of the L p -norms for p. It turns out that this limit is equivalent to the following definition: x = max { x 1, x 2,..., x n }

33 Illustration of unit circles Illustration of unit circles in different p-norms every vector from the origin to the unit circle has a length of one.

34 Other approaches Two examples of direct-regression methods: Least Absolute Deviations (LAD): L 1 regression. Minimize ei = y i ŷ i Least Squares (LS): L 2 regression. Minimize e 2 i = (y i ŷ i ) 2