Statistical View of Least Squares

Transcription

1 Basic Ideas Some Examples Least Squares May 22, 2007

2 Basic Ideas Simple Linear Regression Basic Ideas Some Examples Least Squares Suppose we have two variables x and y

3 Basic Ideas Simple Linear Regression Basic Ideas Some Examples Least Squares Suppose we have two variables x and y Assume x is the independent (explanatory) variable

4 Basic Ideas Simple Linear Regression Basic Ideas Some Examples Least Squares Suppose we have two variables x and y Assume x is the independent (explanatory) variable Assume y = f (x)

5 Basic Ideas Some Examples Least Squares The simplest scenario y = ax + b

6 Basic Ideas Some Examples Least Squares The simplest scenario y = ax + b Assume we have data (x 1, y 1 ), (x 2, y 2 ),..., (x n, y n )

7 Basic Ideas Some Examples Least Squares The simplest scenario y = ax + b Assume we have data (x 1, y 1 ), (x 2, y 2 ),..., (x n, y n ) Which would be good guesses (â and b) for a and b, based on the data?

8 Some Examples Simple Linear Regression Basic Ideas Some Examples Least Squares example 1. Number of cars in a city and amount of ozone per m 3 of air

9 Some Examples Simple Linear Regression Basic Ideas Some Examples Least Squares example 1. Number of cars in a city and amount of ozone per m 3 of air example 2. Number of iterations set in matlab code and time to finish computation

10 Some Examples Simple Linear Regression Basic Ideas Some Examples Least Squares example 1. Number of cars in a city and amount of ozone per m 3 of air example 2. Number of iterations set in matlab code and time to finish computation example 3. Size and weight for some animal species

11 Adding Errors Simple Linear Regression Basic Ideas Some Examples Least Squares If data were observed so that y = ax + b holds exactly

12 Adding Errors Simple Linear Regression Basic Ideas Some Examples Least Squares If data were observed so that y = ax + b holds exactly Only two points needed to fit the line!!!

13 Basic Ideas Some Examples Least Squares Exact Relationship: y = 2+x y x

14 Basic Ideas Some Examples Least Squares Usually real data is of the form y i = ax i + b + ε i, i {1, 2,..., N} Here ε i is the error associated with observation i. Key assumptions E[ε i ] = 0, var[ε i ] = σ 2 We will assume σ 2 is known.

15 Example Simple Linear Regression Basic Ideas Some Examples Least Squares Dr. Crooks gets into gardening

16 Example Simple Linear Regression Basic Ideas Some Examples Least Squares Dr. Crooks gets into gardening Let x be the number of tulip seeds he plants

17 Example Simple Linear Regression Basic Ideas Some Examples Least Squares Dr. Crooks gets into gardening Let x be the number of tulip seeds he plants Let y be the number of plants that bloom

18 Example Simple Linear Regression Basic Ideas Some Examples Least Squares Dr. Crooks gets into gardening Let x be the number of tulip seeds he plants Let y be the number of plants that bloom (Assuming he read the instructions carefully...)

19 Basic Ideas Some Examples Least Squares No Exact Relationship y x

20 Least Squares Simple Linear Regression Basic Ideas Some Examples Least Squares Different people might draw different lines using eye estimation We need a systematic way to proceed

21 Basic Ideas Some Examples Least Squares No Exact Relationship y x

22 Least Squares Simple Linear Regression Basic Ideas Some Examples Least Squares error = observed y - predicted y ε i = y i ŷ i = y i (x i â + b)

23 Least Squares Simple Linear Regression Basic Ideas Some Examples Least Squares error = observed y - predicted y ε i = y i ŷ i = y i (x i â + b) A sensitive criteria is to minimize the aggregate error

24 Least Squares Simple Linear Regression Basic Ideas Some Examples Least Squares error = observed y - predicted y ε i = y i ŷ i = y i (x i â + b) A sensitive criteria is to minimize the aggregate error One common way: Least Squares to minimize the sum of squared errors

25 Mathematical Representation Suppose we have N observations y i = β 0 + β 1 x i + ε i (1) y i : value of the response for the i th observation x i : value of the predictor variable for the i th observation β 0 : intercept β 1 : slope ε i : random error term corresponding to i th observation

26 Estimation Using Least Squares We want to minimize the sum of squared errors SSE(β 0, β 1 ) = n (y i β 0 β 1 x i ) 2 (2) i=1 Find that pair of (β 0, β 1 ) for which SSE is smallest

27 Estimation Using Least Squares We want to minimize the sum of squared errors SSE(β 0, β 1 ) = n (y i β 0 β 1 x i ) 2 (2) i=1 Find that pair of (β 0, β 1 ) for which SSE is smallest We can call them ( ˆβ 0, ˆβ 1 )

28 Estimation Using Least Squares We want to minimize the sum of squared errors SSE(β 0, β 1 ) = n (y i β 0 β 1 x i ) 2 (2) i=1 Find that pair of (β 0, β 1 ) for which SSE is smallest We can call them ( ˆβ 0, ˆβ 1 ) Let s find them analytically

29 Normal Equations To find ( ˆβ 0, ˆβ 1 ) we take the partial derivatives of SSE with respect to β 0 and β 1 and get the following equations called normal equations: dsse dβ 0 = 0 dsse dβ 1 = 0 problem 1. Solve these two equations simultaneously to obtain ( ˆβ 0, ˆβ 1 )

30 Normal Equations The answer is... ˆβ 1 = (yi y)(x i x) (xi x) 2 ˆβ 0 = y ˆβ 1 x

31 Evaluating Our Procedure Why is this a sensible way to proceed?

32 Evaluating Our Procedure Why is this a sensible way to proceed? ( ˆβ 0, ˆβ 1 ) is known as the BLUE

33 Evaluating Our Procedure Why is this a sensible way to proceed? ( ˆβ 0, ˆβ 1 ) is known as the BLUE Best Linear Unbiased Estimator

34 Evaluating Our Procedure Why is this a sensible way to proceed? ( ˆβ 0, ˆβ 1 ) is known as the BLUE Best Linear Unbiased Estimator What????

35 Evaluating Our Procedure Why is this a sensible way to proceed? ( ˆβ 0, ˆβ 1 ) is known as the BLUE Best Linear Unbiased Estimator What???? This means E[ ˆβ i ] = β i, i {0, 1} var[ ˆβ i ] is the smallest attainable by an estimator of the form α + α i y i

36 This is the only place where we use E[ε i ] = 0, var[ε i ] = σ 2

37 Assumptions made for doing Statistical Inference Maybe we want to perform further inferences: like proposing a range of plausible values for ˆβ 0 and/or ˆβ 1 (confidence interval)

38 Assumptions made for doing Statistical Inference Maybe we want to perform further inferences: like proposing a range of plausible values for ˆβ 0 and/or ˆβ 1 (confidence interval) This can be done if we make some more assumptions

39 Assumptions made for doing Statistical Inference Maybe we want to perform further inferences: like proposing a range of plausible values for ˆβ 0 and/or ˆβ 1 (confidence interval) This can be done if we make some more assumptions Remember our model was y i = β 0 + β 1 x i + ε i (3)

40 Assumptions made for doing Statistical Inference Maybe we want to perform further inferences: like proposing a range of plausible values for ˆβ 0 and/or ˆβ 1 (confidence interval) This can be done if we make some more assumptions Remember our model was y i = β 0 + β 1 x i + ε i (3) We now assume ε i N(0, σ 2 ), i.e., Normally distributed with constant variance

41 Assumptions made for doing Statistical Inference Maybe we want to perform further inferences: like proposing a range of plausible values for ˆβ 0 and/or ˆβ 1 (confidence interval) This can be done if we make some more assumptions Remember our model was y i = β 0 + β 1 x i + ε i (3) We now assume ε i N(0, σ 2 ), i.e., Normally distributed with constant variance We also assume ε i and ε j are independent for i j

42 Matrix Representation The regression model y i = β 0 + β 1 x i + ɛ i can also be written using matrix notation as: y = X β + ɛ (4) How does X look like here?

43 Matrix Representation The normal equations are X X β = X y (5) estimate of β : residuals: ˆβ = (X X ) 1 X y (6) ê = Y ŷ = y X ˆβ (7)

44 What about σ 2 In real applications σ is not known If we optimize SSE with respect to (β 0, β 1, σ) Our expressions for ˆβ 0 and ˆβ 1 do not change!! The Least Squares estimate for σ is given by σ 2 = n i=1 (y i β 0 β 1 x i ) 2 n 2 (8)

45 Checking Model Assumptions Always do some exploratory data analysis before making inference

46 Checking Model Assumptions Always do some exploratory data analysis before making inference The results will not make sense if the assumptions are terribly violated

47 Checking Model Assumptions Always do some exploratory data analysis before making inference The results will not make sense if the assumptions are terribly violated ŷ i s are called fitted values ŷ i = ˆβ 0 + ˆβ 1 x i (9)

48 Checking Model Assumptions Always do some exploratory data analysis before making inference The results will not make sense if the assumptions are terribly violated ŷ i s are called fitted values ŷ i = ˆβ 0 + ˆβ 1 x i (9) ê i = y i ˆβ 0 + ˆβ 1 x i (10) ê i s are called residuals

49 Checking Model Assumptions Always do some exploratory data analysis before making inference The results will not make sense if the assumptions are terribly violated ŷ i s are called fitted values ŷ i = ˆβ 0 + ˆβ 1 x i (9) ê i = y i ˆβ 0 + ˆβ 1 x i (10) ê i s are called residuals The residuals are used to check if any of the model assumptions have been violated

50 Checking Model Assumptions: Independence We will look at the plot of the residuals vs. the fitted values

51 Checking Model Assumptions: Independence We will look at the plot of the residuals vs. the fitted values If the assumption of independence really holds we would expect the residuals to fluctuate in a random pattern around 0

52 Checking Model Assumptions: Independence We will look at the plot of the residuals vs. the fitted values If the assumption of independence really holds we would expect the residuals to fluctuate in a random pattern around 0 A curved pattern may indicate that the relationship between y and x is not linear

53 Checking Model Assumptions: Independence Residual Plot Where y is a Quadratic Function of x Residuals Fitted Values

54 Checking Model Assumptions: Independence Residual Plot After Making the Explanatory Variable x^2 Residuals Fitted Values

55 Checking Model Assumptions: Independence If there s some distinct pattern like a sinusoidal curve we ll start worrying that they are correlated (independent implies uncorrelated, but the converse is not true)

56 Checking Model Assumptions: Independence Residual Plot Where the Observations are Correlated Residuals Observation Statistical Number View of Least Squares

57 Checking Model Assumptions: Constant Variance To check whether the assumption of constant variance holds we also look at the residual plots

58 Checking Model Assumptions: Constant Variance To check whether the assumption of constant variance holds we also look at the residual plots If the cloud of residual points has a vertical width that changes with x the data may be heteroskedastic, i.e., it may not have constant variance

59 Checking Model Assumptions: Constant Variance Residual Plot for Non constant Variance Residuals Fitted Values

60 Checking Model Assumptions: Normality To check normality we plot the quantiles (AKA percentiles) of the data vs. the quantiles of the standard normal distribution

61 Checking Model Assumptions: Normality To check normality we plot the quantiles (AKA percentiles) of the data vs. the quantiles of the standard normal distribution On this QQ-plot perfectly normal data form a straight diagonal line

62 Checking Model Assumptions: Normality To check normality we plot the quantiles (AKA percentiles) of the data vs. the quantiles of the standard normal distribution On this QQ-plot perfectly normal data form a straight diagonal line Large deviations from a straight line may mean your data is not normally distributed

63 Checking Model Assumptions: Normality Normal Q Q Plot Normal Q Q Plot Sample Quantiles Sample Quantiles Simon Lunagomez Theoretical Quantiles and Dr. Jim Crooks Statistical View oftheoretical Least Squares Quantiles

64 Problem 2. Now we have all the tools to actually implement these using matlab Download the dataset smoke.txt Look at the scatterplot first to see if doing linear regression seems appropriate Find the least square estimates of the model parameters Look at residual and QQ- plots Do the assumptions seem valid here? Look at the graph of fitted values superimposed on the observed values How does the fit look?