Lecture 11 Simple Linear Regression

Transcription

1 Lecture 11 Simple Liear Regressio Fall 2013 Prof. Yao Xie, H. Milto Stewart School of Idustrial Systems & Egieerig Georgia Tech

2 Midterm 2 mea: 91.2 media: std: 6.5 2

3 Meddicorp Sales Meddicorp Compay sells medical supplies to hospitals, cliics, ad doctor's offices. Meddicorp's maagemet cosiders the effectiveess of a ew advertisig program. Maagemet wats to kow if the advertisemet i 1999 is related to sales. 3

4 Data The compay observes for 25 offices the yearly sales (i thousads) ad the advertisemet expediture for the ew program (i hudreds) SALES ADV

5 Regressio aalysis Step 1: graphical display of data scatter plot: sales vs. advertisemet cost 5

6 Step 2: fid the relatioship or associatio betwee Sales ad Advertisemet Cost Regressio 6

7 Regressio Aalysis The collectio of statistical tools that are used to model ad explore relatioships betwee variables that are related i odetermiistic maer is called regressio aalysis Occurs frequetly i egieerig ad sciece 7

8 Scatter Diagram May problems i egieerig ad sciece ivolve explorig the relatioships betwee two or more variables. Regressio aalysis is a statistical techique that is very useful for these types of problems 8 = = = = i i i i i i i y y x x y y x x ) ( ) ( ) )( ( ˆρ 1 ˆ 1 ρ

9 Basics of Regressio We observe a respose or depedet variable (Y) With each (Y), we also observe regressors or predictors {X 1,, X } Goal: determie the mathematical relatioship betwee respose variables ad regressors Y = h(x 1,, X ) 9

10 Fuctio ca be o- liear I this class, we will focus o the case where Y is a liear fuctio of {X 1,, X } Y = h(x1,...,x) = β0+β1x1+...+βx C

11 Differet forms of regressio Simple liear regressio Y = β0 + β1x + ε Multiple liear regressio Y = β0 + β1x1 + β2x2+ ε Polyomial regressio Y = β0 + β1x + β2x 2 + ε

12 Basics of regressios Which is the RESPONSE ad which is the PREDICTOR? The respose or depedet variable varies with differet values of the regressor/predictor. The predictor values are fixed: we observe the respose for these fixed values The focus is i explaiig the respose variable i associatio with oe or more predictors 12

13 Simple liear regressio Our goal is to fid the best lie that describes a liear relatioship: Fid (β0,β1) where Y = β0 + β1x + ε Ukow parameters: 3 1. β0 Itercept (where the lie crosses y-axis) 2. β1 Slope of the lie Basic idea a. Plot observatios (X,Y) b. Fid best lie that follows plotted poits C

14 Class activity 1. I the Meddicorp Compay example, the respose is: A. Sales B. Advertisemet Expediture 2. I the Meddicorp Compay example, the predictor is: A. Sales B. Advertisemet Expediture 3. To lear about the associatio betwee sales ad the advertisemet expediture we ca use simple liear regressio: A. True Β. False 4. If the associatio betwee respose ad predictor is positive the the slope is A. Positive Β. Negative C. We caot idetify the slope sig 14

15 Simple liear regressio: model With observed data {(X1,Y1),.,(X,Y)}, we model the liear relatioship E(εi) = 0 Var(εi) = σ 2 Yi = β0 + β1xi + εi, i =1,, {ε1,, ε} are idepedet radom variables (Later we assume εi ~ Normal) Later, we will check these assumptios whe we check model adequacy 15

16 Summary: simple liear regressio Based o the scatter diagram, it is probably reasoable to assume that the mea of the radom variable Y is related to X by the followig simple liear regressio model: Respose Regressor or Predictor ε i Y Itercept i = β + β X i + ε i =1,2,, 0 1 i ( ) ε i Ν 0, σ 2 Slope Radom error where the slope ad itercept of the lie are called regressio coefficiets. The case of simple liear regressio cosiders a sigle regressor or predictor x ad a depedet or respose variable Y. 16

17 Estimate regressio parameters To estimate (β0,β1), we fid values that miimize squared error: ( ) 2 y i ( β + β 0 1x i ) i= 1 derivatio: method of least squares 17

18 Method of least squares y i 0 1 x i i, i 1, 2, p, y Observed value Data (y) ` To estimate (β0,β1), we fid values that miimize squared error: L a 2 i a a 1 a 1 y i x i The least squares estimators of 0 ad 1, say, ˆ 0 ad ˆ 1, must satisfy Figure 11-3 Estimated regressio lie x L ` 0 ˆ ` a 1 L 2 ` 1 2 a 2 1 ` ˆ 0, ˆ 1 0, ˆ 1 2 a ` 1 2 ` 1 2 Least square ormal equatios 1 y i ˆ 0 ˆ 1x i y i ˆ 0 ˆ 1x i 2 x i 0 ˆ 0 a ˆ 0 ˆ 1 a x i a x i ˆ 1 a x i 2 a y i y i x i 18

19 Least square estimates The least squares estimates of the itercept ad slope i the simple liear regressio model are ˆ 0 y ˆ 1x (11-7) ˆ 1 a y i x i a a a x 2 i a a y i b a a 2 x i b x i b (11-8) where y 11 2 g y i ad x 11 2 g x i. 19

20 Alterative otatio S x x a 1 2 1x i x2 2 a 2 x i a a a x i b 2 b (11-10) S x y a 1y i y21x i x2 a a a a a x i b a a x i ay ia b a a b y i b (11-11) ˆ β 0 = y ˆ β1x 1 ˆβ = S S xy xx ˆ ˆ ˆ yi = β 0 + β1x i Fitted (estimated) regressio model 20

21 Example: oxyge ad hydrocarco level Table 11-1 Oxyge ad Hydrocarbo Levels Observatio Hydrocarbo Level Purity Number x (%) y (%) Purity (y) Questio: fit a simple regressio model to related purity (y) to hydrocarbo level (x) Figure 11-1 Scatter diagram of oxyge purity versus hydrocarbo level from Table Hydrocarbo level ( x) 21

22 a x i a x y y i 1, a y i 2 170, a a x i y i 2, x i S x x a 20 x i 2 a a x i b ad S x y a 20 x i y i a a 20 x i b a a y i b 2, ,

23 Therefore, the least squares estimates of the slope ad itercept are ˆ 1 S x y S x x ad ˆ 0 y ˆ 1x The fitted simple liear regressio model (with the coefficiets reported to three decimal places) is 102 ŷ x Oxyge purity y (%) Hydrocarbo level (%) x

24 Iterpretatio of regressio model Regressio model ŷ 1 2 ŷ x 89.23% whe the This may be iterpreted as a estimate of the true populatio mea purity whe x 1.00%, The estimates are subject to error hydrocarbo level is x 1.00%. T later: we will use cofidece itervals to describe the error i estimatio from a regressio model 24

25 Estimatio of variace Usig the fitted model, we ca estimate value of the respose variable for give predictor Residuals: Our model: Y i = β 0 + β 1 X i + ε i, i =1,,, Var(ε i ) = σ 2 Ubiased estimator (MSE: Mea Square Error) ˆ σ 2 = ˆ ˆ yi = β 0 + β1x i r i = MSE y = i oxyge ad hydrocarco level example i= 1 ˆ yˆ r i 2 i 2 ˆ , 25

26 Example: Oil Well Drillig Costs Estimatig the costs of drillig oil wells is a importat cosideratio for the oil idustry. Data: the total costs ad the depths of 16 off-shore oil wells located i Philippies. Depth Cost Depth Cost

27 Step 1: graphical display of the data R code: plot(depth, Cost, xlab= Depth, ylab = Cost ) 27

28 Class activity 1. I this example, the respose is: A. The drillig cost B. The well depth 2. I this example, the depedet variable is: A. The drillig cost B. The well depth 3. Is there a liear associatio betwee the drillig cost ad the well depth? A. Yes ad positive Β. Yes ad egative C. No 28

29 Step 2: fid the relatioship betwee Depth ad Cost 29

30 Results ad use of regressio model 1. Fit a liear regressio model: Estimates (β 0,β 1 ) are ( , ) 2. What does the model predict as the cost icrease for a additioal depth of 1000 ft? If we icrease X by 1000, we icrease Y by 1000β 1 = $ What cost would you predict for a oil well of 10,000 ft depth? X = 10,000 ft is i the rage of the data, ad estimate of the lie at x=10,000 is ˆ β + (10,000) ˆ β = ,033 = $ What is the estimate of the error variace? Estimate σ 2 774,211 5.What could you say about the cost of a oil well of depth 20,000 ft? X=20,000 ft is much greater tha all the observed values of X We should ot extrapolate the regressio out that far

31 Summary Simple liear regressio Estimate coefficiets from data: method of least squares ˆ β = y ˆ β1x Y = β0 + β1x Estimate of variace 0 ˆβ 1 = S S xy xx ˆ ˆ ˆ yi = β 0 + β1x i y Fitted (estimated) regressio ` model Observed value Data (y) ` Estimated regressio lie x 31