Simple Regression. Acknowledgement. These slides are based on presentations created and copyrighted by Prof. Daniel Menasce (GMU) CS 700

Transcription

1 Simple Regressio CS 7 Ackowledgemet These slides are based o presetatios created ad copyrighted by Prof. Daiel Measce (GMU)

2 Basics Purpose of regressio aalysis: predict the value of a depedet or respose variable from the values of at least oe explaatory or idepedet variable (also called predictors or factors). Purpose of correlatio aalysis: measure the stregth of the correlatio betwee two variables. 3 Liear Relatioship

3 No Relatioship Negative Curviliear

4 Simple Liear Regressio Residual Error y error estimated y x 7 Simple Liear Regressio Selectig the best lie y Miimize the sum of the squares of the errors: least squares criterio. error estimated y x 8 4

5 Liear Regressio Yˆi X i b SSE = : : ˆ + Yi = b b X i predicted value of Y for observatio i. value of observatio i. ad b i= e Subject to: e i = i= i = are chose to miimize: ( Y ˆ i Yi ) = i= i= [ Y ( b + b X )] i i 9 Method of Least Squares b = i= i= b = Y b X X i Y i X Y X i ( X ) 5

6 Liear Regressio Example Number of I/Os (x) CPU Time (y) Estimate (.48*x +.58) Error Error Squared Xbar 5.5 Ybar.75 Sum x 385 Sum xy b.48 b.58 Liear Regressio Example CPU Time (sec) CPU time =.48*No. I/Os +.58 R = Number of I/Os 6

7 Allocatio of Variatio No regressio model: use mea as predicted value. SSE is: SST = ( Yi Y ) i= Sum of squares total SSR = SST SSE Sum of squares explaied by the regressio. Variatio ot explaied by regressio 3 Allocatio of Variatio Coefficiet of determiatio (R ): fractio of variatio explaied by the regressio. R SSR SST SSE = = = SST SST SSE SST The closer R is to oe, the better is the regressio model. 4 7

8 Number of I/Os (x) CPU Time (y) Estimate (.48*x +.58) Error Error Squared SSY SST SSR R SST Y Y Y = ( i ) = i Y = SSY SS i= i= SSE = ( Yi Yˆ i ) i= SSR = ( Yˆ i Y ) = SST SSE i= SSR R = SST coefficiet of determiatio. SSE SSY The higher the value of R the better the regressio. 5 Stadard Deviatio of Errors Variace of errors: divide the sum of squares (SSE) by the umber of degrees of freedom (- sice two regressio parameters eed to be computed first). s e = SSE Mea squared error (MSE) 6 8

9 Degrees of freedom of various sum of squares. SST - Need to compute Y SSY SS SSE SSR - Does ot deped o ay other parameter Need to compute two regressio parameters =SST-SSE Degrees of freedom add as sum of squares do. 7 Cofidece Iterval for Regressio Parameters b o ad b were computed from a sample. So, they are just estimates of the true parameters β ad β for the true model. Stadard deviatios for b o ad b. s b = s e + i= ( X ) X i ( X ) s b = s e X i ( X ) i= 8 9

10 Cofidece Iterval for Regressio Parameters (-α)% cofidece iterval for b o ad b b b ± t ± t [ α / ; ] [ α / ; ] s s b b 9 Cofidece Iterval Example Number of I/Os (x) CPU Time (y) Estimate (.48*x +.58) Error Error Squared SSE:.7 Xbar 5.5 Ybar.75 Sum x 385 Sum xy b.48 b.58 se.44 Lower bo.777 se.464 Upper bo.739 sb.7 sb.69 Lower b % cofidece level Upper b alpha.5 t[-alpha/;-].3656 SST SSR.377 R

11 Cofidece Iterval for the Predicted Value The stadard deviatio of the mea of a future sample of m observatios at X = X p is y mp = s e m + + sˆ ( X p X ) X i X As the future sample size (m) decreases, the stadard deviatio for predicted value decreases. i= / Cofidece Iterval for the Predicted Value (-α)% cofidece iterval for the predicted value for a future sample of size m at X p : yˆ p ± t[ α / ; ] s ˆ y mp

12 Liear Regressio Assumptios Liear relatioship betwee the respose (y) ad the predictor (x). The predictor (x) is o-stochastic ad is measured without ay error. Errors are statistically idepedet. Errors are ormally distributed with zero mea ad a costat stadard deviatio. 3 Liear Regressio Assumptios Liear relatioship betwee the respose (y) ad the predictor (x). y liear y piecewise-liear x x y possible outlier y o-liear x x 4

13 Liear Regressio Assumptios Errors are statistically idepedet. residual residual o tred predicted respose residual predicted respose tred predicted respose tred 5 Liear Regressio Assumptios Errors are ormally distributed. residual quatile residual quatile Normal quatile ormally distributed errors Normal quatile o-ormally distributed errors 6 3

14 Liear Regressio Assumptios Errors have a costat stadard deviatio. residual predicted respose o tred i spread residual predicted respose icreasig spread 7 Other Regressio Models 8 4

15 Multiple Liear Regressio Use to predict the value of the respose variable as fuctio of k predictor variables x,, x. Y ˆ = b + b X + b X b i i i Similar to simple liear regressio. MS Excel ca be used to do multiple liear regressio. x X ki 9 CPU Time (yi) I/O Time (xi) Memory Requiremet (xi) Wat to fid: CPUTime = b + b * I/OTime + b * MemoryRequiremet 3 5

16 SUMMARY OUTPUT Regressio Statistics Multiple R.987 R Square.974 Adjusted R Square.964 Stadard Error.5 Observatios 7 R Coefficiets Stadard Error t Stat Lower 95% Upper 95% Lower 9.% Upper 9.% Itercept (b) X Variable (b) X Variable (b) Curviliear Regressio Approach: plot a scatter plot. If it does ot look liear, try o-liear models: No-liear Liear y = a + b / x y = a + b(/ x) y = /( a + bx) (/ y) = a + bx y = x /( a + bx) ( x / y) = a + bx x y = a b l y = l a + x l b y = a + bx y = a + b( x ) 3 6