Regression Analysis. Regression Analysis

Transcription

1 Regresson Analyss Smple Regresson Multvarate Regresson Stepwse Regresson Replcaton and Predcton Error 1 Regresson Analyss In general, we "ft" a model by mnmzng a metrc that represents the error. n mn (y - y ) 2 =1 The sum of squares gves closed form solutons and mnmum varance for lnear models. 2

2 The Smplest Regresson Model Lne through the orgn: y y=bx x y u =βx u +ε u u=1,2,...,n ε u ~N(0, σ R 2 ) n mn S = mn (y u - βx u ) 2 : estmate of 2 σ R u=1 y=bx η u =βx u b: estmate of β y: estmate of η u, the true value of the model. 3 Usng the Normal Equaton mn (y-y) 2 y2 y y=bx (1 d.f.) y1 4

3 Usng the Normal Equaton (cont) Choose b so that the resdual vector s perpendcular to the model vector... (y-y) x =0 (y - bx) x = 0 b= xy x (est. of β) 2 s2 = S R n-1 (est. of σ R 2 ) V(b) = s2 67% conf: b ± s 2 x 2 x 2 Sgnf. test: t= b-β* s 2 x 2 ~ t n-1 5 Etch tme vs removed materal: y = bx 500 R e m o v ed ( n m ) Data Fle: regresson Varable Name Etch Tme (sec) x 10^3 Coeffcent Dependent Varable : Removed (nm) Std. Err. Estmate t Statstc Prob > t Etch Tme (sec) e e e e-8 6

4 Model Valdaton through ANOVA The dea s to decompose the sum of squares nto orthogonal components. Assumng that there s no dependence: H 0 : β * =0 y2 u = y2 u + (y u - y u ) 2 n p n-p total model resdual 7 Model Valdaton through ANOVA (cont) Assumng a specfc model: H 0 : β * = b (y u - β * x u ) 2 = (y u - β * x u ) 2 + (y u - y u ) 2 n p n-p total model resdual The ANOVA table wll answer the queston: Is Is there a relatonshp between x and y? y? 8

5 Data Fle: ANOVA table and Resdual Plot regresson Source Sum of Squares Deg. of Freedom Mean Squares F-Rato Prob>F Model Error e e e e e e-6 Total R es d u a l s e Coeffcent 40 of Determnaton Coeffcent 20 of Correlaton Standard Error of Estmate 0 Durbn-Watson Statstc e e e e Etch Tme (sec) x 10^ A More Complex Regresson Equaton actual estmated η = α + β (x - x ) y = a + b (x - x ) y ~ N (η, σ 2 ) Mnmze R = (y -y ) 2 to estmate α and β a=y b= (x -x)y (x -x) 2 =(x -x)(y -y) (x -x) 2 Are a and b good estmators of α and β? E[a] = α E[b] = (x -x)e[y ] (x -x) 2 = β 10

6 Varance Estmaton: Note that all varablty comes from y! V[a] = V V[b] = V y = 1 2 V[ y ] = σ 2 (x -x)y (x -x) 2 = σ 2 (x -x) 2 mn var. thans to to least squares! 11 LTO thcness vs deposton tme: y = a + bx L T O t h c A x ^ Dep tme x 10^3 Data Fle: regresson Dependent Varable: LTO thc A Varable Name Coeffcent Std. Err. Estmate t Statstc Prob > t Constant Dep tme e e e e e e e e-17 12

7 Data Fle: Source regresson Anova table and Resdual Plot Sum of Squares Deg. of Freedom Mean Squares F-Rato Prob>F Model Error e e e e e e-17 Total e+6 17 Coeffcent 100 of Determnaton R Coeffcent of Correlaton es Standard Error of Estmate Durbn-Watson 0 Statstc d u a l s e e e e Dep tme x 10^3 13 ANOVA Representaton (x,y ) (y -y ) y (y -η ) b(x -x) (y -η ) (a-α) y = a+b(x -x) η = α+β(x -x) β(x -x) x x x Note dfferences between "true" and "estmated" model. 14

8 ANOVA Representaton (cont) (y -η ) = (a- α ) + (b- β )(x -x) + ( y - y ) (y -η ) 2 = (a-α ) 2 + (b-β) 2 (x -x)+ () (1) (1) ~σ 2 χ 2 () ~σ 2 χ 2 (1) ~ σ 2 χ 2 (1) (y -y ) 2 (-2) ~σ 2 χ 2 (-2) In In ths way, the sgnfcance of of the model can be be analyzed n n detal. 15 Confdence Lmts of an Estmate y0= y+b(x0 -x ) V(y0) = V(y)+(x0 -x ) 2 V(b) V(y0) = 1 n (x0 -x )2 + (x -x ) 2 s2 predcton nterval: y 0 +/- tα 2 V(y 0 ) 16

9 L T O Confdence Interval of Predcton (all ponts) p 3000 T h c n e s s Dep tme Leverage 17 Confdence Interval of Predcton (half the ponts) L T O T h c n e s s Dep tme Leverage 18

10 Confdence Interval of Predcton (1/4 of ponts) L T O T h c n e s s Dep tme Leverage 19 Predcton Error vs Expermental Error y Expermental Error Predcton error Estmated Model True model x Expermental Error Error Does Does not not depend on on locaton or or sample sample sze. sze. Predcton Error Error depends on on locaton gets gets smaller smaller as as sample sample sze sze ncreases. 20

11 Multvarate Regresson η = β 1 x 1 +β 2 x 2 β 2 y y x 2 R The Resdual s s to to y,, x 1,, x 2.. β 1 x 1 Coeffcent Estmaton: (y-y)x 1 =0 (y-y)x 2 =0 yx 1 -b 1 x 1 2 -b 2 x 1 x 2 = 0 yx 2 -b 2 x 2 2 -b 1 x 1 x 2 = 0 21 Varance Estmaton: s 2 = S R n-p V(b 1 ) = 1 s 2 1-ρ 2 x2 1 V(b 2 ) = 1 1-ρ 2 s 2 x 2 2 ρ = -x 1x 2 x 12 x

12 Thcness vs tme, temp: y = a + b1 x1 + b2 x2 Data Fle: regresson Varable Name Coeffcent Dependent Varable : tox nm Std. Err. Estmate t Statstc Prob > t Constant temp tme mn e e e e e e e e e e e e-9 23 Data Fle: Anova table and Correlaton of Estmates regresson Source Sum of Squares Deg. of Freedom Mean Squares F-Rato Prob>F Model Error e e e e e e-14 Total e+4 20 Coeffcent of Determnaton e-1 Coeffcent of Correlaton e-1 Standard Error of Estmate e+0 Data Fle: regresson Durbn-Watson Statstc Tox Temp e-1 Tme tox nm temp tme mn

13 Multple Regresson n General x 1 x 2 x n b = y + e mnmze Xb - y 2 = e 2 = ( y - Xb ) T ( y - Xb ) or, mn -e T Xb + e T y whch s equv. to: ( y - Xb ) T Xb = 0 X T Xb = X T y b = ( X T X ) -1 X T y V(b) = ( X T X ) -1 σ 2 25 Jont Confdence Regon for x 1 x 2 S = S R 1 + p n-p F α(p, n-p) 2 β 1 -b 1 x β 1 -b 1 β 2 -b 2 x 1 x 2 + β 2 -b 2 x 2 2= S-S R 26

14 What f a lnear model s not enough? 300 d e p r a t e nlet temp Data Fle: Varable Name regresson Coeffcent Dependent Varable: dep rate Std. Err. Estmate t Statstc Prob > t Constant nlet temp e e e e e e e e-9 27 Data Fle: ANOVA table and Resdual Plot regresson Source Sum of Squares Deg. of Freedom Mean Squares F-Rato Prob>F Model Error e e e e e e+0 Total e Coeffcent of Determnaton Coeffcent of Correlaton R 10 Standard es Error of Estmate Durbn-Watson Statstc 0 d u a -10 l s e e e e nlet temp 28

15 Multple Regresson wth Replcaton S E = 1 2 (y 1 -y 2 ) 2 S LF =S R -S E (a-α) 2 η v n (y v -η ) 2 = η + (b-β) 2 η (x -x) 2 + η (y. -y ) 2 + (y v -y. ) 2 n η - v v n v n (y v -y) 2 = (y v -y. ) 2 + η (y. -y ) 2 + η (y-y ) 2 29 Pure Error vs. Lac of Ft Example Lac Of Ft Source Lac Of Ft Pure Error Total Error DF Sum of Squares Mean Square F Rato Prob > F Parameter Estmates Term Intercept nlet temp Estmate Std Error t Rato Prob> t Effect Test Source nlet temp Nparm 1 DF 1 Sum of Squares F Rato Prob > F

16 Dep. rate vs temperature: y = a + bx + cx d e p r a t e Data Fle: regresson nlet Dependent temp Varable : dep rate Varable Name Coeffcent Std. Err. Estmate t Statstc Prob > t Constant nlet temp nlet temp ^ e e e e e e e e e e e e-5 31 Pure Error vs. Lac of Ft Example (cont) Lac Of Ft Source Lac Of Ft Pure Error Total Error DF Sum of Squares Mean Square F Rato Prob > F Parameter Estmates Term Intercept nlet temp^1 nlet temp^2 Estmate Std Error t Rato Prob> t Effect Test Source Poly(nlet temp,2) Nparm 2 DF 2 Sum of Squares F Rato Prob > F

17 Data Fle: Source ANOVA table and Resdual Plot regresson Sum of Squares Deg. of Freedom Mean Squares F-Rato Prob>F Model Error e e e e e e+0 Total e+4 22 Coeffcent 6 of Determnaton e-1 Coeffcent 4 of Correlaton e-1 RStandard Error of Estmate e+0 es 2 Durbn-Watson Statstc e+0 0 d u -2 a l s nlet temp 33 Use regresson lne to predct LTO thcness... y = x R 2 = y = x R 2 = LTO Thc A 90%LmtLow 90%LmtHgh Dep Tme Sec LTO Thc A

18 Response Surface Methodology Objectves: get a feel of I/O relatonshps fnd settng(s) that satsfy multple constrants fnd settngs that lead to optmum performance Observatons: Functon s nearly lnear away from the pea Functon s nearly quadratc at the pea 35 Buldng the planar model A Factoral experment wth center ponts s enough to buld and confrm a planar model. b1, b2, b12 = /-0.75 b11+b22=1/4p+1/3c= /

19 Quadratc Model and Confrmaton Run Close to the pea, a quadratc model can be bult and confrmed by an expanded two-phase experment. 37 Response Surface Methodology RSM conssts of creatng models that lead to vsual mages of a response. The models are usually lnear or quadratc n nature. Ether expanded factoral experments, or regresson analyss can be used. All emprcal models have a random predcton error. In RSM, the average varance of the model s: V(y) = 1 n n =1 V(y ) = pσ2 n where p s the number of model parameters and n s the number of experments. 38

20 Response Surface Exploraton 39 "Popular" RSM Use snge-stage Box-B or Box-W desgns Use computer (smulated) experments Rely on "goodness of ft" measures Automate model structure generaton Problems? 40