Regression Analysis. Simple Regression Multivariate Regression Stepwise Regression Replication and Prediction Error EE290H F05

Transcription

1 Regression Analysis Simple Regression Multivariate Regression Stepwise Regression Replication and Prediction Error 1

2 Regression Analysis In general, we "fit" a model by minimizing a metric that represents the error. n Σ min (y i - y i ) 2 i=1 The sum of squares gives closed form solutions and minimum variance for linear models. 2

3 The Simplest Regression Model Line through the origin: y y=bx x y u =βx u +ε u u=1,2,...,n ε u ~N(0, σ R 2 ) n min S = min Σ (y u - βx u ) 2 : estimate of 2 σ R u=1 y=bx η u =βx u b: estimate of β y: estimate of η u, the true value of the model. 3

4 Using the Normal Equation to fit line through the origin model Our model only has one degree of freedom This is why our choices are confined on this line min Σ (y-y) 2 y2 y y=bx (1 d.f.) y1 4

5 Using the Normal Equation (cont) (fitting line through the origin model) Choose b so that the residual vector is perpendicular to the model vector... Σ (y-y) x =0 Σ (y - bx) x = 0 b= Σ xy Σx 2 (est. of β) s 2 = S R n-1 (est. of σ R 2 ) V(b) = s2 Σx 2 67% conf: b ± s 2 Σx 2 Significance test: t = b-β* s 2 Σx 2 ~ t n-1 5

6 Etch time vs. removed material: y = bx 500 R e m o v ed ( n m ) Etch Time (sec) x 10^3 Variable Name Coefficient Std. Err. Estimate t Statistic Prob > t Etch Time (sec)

7 Model Validation through ANOVA The idea is to decompose the sum of squares into orthogonal components. Assuming that there is no need for a model at all* (always a good null Hypothesis!): H 0 : β * =0 Σ y u 2 = Σ y u 2 + Σ (y u - y u ) 2 n p n-p total model residual * This is equivalent to saying that y~n(μ,σ 2 ), where μ and σ are constants, independent of x. 7

8 Model Validation through ANOVA (cont) Assuming a specific model: H 0 : β * = b Σ(y u - β * x u ) 2 = Σ (y u - β * x u ) 2 + Σ (y u - y u ) 2 n p n-p total model residual The ANOVA table will answer the question: Is Is there a relationship between x and y? y? 8

9 ANOVA table and Residual Plot Source Sum of Squares Deg. of Freedom Mean Squares F-Ratio Prob>F Model 1.83e e e e-6 Error 6.47e e+2 Total 1.89e+5 8 R es i d u a l s Etch Time (sec) x 10^

10 A More Complex Regression Equation - a straight line with two parameters actual estimated η = α + β (x - x ) y = a + b (x - x ) y i ~ N (η i, σ 2 ) Minimize R = Σ(y i -y i ) 2 to estimate α and β a=y b= Σ(x i -x)y i Σ(x i -x) 2 =Σ(x i -x)(y i-y) Σ(x i -x) 2 Are a and b good estimators of α and β? E[a] = α E[b] = Σ(x i -x)e[y i] Σ(x i -x) 2 = β 10

11 Variance Estimation: Note that all variability comes from y i! V[a] = V V[b] = V Σ y i k = 1 k 2 Σ V[ y i] = σ 2 k Σ (x i -x)y i Σ (x i -x) 2 = σ 2 Σ (x i -x) 2 min var. thanks to to least squares! 11

12 LTO thickness vs deposition time: y = a + bx L T O t h i c k A x 1 0^ Dep time x 10^3 Variable Name Coefficient Std. Err. Estimate t Statistic Prob > t Constant Dep time 6.04e e e e e e

13 Source Anova table and Residual Plot Sum of Squares Deg. of Freedom Mean Squares F-Ratio Prob>F Model 4.77e e e Error 5.09e e+3 Total 4.82e R es i d u a l s Dep time x 10^3 13

14 ANOVA Representation (x i,y i ) (y i -y i ) y (y i -η i ) b(x i -x) (y i -η i ) (a-α) y i = a+b(x i -x) η i = α+β(x i -x) β(x i -x) x x i x Note differences between "true" and "estimated" model. 14

15 ANOVA Representation (cont) (y i -η i ) = (a- α ) + (b- β )(x i -x) + ( y i - y i ) Σ(y i -η i ) 2 = k(a-α ) 2 + (b-β) 2 Σ(x i -x)+ (k) (1) (1) ~σ 2 χ 2 (k) ~σ 2 χ 2 (1) ~ σ 2 χ 2 (1) Σ(y i -y i ) 2 (k-2) ~σ 2 χ 2 (k-2) In In this way, the significance of of the model can be be analyzed in in detail. 15

16 Confidence Limits of an Estimate y0= y+b(x0 -x ) V(y0) = V(y)+(x0 -x ) 2 V(b) V(y0) = 1 n + (x 0 -x ) 2 Σ (x -x ) 2 s2 prediction interval: y0 +/- tα 2 V(y0) 16

17 Confidence Interval of Prediction (all points) p L T O T h i c k n e s s Dep time Leverage 17

18 Confidence Interval of Prediction (half the points) L T O 3000 T h i c k n e s s Dep time Leverage 18

19 Confidence Interval of Prediction (1/4 of points) L T O T h i c k n e s s Dep time Leverage 19

20 Prediction Error vs Experimental Error Experimental Error y Prediction error Estimated Model True model x Experimental Error Error Does Does not not depend on on location or or sample sample size. size. Prediction Error Error depends on on location gets gets smaller smaller as as sample sample size size increases. 20

21 Multivariate Regression η = β 1 x 1 +β 2 x 2 β 2 y y x 2 R The Residual is is to to y,, x 1,, x 2.. β 1 x 1 Coefficient Estimation: Σ(y-y)x 1 =0 Σ(y-y)x 2 =0 Σyx 1 -b 1 Σx 1 2 -b 2 Σx 1 x 2 = 0 Σyx 2 -b 2 Σx 2 2 -b 1 Σx 1 x 2 = 0 21

22 Variance Estimation: s 2 = S R n-p V(b 1 ) = 1 1-ρ 2 s 2 Σx 1 2 V(b 2 ) = 1 1-ρ 2 s 2 Σx 2 2 ρ = - Σx 1 x 2 Σx 12 Σx

23 Thickness vs time, temp: y = a + b1 x1 + b2 x2 Variable Name Coefficient Std. Err. Estimate t Statistic Prob > t Constant temp time min -7.04e e e e e e e e e

24 Anova table and Correlation of Estimates Source Sum of Squares Deg. of Freedom Mean Squares F-Ratio Prob>F Model 2.58e e e Error 7.71e e+1 Total 2.66e+4 20 Data File: tox nm temp time min regression Tox Temp Time

25 Multiple Regression in General x 1 x 2 x n b = y + e minimize Xb - y 2 = e 2 = ( y - Xb ) T ( y - Xb ) or, min -e T Xb + e T y which is equiv. to: ( y - Xb ) T Xb = 0 X T Xb = X T y b = ( X T X ) -1 X T y V(b) = ( X T X ) -1 σ 2 25

26 Joint Confidence Region for x 1 x 2 S = S R 1 + p n-p F α(p, n-p) Σ 2 β 1 -b 1 Σ x β 1 -b 1 β 2 -b 2 Σ 2 x 1 x 2 + β 2 -b 2 Σ x 2 2= S-S R 26

27 What if a linear model is not enough? 300 d e p r a t e inlet temp Variable Name Coefficient Std. Err. Estimate t Statistic Prob > t Constant inlet temp -1.85e e e e e e

28 ANOVA table and Residual Plot Source Sum of Squares Deg. of Freedom Mean Squares F-Ratio Prob>F Model 3.65e e e Error 4.06e e+1 Total 3.69e R es i d u a l s inlet temp 28

29 Multiple Regression with Replication S E = 1 2 Σ (y i1 -y i2 ) 2 S LF =S R -S E (a-α) 2 k Σ i η i k k i Σ v n i (y iv -ηi) 2 k Σ i η i + (b-β) 2 Σ η i (x i -x) 2 + η i (y i. -y i ) 2 + (y iv -y i. ) 2 i Σ k Σ i = k n i Σ Σ 1 1 k-2 η i -k i v k Σ i k Σ i Σ v n i (y iv -y) 2 k Σ Σ v n i = (y iv -y i. ) 2 i k + Σ η i i k (y i. -y i ) 2 + Σ η i i (y-y i ) 2 29

30 Pure Error vs. Lack of Fit Example Lack Of Fit Source Lack Of Fit Pure Error Total Error DF Sum of Squares Mean Square F Ratio Prob > F Parameter Estimates Term Intercept inlet temp Estimate Std Error t Ratio Prob> t Model Test Source inlet temp DF 1 Sum of Squares F Ratio Prob > F

31 Dep. rate vs temperature: y = a + bx + cx d e p r a t e 200 Variable Name inlet temp Std. Err. t Coefficient Estimate Statistic Prob > t Constant inlet temp inlet temp ^2 8.34e e e e e e e e e

32 Pure Error vs. Lack of Fit Example (cont) Lack Of Fit Source Lack Of Fit Pure Error Total Error DF Sum of Squares Mean Square F Ratio 8.37 Prob > F Parameter Estimates Term Intercept inlet temp^1 inlet temp^2 Estimate Std Error t Ratio Prob> t Model Test Source Poly(inlet temp,2) DF 2 Sum of Squares F Ratio Prob > F

33 Source ANOVA table and Residual Plot Sum of Squares Deg. of Freedom Mean Squares F-Ratio Prob>F Model 3.67e e e Error 1.55e e+0 Total 3.69e+4 22 R es i d u a l s inlet temp 33

34 Use regression line to predict LTO thickness... y = x R 2 = y = x R 2 = LTO Thick A 90%LimitLow 90%LimitHigh Dep Time Sec LTO Thick A

35 Response Surface Methodology Objectives: get a feel of I/O relationships find setting(s) that satisfy multiple constraints find settings that lead to optimum performance Observations: Function is nearly linear away from the peak Function is nearly quadratic at the peak 35

36 Building the planar model A Factorial experiment with center points is enough to build and confirm a planar model. b1, b2, b12 = /-0.75 b11+b22=1/4σp+1/3σc= /

37 Quadratic Model and Confirmation Run Close to the peak, a quadratic model can be built and confirmed by an expanded two-phase experiment. 37

38 Response Surface Methodology RSM consists of creating models that lead to visual images of a response. The models are usually linear or quadratic in nature. Either expanded factorial experiments, or regression analysis can be used. All empirical models have a random prediction error. In RSM, the average variance of the model is: V(y) = 1 n n Σ i=1 V(y i ) = pσ2 n where p is the number of model parameters and n is the number of experiments. 38

39 Response Surface Exploration 39

40 "Popular" RSM Use singe-stage Box-B or Box-W designs Use computer (simulated) experiments Rely on "goodness of fit" measures Automate model structure generation Problems? 40