Lecture # 37. Prof. John W. Sutherland. Nov. 28, 2005

Transcription

1 Lecture # 37 Prof. John W. Sutherland Nov. 28, 2005

2 Linear Regression 8 y x

3 Modeling To describe the data above, propose the model: y = B 0 + B 1 x + ε Fitted model will then be ŷ = b 0 + b 1 x Want to select values for n = 6 ( y i ŷ i ) 2 i = 1 b 0 & b 1 that minimize

4 Modeling (Cont.) Define n = 6 Sb ( 0, b 1 ) = ( y i ŷ i ) 2 i = 1 the model residual Sum of Squares. Minimize n = 6 Sb ( 0, b 1 ) ( y i ŷ i ) 2 n = 6 = = ( y i b 0 b 1 x 1 ) 2 i = 1 i = 1

5 Modeling (Cont.) To find minimum S, take partial derivatives of S with respect to solve for b 0 & b 1 b 0 & b 1, set these equal to zero, and Sb0 (, b b 1 ) = 2Σ( y i b 0 b 1 x i )( 1) = 0 0 Sb0 (, b b 1 ) = 2Σ( y i b 0 b 1 x i )( x i ) = 0 1

6 Σy i + Σb o + Σb 1 x i = 0 Modeling (Cont.) 2 Σx i y i + Σb o x i + Σb 1 x i = 0 Simplifying, we obtain: nb 0 + b 1 Σx i = Σy i 2 b 0 Σx i + b 1 Σx i = Σx i y i

7 Modeling (Cont.) These two equations are known as Normal Equations. The values of b 0 & b 1 that satisfy the Normal Equations are the least squares estimates -- they are the values that give a minimum S.

8 Matrix Form N Σx i 2 Σx i Σx i b 0 Σ y = i b 1 Σ x i y i * b 0 * b 1 = Least Squares Estimates = n Σx Σx Σx 2 1 Σ y Σ xy

9 Matrix Form (Cont.) * b 0 * b 1 = nσx 2 ( Σx) 2 Σx 2 Σx Σx n Σ y Σ xy = nσx 2 ( Σx) 2 Σx 2 Σy ΣxΣxy ΣxΣy + nσxy

10 Matrix Form (Cont.) * b 0 * b 1 = * * b 0, b1 are the values of b 0 & b 1 that minimize S, the Residual Sum of Squares. * b 0 = Bˆ 0 = an estimate of B 0 * b 1 = Bˆ 1 = an estimate of B 1

11 Fitted Line Y X

12 Matrix Approach x ỹ : Vector of Observations =, : Matrix of Independent Variables, i.e., the Design Matrix =

13 Matrix Approach (Cont) ŷ 1 ŷ ŷ 2 : ŷ n = Vector of Predictions = = x b b coefficients = b 0 b 1

14 Matrix Approach (Cont.) e 1 ẽ = Vector of Prediction Errors = e 2 : e n ẽ = ỹ ŷ Want to Min ẽ T ẽ or Min ( ỹ ) x b T ( ỹ ) x b

15 Matrix Approach (Cont.) Take derivative with respect to b s and set = 0 x T T ( ỹ ) = 0 = ỹ + ( x b x x T x )b ( x T x )b = x T ỹ

16 Matrix Approach (Cont.) Therefore, = (. b x T ) x 1 x T ỹ It is analogous to b 0 = n Σx b 1 Σx Σx 2 1 Σy Σxy

17 Matrix Approach (Cont.) Re-run experiments several times b =, b b =, b b = b If true model is y = B 0 +B 1 x + ε Then E(b 0 ) = B 0, E(b 1 )=B 1, E[ ] = b B

18 Matrix Approach (Cont.) B 0 b 0 T Var( ) ( ) b x x 1 2 = σ y where, σ y 2 the y s ( ). σ ε 2 describes the experimental error variation in

19 Matrix Approach (Cont.) Var( b For our example, Var( ) 0 ) Cov( b 0, b 1 ) =. b Cov( b 0, b 1 ) Var( b 1 ) 2 2 If σ y ( or σ ε ) is unknown, we can estimate it with s ( y ŷ)t ( y ŷ) e T e = = = ( n - # of parameters) n p S res ν

20 Matrix Approach (Cont.) For the example, n = 6 ˆ ( y i y i ) 2 2 i = 1 s y = = n T Var( ) ( ) b x 1 2 = s x y = ) 2 s b0 2 s b1 = 0.586, s b0 = = 0.039, s b1 = standard error of b 0 standard error of b 1

21 Dynamic Systems Many processes have dynamic characteristics -- data are produced as a result of dynamic behavior within the process - Chemical processes - Vibrating systems -? Process Data, X s

22 More on Dynamic Systems Because of our experience with differential equations and vibrations, we tend to think of dynamic behavior as in the figure below. response tim e

23 Common Cause Variability With the addition of process noise, however, we often see behavior like that below. response tim e

24 Time Series Analysis For situations like that shown in the previous figure, we can use time series analysis to extract information about the process. From a time series model we can back out information about the unknown underlying system dynamics. Simple autoregressive model [ AR(1) ] X i = φx i 1 + a i

25 Interpreting Phi data (X) φ = sample number X(i) X(i-1)

26 Interpreting Phi data (X) sample number φ = X(i) X(i-1)

27 Interpreting Phi φ = 1.02 X(i) Sample # 100 Behavior is unstable because φ>1 X(i) X(i-1)

28 Finding φ Let s say that we have a series of data such as that below X(i) Sample # X(i) X(i-1)

29 Finding φ Let s apply this model to data: X i = φx i 1 + a i Same form as y i = β 1 x i + ε i We now know how to estimate the value for β 1 (called estimate b 1 ) n S ( X i Xˆ i) 2 n = = ( X i φx i 1 ) 2 i = 1 i = 1 ds = dφ 2Σ ( X i φx i 1 )( 1) = 0

30 φˆ = n X i X i 1 X i 1 i = 2 n i = 2 Finding φ 2 For data shown previously, n X i X i i = 2 n 2 = X i 1 i = 2 = φˆ = So, Xˆ i = φˆ X i 1 -- also can be thought of as a forecast

31 Define backshift operator as X t 1 = BX t In general, X t j B j = Xt Backshift Operator Previous equation: X i = φx i 1 + a i, can be rewritten as X i = φx i 1 + a i X i = φbx i + a i X i ( 1 φb) = a i X i = a i ( 1 φb) -- denominator is characteristic eqn

32 More on AR(1) X i = φx i 1 + a i, or since X i 1 = + φx i 2 a i 1 X i = φφx ( i 2 + a i 1 ) + a i or X i = φ 2 X i 2 + φa i 1 + a i or X i = k j = 0 φ j a i j Note similar form to EWMA

33 AR(2) Model AR(2) model: X t = φ 1 X t 1 + φ 2 X t 1 a + t or X t φ 1 X t 1 φ 2 X t 2 = a t a t ~NID 2 0, σ a