Statistical Techniques II

Transcription

1 Statistical Techniques II EST705 Regression with atrix Algebra 06a_atrix SLR

2 atrix Algebra We will not be doing our regressions with matrix algebra, except that the computer does employ matrices. In fact, there is really no other way to do the basic calculations. ou will be responsible for knowing about matrices only to the extent that ROC REG or ROC GL produces information. This is primarily the initial and final matrices. 06a_atrix SLR 2

3 So, what is a matrix? A matrix is a rectangular arrangement of numbers, usually represented by an upper case letter (A, B, C, D, etc.) A = D = L N O Q L N O Q 06a_atrix SLR 3

4 The dimensions of a matrix are given by the number of rows and columns in the matrix (i.e. the dimensions are r by c). For the matrices above, A is 2 by 2 D is 4 by 3 06a_atrix SLR 4

5 For a simple linear regression the matrices of initial interest would be the data matrices, a matrix of values of the dependent variable and an matrix of values of the independent variable. The matrix also has a column of ones added to fit the intercept. 06a_atrix SLR 5

6 = L N O Q = L N O Q 06a_atrix SLR 6

7 As with our algebraic calculations we need some intermediate values; sums, sums of squares and cross-products. These are obtained by calculating First, a transpose matrix for both and. This is simply the matrix turned on its side so the rows of the original matrix become the columns of the transpose. These are denoted ' and '. 06a_atrix SLR 7

8 L = N O Q = a_atrix SLR 8

9 We now calculate 3 matrices, ', ' and '. This requires matrix multiplication. = L N O Q L N O Q a_atrix SLR 9

10 Calculate ', ' and ' (continued). = L N O Q a_atrix SLR 0

11 Calculate ', ' and ' (continued). = L N O Q L N O Q a_atrix SLR

12 n i i i 2 i n i n i n = = = L N O Q i i i i n i n = = L N O Q i 2 i n = L N O Q The results of these 3 calculations are; ' = ' = ' = 06a_atrix SLR 2

13 Notice that the contents of these 3 matrices are the same as the values we used for the algebraic solution. Normal equations - when the equations needed to solve a simple linear regression are derived, the result is two equations with two unknowns that must be resolved. These are called the normal equations. 06a_atrix SLR 3

14 The normal equations are b 0 n + b Σ i = Σ i b 0 Σ i + b Σ 2 i = Σ i i If you solve these algebraically, you get the two equations we use to solve for b 0 and b. 06a_atrix SLR 4

15 When expressed as matrices this factors out to n n i b i i= 0 i= = n n b n 2 i i i i i= i= i= n 06a_atrix SLR 5

16 In simple matrix notation, a B matrix (vector) times the ' equals the '. (')B=' As with the algebraic equations we need to solve for B (i.e. b 0 and b ). If we do this with algebra, we get the usual equations. Solving the matrix equations we get, B=(') - ' 06a_atrix SLR 6

17 This equation is the matrix algebra solution for a simple linear regression. B=(') - ' Note that there is not such thing as matrix "division". As with the algebraic values, if we multiply the B values (B matrix) by the values we get the predicted values. B=(') - ' = hat vector 06a_atrix SLR 7

18 What do we need to know about these matrix calculations? We need to know that the solution to the problem using matrix algebra involves the same values as for the simple linear regression. We need to know that the (') - is a key component to this solution. We need to know that the predicted values require the matrix segment (') - ' times the vector (AIN DIAGONAL). 06a_atrix SLR 8

19 Why? We can get the matrices from SAS, but we want to understand what we have. (') - is a key component not only of the solution for the regression coefficients, but also for the variance-covariance matrix. The (') - ' matrix main diagonal is a diagnostic that we will use (hat diag). 06a_atrix SLR 9

20 But the most important reason for using matrices is that the solution for simple linear and multiple regression are the same. Basically, matrix algebra is the ONL way to solve multiple regressions. So, what do we get from SAS? If the options and I are placed on the model statement, we can get the ' matrix and the (') - matrix. 06a_atrix SLR 20

21 For the simple linear regression that we saw for the tree weights and diameters, these options produce the following output. odel Cross-products ' ' ' ' INTERCE DBH WEIGHT INTERCE DBH WEIGHT ' Inverse, arameter Estimates, and SSE INTERCE DBH WEIGHT INTERCE DBH WEIGHT a_atrix SLR 2

22 The first two rows and columns of numbers contain the ' matrix, which has the values for n, Σ i and Σ 2 i odel Cross-products ' ' ' ' INTERCE DBH INTERCE DBH a_atrix SLR 22

23 The last column has ' (values for Σ i and Σ i i ) and the last value is ' (Σ 2 i). odel Cross-products ' ' ' ' WEIGHT INTERCE 7359 DBH WEIGHT a_atrix SLR 23

24 In the ' inverse matrix section, the first two rows and columns of numbers contain the (') - matrix and the value in the third row and third column is the SSE. The other values are b 0 and b. ' Inverse, arameter Estimates, and SSE INTERCE DBH WEIGHT INTERCE DBH WEIGHT a_atrix SLR 24

25 ou will be responsible only for knowing where the 6 intermediate values are for simple linear regression, and where to find the (') - matrix. odel Cross-products ' ' ' ' INTERCE DBH WEIGHT INTERCE DBH WEIGHT ' Inverse, arameter Estimates, and SSE INTERCE DBH WEIGHT INTERCE DBH WEIGHT a_atrix SLR 25

26 ultiple Regression The only difference between simple linear regression and multiple regression is the fact that multiple regression has several independent variables ( i variables). There for the matrix ' will be larger. For a simple linear regression, ' is 2x2. For a 3 factor multiple regression (, 2, 3 and an intercept) the ' matrix will be 4x4. 06a_atrix SLR 26

27 = L N O Q = L N O Q 06a_atrix SLR 27

28 ultiple Regression (continued) The values contained in the matrix include all of the sums, sums of squares and cross-products for all of the i variables (plus n in the upper left hand corner). 06a_atrix SLR 28

29 ultiple Regression (continued) ' = n Σ i Σ 2i Σ 3i Σ i Σ 2 i Σ i 2i Σ i 3i Σ 2i Σ i 2i Σ 2 2i Σ 2i 3i Σ 3i Σ i 3i Σ 2i 3i Σ 2 3i 06a_atrix SLR 29

30 ultiple Regression (continued) ' = Σ i Σ i i Σ 2i i Σ 3i i 06a_atrix SLR 30

31 For the multiple regression the solution is still given by; B=(') - ' The predicted values are still given by; B=(') - ' = hat vector The residuals are given by; hat= i - (') - ' The (') - ' matrix main diagonal is a diagnostic that we will use (hat diag). 06a_atrix SLR 3

32 So basically everything works the same in multiple regression as in simple linear regression if we use matrix algebra. One last piece of the puzzle. We will need some estimates of variances and covariances. As usual these will all involve the SE (ean Square Error). We need all of the variances and covariances for the regression coefficients 06a_atrix SLR 32

33 These variances and covariances are obtained by multiplying the (') - matrix by the SE. The resulting matrix contains all of the variances and covariances for the regression coefficients. The variances of regression coefficients are on the main diagonal. The square root of these values gives the standard error used for confidence intervals and testing of the b i values. 06a_atrix SLR 33

34 ultiple Regression (continued) The (') - matrix can be obtained from SAS. When multiplied by the SE value this gives the Variance-Covariance matrix. This can also be obtained from SAS. 06a_atrix SLR 34

35 ultiple Regression (continued) A note on the assumption of independence. We assume that the e i values are independent of each other. We assume that the e i are independent of the hat values (b 0 +b i ). But WE DO NOT ASSUE THAT THE VARIOUS REGRESSION COEFFICIENTS ARE INDEENDENT OF EACH OTHER. Calculations do not assume zero covariances, they employ the Variance- Covariance matrix. 06a_atrix SLR 35

36 Summary atrix solutions to regression begin with the matrices ', ' and '. The values in these matrices are the sums, sums of squares and cross-products, just as with simple linear regression. The normal equations are solved just as for SLR. The solution is B=(') - '. Using matrix algebra we can obtain predicted values and residuals, diagnostics, variances and covariances and all of the other values needed for testing and interpretation of the multiple regression. 06a_atrix SLR 36