Review from Bootcamp: Linear Algebra

Transcription

1 Review from Bootcamp: Linear Algebra D. Alex Hughes October 27, 2014

2 1 Properties of Estimators 2 Linear Algebra Addition and Subtraction Transpose Multiplication Cross Product Trace 3 Special Matrices Matrix Inversion Determinants D. Alex Hughes Linear Algebra Review October 27, / 46

3 Properties of Estimators 1 Unbiasedness: 2 Asymptotic Unbiasedness: E(ˆθ) θ = 0 lim P( E[ˆθ] θ > ɛ) 0; ɛ > 0 n 3 Efficiency 4 Consistency 1 MSE = 1 E[(ˆθ θ) 2 ] lim n P( ˆθ θ > ɛ) 0 ɛ > 0 D. Alex Hughes Linear Algebra Review October 27, / 46

4 Linear algebra Motivation: Linear algebra or matrix algebra avoids the mess and lets us solve for things we care about quickly, cleanly and easily. This is no different than algebra. Consider the difference in the following formulas for the mean: x = x 1 + x 2 n x = x 1 + x 2 + x 3 n x = x 1 + x 2 + x 3 + x 4 n xi x = n D. Alex Hughes Linear Algebra Review October 27, / 46

5 Linear algebra Similarly, matrix algebra is a form of notation that cleans up the mess when working with more complex formulas. So suspend disbelief and concern, and treat this as a new language you are learning. Think of this as algebra on steroids. D. Alex Hughes Linear Algebra Review October 27, / 46

6 Motivation Why are we studying matrix algebra? We will use matrix algebra to derive the least squares estimator Matrices are an intuitive way to think about data. We have a set of observations (perhaps individuals) on the row, and observe many different characteristics (such as race, gender, PID, etc.) corresponding to columns Matrices are useful for solving systems of equations, like multiple regression Notation is much more compact and concise D. Alex Hughes Linear Algebra Review October 27, / 46

7 Definition of Matrices and Vectors Definition A matrix is simply an arrangement of numbers in rectangular form. Generally, a (j k) matrix A can be written as follows: a 11 a 12 a 1k a 21 a 22 a 2k A = a j1 a j2 a jk Note that there are j rows and k columns, defining the dimensionality (order) of the matrix. Also note that the elements are double sub-scripted, with the row number first, and the column number second. In general terms, the A above is of order (j, k). D. Alex Hughes Linear Algebra Review October 27, / 46

8 Examples Example W = [ is of order (2, 2). This is also called a square matrix. There are also rectangular matrices (j k), such as: ] Example which is of order (4,2). Γ = D. Alex Hughes Linear Algebra Review October 27, / 46

9 Notation Matrices are usually written using capital, bold-faced Roman or Greek letters. Roman is typically data, and Greek is typically parameters. This is not universal. D. Alex Hughes Linear Algebra Review October 27, / 46

10 Vector Definition Vectors are matrices that have either one row or one column. is the same as a scalar a regular number. Row vectors have a single row and multiple columns. α = [ ] α 1 α 2 α 3 α k Column vectors are those that have a single column and multiple rows. y 1 y 2 y =. y k D. Alex Hughes Linear Algebra Review October 27, / 46

11 Operations on Matrices Addition and Subtraction Scalar addition is simply: m + n = = 7 Addition is similarly defined for matrices. If matrices or vectors are of the same order, then they can be added. One performs the addition element by element. D. Alex Hughes Linear Algebra Review October 27, / 46

12 Addition A + B = C: [ a11 a 12 a 21 a 22 ] [ b11 b + 12 b 21 b 22 ] [ a11 + b = 11 a 12 + b 12 a 21 + b 21 a 22 + b 22 ] [ c11 c = 12 c 21 c 22 ] D. Alex Hughes Linear Algebra Review October 27, / 46

13 Subtraction A B = D: [ ] [ ] = [ ] D. Alex Hughes Linear Algebra Review October 27, / 46

14 Properties of Matrix Addition A + B = B + A. Matrix addition is commutative. (A + B) + C = A + (B + C). Matrix addition is associative. D. Alex Hughes Linear Algebra Review October 27, / 46

15 Transpose Definition To transpose a matrix is to exchange order subscripts. An order (j, k) matrix becomes an order (k, j) matrix. Transposition is denoted by placing a prime after a matrix or by placing a superscript T. Q = q 1,1 q 1,2 q 2,1 q 2,2 q 3,1 q 3,2 [ ] Q q1,1 q = 2,1 q 3,1 q 1,2 q 2,2 q 3,2 Note that the subscripts in the transpose remain the same, they are just exchanged. D. Alex Hughes Linear Algebra Review October 27, / 46

16 Example Example ω = ω = [ ] D. Alex Hughes Linear Algebra Review October 27, / 46

17 Some Definitions There are a few results regarding transposition that are important to remember: An order (j, j) matrix A is said to be symmetric iff A = A. (A ) = A W = For a scalar k, (ka) = ka. W = For two matrices of the same order, the transpose of the sum is equal to the sum of the transposes. (A + B) = A + B D. Alex Hughes Linear Algebra Review October 27, / 46

18 Matrices and Multiplication Scalar times a matrix. In words, a scalar α times a matrix A equals the scalar times each element of A. Thus, [ ] [ ] a1,1 a αa = α 1,2 αa1,1 αa = 1,2 a 2,1 a 2,2 αa 2,1 αa 2,2 So, for: A = [ ] [ A = ] D. Alex Hughes Linear Algebra Review October 27, / 46

19 Matrices and Multiplication Definition Given A of order (m, n) and B of order (n, r), then the product AB = C is the order (m, r) matrix whose entries are defined by: c i,j = n a i,k b k,j k=1 where i = 1,..., m and j = 1,..., r and k 1 = n 2 D. Alex Hughes Linear Algebra Review October 27, / 46

20 Matrices and Multiplication A = [ ] B = AB = = [ ( 2) ( 3) ( 2) ( 3) [ ] ] D. Alex Hughes Linear Algebra Review October 27, / 46

21 Matrices and Multiplication Is multiplication of matrices commutative? A = [ ] B = D. Alex Hughes Linear Algebra Review October 27, / 46

22 Matrices and Multiplication BA = No: Multiplication of matrices is not commutative. In other words: AB BA. D. Alex Hughes Linear Algebra Review October 27, / 46

23 Matrices and Multiplication Important results Matrix multiplication is not commutative: AB BA. Matrix multiplication is associative: (AB)C = A(BC) Matrix multiplication is distributive: A(B + C) = AB + AC The transpose of a product can be written as (AB) = B A D. Alex Hughes Linear Algebra Review October 27, / 46

24 Vectors and Multiplication Inner product of vectors e e = [ ] e 1 e 2 e N e 1 e 2. e N Alt: outer product e e = e 1 e 1 + e 2 e e N e N = N i=1 e 2 i D. Alex Hughes Linear Algebra Review October 27, / 46

25 Other Useful Vector Products Let i denote an order (N, 1) vector of ones, and x denote an order (N, 1) vector of data. i x = (x 1 + x 2 + x N ) = x i From this, it follows that: 1 N i x = 1 xi = x N D. Alex Hughes Linear Algebra Review October 27, / 46

26 Cross Product A B = ˆn A B cos(θ) ˆn: perpendicular unit vector A : Length of A θ: angle between A & B D. Alex Hughes Linear Algebra Review October 27, / 46

27 Trace Sum of the diagonal elements of a square matrix. A = a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn tr(a) = a ii = a 11 + a a nn D. Alex Hughes Linear Algebra Review October 27, / 46

28 Special Matrices and Their Properties When performing scalar algebra, we know that x 1 = x, which is known as the identity relationship. There is a similar relationship in matrix algebra: AI = A. What is I? It can be shown that I is a diagonal, square matrix with ones on the main diagonal, and zeros on the off diagonal. For example, the order three identity matrix is: I 3 = D. Alex Hughes Linear Algebra Review October 27, / 46

29 Special Matrices and Their Properties Notice that I is oftentimes subscripted to denote its dimensionality. Here is an example of the use of an identity matrix: [ ] [ ] [ ] = D. Alex Hughes Linear Algebra Review October 27, / 46

30 Special Matrices and Their Properties One of the nice properties of the identity matrix is that it is commutative with respect to multiplication. That is, AIB = IAB = ABI = AB An identity in scalar algebra is x + 0 = x. This generalizes to matrix algebra, with the definition of the null matrix, which is simply a matrix of zeros, denoted 0 j,k. Here is an example: A + 0 2,2 = [ ] [ ] = [ ] = A D. Alex Hughes Linear Algebra Review October 27, / 46

31 Matrix Inversion Definition: ZZ 1 = I This is roughly akin to division in non-matrix algebra. Actually calculating the inverse of a matrix takes several steps and has several prerequisites. D. Alex Hughes Linear Algebra Review October 27, / 46

32 Matrix Inversion General solution for a square matrix A: A 1 = 1 A adja So we need to figure out 1 A and adj(a). D. Alex Hughes Linear Algebra Review October 27, / 46

33 Matrix Inversion The first of these, A, is called the determinant. There s lots to learn about determinants, but we ll stick to the basics. Most importantly, if the determinant is NOT zero, a square matrix is invertible. The determinant is a scalar, that is, a single number, like 5. For a two by two matrix, the determinant is: [ ] a11 a A = 12 = a a 21 a 11 a 22 a 21 a So you multiply the corners and subtract one product from the other. D. Alex Hughes Linear Algebra Review October 27, / 46

34 Graphical Intuition for Determinant of a 2x2 Matrix D. Alex Hughes Linear Algebra Review October 27, / 46

35 D. Alex Hughes Linear Algebra Review October 27, / 46

36 D. Alex Hughes Linear Algebra Review October 27, / 46

37 Graphical Linear Algebra For a 2x2 matrix, the determinant is two times the area of the triangle defined by the row vectors. Think about this for a matrix like: [ ] 1 2 A = 1 2 Or... A = [ Most linear algebra functions can be represented graphically. Ask me for citations if you want a book that illustrates all these. ] D. Alex Hughes Linear Algebra Review October 27, / 46

38 Calculating Determinants With three by three matrices, the determinant is still quite manageable: a 11 a 12 a 13 A = a 21 a 22 a 23 a 31 a 32 a 33 = a 11 a 22 a 33 + a 12 a 23 a 31 + a 13 a 21 a 32 a 11 a 23 a 32 a 12 a 21 a 33 a 13 a 22 a 31 Graphically, this means adding the diagonal products from left to right, and subtracting the diagonal products from right to left. (example) D. Alex Hughes Linear Algebra Review October 27, / 46

39 More Determinants For bigger matrices, we have to use alternative methods, typically the Laplace Expansion. Basically, we break the matrix into sub-matricies, and calculate determinants of these submatricies, then combine our results. Steps: 1 Pick a row or column to work with. 2 For each element in that row, calculate the subdeterminant, also called the minor. 3 Multiply each element by its subdeterminant, determine signs, and add. D. Alex Hughes Linear Algebra Review October 27, / 46

40 Higher Order Determinants - Example = D. Alex Hughes Linear Algebra Review October 27, / 46

41 Properties of Determinants A = A Interchanging any two rows or columns will alter sign but not value of determinant. Multiplication of one row by k will change A to k A. Addition/subtraction of a multiple of any row to another row will leave the value of the determ unaltered (works for col too). If one row or columns is a multiple of another, the value of the determinant will be zero - matrix is singular D. Alex Hughes Linear Algebra Review October 27, / 46

42 Tricks for Determinants Pick a good row or column = = D. Alex Hughes Linear Algebra Review October 27, / 46

43 Tricks for Determinants Manipulate rows if possible = = D. Alex Hughes Linear Algebra Review October 27, / 46

44 Back to inverting a matrix A 1 = 1 A adj(a) (adj means adjugate) where C 11 C 12 C 1n C 21 C 22 C 2n adj(a) = C n1 C n2 C nn D. Alex Hughes Linear Algebra Review October 27, / 46

45 Cofactors C ij is a matrix cofactor - the determinant of the matrix when excluding row i and column j, and adj(a) is the transpose of the matrix of cofactors. The determinant of the cofactor submatrix is multiplied by 1 when i + j is even, and by -1 when i + j is odd. D. Alex Hughes Linear Algebra Review October 27, / 46

46 Easy Inversion - 2X2 A 1 = 1 A adj(a) [ ] 1 2 X = 1 3 X 1 = 1 [ ] D. Alex Hughes Linear Algebra Review October 27, / 46