Topic 7 - Matrix Approach to Simple Linear Regression Review of Matrices Outline Regression model in matrix form - Fall 03 Calculations using matrices Topic 7 Matrix Collection of elements arranged in rows and columns Elements will be numbers or symbols For example: A = [ 3 5 6 Rows denoted with the i subscript Columns denoted with the j subscript The element in row col is 3 The element in row 3 col is Matrix Elements often expressed using symbols A = a a a 3 a c a a a 3 a c a r a r a r3 a rc Matrix A has r rows and c columns Said to be of dimension r c Element a ij is in i th row and j th col A matrix is square if r = c Called a column vector is c= Topic 7 3 Topic 7 4
Transpose enoted as A Matrix Operations Row becomes Col, Row r becomes Col r Col becomes Row, Col c becomes Row c If A = [a ij then A = [a ji If A is r c then A is c r Addition and Subtraction Matrices must have the same dimension Addition/subtraction done on element by element basis a + b a + b a c + b c A + B = a r + b r a r + b r a rc + b rc Multiplication Matrix Operations If scalar then λa = [λa ij If multiplying two matrices (AB) Cols of A must equal Rows of B Resulting matrix of dimension Rows(A) Col(B) Elements obtained by taking cross products of rows of A with cols of B [ 4 3 3 [ 4 = [ 6 6 3 7 0 5 5 6 Topic 7 5 Topic 7 6 Regression Matrices Consider example with n = 4 Y Y Y 3 Y 4 Y Y Y 3 Y 4 Y = β 0 + β X +ε Y = β 0 + β X +ε Y 3 = β 0 + β X 3 +ε 3 Y 4 = β 0 + β X 4 +ε 4 = = β 0 + β X β 0 + β X + β 0 + β X 3 β 0 + β X 4 X X X 3 X 4 [ β0 β + Y = Xβ + ε ε ε ε 3 ε 4 ε ε ε 3 ε 4 Special Regression Examples Using multiplication and transpose Y Y = Y i X X = X Y = [ n Xi Xi X i [ Yi Xi Y i Will use these to compute ˆβ etc Topic 7 7 Topic 7 8
Special Types of Matrices Symmetric matrix When A = A Requires A to be square Example: X X iagonal matrix Square matrix with off-diagonals equal to zero Important example: Identity matrix IA = AI = A 0 0 0 0 0 0 0 0 0 0 0 0 Linear ependence If there is a relationship between the column(row) vectors of a matrix such that λ C + + λ c C c = 0 and not all λ s are 0, then the set of column(row) vectors are linearly dependent If such a relationship does not exist then the set of column(row) vectors are linearly independent Consider the matrix Q with column vectors C C 3 C = [ 5 Q = [ 5 3 0 [ 3 C = C 3 = [ 0 Topic 7 9 Topic 7 0 Rank of a Matrix Using λ =, λ = 0, λ 3 = [ 5 + 0 [ 3 + [ 0 = [ 0 0 0 The columns of Q are linearly dependent The rank of a matrix is number of linear independent columns (or rows) Rank of a matrix cannot exceed min(r,c) Full Rank all columns are linearly independent In this example: The rank of Q is Inverse of a matrix Inverse similar to the reciprocal of a scalar Inverse defined for square matrix of rank r Want to find the inverse of S, such that S S = I Easy example: iagonal matrix Let S = [ 0 0 4 then [ S 0 = 0 4 inverse of each element on the diagonal Topic 7 Topic 7
Inverse of a matrix General procedure for matrix Consider: [ a b A = c d Calculate the determinant = a d b c If = 0 then the matrix has no inverse In A, switch a and d; make c and b negative; multiply each element by A = Steps work only for a matrix Algorithm for 3 3 given in book [ d c b a Use of Inverse Consider equation x = 3 x = 3 Inverse similar to using reciprocal of a scalar Pertains to a set of equations Assuming A has an inverse: A X = C (r r) (r ) (r ) A AX = A C X = A C Topic 7 3 Topic 7 4 Random Vectors and Matrices Contain elements that are random variables Can compute expectation and (co)variance In regression set up, Y = Xβ + ε, both ε and Y are random vectors Expectation vector: E(A) = [E(A i ) Covariance matrix: symmetric σ (A) = σ (A ) σ(a, A ) σ(a, A r ) σ(a, A ) σ (A ) σ(a, A r ) σ(a r, A ) σ(a r, A ) σ (A r ) Basic Theorems Consider random vector Y Consider constant matrix A Suppose W = AY W is also a random vector E(W) = AE(Y) σ (W) = Aσ (Y)A Topic 7 5 Topic 7 6
Regression Matrices Can express observations Y = Xβ + ε Both Y and ε are random vectors E(Y) = Xβ + E(ε) = Xβ σ (Y) = 0 + σ (ε) = σ I Express quantity Q Least Squares Q = (Y Xβ) (Y Xβ) = Y Y β X Y Y Xβ + β X Xβ Taking derivative X Y + X Xβ = 0 This means b = (X X) X Y Topic 7 7 Topic 7 8 Fitted Values The fitted values Ŷ = Xb = X(X X) X Y Residual matrix Residuals e = Y Ŷ = Y HY = (I H)Y Matrix H = X(X X) X called hat matrix Equivalently write Ŷ = HY H symmetric and idempotent (HH = H) Matrix H used in diagnostics (chapter 9) e a random vector E(e) = (I H)E(Y) = (I H)Xβ = Xβ Xβ = 0 σ (e) = (I H)σ (Y)(I H) = (I H)σ I(I H) = (I H)σ Topic 7 9 Topic 7 0
Quadratic form defined as ANOVA Y AY = where A is symmetric n n matrix i j a ij Y i Y j Sums of squares can be shown to be quadratic forms (page 07) Quadratic forms play significant role in the theory of linear models when errors are Normally distributed Inference Vector b = (X X) X Y = AY The mean and variance are E(b) = (X X) X E(Y) = (X X) X Xβ = β σ (b) = Aσ (Y)A = Aσ IA = σ AA = σ (X X) Thus, b is multivariate Normal(β, σ (X X) ) Topic 7 Topic 7 Inference Continued Consider X h = [ X h Mean response Ŷh = X hb E(Ŷh) = X hβ Var(Ŷh) = X hσ (b)x h = σ X h(x X) X h Prediction E(Ŷh) = X hβ Var(Ŷh) = σ ( + X h(x X) X h ) KNNL Chapter 5 Background Reading KNNL Sections 6-65 Topic 7 3 Topic 7 4