Econ 620. Matrix Differentiation. Let a and x are (k 1) vectors and A is an (k k) matrix. ) x. (a x) = a. x = a (x Ax) =(A + A (x Ax) x x =(A + A )

Transcription

1 Econ 60 Matrix Differentiation Let a and x are k vectors and A is an k k matrix. a x a x = a = a x Ax =A + A x Ax x =A + A x Ax = xx A We don t want to prove the claim rigorously. But a x = k a i x i i= If you want to differentiate the function with respect to x, you have to differentiate the function with respect to each element of vector x and form a vector -called gradient- with the result. a x a x a x a = = a = a a x a k k x x You can understand a simply as the transpose of a. For the differentiation of the quadratic form, consider the summation expression; x Ax = k i= j= k x i a ij x j = x a x + x a x + x a 3 x x a k x k + x a x + x a x + x a 3 x x a k x k + x 3 a 3 x + x 3 a 3 x + x 3 a 33 x x 3 a 3k x k + Now, we have + x k a k x + x k a k x + x k a k3 x x k a kk x k x Ax =a x + a x + a 3 x a k x k + x a + x 3 a x k a k = a x + a x + a 3 x a k x k + a x + a x + a 3 x a k x k = A x + A x = A + A x where A is the first row of the matrix A and A is the first column of the matrix A. Similarly, x Ax = a x +a x + a 3 x a k x k + x a + x 3 a x k a k = a x + a x + a 3 x a k x k + a x + a x + a 3 x a k x k = A x + A x = A + A x

2 You see the pattern emerging from the calculation. In general, x Ax i = A i + A i x i =,,,k We stack the vectors to get; x Ax = Ax Ax Ax k = A A A k + A A A k x =A + A x You can verify the result for Ax A Consider the least squares problem; = xx with a similar argument. S b =y Xb y Xb=y b X y Xb = y y y Xb b X y + b X Xb = y y y Xb+ b X Xb Note that y X is a vector, b is x vector and X X is A matrix in the formula above. Hence, S b b = X y + X X+X X b = X y +X Xb Least Squares Estimator in Matrix Form The model is given by In matrix notation y i = β + β x i + β 3 x i3 + + β k x ik + ε i E ε i =0,E ε i = σ,eε i ε j =0wheni j y = Xβ + ε E ε =0,Eεε =σ I The least squares estimator is β =X X X y Unbiasedness of β E β X X X y β +X X X ε X X X Xβ + ε = β +X X X E ε =β iance of β β β E β β E β β β β β X X X εε X X X =X X X E εε X X X = σ X X X IX X X = σ X X

3 Residual vector and M matrix e = y X β = y X X X X y = = My I X X X X y The matrices P = X X X X and M =I P are called projection matrix. Especially, P is the projection matrix onto space spanned by columns of X and M is the projection onto the space orthogonal to the space spanned by columns of X. When people simply say the projection matrix, they mean P. P and M have a nice interpretation in terms of geometry.. Properties of P and M matrix iboth P and M are symmetric and idempotent. - proof is easy. ii ρ P =k and ρ M =N k. ρ P =ρ X X X X =min ρ X,ρ X X,ρX =mink, k, k =k ρ M =tr M =tr I P =tr I tr P =tr I ρ P =N k Note that the rank of an idempotent matrix is its trace and both P and M are idempotent. iii MX = 0 and P + M = I MX = I X X X X X = X X X X X X = X X = 0 P + M = X X X X + I X X X X = I Estimation of σ Since ε is unobservable by definition, we do not know its variance σ, either. However, we can estimate it using the sum of squared residuals. Note that e = N y i β β N x i β k x ik = e i = e e i= y X β = y X X X X y = I X X X X y = My = M Xβ + ε =MXβ + Mε = Mε i= Hence, e e =Mε Mε=ε M Mε = ε MMε = ε Mε Now, taking expectation on both sides, E e e=e ε Mε tr ε Mε since ε Mε is scalar tr Mεε since tr AB =tr BA = tr E Mεε since expectation is a linear operator = tr MEεε since M is non-stochastic = tr Mσ I = σ tr M sincetr aa =atr A when a is a scalar = σ ρ M since M is idempotent = σ N k from the argument above Therefore, to get an unbiased estimator of σ, we propose; s = 3 e e N k

4 Then, E s = N k E e e= σ N k = σ N k Distribution of s Fact-you can actually prove this, try-. Then, N k s σ = e e σ χ N k e e E σ =N k E e e=σ N k e e σ =N k e e=σ 4 N k A matrix A I where is an N vector whose elements are all. If we postmultiply A matrix with a vector, say y, it will results in a vector in mean deviation form; Ay = I y = y y y y = y y y N y N y y = y y N y N y N N y y y = y y N = y i= y i N i= N y i y N y N y N N i= y i y y y y = y y = y y y N y y N y Why do we introduce the matrix A? There is a good reason for it. Consider the classical multiple regression model in the following form; y = Xβ + ε = β X + ε = β β +X β + ε where we partitioned X matrix into the column corresponding to the constant term,, and the columns corresponding to all the other regressors,x. Then, β β = =X β X X y = X X X y = X y X X X X y 4

5 What is the lower right block of the inverse matrix? From the formula for the inverse of the partitioned matrix, β = X X X X X y + X X X X X y = I X X y + X X I X X y = X AX X y +X AX X y =X AX X I y =X AX X Ay =X A AX X A Ay = AX AX AX Ay Now consider another approach to the estimation; Premultiplying both sides with A gives; y = Xβ + ε = X β β + ε = β +X β + ε Ay = β A+AX β + Aε = AX β + Aε since A = I = 0 Now, define Ay = y,ax = X, and Aε = ε to get The least squares estimator is given by; y = X β + ε β =X X X y = AX AX AX Ay =X A AX X A Ay =X AX X Ay which is identical to the least squares estimator for β in the original model. The transformed regression does not include a constant term and the data used in the transformed regression is in mean deviation formsasshownabove-ay and AX. In sum, the slope estimates from the original regression - one with a constant term and untransformed data- is identical to those from the transformed regression - one without a constant term and with data in mean deviation forms. Then, what about the constant term? The least squares estimator for the constant term is given by; β = y β x β 3 x 3 β k x k which can be derived easily from the first order condition. iance matrix from the two regressions In model without transformation, we know that β = β Cov β, β Cov β, β β = σ X X = σ X X X X 5

6 Therefore, β = σ X X X X = σ X I X = σ X AX Thevariancematrixof β is identical to that from the regression in mean deviation forms since β = σ X X = σ X AX Therefore, the two regressions result in the same estimates of the slope coefficients and variances of the estimates. R in the multiple regression analysis; R is defined as the ratio between the explained sum of squares and the total sum of squares; R SS TSS = RSS TSS TSS is the sum of squares of variations in the dependent variable around the mean; On the other hand, N N TSS = y i y = y i yy i y =Ay Ay =y Ay i= i= y Ay =Ay Ay =Aŷ + Ae Aŷ + Ae =Aŷ + e Aŷ + e = ŷ Aŷ + e e Hence, R = ŷ Aŷ y Ay = = e e y Ay X β A X β y Ay = β X AX β y Ay 6