Matrix Differential Calculus with Applications in Statistics and Econometrics

Transcription

1 Matrix Differential Calculus with Applications in Statistics and Econometrics Revised Edition JAN. R. MAGNUS CentERjor Economic Research, Tilburg University and HEINZ NEUDECKER Cesaro, Schagen JOHN WILEY & SONS Chichester New York Weinheim Brisbane Singapore Toronto

2 Contents Preface Preface to the fast revised printing Preface to the second revised printing Part One Matrices 1 Basic properties of vectors and matrices 1 Introduction, 3 2 Sets, 3 3 Matrices: addition and multiplication, 4 4 The transpose of a matrix, 5 5 Square matrices, 6 6 Linear forms and quadratic forms, 7 7 The rank ofa matrix, 8 8 The inverse, 9 9 The determinant, 9 10 The trace, Partitioned matrices, Complex matrices, cigenvalues and eigenvectors, Schur's decomposition theorem, The Jordan decomposition, The singular-value decomposition, Furt her results concerning eigenvalues, Positive (semi)definite matrices, Threefurther resultsfor positive definite matrices, A useful result, 24 Miscellaneous exercises, 25 Bibiliographical notes, 26 2 Kronecker products, the vec Operator and the Moore-Penrose inverse 1 Introduction, 27 2 The Kronecker product, 27

3 3 Eigenvalues of a Kronecker product, 28 4 The vec Operator, 30 5 The Moore-Penrose (MP) inverse, 32 6 Existence and uniqueness of the MP inverse, 32 7 Some properties ofthe MP inverse, 33 8 Further properties, 34 9 The Solution of linear equation Systems, 36 Miscellaneous exercises, 38 Bibliographical notes, 39 3 Miscellaneous matrix results 1 Introduction, ie adjoint matrix, 40 3 Proof of Theorem 7, \vo results concerning bordered determinants, ie matrix equation AX = 0, ie Hadamard product, »e commutation matrix K mn, ie duplication matrix D n, 48 9 Relationship between D +i and D, I, Relationship between D +l and D, II, Conditions for a quadratic form to be positive (negative) subject linear constraints, Necessary and sufficient conditions for r(a:b) = r{a) + r(b), The bordered Gramian matrix, The equations X 1 A + X 2 B' = G i,x l B = G 2, 60 Miscellaneous exercises, 62 Bibliographical notes, 62 Part Two Differentials: the theory 4 Mathematical preliminaries 1 Introduction, 65 2 Interior points and accumulation points, 65 3 Open and ciosed sets, ie Bolzano-Weierstrass theorem, 69 5 Functions, ie /imit of afunction, 70 7 Continuous functions and compactness, 71 8 Conuex sets, 72 9 Conuex and concave functions, 75 Bibliographical notes, 77

4 Contents vu 5 Differentials and differentiability 78 1 Introduction, 78 2 Continuity, 78 3 Differentiability and linear approximation, 80 4 The differential of a vector function, 82 5 Uniqueness of the differential, 84 6 Continuity of differentiable functions, 84 7 Partial derivatives, 85 8 Thefirst identification theorem, 87 9 Existence of the differential, I, Existence of the differential, II, Continuous differentiability, The chain rule, Cauchy invariance, The mean-value theorem for real-valued functions, Matrix functions, Some remarks on notation, 96 Miscellaneous exercises, 98 Bibliographical notes, 98 6 The second differential 99 1 Introduction, 99 2 Second-order partial derivatives, ie Hessian matrix, Twice differentiability and second-order approximation, 1, Definition of twice differentiability, The second differential, (Column) symmetry of the Hessian matrix, The second identification theorem, Twice differentiability and second-order approximation, II, Chain rule for Hessian matrices, The analogue for second differentials, Taylor's theorem for real-valued functions, Higher-order differentials, Matrix functions, 114 Bibliographical notes, Static optimization Introduction, Unconstrained optimization, The existence of absolute extrema, Necessary conditions for a local minimum, 119

5 5 Sufficient conditions for a local minimum: first-derivative test, Sufficient conditions for a local minimum: second-derivative test, Characterization of differentiable convex functions, Characterization of twice differentiable convex functions, Sufficient conditions for an absolute minimum, Monotonie transformations, Optimization subjeet to constraints, Necessary conditions for a local minimum under constraints, Sufficient conditions for a local minimum under constraints, Sufficient conditions for an absolute minimum under constraints, v4 note on constraints in matrix form, Economic interpretation of Lagrange multipliers, 141 Appendix: the implicit funetion theorem, 142 Bibliographical notes, 144 Part Three Differentials: the practice 8 Some important differentials 1 Introduction, Fundamental rules of differential calculus, The differential of a determinant, The differential of an inverse, The differential of the Moore-Penrose inverse, The differential of the adjoint matrix, On differentiating eigenvalues and eigenvectors, The differential of eigenvalues and eigenvectors: the real Symmetrie case, The differential of eigenvalues and eigenvectors: the general complex case, Two alternative expressions for <U, The second differential of the eigenvalue funetion, Multiple eigenvalues, 167 Miscellaneous exercises, 167 Bibliographical notes, First-order differentials and Jacobian matrices 1 Introduction, Classification, Bad notation, Good notation, Identification of Jacobian matrices, The first identification table, 175

6 7 Partitioning of the derivative, Scalar functions of a vector, Scalar functions of a matrix, I: trace, Scalar functions of a matrix, II: determinant, Scalar functions of a matrix, 111: eigenvalue, Two examples of vector functions, Matrix functions, Kronecker products, Some other problems, 185 Bibliographical notes, Second-order differentials and Hessian matrices 1 Introduction, The Hessian matrix of a matrix function, Identification of Hessian matrices, ie second identification table, An explicit formula for the Hessian matrix, Scalar functions, Vector functions, Matrix functions, I, Matrix functions, II, 195 Part Four Inequalities 11 Inequalities 1 Introduction, ie Caudry-Sc/iwarz inequality, Matrix analogues of the Cauchy-Schwarz inequality, ie theorem of the arithmetic and geometric means, ie Rayleigh quotient, Concavity of X t, convexity of X, Variational description of eigenvalues, Fischefs min-max theorem, Monotonicity of the eigenvalues, ie Poincare Separation theorem, Two corollaries of Poincare's theorem, Further consequences of the Poincare theorem, Multiplicative version, je maximum of a bilinear form, Hadamard's inequality, /In interlude: Karamata's inequality, Karamata's inequality applied to eigenvalues, 217

7 18 An inequality concerning positive semidefinite matrices 19 A representation theoremfor (Za p ) llp, A representation theorem for (tr A") ilp, Holder's inequality, Concavity of log \A\, Minkowskis inequality, Quasilinear representation of\a\ { ", Minkowski's determinant theorem, Weighted means of order p, Schlömilch's inequality, Curvature properties of M {x, a), Least Squares, Generalized least Squares, Restricted least Squares, Restricted least Squares: matrix Version, 235 Miscellaneous exercises, 236 Bibliographical notes, 240 Part Five The linear model 12 Statistical preliminaries 1 Introduction, ie cumulative distribution function, ie yoim density function, Expectations, Variance and covariance, Independence oftwo random variables (vectors), Independence of n random variables (vectors), Sampling, ie one-dimensional normal distribution, The multivariate normal distribution, Estimation, 252 Miscellaneous exercises, 253 Bibliographical notes, The linear regression model 1 Introduction, Affine minimum-trace unbiased estimation, ie Gauss-Markov theorem, ie method of least Squares, Aitken's theorem, Multicollinearity, Estimable functions, 263

8 Contents XI 8 Linear constraints: the case M{R')<^J1 {X'\ Linear constraints: the general case, Linear constraints: the case Jt(R')c^J((X') = {0}, A singular variance matrix: the case J((X)<^M(V), A singular variance matrix: the case r(x' V + X) = r(x), A singular variance matrix: the general case, I, Explicit and implicit linear constraints, The general linear model, I, A singular variance matrix: the general case, II, The general linear model, II, Generalized least Squares, Restricted least Squares, 283 Misceilaneous exercises, 285 Bibliographical notes, Further topics in the linear model Introduction, Best quadratic unbiased estimation of a 2, ie best quadratic and positive unbiased estimator of a 2, The best quadratic unbiased estimator ofa 2, Best quadratic invariant estimation ofa 2, The best quadratic and positive invariant estimator of a 2, The best quadratic invariant estimator ofa 2, Best quadratic unbiased estimation in the multivariate normal case, Boundsfor the bias of the least Squares estimator ofa 2,1, Boundsfor the bias of the least Squares estimator ofa 2, II, The prediction of disturbances, Predictors that are best linear unbiased with scalar variance matrix (BLUS), Predictors that are best linear unbiased with fixed variance matrix (BLUF), l, Predictors that are best linear unbiased with fixed variance matrix (BLUF), II, Local sensitivity of the posterior mean, Local sensitivity of the posterior precision, 308 Bibliographical notes, 309 Part Six Applications to maximum likelihood estimation 15 Maximum likelihood estimation Introduction, The method of maximum likelihood (ML), ML estimation of the multivariate normal distribution, 314

9 Xll 4 Implicit versus explicit treatment of symmetry, The treatment of positive definiteness, The information matrix, ML estimation of the multivariate normal distribution with distinct means, The multivariate linear regression model, The errors-in-variables model, The nonlinear regression model with normal errors, A special case: functional independence of mean parameters and variance parameters, Generalization of Theorem 6, 327 Miscellaneous exercises, 329 Bibliographical notes, Simultaneous equations 1 Introduction, The simultaneous equations model, The Identification problem, Identification with linear constraints on B and V only, Identification with linear constraints on B, Y and L, Nonlinear constraints, Full-information maximum likelihood (FIML): the information matrix (general case), Full-information maximum likelihood (FIML): the asymptotic variance matrix (special case), Limited-information maximum likelihood (LIML): the first-order conditions, Limited-information maximum likelihood (LIML): the information matrix, Limited-information maximum likelihood (LIML): the asymptotic variance matrix, 346 Bibliographical notes, Topics in psychometrics 1 Introduction, Population principal components, Optimality of principal components, A related result, Sample principal components, Optimality of sample principal components, Sample analogue of Theorem 3, One-mode component analysis, 358

10 Contents 9 Relationship between one-mode component analysis and sample principal components, Two-mode component analysis, Multimode component analysis, Factor analysis, A zigzag routine, A Newton-Raphson routine, Kaiser's varimax method, Canonical correlations and variates in the population, 376 Bibliographical notes, 378 Bibliography Index of Symbols Subject Index