Matrix Algebra and Its Applications to Statistics and Econometrics

Matrix Algebra and Its Applications to Statistics and Econometrics

This page is intentionally left blank

Matrix Algebra and Its Applications to Statistics and Econometrics c. Radhakrishna Rao Pennsylvania State University, USA M. Bhaskara Rao North Dakota State University, USA, World Scientific Singapore New Jersey London Hong Kong

Published by World Scientific Publishing Co. Pte. Ltd. POBox 128, Farrer Road, Singapore 912805 USA office: 27 Warren Street, Suite 401-402, Hackensack, NJ 07601 UK office: 57 Shelton Street, Covent Garden, London WC2H 9HE Library of Congress Cataloging-in-Publication Data Rao, C. Radhakrishna (Calyampudi Radhakrishna), 1920- Matrix algebra and its applications to statistics and econometrics / C. Radhakrishna Rao and M. Bhaskara Rao. p. cm. Includes bibliographical references and index. ISBN 9810232683 (alk. paper) I. Matrices. 2. Statistics. 3. Econometrics. QA188.R36 1998 512.9'434--dc21 I. Bhaskara Rao, M. British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library. First published 1998 Reprinted 2001, 2004 Copyright 1998 by World Scientific Publishing Co. Pte. Ltd. 98-5596 CIP All rights reserved. This book, or parts thereof, nwy not be reproduced in any form or by any means, electronic or mechanical, including photocopying, recording or any infornwtion storage and retrieval system now known or to be invented, without written permission from the Publisher. For photocopying of material in this volume, please pay a copying fee through the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA. In this case permission to photocopy is not required from the publisher. Printed in Singapore by Utopia Press Pte Ltd

To our wives BHARGAVl (Mrs. C.R. Rao) JAYASRI (Mrs. M.B. Rao)

This page is intentionally left blank

PREFACE Matrix algebra and matrix computations have become essential prerequisites for study and research in many branches of science and technology. It is also of interest to know that statistical applications motivated new lines of research in matrix algebra, some examples of which are generalized inverse of matrices, matrix approximations, generalizations of Chebychev and Kantorovich type inequalities, stochastic matrices, generalized projectors, Petrie matrices and limits of eigenvalues of random matrices. The impact of linear algebra on statistics and econometrics has been so substantial, in fact, that a number of books devoted entirely to matrix algebra oriented towards applications in these two subjects are now available. It has also become a common practice to devote one chapter or a large appendix on matrix calculus in books on mathematical statistics and econometrics. Although there is a large number of books devoted to matrix algebra and matrix computations, most of them are somewhat specialized in character. Some of them deal with purely mathematical aspects and do not give any applications. Others discuss applications using limited matrix theory. We have attempted to bridge the gap between the two types. We provide a rigorous treatment of matrix theory and discuss a variety of applications especially in statistics and econometrics. The book is aimed at different categories of readers: graduate students in mathematics who wish to study matrix calculus and get acquainted with applications in other disciplines, graduate students in statistics, psychology, economics and engineering who wish to concentrate on applications, and to research workers who wish to know the current developments in matrix theory for possible applications in other areas. This book provides a self-contained, updated and unified treatment of the theory and applications of matrix methods in statistics and econ~ metrics. All the standard results and the current developments, such as the generalized inverse of matrices, matrix approximations, matrix vii

viii MATRIX ALGEBRA THEORY AND APPLICATIONS differential calculus and matrix decompositions, are brought together to produce a most comprehensive treatise to serve both as a text in graduate courses and a reference volume for research students and consultants. It has a large number of examples from different applied areas and numerous results as complements to illustrate the ubiquity of matrix algebra in scientific and technological investigations. It has 16 chapters with the following contents. Chapter 1 introduces the concept of vector spaces in a very general setup. All the mathematical ideas involved are explained and numerous examples are given. Of special interest is the construction of orthogonal latin squares using concepts of vector spaces. Chapter 2 specializes to unitary and Euclidean spaces, which are vector spaces in which distances and angles between vectors are defined. They playa special role in applications. Chapter 3 discusses linear transformations and matrices. The notion of a transformation from one vector space to another is introduced and the operational role of matrices for this purpose is explained. Thus matrices are introduced in a natural way and the relationship between transformations and matrices is emphasized throughout the rest of the book. Chapters 4, 5, 6 and 7 cover all aspects of matrix calculus. Special mention may be made of theorems on rank of matrices, factorization of matrices, eigenvalues and eigenvectors, matrix derivatives and projection operators. Chapter 8 is devoted to generalized inverse of matrices, a new area in matrix algebra which has been found to be a valuable tool in developing a unified theory of linear models in statistics and econometrics. Chapters 9, 10 and 11 discuss special topics in matrix theory which are useful in solving optimization problems. Of special interest are inequalities on singular values of matrices and norms of matrices which have applications in almost all areas of science and technology. Chapters 12 and 13 are devoted to the use of matrix methods in the estimation of parameters in univariate and multivariate linear models. Concepts of quadratic subspaces and new strategies of solving linear equations are introduced to provide a unified theory and computational techniques for the estimation of parameters. Some modern developments in regression theory such as total least squares, estimation of parameters in mixed linear models and minimum norm quadratic estimation are discussed in detail using matrix methods. Chapter 14

Preface ix deals with inequalities which are useful in solving problems in statistics and econometrics. Chapter 15 is devoted to non-negative matrices and Perron-Frobenius theorem which are essential for the study of and research in econometrics, game theory, decision theory and genetics. Some miscellaneous results not covered in the main themes of previous chapters are put together in Chapter 16. It is a pleasure to thank Marina Tempelman for her patience in typing numerous revisions of the book. March 1998 C.R. Rao M.B. Rao

This page is intentionally left blank

NOTATION The following symbols are used throughout the text to indicate certain elements and the operations based on them. Scalars R C F x = Xl + ix2 X = Xl - ix2 Ixi = (xi + X~)1/2 General {an} A,B,... ACB xea A+B AuB AnB Vector Spaces (V, F) dimv at, a2, Sp(at,..., ak) Fn Rn C n real numbers complex numbers general field of elements a complex number conjugate of X modulus of X a sequence of elements sets of elements set A is contained in set B x is an element of set A {Xl + X2 : Xl E A, X2 E B} {x : X E A and/or B} {x : X E A and x E B} vector space over field F dimension of V vectors in V the set {alai +... + akak : al,.., ak E C} n dimensional coordinate (Euclidean) space same as Fn with F = R same as Fn with F = C xi

xii El1 <.,. > (.,.) MATRIX ALGEBRA THEORY AND APPLICATIONS direct sum, {x + y : x E V, yew; V n W = O} inner product semi inner product Transformations T:V-+W R(T) K(T) v{t) Matrices A,B,C,... A mxn Mm,n Mm,n(-) Mn A = [aij] A E Mm,n Sp(A) A* =.A' A* = A A*A = AA* = I A*A = AA* A# A-I A- A+ ALMN In I transformation from space V to space W the range of T, i.e., the set {Tx : x E V} the kernel of T, i.e., the set {Tx = 0 : x E V} nullity (dimension of K{T)) general matrices or linear transformations m x n order matrix the class of matrices with m rows and n columns m x n order matrices with specified property 0 the class of matrices with n rows and n columns aij is the (i,j) the entry of A (i-th row and j-th column) A is a matrix with m rows and n columns the vector space spanned by the column vectors of A, also indicated by R(A) considering A as transformation iiij is complex conjugate of aij obtained from A interchanging rows and columns, i.e., if A = (aij) then A' = (aji) Conjugate transpose or transpose of.a defined above Hermitian or self adjoint unitary normal adjoint (A E Mm,n, < Ax, Z >m= < x, A#z >n) inverse of A E Mn such that AA-I = A-I A = I generalized or g-inverse of A E Mm,n, (AA- A = A) Moore-Penrose inverse Rao-Yanai (LMN) inverse identity matrix of order n with all diagonal elements as unities and the rest as zeros identity matrix when the order is implicit

o p(a) p.. (A) vec A (all. Ian) [AIIA2J tr A IAI or det A A B A B A0B AoB IIxll IIxll.. e II All IIAIIF II Allin IIAlls IIAllui IIAllwui IIAIIMNi m(a) pd nnd s.v.d. B~L A Notation zero scalar, vector or matrix rank of matrix A spectral radius of A vector of order mn formed by writing the columns of A E Mm,n one below the other matrix partitioned by colwnn vectors ai,...,an matrix partitioned by two matrices Al and A2 trace A, the sum of diagonal elements of A E Mn determinant of A Hadamard-Schur product Kronecker product Khatri-Hao product matrix with < b i, aj > as the (j, i)-th entry, where A = (all... la ftl ), B = (bll. Ibn ) norm of vector x semi-norm of vector x norm or matrix norm of A Frobenius norm of A = ([tr(a* A)Jl/2 induced matrix norm: max II Axil for IIxll = 1 spectral norm of A unitarily invariant norm, IIU* AVII = IIAII for all unitary U and V, A E Mm,n weakly unitarily invariant norm, IIU* AUII = IIAII for all unitary U, A E Mn M, N invariant norm xiii matrix obtained from A = (aij) by replacing aij by laijl, the modulus of the number aij E C positive definite matrix (x* Ax > 0 for x i= 0) non-negative definite matrix (x" Ax ~ 0) singular value decomposition or simply B ~ A to indicate (LOwner partial order) A - B is nnd

xiv MATRIX ALGEBRA THEORY AND APPLICATIONS X ~e Y xi ~ Yi, i = 1,...,n, where x' = (x},...,xn ) and y' = (y},...,yn) B ~e A entry wise inequality bij ~ aij, A = (aij), B = (bij ) B ~e A A ~e 0 A >e 0 Y «x y «w x y «8 x {Ai(A)} {O'i(A)} entry wise inequality bij ~ aij non-negative matrix (all elements are non-negative) positive matrix (all elements are positive) vector x majorizes vector y vector x weakly majorizes vector y vector x soft majorizes vector y eigenvalues of A E M n, [A} (A) ~... ~ An(A)] singular values of A E Mm,n, [O'}(A) ~... ~ O'r(A)], r = min{m,n}

CONTENTS Preface........................................... vii Notation........................................................ xi CHAPTER 1. VECTOR SPACES 1.1 Rings and Fields.............................................. 1 1.2 Mappings............................. 14 1.3 Vector Spaces............................................... 16 1.4 Linear Independence and Basis of a Vector Space................ 19 1.5 Subspaces..................................... 24 1.6 Linear Equations............................................ 29 1. 7 Dual Space.............................. 35 1.8 Quotient Space.............................................. 41 1.9 Projective Geometry......................................... 42 CHAPTER 2. UNITARY AND EUCLIDEAN SPACES 2.1 Inner Product............................................... 51 2.2 Orthogonality......................... 56 2.3 Linear Equations.................................... 66 2.4 Linear Functionals........................................... 71 2.5 Semi-inner Product................................ 76 2.6 Spectral Theory............................................. 83 2.7 Conjugate Bilinear Functionals and Singular Value Decomposition................................... 101 CHAPTER 3. LINEAR TRANSFORMATIONS AND MATRICES 3.1 Preliminaries...................................... 107 3.2 Algebra of Transformations............................... 110 3.3 Inverse Transformations..................................... 116 3.4 Matrices......................................... 120 xv

XVI MATRIX ALGERBA THEORY AND APPLICATIONS CHAPTER 4. CHARACTERISTICS OF MATRICES 4.1 Rank and Nullity of a Matrix......... 128 4.2 Rank and Product of Matrices................................ 131 4.3 Rank Factorization and Further Results............ 136 4.4 Determinants...................... 142 4.5 Determinants and Minors.................................... 146 CHAPTER 5. FACTORIZATION OF MATRICES 5.1 Elementary Matrices................................ 157 5.2 Reduction of General Matrices.................... 160 5.3 Factorization of Matrices with Complex Entries................. 166 5.4 Eigenvalues and Eigenvectors................................. 177 5.5 Simultaneous Reduction of Two Matrices...................... 184 5.6 A Review of Matrix Factorizations............................ 188 CHAPTER 6. OPERATIONS ON MATRICES 6.1 Kronecker Product.......................................... 193 6.2 The Vec Operation.......................................... 200 6.3 The Hadamard-Schur Product................................ 203 6.4 Khatri-Roo Product......................................... 216 6.5 Matrix Derivatives.......................................... 223 CHAPTER 7. PROJECTORS AND IDEMPOTENT OPERATORS 7.1 Projectors.......................... 239 7.2 Invariance and Reducibility.................................. 245 7.3 Orthogonal Projection....................................... 248 7.4 Idempotent Matrices........................................ 250 7.5 Matrix Representation of Projectors........................... 256 CHAPTER 8. GENERALIZED INVERSES 8.1 Right and Left Inverses...................................... 264 8.2 Generalized Inverse (g-inverse)....................... 265 8.3 Geometric Approach: LMN-inverse................. 282 8.4 Minimum Norm Solution.................................... 288

Contents xvii 8.5 Least Squares Solution........................ 289 8.6 Minimum Norm Least Squares Solution........................ 291 8.7 Various Types of g-inverses....... 292 8.8 G-inverses Through Matrix Approximations.................... 296 8.9 Gauss-Markov Theorem.................................... 300 CHAPTER 9. MAJORIZATION 9.1 Majorization.............. 303 9.2 A Gallery of Functions..................... 307 9.3 Basic Results............................................... 308 CHAPTER 10. INEQUALITIES FOR EIGENVALUES 10.1 Monotonicity Theorem.................................... 322 10.2 Interlace Theorems............... 328 10.3 Courant-Fischer Theorem.................................. 332 loa Poincare Separation Theorem..................... 337 10.5 Singular Values and Eigenvalues............................ 339 10.6 Products of Matrices, Singular Values, and Horn's Theorem................. 340 10.7 Von Neumann's Theorem.................................. 342 CHAPTER 11. MATRIX APPROXIMATIONS 11.1 Norm on a Vector Space................................... 361 11.2 Norm on Spaces of Matrices................................ 363 11.3 Unitarily Invariant Norms....... 374 11.4 Some Matrix Optimization Problems........................ 383 11.5 Matrix Approximations.................................... 388 11.6 M, N-invariant Norm and Matrix Approximations............. 394 11.7 Fitting a Hyperplane to a Set of Points...................... 398 CHAPTER 12. OPTIMIZATION PROBLEMS IN STATISTICS AND ECONOMETRICS 12.1 Linear Models............................................ 403 12.2 Some Useful Lemmas...................................... 403 12.3 Estimation in a Linear Model........... 406 1204 A Trace Minimization Problem............................. 409 12.5 Estimation of Variance.................................... 413

xviii MATRIX ALGEBRA THEORY AND APPLICATIONS 12.6 The Method of MIN QUE: A Prologue....................... 415 12.7 Variance Components Models and Unbiased Estimation....... 416 12.8 Normality Assumption and Invariant Estimators............. 419 12.9 The Method of MIN QUE.................................. 422 12.10 Optimal Unbiased Estimation.............................. 425 12.11 Total Least Squares....................................... 428 CHAPTER 13. QUADRATIC SUBSPACES 13.1 Basic Ideas 433 13.2 The Structure of Quadratic Subspaces....................... 438 13.3 Commutators of Quadratic Subspaces....................... 442 13.4 Estimation of Variance Components......................... 443 CHAPTER 14. INEQUALITIES WITH APPLICATIONS IN STATISTICS 14.1 Some Results on nnd and pd Matrices....................... 449 14.2 Cauchy-Schwartz and Related Inequalities............... 454 14.3 Hadamard Inequality...................................... 456 14.4 Holder's Inequality........................................ 457 14.5 Inequalities in Information Theory.......................... 458 14.6 Convex Functions and Jensen's Inequality.................... 459 14.7 Inequalities Involving Moments............................. 461 14.8 Kantorovich Inequality and Extensions...................... 462 CHAPTER 15. NON-NEGATIVE MATRICES 15.1 Perron-Frobenius Theorem................................. 467 15.2 Leontief Models in Economics.............................. 477 15.3 Markov Chains........................................... 481 15.4 Genetic Models........................................... 485 15.5 Population Growth Models................................. 489 CHAPTER 16. MISCELLANEOUS COMPLEMENTS 16.1 Simultaneous Decomposition of Matrices..................... 493 16.2 More on Inequalities........................... 494 16.3 Miscellaneous Results on Matrices.................... 497 16.4 Toeplitz Matrices......................................... 501 16.5 Restricted Eigenvalue Problem..................... 506

Contents xix 16.6 Product of Two Raleigh Quotients.......................... 507 16.7 Matrix Orderings and Projection......... 508 16.8 Soft Majorization......................................... 509 16.9 Circulants............................................... 511 16.10 Hadamard Matrices....................................... 514 16.11 Miscellaneous Exercises.................................... 515 REFERENCES................................................. 519 INDEX................................................. 529