Statistics for Social and Behavioral Sciences Advisors: S.E. Fienberg W.J. van der Linden For other titles published in this series, go to http://www.springer.com/series/3463
Haruo Yanai Kei Takeuchi Yoshio Takane Projection Matrices, Generalized Inverse Matrices, and Singular Value Decomposition
Haruo Yanai Department of Statistics St. Luke s College of Nursing 10-1 Akashi-cho Chuo-ku Tokyo 104-0044 Japan hyanai@slcn.ac.jp Kei Takeuchi 2-34-4 Terabun Kamakurashi Kanagawa-ken 247-0064 Japan kei.takeuchi@wind.ocn.ne.jp Yoshio Takane Department of Psychology McGill University 1205 Dr. Penfield Avenue Montreal Québec H3A 1B1 Canada takane@psych.mcgill.ca ISBN 978-1-4419-9886-6 e-isbn 978-1-4419-9887-3 DOI 10.1007/978-1-4419-9887-3 Springer New York Dordrecht Heidelberg London Library of Congress Control Number: 2011925655 Springer Science+Business Media, LLC 2011 All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)
Preface All three authors of the present book have long-standing experience in teaching graduate courses in multivariate analysis (MVA). These experiences have taught us that aside from distribution theory, projections and the singular value decomposition (SVD) are the two most important concepts for understanding the basic mechanism of MVA. The former underlies the least squares (LS) estimation in regression analysis, which is essentially a projection of one subspace onto another, and the latter underlies principal component analysis (PCA), which seeks to find a subspace that captures the largest variability in the original space. Other techniques may be considered some combination of the two. This book is about projections and SVD. A thorough discussion of generalized inverse (g-inverse) matrices is also given because it is closely related to the former. The book provides systematic and in-depth accounts of these concepts from a unified viewpoint of linear transformations in finite dimensional vector spaces. More specifically, it shows that projection matrices (projectors) and g-inverse matrices can be defined in various ways so that a vector space is decomposed into a direct-sum of (disjoint) subspaces. This book gives analogous decompositions of matrices and discusses their possible applications. This book consists of six chapters. Chapter 1 overviews the basic linear algebra necessary to read this book. Chapter 2 introduces projection matrices. The projection matrices discussed in this book are general oblique projectors, whereas the more commonly used orthogonal projectors are special cases of these. However, many of the properties that hold for orthogonal projectors also hold for oblique projectors by imposing only modest additional conditions. This is shown in Chapter 3. Chapter 3 first defines, for an n by m matrix A, a linear transformation y = Ax that maps an element x in the m-dimensional Euclidean space E m onto an element y in the n-dimensional Euclidean space E n. Let Sp(A) = {y y = Ax} (the range or column space of A) and Ker(A) = {x Ax = 0} (the null space of A). Then, there exist an infinite number of the subspaces V and W that satisfy E n = Sp(A) W and E m = V Ker(A), (1) where indicates a direct-sum of two subspaces. Here, the correspondence between V and Sp(A) is one-to-one (the dimensionalities of the two subspaces coincide), and an inverse linear transformation from Sp(A) to V can v
vi PREFACE be uniquely defined. Generalized inverse matrices are simply matrix representations of the inverse transformation with the domain extended to E n. However, there are infinitely many ways in which the generalization can be made, and thus there are infinitely many corresponding generalized inverses A of A. Among them, an inverse transformation in which W = Sp(A) (the ortho-complement subspace of Sp(A)) and V = Ker(A) = Sp(A ) (the ortho-complement subspace of Ker(A)), which transforms any vector in W to the zero vector in Ker(A), corresponds to the Moore-Penrose inverse. Chapter 3 also shows a variety of g-inverses that can be formed depending on the choice of V and W, and which portion of Ker(A) vectors in W are mapped into. Chapter 4 discusses generalized forms of oblique projectors and g-inverse matrices, and gives their explicit representations when V is expressed in terms of matrices. Chapter 5 decomposes Sp(A) and Sp(A ) = Ker(A) into sums of mutually orthogonal subspaces, namely and Sp(A) = E 1 E 2 E r Sp(A ) = F 1 F 2 F r, where indicates an orthogonal direct-sum. It will be shown that E j can be mapped into F j by y = Ax and that F j can be mapped into E j by x = A y. The singular value decomposition (SVD) is simply the matrix representation of these transformations. Chapter 6 demonstrates that the concepts given in the preceding chapters play important roles in applied fields such as numerical computation and multivariate analysis. Some of the topics in this book may already have been treated by existing textbooks in linear algebra, but many others have been developed only recently, and we believe that the book will be useful for many researchers, practitioners, and students in applied mathematics, statistics, engineering, behaviormetrics, and other fields. This book requires some basic knowledge of linear algebra, a summary of which is provided in Chapter 1. This, together with some determination on the part of the reader, should be sufficient to understand the rest of the book. The book should also serve as a useful reference on projectors, generalized inverses, and SVD. In writing this book, we have been heavily influenced by Rao and Mitra s (1971) seminal book on generalized inverses. We owe very much to Professor
PREFACE vii C. R. Rao for his many outstanding contributions to the theory of g-inverses and projectors. This book is based on the original Japanese version of the book by Yanai and Takeuchi published by Todai-Shuppankai (University of Tokyo Press) in 1983. This new English edition by the three of us expands the original version with new material. January 2011 Haruo Yanai Kei Takeuchi Yoshio Takane
Contents Preface v 1 Fundamentals of Linear Algebra 1 1.1 Vectors and Matrices....................... 1 1.1.1 Vectors.......................... 1 1.1.2 Matrices.......................... 3 1.2 Vector Spaces and Subspaces.................. 6 1.3 Linear Transformations..................... 11 1.4 Eigenvalues and Eigenvectors.................. 16 1.5 Vector and Matrix Derivatives.................. 19 1.6 Exercises for Chapter 1..................... 22 2 Projection Matrices 25 2.1 Definition............................. 25 2.2 Orthogonal Projection Matrices................. 30 2.3 Subspaces and Projection Matrices............... 33 2.3.1 Decomposition into a direct-sum of disjoint subspaces.................... 33 2.3.2 Decomposition into nondisjoint subspaces....... 39 2.3.3 Commutative projectors................. 41 2.3.4 Noncommutative projectors............... 44 2.4 Norm of Projection Vectors................... 46 2.5 Matrix Norm and Projection Matrices............. 49 2.6 General Form of Projection Matrices.............. 52 2.7 Exercises for Chapter 2..................... 53 3 Generalized Inverse Matrices 55 3.1 Definition through Linear Transformations........... 55 3.2 General Properties........................ 59 3.2.1 Properties of generalized inverse matrices....... 59 ix
x CONTENTS 3.2.2 Representation of subspaces by generalized inverses.................... 61 3.2.3 Generalized inverses and linear equations....... 64 3.2.4 Generalized inverses of partitioned square matrices...................... 67 3.3 A Variety of Generalized Inverse Matrices........... 70 3.3.1 Reflexive generalized inverse matrices......... 71 3.3.2 Minimum norm generalized inverse matrices...... 73 3.3.3 Least squares generalized inverse matrices....... 76 3.3.4 The Moore-Penrose generalized inverse matrix.... 79 3.4 Exercises for Chapter 3..................... 85 4 Explicit Representations 87 4.1 Projection Matrices........................ 87 4.2 Decompositions of Projection Matrices............. 94 4.3 The Method of Least Squares.................. 98 4.4 Extended Definitions....................... 101 4.4.1 A generalized form of least squares g-inverse..... 103 4.4.2 A generalized form of minimum norm g-inverse.... 106 4.4.3 A generalized form of the Moore-Penrose inverse... 111 4.4.4 Optimal g-inverses.................... 118 4.5 Exercises for Chapter 4..................... 120 5 Singular Value Decomposition (SVD) 125 5.1 Definition through Linear Transformations........... 125 5.2 SVD and Projectors....................... 134 5.3 SVD and Generalized Inverse Matrices............. 138 5.4 Some Properties of Singular Values............... 140 5.5 Exercises for Chapter 5..................... 148 6 Various Applications 151 6.1 Linear Regression Analysis................... 151 6.1.1 The method of least squares and multiple regression analysis.................... 151 6.1.2 Multiple correlation coefficients and their partitions...................... 154 6.1.3 The Gauss-Markov model................ 156 6.2 Analysis of Variance....................... 161 6.2.1 One-way design...................... 161 6.2.2 Two-way design..................... 164
CONTENTS xi 6.2.3 Three-way design..................... 166 6.2.4 Cochran s theorem.................... 168 6.3 Multivariate Analysis....................... 171 6.3.1 Canonical correlation analysis.............. 172 6.3.2 Canonical discriminant analysis............. 178 6.3.3 Principal component analysis.............. 182 6.3.4 Distance and projection matrices............ 189 6.4 Linear Simultaneous Equations................. 195 6.4.1 QR decomposition by the Gram-Schmidt orthogonalization method................ 195 6.4.2 QR decomposition by the Householder transformation............... 197 6.4.3 Decomposition by projectors.............. 200 6.5 Exercises for Chapter 6..................... 201 7 Answers to Exercises 205 7.1 Chapter 1............................. 205 7.2 Chapter 2............................. 208 7.3 Chapter 3............................. 210 7.4 Chapter 4............................. 214 7.5 Chapter 5............................. 220 7.6 Chapter 6............................. 223 8 References 229 Index 233