Parameter Estimation and Hypothesis Testing in Linear Models

Transcription

1 Parameter Estimation and Hypothesis Testing in Linear Models

2 Springer-Verlag Berlin Heidelberg GmbH

3 Karl-Rudolf Koch Parameter Estimation and Hypothesis Testing in Linear Models Second, updated and enlarged Edition With 20 Figures " Springer

4 Prof. Dr.-Ing., Dr.-Ing. E. h. Karl-Rudolf Koch Institute of Theoretical Geodesy of the University of Bonn Nussallee 17 D Bonn Gennany This is a translation of the 3 n1 German Edition "Parameterschiitzung und Hypothesentests in!inearen Modellen" Ferd. Dummlers Verlag, 1997 ISBN CIP data applied for Die Deutsche Bibliothek - CIP-Einheitsaufnahme Koch, Karl-Rudolf: Parameter estimation and hypothesis testing in linear models I Karl-Rudolf Koch. - 2., upd. and enl. ed. Einheitssacht.: Parameterschlltzung und Hypothesentests in Iinearen Modellen <eng!.> I S B N I S B N ( e B o o k ) D O I / This work is subject to copyright. All rights are reserved, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilm or in other ways, and storage in data banks. Duplication of this publication or parts thereof is permitted only under the provisions of the German Copyright Law of September 9, 1965, in its current version, and permission for use must always be obtained from Springer-Verlag Berlin Heidelberg GmbH. Copyright Law. Violations are liable for prosecution act under German Springer-Verlag Berlin Heidelberg 1999 Originally published by Springer-Verlag Berlin Heidelberg New Yark m 1999 Tht use of general descriptive names, registered names, trademarks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. Typesetting: Camera ready by author Cover design: E. Kirchner, Heidelberg SPIN: Printed on acid -free paper

5 Preface to the Second Edition The necessity to publish the second edition of this book arose when its third German edition had just been published. This second English edition is therefore a translation of the third German edition of Parameter Estimation and Hypothesis Testing in Linear Models, published in It differs from the first English edition by the addition of a new chapter on robust estimation of parameters and the deletion of the section on discriminant analysis, which has been more completely dealt with by the author in the book Bayesian Inference with Geodetic Applications, Springer-Verlag, Berlin Heidelberg New York, Smaller additions and deletions have been incorporated, to improve the text, to point out new developments or to eliminate errors which became apparent. A few examples have been also added. I thank Springer-Verlag for publishing this second edition and for the assistance in checking the translation, although the responsibility of errors remains with the author. I also want to express my thanks to Mrs. Ingrid Wahl and to Mrs. Heidemarlen Westhiiuser who prepared the second edition. Bonn, January 1999 Karl-Rudolf Koch

6 Preface to the First Edition This book is a translation with slight modifications and additions of the second German edition of Parameter Estimation and Hypothesis Testing in Linear Models, published in As already mentioned in the preface to the first and second German edition, the book intends to give a self-contained presentation of the multidimensional methods of estimating parameters, testing hypotheses and estimating intervals. In addition to these topics, vector and matrix algebra as well as probability theory are therefore presented. To deal with all this material, a condensed writing style is necessary which, however, is interspersed with many examples. The book is addressed especially to engineers, to those still studying and to those applying statistical methods to practical problems. But also members of other professions should find advice in this book helpful when dealing with statistical inference. I am indebted to Springer-Verlag for publishing the English translation. Assistance of Springer-Verlag in checking the translation is gratefully acknowledged, although the responsibility of errors remains with the author. I would like to thank colleagues and students at the Department of Surveying Engineering of the University of Calgary for eliminating errors in the manuscript, which was used as the basis for a course in statistical inference in the autumn of I also would like to express my gratitude to Mrs. Karin Bauer who prepared the copy of the book. Bonn, December 1987 Karl-Rudolf Koch

7 Preface to the Third (German) Edition For the third edition, which became necessary, this book has again been completely revised. A section on robust estimation of parameters has been added and the chapter on discriminant analysis has been deleted, since this topic was dealt with more completely by the author in the book Bayesian Inference with Geodetic Applications, Springer-Verlag, Berlin Heidelberg New York, Some additional examples have been included, to give a clearer idea of the presentation. Smaller additions or deletions have been incorporated to improve the text, to point out newer developments or to remove errors which became apparent. I want to thank all students of our university, who made suggestions to improve the book. I also express my thanks to the co-workers of the Institute of Theoretical Geodesy, who assisted in the new edition. My special thanks go to Mr. Robert Blinken, Dipl.-Ing., who made several suggestions for the new edition, and to Mrs. Ingrid Wahl for writing and designing the text and the figures of this third edition. Finally I would like to mention the good cooperation with the publisher. Bonn, March 1997 Karl-Rudolf Koch

8 Preface to the Second (German) Edition The necessity to come out with a second edition of this book was used to revise it completely. Again, the endeavor was made to give a self-contained presentation of the methods of estimating unknown parameters, of testing hypotheses and of interval estimation and to add necessary knowledge on linear algebra and probability theory. The structure of the book therefore has not been changed. But several examples have been added, in order to loosen up the generally compact presentation of the methods. Where the text had to be improved or where new develoments had to be pointed out, passages have been added or deleted and errors, which became apparent, were removed. More extensive additions to be mentioned are the direct computation of the Cholesky factorization, the properties of a special symmetrical reflexive generalized inverse, the non-central Wishart distribution for the matrices of quadratic forms, the interpretation of the constraints to remove the rank deficiency in a free network, the datum transformation, the Gauss-Markoff model for outliers in the observations, the simplified iterated estimates of the variance and covariance components, the less sensitive tests of hypotheses in the univariate and multivariate model of estimating parameters, the computation of the probability of Type II errors for tests in the multivariate model, the confidence region for several linear functions of the unknown parameters, the derivation of the test for outliers as a special case of the test of a general hypothesis in the univariate model, the measures of reliability and their interpretation. I want to thank all the colleagues and students, who have contributed to this edition by their suggestions for improvements. I also convey my thanks to my co-workers in the Institute of Theoretical Geodesy, who have assisted in the preparation of the new edition. In particular, I thank Mrs. Stefanie Schulte, Dipl.-Ing., for her suggestions to this edition and Mrs. Karin Bauer, who typed most of the copy. Finally, I would like to mention the cooperation I received from my publishing house. Bonn, July 1986 Karl-Rudolf Koch

9 Preface to the First (German) Edition This book shall explain and substantiate the methods for estimating parameters, for testing hypotheses and for interval estimation. It therefore deals with the statistical inference for parameters. In order to address a large group of readers, representing different training and special fields, the book was planned such that no requirements are necessary for its understanding except for some basic acquaintance with the analysis. The knowledge which is needed from linear algebra and probability theory is presented in the first two chapters of the book. As it is the aim to discuss the inference for parameters with respect to the application, more emphasis was placed on conveying the understanding for the methods to be discussed, than on presenting general definitions and proofs. Linear models are applied for estimating the parameters and testing the hypotheses. This, however, does not mean a restriction of the generality, since under assumptions, which in general are readily fulfilled, the non-linear models can be transformed into linear models, as shall be shown. On the other hand, the linear models have the advantage that the methods of linear algebra can be applied. The definitions and theorems, which are needed, are therefore presented in Chapter 1. The theorems are proved with only a few exceptions. Chapter 1 also contains the generalized inverses, which are used for estimating parameters in models which are not of full rank, and presents the projections, which serve the geometrical interpretation of the estimation. Besides the estimation of parameters, which is applied to univariate and multivariate models, interval estimation, hypothesis testing and discriminant analysis are also treated. Hence, Chapter 2 deals with probability theory. The random variables are defined and the most important univariate and multivariate distributions are derived together with the test distributions for the univariate and multivariate models of estimating parameters. For each distribution a method is given to numerically compute its cumulative distribution function. Thus, printing tables of the distribution function could be dispensed with. Chapter 3 deals with a univariate and multivariate model to estimate parameters, which are defined as fixed quantities. Fixed parameters, together with random parameters, are estimated in the mixed model. The analysis of variance and the estimation of variance and covariance components are also presented in Chapter 3. Chapter 4 is devoted to the testing of hypotheses, to interval estimation and to testing for outliers, while Chapter 5 finally gives a short review of discriminant analysis. If theorems, which have been already presented, are needed for proofs,

10 XIV Preface they will be referred to by means of the appropriate numbers. Hence, Chapter 1 and also Chapter 2 do not need to be read before the remaining sections, since on the basis of the references to the theorems, the lacking knowledge can be found easily in the first two chapters. If a page number is given with a book cited as a reference, it only denotes the first page of interest; the following pages, which might also be of importance, are not pointed out. I want to thank all the co-workers in the Institute of Theoretical Geodesy, who contributed to the publication of this book. In particular, I thank Mr. Burkhard Schaffrin, Dipl.-Math., Dipl.-Ing., who made many suggestions. Finally, I would like to mention the cooperation I received from my publishing house during the preparation of the book. Bonn, September 1979 Karl-Rudolf Koch

11 Contents Introduction 1 1 Vector and Matrix Algebra 1.1 Sets and Fields Notion of Sets Composition of Sets Relations Field of Real Numbers. 1.2 Vector Algebra Definition of Vectors and Vector Space Linear Dependence and Basis of a Vector Space Inner Product and Euclidian Space Orthogonal Subspaces Matrices Matrix Definition and Matrix Composition Rank of a Matrix Computation of Inverse Matrices Matrix Identities Column Space and Null Space of a Matrix Determinants Trace of a Matrix and Vector Representation 1.4 Quadratic Forms Transformations Eigenvalues and Eigenvectors Definite Matrices Generalized Inverses Right Inverse and Left Inverse Idempotent Matrices Generalized Inverse, Reflexive Generalized Inverse and Pseudoinverse Systems of Linear Equations Generalized Inverses of Symmetrical Matrices Properties of the Pseudoinverse and of a Special Symmetrical Reflexive Generalized Inverse Projections General Projections Orthogonal Projections Differentiation and Integration of Vectors and Matrices

12 XVI Contents Extrema of Functions Derivatives of Special Functions Integration and Transformation of Variables Probability Theory 2.1 Probability Introduction Random Events Axioms of Probability Conditional Probability and Bayes' Formula Independent Events 2.2 Random Variable Definition Distribution Function Discrete and Continuous Random Variable Binomial Distribution and Poisson Distribution Multidimensional Continuous Random Variable Marginal Distribution Conditional Distribution Independent Random Variables Transformation of Variables Expected Values and Moments of Random Variables Expectation Multivariate Moments Covariance Matrix and Covariances of Random Vectors Moment Generating Function 2.4 Univariate Distributions Normal Distribution Derivation of the Normal Distribution as Distribution of Observational Errors Gamma Distribution Derivation of the Gamma Distribution as Waiting-Time Distribution Beta Distribution Multivariate Normal Distribution Definition and Derivation Moment Generating Function for the Normal Distribution Marginal Distribution and Conditional Distribution Independence of Normally Distributed Random Variables Linear Functions of Normally Distributed Random Variables Sum of Normally Distributed Random Variables Test Distributions for Univariate Models

13 Contents XVII X2-Distribution Non-Central x2-distribution F-Distribution Non-Central F-Distribution t-distribution.... Quadratic Forms Expected Value and Covariance Distribution of the Quadratic Form Independence of Two Quadratic Forms Independence of a Linear and a Quadratic Form Test Distributions for Multivariate Models Wishart Distribution Derivation of the Wishart Distribution Distribution of the Sum of Wishart Matrices Distribution of the Transformed Wishart Matrix Distribution of Matrices of Quadratic Forms and Independence of Wishart Matrices Distribution of the Ratio of the Determinants of Two Wishart Matrices Distribution of Special Functions of Wishart Matrices Parameter Estimation in Linear Models 3.1 Methods of Estimating Parameters Point Estimation Best Unbiased Estimation Method of Least Squares Maximum-Likelihood Method. 3.2 Gauss-Markoff Model Definition and Linearization Best Linear Unbiased Estimation Method of Least Squares Maximum-Likelihood Method Unbiased Estimation of the Variance of Unit Weight Numerical Computation of the Estimates and Their Covariances Gauss-Markoff Model with Constraints Recursive Parameter Estimation Deviations From the Model Gauss-Markoff Model not of Full Rank Method of Least Squares and Maximum-Likelihood Method Estimable Functions Projected Parameters as Estimable Functions Gauss-Markoff Model not of Full Rank with Constraints

14 XVIII Contents 3.4 Special Gauss-Markoff Models Polynomial Model Analysis of Variance Parameter Estimation for the Analysis of Variance by a Symmetrical Reflexive Generalized Inverse Analysis of Covariance Gauss-Markoff Model for Outliers in the Observations Generalized Linear Models Regression Model Mixed Model Best Linear Unbiased Estimation in the Mixed Model Method of Least Squares and Maximum-Likelihood Method for the Mixed Model Model of the Adjustment with Condition Equations Prediction and Filtering.... Estimation of Variance and Covariance Components Best Invariant Quadratic Unbiased Estimation Locally Best Estimation Iterated Estimates Best Unbiased Estimation of the Variance of Unit Weight.... Multivariate Parameter Estimation Multivariate Gauss-Markoff Model Estimation of the Vectors of Parameters Estimation of the Covariance Matrix Numerical Computation of the Estimates and Incomplete Multivariate Models Special Model for Estimating Covariance Matrices and Estimation of Covariances for Stochastic Processes Multivariate Model with Constraints Robust Parameter Estimation Choice of the Score Function Robust M-Estimation M-Estimation of Huber Lp-Norm Estimation.. Leverage Points..... Modified M-Estimation of Huber Method of Rousseeuw Hypothesis Testing, Interval Estimation and Test for Outliers Distributions Based on Normally Distributed Observations Distributions of Functions of the Residuals in the Univariate Model

15 Contents XIX Distributions of the Covariance Matrices Estimated in the Multivariate Model Test of Hypotheses Methods for Hypothesis Testing and Likelihood Ratio Test Test of a General Hypothesis in the Univariate Gauss-Markoff Model Special Hypotheses Hypothesis Testing for the Variance of Unit Weight Test of a General Hypothesis in the Multivariate Gauss-Markoff Model Hypothesis of the Identity of a Covariance Matrix with a Given Matrix Interval Estimation Confidence Intervals Confidence Intervals for Parameters, for Linear Functions of the Parameters and Confidence Hyperellipsoids Confidence Interval for the Variance of Unit Weight Test for Outliers Hypothesis Testing for Outliers Test for One Outlier Percentage Points of the Test for Outliers Measures of Reliability References 311 Index