INFORMATION THEORY AND STATISTICS

Transcription

1 INFORMATION THEORY AND STATISTICS Solomon Kullback DOVER PUBLICATIONS, INC. Mineola, New York

2 Contents 1 DEFINITION OF INFORMATION 1 Introduction 1 2 Definition 3 3 Divergence 6 4 Examples 7 5 Problems...'' PROPERTIES OF INFORMATION 1 Introduction 12 2 Additivity 12 3 Convexity 14 4 Invariance 18 5 Divergence 22 6 Fisher's information 26 7 Information and sufficiency 28 8 Problems 31 3 INEQUALITIES OF INFORMATION THEORY 1 Introduction 36 2 Minimum discrimination information 36 3 Sufficient statistics 43 4 Exponential family 45 5 Neighboring parameters 55 6 Efficiency 63 7 Problems 66 4 LIMITING PROPERTIES 1 Introduction 70 2 Limiting properties 70 3 Type I and type II errors 74 4 Problems 78

3 XU CONTENTS 5 INFORMATION STATISTICS 1 Estimate of/(*: 2) 81 2 Classification 83 3 Testing hypotheses 85 4 Discussion 94 5 Asymptotic properties 97 6 Estimate of /(*, 2) Problems MULTINOMIAL POPULATIONS 1 Introduction Background Conjugate distributions Ill 4 Single sample Basic problem Analysis of 7(*:2;OJV) Parametric case "One-sided" binomial hypothesis "One-sided" multinomial hypotheses Summary Illustrative values Two samples Basic problem "One-sided" hypothesis for the binomial r samples Basic problem Partition Parametric case Problems POISSON POPULATIONS 1 Background Conjugate distributions r samples Basic problem Partition "One-sided" hypothesis, single sample "One-sided" hypothesis, two samples Problems : CONTINGENCY TABLES 1 Introduction Two-way tables 155

4 CONTENTS Xlll 3 Three-way tables Independence of the three classifications Row classification independent of the other classifications Independence hypotheses Conditional independence Further analysis Homogeneity of two-way tables Conditional homogeneity Homogeneity Interaction Negative interaction Partitions Parametric case Symmetry Examples Problems MULTIVARIATE NORMAL POPULATIONS 1 Introduction...,.'' Components of information Canonical form Linear discriminant functions Equal covariance matrices Principal components Canonical correlation Covariance variates General case Problems THE LINEAR HYPOTHESIS 1 Introduction Background The linear hypothesis The minimum discrimination information statistic Subhypotheses Two-partition subhypothesis Three-partition subhypothesis Analysis of regression: one-way classification, k categories Two-partition subhypothesis One-way classification, k categories Carter's regression case Example Reparametrization Hypotheses not of full rank Partition 238

5 XIV CONTENTS 10 Analysis of regression, two-way classification Problems MULTIVARIATE ANALYSIS; THE MULTIVARIATE LINEAR HYPOTHESIS 1 Introduction Background The multivariate linear hypothesis Specification Linear discriminant function The minimum discrimination information statistic Subhypotheses Two-partition subhypothesis Three-partition subhypothesis Special cases Hotelling's generalized Student ratio (Hotelling's F 2 ) Centering Homogeneity of r samples r samples with covariance Test of regression ' Test of homogeneity of means and regression Test of homogeneity, assuming regression Canonical correlation Linear discriminant functions Homogeneity of r samples Canonical correlation Hotelling's generalized Student ratio (Hotelling's T 2 ) Examples Homogeneity of sample means Canonical correlation Subhypothesis Reparametrization Hypotheses not of full rank Partition Remark Problems MULTIVARIATE ANALYSIS: OTHER HYPOTHESES 1 Introduction Background Single sample Homogeneity of the sample The hypothesis that a A-variate normal population has a specified covariance matrix The hypothesis of independence Hypothesis on the correlation matrix 304

6 CONTENTS 3.5 Linear discriminant function Independence of sets of variates Independence and equality of variances Homogeneity of means Two samples Linear discriminant function r samples Homogeneity of covariance matrices Two samples Linear discriminant function r samples Correlation matrices Asymptotic distributions Homogeneity of covariance matrices Single sample The hypothesis of independence Roots of determinantal equations Stuart's test for homogeneity of the marginal distributions in a two-way classification A multivariate normal hypothesis The contingency table problem Problems LINEAR DISCRIMINANT FUNCTIONS 1 Introduction Iteration Example Remark Other linear discriminant functions Comparison of the various linear discriminant functions Problems 352 REFERENCES 353 TABLE I. Log, n and n log, n for values of n from 1 through XV TABLE II. F{p lt pi) - pi log + 2i log -, P% gs pi + 2i = 1 = Pi TABLE III. Noncentral x! 380 GLOSSARY...., 381 APPENDIX 389 INDEX 393