Advanced Series on Statistical Science & Applied Probability PRINCIPLES OF STATISTICAL INFERENCE from a Neo-Fisherian Perspective Luigi Pace Department of Statistics University ofudine, Italy Alessandra Salvan Department of Statistics University of Padua, Italy World Scientific Singapore'New Jersey London'Hong Kong
CONTENTS PREFACE LIST OF SYMBOLS xv xvii 1 STATISTICAL MODELS 1 1.1 The Theory of Statistical Inference 1 1.2 Four Paradigms of Inference 3 1.3 Model Specification 6 1.3.1 Levels of Specification 8 1.3.2 Notes on the Specification of a Parametric Model.... 9 1.4 Parametric Statistical Models and Likelihood 11 1.4.1 General Formulation of a Statistical Model 11 1.4.2 Likelihood and Related Quantities 13 1.4.3 Reparameterizations 18 1.5 Examples of Likelihood Functions 21 1.6 Bibliographic Note 27 1.7 Problems 29 2 DATA AND MODEL REDUCTION 33 2.1 Introduction 33 2.2 Statistics 34 2.3 Distribution Constant Statistics 35 2.4 Sufficient Statistics > 38 2.5 Completeness 44 2.6 Conditioning on Distribution Constant Statistics 46 2.7 Discussion and Examples 48 2.8 Ancillary Statistics 52 2.9 Relevant Subsets 56 2.10 Combinants and Pivotal Quantities 58 vii
viii CONTENTS 2.11 The Principle of Parameterization In variance 60 2.11.1 Consequences for Parameter Estimation 61 2.11.2 Consequences for Hypothesis Testing 63 2.11.3 Uninformative Prior Distributions and Invariance... 64 2.12 Bibliographic Note 66 2.13 Problems 67 3 SURVEY OF SOME BASIC CONCEPTS AND TECHNIQUES 71 3.1 Introduction 71 3.2 Moments, Cumulants and their Generating Functions 72 3.2.1 The Moment Generating Function 72 3.2.2 The Characteristic Function 74 3.2.3 Convergence Results 76 3.2.4 Generating Functions for Sums 77 3.2.5 The Cumulant Generating Function 78 3.2.6 Infinitely Divisible, Stable, Selfdecomposable Laws... 80 3.2.7 Multivariate Extensions 82 3.3 Basic Notions of Asymptotic Methods 84 3.3.1 Orders of Magnitude of Sequences 85 3.3.2 Convergence of Sums and Extremes 86 3.3.3 Orders in Probability: Examples 87 3.4 Likelihood and First-Order Asymptotic Theory 88 3.4.1 Null Asymptotic Distributions 89 3.4.2 Non-null Asymptotic Distributions 93 3.4.3 Robustness of Likelihood Methods 94 - - 3.5 Inference in the Frequency-Decision Paradigm 99 3.5.1 General Framework 99 3.5.2 Point Estimation 101 3.5.3 Testing Hypotheses 104 3.5.4 Confidence Regions": 109 3.5.5 Comments on the Relations with Likelihood Inference. 110 3.6 The Empirical Distribution Function 110 3.6.1 Basic Properties 110 3.6.2 Nonparametric Maximum Likelihood Estimate Ill 3.6.3 Statistical Functional 112 3.7 Bibliographic Note 116 3.8 Problems 118
CONTENTS ix 4 NUISANCE PARAMETERS AND PSEUDO-LIKELIHOODS 123 4.1 Nuisance Parameters 123 4.2 Data and Model Reduction with Nuisance Parameters 125 4.2.1 Lack of Information: No Nuisance Parameters 126 4.2.2 Lack of Information in the Presence of Nuisance Parameters 127 4.2.3 Weaker Concepts of Lack of Information with Nuisance Parameters 129 4.2.4 Nuisance Parameters and Reparameterizations 131 4.3 The Notion of a Pseudo-Likelihood 131 4.4 Marginal Likelihood: Examples 134 4.5 Conditional Likelihood: Examples 140 4.6 Profile Likelihood 143 4.7 Orthogonal Parameterization and Approximate Conditional Likelihood 148 4.8 Partial Likelihood 153 4.9 Quasi-Likelihood 157 4.10 Empirical Likelihood 159 4.11 Bibliographic Note 163 4.12 Problems 166 5 EXPONENTIAL FAMILIES 171 5.1 Introduction 171 5.2 Exponential Families of Order 1 172 5.3 Mean Value Mapping and Variance Function 176 5.4 Multiparameter Exponential Families 184 5.4.1 Definitions and Basic Results 184 5.4.2 Independence, Marginal and Conditional Distributions. 188 5.5 Sufficiency and Completeness 194 5.6 Likelihood and Exponential Families 196 5.7 Profile Likelihood and Mixed Parameterization 197 5.8 Procedures with Finite-Sample Optimality Properties 201 5.8.1 Testing Hypotheses: One-Parameter Case 202 5.8.2 Testing Hypotheses: Multiparameter Case 203 5.9 First-Order Asymptotic Theory 208 5.10 Curved Exponential Families 211 5.11 Bibliographic Note. 217
x CONTENTS 5.12 Problems 218 6 EXPONENTIAL DISPERSION FAMILIES AND GENERALIZED LINEAR MODELS 225 6.1 Introduction 225 6.2 Exponential Dispersion Families 226 6.3 Parameterization (/z, a 2 ) and Convolution Properties 230 6.4 Generalized Linear Models 234 6.4.1 Likelihood and Sufficiency 236 6.4.2 Quasi-Likelihood 242 6.4.3 Deviance Tests 244 6.5 Generalized Linear Models for Binary Data 245 6.6 Bibliographic Note 252 6.7 Problems 253 7 GROUP FAMILIES 259 7.1 Introduction 259 7.2 Groups of Transformations 263 7.3 Orbits and Maximal Invariants 266 7.4 Simple Group Families 272 7.5 Composite Group Families 276 7.6 Inference in Simple Group Families 278 7.6.1 Data and Model Reduction 278 7.6.2 Likelihood and Scale and Location Families 282 7.7 Inference in Composite Group Families 289 7.7.1 Data and Model Reduction 289 --- ' 7.7.2 Marginal Likelihood 293 7.8 Bibliographic Note 301 7.9 Problems 302 8 ASYMPTOTIC METHODS:" INTRODUCTION AND ELEMENTARY TECHNIQUES 309 8.1 Introduction 309 8.2 Evaluating the Accuracy of an Approximation 312 8.2.1 Berry-Esseen Inequality 312 8.2.2 Other Methods for Evaluating the Accuracy of a Normal Approximation for a Fixed n 314 8.2.3 Chi-Squared Approximations 318
CONTENTS xi 8.3 Improvements on First-Order Approximations: Historical Notes 320 8.4 Variance Stabilizing Transformations 324 8.5 Skewness Reducing Transformations 328 8.6 Bibliographic Note 331 8.7 Problems 332 9 ASYMPTOTIC EXPANSIONS FOR STATISTICS 335 9.1 Index Notation 335 9.2 Likelihood Quantities 342 9.2.1 Null Moments 343 9.2.2 Asymptotic Orders 344 9.3 Some Basic Tools 348 9.3.1 The Stochastic Taylor Formula 348 9.3.2 Inversion of Asymptotic Series 349 9.3.3 Laplace Expansion 352 9.4 Fundamental Asymptotic Expansions 357 9.4.1 Expansion of 9-9 357 9.4.2 Asymptotic Bias of 9 359 9.4.3 Variance and Other Cumulants of 9 361 9.4.4 Expansion of 1(9) - 1(9) 364 9.4.5 Expansion of E»(W) 365 9.4.6 Expansion of the Profile Score 367 9.5 Parameterization Invariance and Asymptotic Expansions... 369 9.5.1 Tensors 370 9.5.2 Invariance of the Expansion of 1(6) - 1(9) 371 9.5.3 Tensorial Behaviour of the Expansion of the Expected Value of the Profile Score 373 9.6 Bibliographic Note 374 9.7 Problems 376 10 ASYMPTOTIC EXPANSIONS FOR DISTRIBUTIONS 381 10.1 Generating Functions for a Standardized Sum 381 10.2 Hermite Polynomials 382 10.3 Edgeworth Expansion for Density Functions 384 10.4 Edgeworth Expansion for Distribution Functions 386 10.5 Anomalies in Edgeworth Approximations 392 10.6 Cornish-Fisher Expansion and Polynomial Normalizing Transformation 394
xii CONTENTS 10.7 Saddlepoint Expansion for Density Functions 398 10.8 Lugannani-Rice Expansion for Distribution Functions 404 10.9 Multivariate Edgeworth and Saddlepoint Expansions 410 10.9.1 Multivariate Edgeworth Expansion 411 10.9.2 Multivariate Saddlepoint Expansion 412 10.9.3 Mixed Expansion 413 10.10 Asymptotic Expansions for Conditional Distributions 415 10.10.1 Three Methods 415 10.10.2 Exponential Families and Approximate Conditional Inference 418 10.11 Bibliographic Note 424 10.12 Problems 426 11 LIKELIHOOD AND HIGHER-ORDER ASYMPTOTICS 431 11.1 Introduction 431 11.2 Approximation for the Distribution of 9: the p* Formula.... 432 11.3 Null Distribution of W 438 11.4 Effect of the Bartlett Correction 442 11.5 Modified Versions of Z and Z p 446 11.6 Modified Profile Likelihood 455 11.7 Bibliographic Note 462 11.8 Problems 463 A LAWS OF LARGE NUMBERS AND CENTRAL LIMIT THEOREMS 469 A.I Sumsofi.i.d. Random Variables 469 A.2 Sums of Independent Random Variables 471 A.3 Smooth Functions of Converging Sequences 472 A.4 Bibliographic Note 474 B ASYMPTOTIC DISTRIBUTION OF EXTREMES 475 B.I Basic Results ". 475 B.2 Bibliographic Note 476 B.3 Problems 477 C PARAMETRIC INFERENCE: BASIC TERMINOLOGY 479 C.I Point Estimation 479 C.2 Hypothesis Testing 479 C.3 Confidence Regions 480
CONTENTS xiii D RELATIONS BETWEEN THE FREQUENCY-DECISION AND FISHERIAN PARADIGMS 483 REFERENCES 489 AUTHOR INDEX 521 SUBJECT INDEX 527