Nonparametric Statistics with Applications to Science and Engineering

Size: px

Start display at page:

Download "Nonparametric Statistics with Applications to Science and Engineering"

Jesse Shepherd
5 years ago
Views:

1 Nonparametric Statistics with Applications to Science and Engineering

2 THE WILEY BICENTENNIAL-KNOWLEDGE FOR GENERATIONS Gach generation has its unique needs and aspirations. When Charles Wiley first opened his small printing shop in lower Manhattan in 1807, it was a generation of boundless potential searching for an identity. And we were there, helping to define a new American literary tradition. Over half a century later, in the midst of the Second Industrial Revolution, it was a generation focused on building the future. Once again, we were there, supplying the critical scientific, technical, and engineering knowledge that helped frame the world. Throughout the 20th Century, and into the new millennium, nations began to reach out beyond their own borders and a new international community was born. Wiley was there, expanding its operations around the world to enable a global exchange of ideas, opinions, and know-how. For 200 years, Wiley has been an integral part of each generation s journey, enabling the flow of information and understanding necessary to meet their needs and fulfill their aspirations. Today, bold new technologies are changing the way we live and learn. Wiley will be there, providing you the must-have knowledge you need to imagine new worlds, new possibilities, and new opportunities. Generations come and go, but you can always count on Wiley to provide you the knowledge you need, when and where you need it! 4 WILLIAM J. PESCE PRESIDENT AND CHIEF EXECUTIVE OmCER PETER BOOTH WlLEV CHAIRMAN OF THE BOARD

3 Nonparametric Statistics with Applications to Science and Engineering Paul H. Kvam Georgia Institute of Technology The H. hlilton Stewart School oflndustrial and Systems Engineering Atlanta. GA Brani Vidakovic Georgia Institute of Technology and Emory University School of Medicine The Wallace H. Coulter Department of Biomedical Engineering Atlanta, GA BICENTENNIAL BICENTENNIAL WILEY-INTERSCIENCE A John Wiley & Sons, Inc., Publication

4 Copyright by John Wiley & Sons, Inc. All rights reserved. Published by John Wiley & Sons, Inc., Hoboken, New Jersey Published simultaneously in Canada. No part of this publication may be reproduced. stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) , fax (978) , or on the web at Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 11 1 River Street, Hoboken, NJ 07030, (201) , fax (201) , or online at Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) , outside the United States at (317) or fax (317) Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic format. For information about Wiley products, visit our web site at Wiley Bicentennial Logo: Richard J. Pacific0 Library of Congress Cataloging-in-Publication Data is available. ISBN Printed in the United States of America I

5 Contents Preface xi 1 Introduction 1.1 Efficiency of Nonparametric Methods 1.2 Overconfidence Bias 1.3 Computing with MATLAB 1.4 Exercises 2 Probability Basics 2.1 Helpful Functions Events, Probabilities and Random Variables Numerical Characteristics of Random Variables Discrete Distributions 2.5 Continuous Distributions 2.6 Mixture Distributions 2.7 Exponential Family of Distributions 2.8 Stochastic Inequalities 2.9 Convergence of Random Variables V

6 vi CONTENTS 2.10 Exercises Statistics Basics 3.1 Estimation 3.2 Empirical Distribution Function 3.3 Statistical Tests 3.4 Exercises Bayesian Statistics 4.1 The Bayesian Paradigm 4.2 Ingredients for Bayesian Inference 4.3 Bayesian Computation and Use of WinBUGS 4.4 Exercises Order Statistics 5.1 Joint Distributions of Order Statistics 5.2 Sample Quantiles 5.3 Tolerance Intervals 5.4 Asymptotic Distributions of Order Statistics 5.5 Extreme Value Theory 5.6 Ranked Set Sampling 5.7 Exercises Goodness of Fit 6.1 Kolmogorov-Smirnov Test Statistic Smirnov Test to Compare Two Distributions Specialized Tests 6.4 Probability Plotting 6.5 Runs Test 6.6 AIeta Analysis 6.7 Exercises

7 CONTENTS vii Rank Tests 7.1 Properties of Ranks 7.2 Sign Test 7.3 Spearman Coefficient of Rank Correlation 7.4 Wilcoxon Signed Rank Test 7.5 Wilcoxon (Two-Sample) Sum Rank Test 7.6 Mann-Whitney U Test 7.7 Test of Variances 7.8 Exercises Designed Experiments 8.1 Kruskal-Wallis Test 8.2 Friedman Test 8.3 Variance Test for Several Populations 8.4 Exercises Categorical Data 9.1 Chi-square and Goodness-of-Fit 9.2 Contingency Tables 9.3 Fisher Exact Test 9.4 MCNemar Test 9.5 Cochran s Test 9.6 Mantel-Haenszel Test 9.7 CLT for Multinomial Probabilities 9.8 Simpson s Paradox 9.9 Exercises Estimating Distribution Functions 10.1 Introduction 10.2 Nonparametric Maximum Likelihood

8 viii CONTENTS 10.3 Kaplan-Meier Estimator 10.4 Confidence Interval for F 10.5 Plug-in Principle 10.6 Semi- P ar ame tric Inference 10.7 Empirical Processes 10.8 Empirical Likelihood 10.9 Exercises Density Estimation 11.1 Histogram 11.2 Kernel and Bandwidth 11.3 Exercises Beyond Linear Regression 12.1 Least Squares Regression 12.2 Rank Regression 12.3 Robust Regression 12.4 Isotonic Regression 12.5 Generalized Linear Models 12.6 Exercises Curve Fitting Techniques 13.1 Kernel Estimators 13.2 Nearest Neighbor Methods 13.3 Variance Estimation 13.4 Splines 13.5 Summary 13.6 Exercises Wavelets 14.1 Introduction to Wavelets

9 CONTENTS ;x 14.2 How Do the Wavelets Work? 14.3 Wavelet Shrinkage 14.4 Exercises Bootstrap 15.1 Bootstrap Sampling 15.2 Nonparametric Bootstrap 15.3 Bias Correction for Nonparametric Intervals 15.4 The Jackknife 15.5 Bayesian Bootstrap 15.6 Permutation Tests 15.7 More on the Bootstrap 15.8 Exercises EM Algorithm 16.1 Fisher s Example 16.2 Mixtures 16.3 EM and Order Statistics 16.4 MAP via EM 16.5 Infection Pattern Estimation 16.6 Exercises Statistical Learning 17.1 Discriminant Analysis 17.2 Linear Classification Models 17.3 Nearest Neighbor Classification 17.4 Neural Networks 17.5 Binary Classification Trees 17.6 Exercises Nonparametric Bayes 349

10 x CONTENTS 18.1 Dirichlet Processes 18.2 Bayesian Categorical Models 18.3 Infinitely Dimensional Problems 18.4 Exercises A MATLAB A.l Using MATLAB A.2 Matrix Operations A.3 Creating Functions in MATLAB A.4 Importing and Exporting Data A.5 Data Visualization A.6 Statistics B WinBUGS B.l Using WinBUGS B.2 Built-in Functions hiatlab Index Author Index Subject Index 413

11 Preface Danger lies not in what we don't know-. but in what we think we know that just ain't so. Mark Twain ( ) As Prefaces usually start. the author(s) explain why they wrote the book in the first place ~ and we will follow this tradition. Both of us taught the graduate course on nonparametric statistics at the School of Industrial and Systems Engineering at Georgia Tech (ISyE 6404) several times. The audience was always versatile: PhD students in Engineering Statistics. Electrical Engineering, Management, Logistics, Physics. to list a few. While comprising a non homogeneous group. all of the students had solid mathematical, programming and statistical training needed to benefit from the course. Given such a nonstandard class. the text selection was all but easy. There are plenty of excellent monographs/texts dealing with nonparametric statistics, such as the encyclopedic book by Hollander and Wolfe. Nonparametrac Statzstzcal Methods. or the excellent evergreen book by Conover. Practacal Nonparametrzc Statastacs, for example. We used as a text the 3rd edition of Conover's book, which is mainly concerned with what most of us think of as traditional nonparametric statistics: proportions. ranks. categorical data. goodness of fit. and so on, with the understanding that the text would be supplemented by the instructor's handouts. Both of us ended up supplying an increasing number of handouts every year, for units such as density and function estimation. wavelets. Bayesian approaches to nonparametric problems. the EM algorithm. splines, machine learning, and other arguably XI

xi/ PREFACE modern nonparametric topics. About a year ago. we decided to merge the handouts and fill the gaps. There are several novelties this book provides.

12 xi/ PREFACE modern nonparametric topics. About a year ago. we decided to merge the handouts and fill the gaps. There are several novelties this book provides. We decided to intertwine informal comments that might be amusing. but tried to have a good balance. One could easily get carried away and produce a preface similar to that of celebrated Barlow and Proschan's, Statastacal Theory of Relaabalzty and Lzfe Testang: Probabzlaty Models, who acknowledge greedy spouses and obnoxious children as an impetus to their book writing. In this spirit. we featured photos and sometimes biographic details of statisticians who made fundamental contributions to the field of nonparametric statistics, such as Karl Pearson. Nathan hfantel, Brad Efron, and Baron Von Munchausen. Computing. Another specificity is the choice of computing support. The book is integrated with MATLAB@ and for many procedures covered in this book. hfatlab's m-files or their core parts are featured. The choice of software was natural: engineers. scientists, and increasingly statisticians are communicating in the "AlATLAB language." This language is, for example, taught at Georgia Tech in a core computing course that every freshman engineering student takes. and almost everybody around us "speaks MATLAB." The book's website: contains most of the m-files and programming supplements easy to trace and download. For Bayesian calculation we used N-inBUGS, a free software from Cambridge's Biostatistics Research Unit. Both MATLAB and WinBUGS are briefly covered in two appendices for readers less familiar with them. Outline of Chapters. For a typical graduate student to cover the full breadth of this textbook, two semesters would be required. For a one-semester course. the instructor should necessarily cover Chapters 1-3, 5-9 to start. Depending on the scope of the class, the last part of the course can include different chapter selections. Chapters 2-4 contain important background material the student needs to understand in order to effectively learn and apply the methods taught in a nonparametric analysis course. Because the ranks of observations have special importance in a nonparametric analysis, Chapter 5 presents basic results for order statistics and includes statistical methods to create tolerance intervals. Traditional topics in estimation and testing are presented in Chapters 7-10 and should receive emphasis even to students who are most curious about advanced topics such as density estimation (Chapter 11). curve-fitting (Chapter 13) arid wavelets (Chapter 14). These topics include a core of rank tests that are analogous to common parametric procedures (e.g.. t-tests, analysis of variance). Basic methods of categorical data analysis are contained in Chapter 9. Al-

13 PREFACE xi;; though most students in the biological sciences are exposed to a wide variety of statistical methods for categorical data. engineering students and other students in the physical sciences typically receive less schooling in this quintessential branch of statistics. Topics include methods based on tabled data. chisquare tests and the introduction of general linear models. Also included in the first part of the book is the topic of "goodness of fit" (Chapter 6), which refers to testing data not in terms of some unknown parameters, but the unknown distribution that generated it. In a way. goodness of fit represents an interface between distribution-free methods and traditional parametric methods of inference, and both analytical and graphical procedures are presented. Chapter 10 presents the nonparametric alternative to maximum likelihood estimation and likelihood ratio based confidence intervals. The term "regression" is familiar from your previous course that introduced you to statistical methods. Konparametric regression provides an alternative method of analysis that requires fewer assumptions of the response variable. In Chapter 12 we use the regression platform to introduce other important topics that build on linear regression. including isotonic (constrained) regression, robust regression and generalized linear models. In Chapter 13. we introduce more general curve fitting methods. Regression models based on wavelets (Chapter 14) are presented in a separate chapter. In the latter part of the book. emphasis is placed on nonparametric procedures that are becoming more relevant to engineering researchers and practitioners. Beyond the conspicuous rank tests, this text includes many of the newest nonparametric tools available to experimenters for data analysis. Chapter 17 introduces fundamental topics of statistical learning as a basis for data mining and pattern recognition. and includes discriminant analysis. nearest-neighbor classifiers, neural networks and binary classification trees. Computational tools needed for nonparametric analysis include bootstrap resampling (Chapter 15) and the ELI Algorithm (Chapter 16). Bootstrap methods. in particular. have become indispensable for uncertainty analysis with large data sets and elaborate stochastic models. The textbook also unabashedly includes a review of Bayesian statistics and an overview of nonparametric Bayesian estimation. If you are familiar with Bayesian methods. you might wonder what role they play in nonparametric statistics. Admittedly. the connection is not obvious, but in fact nonparametric Bayesian methods (Chapter 18) represent an important set of tools for complicated problems in statistical modeling and learning, where many of the models are nonparametric in nature. The book is intended both as a reference text and a text for a graduate course. \Ye hope the reader will find this book useful. All comments, suggestions. updates, and critiques will be appreciated.

14 xiv PREFACE Acknowledgments. Before anyone else we would like to thank our wives, Lori Kvam and Draga Vidakovic. and our families. Reasons they tolerated our disorderly conduct during the writing of this book are beyond us, but we love them for it. We are especially grateful to Bin Shi, who supported our use of MATLAB and wrote helpful coding and text for the Appendix A. We are grateful to MathWorks Statistics team. especially to Tom Lane who suggested numerous improvements and updates in that appendix. Several individuals have helped to improve on the primitive drafts of this book. including Saroch Boonsiripant, Lulu Kang. Hee Young Kim. Jongphil Kim, Seoung Bum Kim, Kichun Lee, and Andrew Smith. Finally, we thank Wiley's team. Melissa Yanuzzi, Jacqueline Palmieri and Steve Quigley, for their kind assistance. PAUL H. KVAM School of Industrial and System Engineering Georgia Institute of Technology BRAN VIDAKOVIC School of Biomedical Engineering Georgia Institute of Technology

AN INTRODUCTION TO PROBABILITY AND STATISTICS

AN INTRODUCTION TO PROBABILITY AND STATISTICS WILEY SERIES IN PROBABILITY AND STATISTICS Established by WALTER A. SHEWHART and SAMUEL S. WILKS Editors: David J. Balding, Noel A. C. Cressie, Garrett M.