Mining Imperfect Data

Transcription

1 Mining Imperfect Data Dealing with Contamination and Incomplete Records Ronald K. Pearson ProSanos Corporation Harrisburg, Pennsylvania and Thomas Jefferson University Philadelphia, Pennsylvania siam. Society for Industrial and Applied Mathematics Philadelphia

2 Preface ix 1 Introduction Data anomalies Outliers Boxplots: A useful comparison tool Missing data Misalignments Unexpected structure Data anomalies need not be bad Materials with anomalous properties Product design: Looking for "good" anomalies "Niches" in business data records Conversely, data anomalies can be very bad The CAMDA'02 normal mouse dataset The influence of outliers on kurtosis Dealing with data anomalies Outlier-resistant analysis procedures Outlier detection procedures Preprocessing for anomaly detection GSA Organization of this book 31 2 Imperfect Datasets: Character, Consequences, and Causes Outliers Univariate outliers Multivariate outliers Time-series outliers Consequences of outliers Moments versus order statistics The effect of outliers on volcano plots Product-moment correlations Spearman rank correlations Sources of data anomalies Gross measurement errors and outliers 52 v

3 2.3.2 Misalignments and Software errors Constraints and hidden symmetries Missing data Nonignorable missing data and sampling bias Special codes, nulls, and disguises Idempotent data transformations , Missing data from file merging 66 Univariate Outlier Detection Univariate outlier modeis Three outlier detection procedures The 3(7 edit rule The Hampel identifier Quartile-based detection and boxplots Performance comparison Formulation of the case study The uncontaminated reference case Results for 1% contamination Results for 5% contamination Results for 15% contamination Brief summary of the results Application to real datasets The catalyst dataset The flow rate dataset The industrial pressure dataset 90 Data Pretreatment Noninformative variables Classes of noninformative variables A microarray dataset Noise variables Occam's hatchet and Omission bias Handling missing data Omission of missing values Single imputation strategies Multiple imputation strategies Unmeasured and unmeasurable variables Cleaning time-series Thenatureof theproblem Data-cleaning filters The center-weighted median filter The Hampel filter Multivariate outlier detection Visual inspection Covariance-based detection 127

4 VII Regression-based detection Depth-based detection Preliminary analyses and auxiliary knowledge What Is a "Good" Data Characterization? A motivating example Characterization via functional equations A brief introduction to functional equations Homogeneity and its extensions Location-invariance and related conditions Outlier detection procedures Quasi-linear means Results for positive-breakdown estimators Characterization via inequalities Inequalities as aids to interpretation Relations between data characterizations Bounds on means and Standard deviations Inequalities as uncertainty descriptions Coda: What is a "good" data characterization? GSA The GSA metaheuristic Thenotion ofexchangeability Choosing scenarios Some general guidelines Managing subscenarios Experimental design and scenario selection Sampling schemes Selecting a descriptor d() Displaying and interpreting the results Normal Q-Q plots Direct comparisons across scenarios The model approximation case study Extensions of the basic GSA framework Iterative analysis procedures Multivariable descriptors Sampling Schemes for a Fixed Dataset Four general strategies Strategy 1: Random selection Correlation and overlap Strategy 2: Subset deletion Strategy 3: Comparisons Strategy 4: Partially systematic sampling Random selection examples Variability of kurtosis estimates The industrial pressure datasets 235

5 viii Contents 7.3 Subset deletion examples The storage tank dataset Dynamic correlation analysis Comparison-based examples Correlation-destroyingpermutations Rank-based dynamic analysis Two systematic selection examples Movipg- window data characterizations The Michigan lung Cancer dataset Concluding Remarks and Open Questions Analyzing large datasets Prior knowledge, auxiliary data, and assumptions Some open questions How prevalent are different types of data anomalies? How should outliers be modelled? How should asymmetry be handled? How should misalignments be detected? 283 Bibliography 287 Index 301