Non-Parametric Statistical Diagnosis

Non-Parametric Statistical Diagnosis

Mathematics and Its Applications Managing Editor: M. HAZEWINKEL Centre tor Mathematics and Computer Science, Amsterdam, The Netherlands Volume 509

Non-Parametric Statistical Diagnosis Problems and Methods by B.E. Brodsky State University, Higher School of Economics, Moscow, Russia and B.S. Darkhovsky Institute for Systems Analysis, Russian Academy of Sciences, Moscow, Russia Springer-Science+Business Media, B.V.

A c.i.p. Catalogue record for this book is available from the Library of Congress. ISBN 978-90-481-5465-4 ISBN 978-94-015-9530-8 (ebook) DOI 10.1007/978-94-015-9530-8 Printed on acid~free paper All Rights Reserved 2000 Springer Science+Business Media Dordrecht Originally published by Kluwer Academic Publishers in 2000. Softcover reprint of the hardcover 1 st edition 2000 No part of the material protected by this copyright notice may be reproduced or utilized in any form or by any means, electronic or mechanical, inciuding photocopying, recording or by any information storage and retrieval system, without written permission from the copyright owner.

Table of Contents Preface... ix Part 1. Theory... 1 Chapter 1 Preliminary considerations... 3 1.1 Necessary results from the theory of random processes... 3 1.1.1 Mixing conditions... 3 1.1.2 Some inequalities. Convergence of random variables... 8 1.1.3 Ergodicity of random sequences... 11 1.1.4 The Wiener process and some functionals related to it... 12 1.1.5 Weak convergence... 16 1.1.6 Probabilistic inequalities for maximum of sums of random variables... 29 1.1.7 Martingales. Markov's moments. Wald's identity... 35 1.2 Necessary results from the theory of random fields... 37 1.3 Necessary results from optimization theory... 49 1.3.1 Optimality conditions for smooth finite-dimensional problems of mathematical programming... 49 1.3.2 Some properties of functionals of the maximum type... 53 1.4 Main ideas of the nonparametric approach to the problems of statistical diagnosis... 57 1.4.1 Random processes... 58 1.4.2 Random fields... 63 1.5 Main assumptions... 66 1.5.1 Random processes... 67 1.5.2 Random fields... 77

Vi CONTENTS Chapter 2 State of the art review... 83 2.1 Introduction... 83 2.2 Retrospective methods of statistical diagnosis... 85 2.2.1 Change-point problems... 85 2.2.2 'Contamination' problems... 107 2.3 Sequential methods of statistical diagnosis... 109 2.4 Problems of statistical diagnosis for random fields... 122 Chapter 3 Retrospective methods of statistical diagnosis for random sequences: change-point problems... 127 3.1 Statement of the general retrospective change-point problem for a random sequence. Variants of problems under consideration... 127 3.2 A single abrupt change-point... 130 3.2.1 An abrupt change of the function cp... 130 3.2.2 An abrupt change of coefficients of the linear functional regression... 134 3.2.3 An abrupt change of (k + l)-th derivative of the function cp... 145 3.3 Multiple abrupt change-points... 146 3.3.1 Abrupt changes of the function cp... 146 3.3.2 Abrupt changes of coefficients of the linear functional regression... 151 3.4 Gradual change-point problems... 155 3.4.1 A single gradual change of the function i.p...... 155 3.5 Asymptotic analysis of change-point estimates... 169 3.6 Apriori low bounds in retrospective change-point problems... 182 3.6.1 The Rao-Cramer type inequality... 182 3.6.2 Asymptotic low bound in the single change-point problem... 186 3.6.3 Asymptotic low bound in the multiple change-point problem... 191 3.6.4 Asymptotic low bound in the single change-point problem for regression relationships... 195 3.7 Conclusion... 197 Bibliographical comments... 200 Chapter 4 Retrospective methods of statistical diagnosis for random processes: 'Contamination' problems... 201 4.1 Introduction... 201 4.2 Problem of mean value 'contamination'... 203 4.3 Generalisations... 209 4.3.1 General remarks... 209 4.3.2 'Contamination' problems for regression models... 212

CONTENTS vii 4.4 Apriori estimates in 'contamination' problems... 213 4.5 Monte Carlo experiments... 216 4.6 Conclusion...,... 218 Chapter 5 Sequential methods of statistical diagnosis... 219 5.1 Introduction... 219 5.2 Change-point problem... 223 5.2.1 Quality characteristics of sequential methods... 224 5.2.2 Apriori estimates of quality of sequential change-point detection methods... 247 5.2.3 Asymptotic comparative analysis of sequential change-point detection methods... 253 5.3 Problem of 'early detection'... 281 5.3.1 Formulation of the problem... 281 5.3.2 Characteristics of methods in the problem of 'early detection'... 283 5.3.3 Apriori estimates of quality for 'early detection' methods... 28.5 5.3.4 Analysis of asymptotic optimality of 'early detection' methods... 290 5.3.5 Robustness of sequential methods... 292 5.4 Conclusion... 297 Chapter 6 Statistical diagnosis problems for random fields 299 6.1 Retrospective diagnosis problems for random fields... 300 6.1.1 Problem A... 303 6.1.2 Problem B... 309 6.1.3 Apriori estimates of quality in retrospective diagnosis problems for random fields... 317 6.2 Sequential diagnosis problems for random fields... 322 6.3 'Contamination' problems for random fields... 324 6.4 Conclusion... 328 Part 2. Applications... 331 Chapter 7 Application of the change-point analysis to investigation of the brain electrical activity... 333 7.1 Introduction... 333 7.2 General description of approaches to quantitative feature extraction from the EEG signal... 336 7.3 Nonstationarities in EEGs. Methods of segmentation of the EEG signal... 337

Vlll CONTENTS 7.3.1 Segmentation of the EEG using fixed intervals... 338 7.3.2 Parametric segmentation of the EEG... 339 7.3.3 Nonparametric approaches to the description of piecewise stationary structure of EEG... 342 7.4 Experimental results... 346 7.4.1 Change-point detection algorithm performance with real EEG data... 349 7.4.2 Change-points in EEG components with different pattern... 350 7.4.3 Detection of change-points at different levels of the hierarchy of EEG segmental descriptions... 352 7.4.4 Change-points in multi-channel EEG... 357 7.4.5 The approach to the analysis of change-point synchronization in multi-channel EEG... 362 7.4.6 Change-point synchronization in pairs of EEG channels... 365 7.4.7 Multichannel analysis of spatial synchronization of the change-points... 372 7.5 Some general theoretical considerations... 378 7.5.1 Unsolved problems and new prospects... 378 7.5.2 Change-point based analysis of the synchronization between signals or signal components... 380 7.6 Other prospects of the change-point based analysis... 384 7.7 Conclusion... 386 Chapter 8 Methods of statistical diagnosis in economic and financial systems... 389 8.1 Introduction... 389 8.2. Econometric models with structural breaks... 390 8.3 Econometric models with 'contamination' effects... 397 8.4 Early detection of crises in economic and financial systems... 399 8.5 Conclusion... 40.5 Appendix. Algorithms of statistical diagnosis... 407 Bibliography... 417 Author Index... 445 Subject Index....447 Main Notations and Abbreviations....451

Preface This book has a distinct philosophy and it is appropriate to make it explicit at the outset. In our view almost all classic statistical inference is based upon the assumption (explicit or implicit) that there exists a fixed probabilistic mechanism of data generation. Unlike classic statistical inference, this book is devoted to the statistical analysis of data about complex objects with more than one probabilistic mechanism of data generation. We think that the existence of more than one data generation process (DGP) is the most important characteristic of com plex systems. When the hypothesis of statistical homogeneity holds true, Le., there exists only one mechanism of data generation, all statistical inference is based upon the fundamentallaws of large numbers. However, the situation is completely different when the probabilistic law of data generation can change (in time or in the phase space). In this case all data obtained must be 'sorted' in subsamples generated by different probabilistic mechanisms. Only after such classification we can make correct inferences about all DGPs. There exists yet another type of problem for complex systems. Here it is important to detect possible (but unpredictable) changes of DGPs on-line with data collection. Since the complex system can change the probabilistic mechanism of data generation, the correct statistical analysis of such data must begin with decisions about possible changes in DGPs. This wide range of problems falls into the scope of statistical diagnosis - a comparatively new field of research in mathematical statistics. The main goal of statistical diagnosis is to correctly answer the question of whether the data obtained are generated by one or by many probabilistic mechanisms. In our view, any statistical research must begin with the statistical diagnosis of data, since the classic methodology of mathematical statistics is effectively applied only in situations of a unique mechanism of data generation. In other words, one should check statistical homogeneity of data obtained before estimation of parameters of statistical models and decision making procedures.

x Preface Nowadays, problems of statistieal diagnosis become very actual owing to the development of technology and the implementation of complex systems in all spheres of human life. Here we would like to mention some important examples. First, economics, econometrics, and analysis of financial systems. In this field the idea of using non-stationary models for the description of complex systems dynamies with unpredietable changes in mechanisms of data generation has become more and more popular. Many types of exogenous and endogenous 'shocks' in economic systems, as weil as financial crises, can be described by such dynamic models with structural changes. More generally, the problem of structural changes in economie models, systems, and data is very real nowadays. The correct estimation of parameters in such models can be done only after the detection of structural changes in data and splitting the whole sampie into stationary segments. In the field of research into the functioning of the human brain on the basis of measurements of its electric activity, it was discovered recently that an electro- encephalographie (EEG) signal cannot be described by a unique mathematical model but only by a rieh enough set of such models. Therefore, the problem of structural analysis of EEG signals is very important for the analysis of the functioning of the human brain and the development of modern diagnostics and theurapy. A completely new and very interesting field of research is the statistical analysis of narrative historical texts. It turns out that such texts can be quantitatively analysed with the aim of statistical identification of dates of historical events described in them. Other related historieo-metrologieal problems actually belong to the field of statistieal diagnosis and include problems of an authentic stylistie structure of a text (whether it was com posed of 'pieces ' written by different authors or represents a compilation of some other texts?). These examples do not exhaust the list of situations in whieh the statistical diagnosis of data is a very important and even crucial research problem. However, they can explain why interest in the problems of statistical diagnosis has been so high all over the world in the last 20-30 years. Nowadays, two large classes of problems of statistieal diagnosis can be observed: (a) retrospective problems, (b) sequential problems, or problems of the fastest detection of disorders. Retrospective problems of statistical diagnosis include research situations in whieh it is necessary to detect disorders and non-stationarities aposteriori, i.e., in the whole of the information received. Sequential problems include situations in which decisions about homo-

Preface Xl geneity or non-homogeneity of processes observed are made on-line with data collection. In its turn, the dass of retrospective problems can be also divided into two subdasses. The first subdass indudes situations in which all data generated by one probabilistic mechanism are located in compact time intervals (of random processes ) or in compact zones of the phase space (of random fields). In these situations one can speak about boundaries that separate data generated by different probabilistic mechanisms. Historically, research into the field of statistical diagnosis has begun from problems of this subdass. In that early period (1950-1960s) the term 'change-point' has appeared in works of E.S. Page, A.N. Kolmogorov, and A.N. Shiryaev. The second dass comprises situations in which data generated by one probabilistic law are 'dispersed' among data generated by (an)other mechanism(s). Research into these 'contamination' problems has been already initiated by founders of the dassic mathematical statistics. In these problems it is necessary to detect and identify 'small contaminations' which are generated by some extraneous probabilistic mechanism. Problems of detection of small 'contaminations' had been already posed in the 1930-1940s. Evidently, the dassic 'contamination problem' belongs to the field of statistical diagnosis and allows for essential generalisations. Thus, considering the contents of this book, we can say that it deals with 'disorder' problems in retrospective and sequential settings, as well as 'contamination' problems for random processes and fields. In 1993 our book Nonparametric Methods in Change-Point Problems was published in Kluwer Academic Publishers. In that book we presented our results in the field of statistical diagnosis obtained before 1993. As far as we know, the book had a positive scientific impact. Our present book is not only a corrected edition of the previous one. Here we generalize many ideas and present new theoretical and practical results of 1993-1999. We would like to emphasize new practical applications of our methods, in particular, the nonparametric analysis of human EEG signals. In this field our long and fruitful cooperation with our colleagues from Moscow State University - Alexandr Kaplan and Sergei Shishkin - has resulted in the creation of statistical program package for the analysis of EEG signals. Kaplan and Shishkin wrote a special chapter for this book devoted to applications of our methods of statistical diagnosis to EEG analysis. We hope this chapter will be informative for professional biologists and neurophysiologists. Far other readers it might be interesting as an important example of practical relevant applications of mathematical statistics. Another important application is the structural analysis of economic mod-

Xll Preface eis and systems. Here we consider the problem of structural breaks in dynamic econometric models, the problem of the detection of outliers in 'contaminated' econometric models, and the problem ofthe early diagnosis of crises in financial and economic systems. We think it is only the first step in creating methods of structural analysis of economic and sodal systems - the rapidly developing branch of modern economic and sodal theory. As in the first book, we consider the nonparametric approach to problems of statistical diagnosis. It means that we propose and analyse methods that do not require apriori knowledge of probabilistic distributions of data observed. We think that such methods are most useful in applications, because structural diagnostic problems must be solved before any parametric analysis of data. Now we list new theoretical results of this book. a) Main ideas 01 the nonparametric approach to statistical diagnosis The general ideas of our approach to 'disorder' and 'contamination' problems for random processes and fields are formulated and it is shown that any diagnostic problem of this kind can be reduced to some standard problem which is analysed on the basis of the nonparametric methodology. b ) Retrospective problems 01 statistical diagnosis - Apriori minimax low boundaries are obtained for the probability of the change-points estimation error for the case of an abrupt change in the mathematical expectation, a break or ajump in any derivative of the regression function for random sequences; - An analog of the Rao-Cramer type inequality for change-point problems is established; - A nonparametric method and an algorithm of detecting abrupt changes in the mathematical expectation of random sequences; - Nonparametric methods and algorithms of detecting abrupt changes in coeffidents of the linear functional regression (inciuding changes in derivatives) ; - Methods of solving 'contamination' problems for random sequences and linear regressions are proposed; - Apriori minimax low boundaries are obtained for the estimation error probability of a discriminating surface which divides the domain of a randorn field into two areas with different mathematical expectations; - Methods of estimation of a (vector) parameter of a discriminating surface that divides the domain of a random field into two areas with different sets of coeffidents of the linear functional regression are proposed and investigated; - Methods of solving 'contamination' problems for random fields are proposed and analysed. c) Sequential problems 01 statistical diagnosis

Preface Xlll - On the basis of the nonparametric approach to the asymptotic comparative analysis of sequential methods of change-point detection the following new results were obtained: - The apriori low boundary for the rate of convergence of the normed delay time of detection to its deterministic limit is established - the Rao-Cramer type inequality; - For the nonparametric analogs of the main sequential change-point detection methods, the analysis of asymptotic optimality based on comparison of real and 'ideal' characteristics of the delay time and the 'false alarm' probability is given. - Areas of the most efficient applications of different sequential methods are investigated; - The problem of 'early detection' of non-stationarities of random sequences is formulated and analysed; - The robust properties of different sequential change-point detection methods for random sequences are analysed and robust modifications ofthese methods are given; - The method of sequential detection of non-stationarities for random fields is proposed. Now we briefly review the contents of the book. The book consists of two parts. The first part (Chapters 1-6) deals with the theory; the second part (Chapters 7, 8 and the Appendix) is devoted to applications. In the first chapter, results from the theory of probability, the theory of random processes, the theory of random fields, and optimisation theory, as weil as some auxiliary results which will be used in the following chapters, are presented. Here we formulate the main ideas of our approach to the problems of statistical diagnosis and the main assumptions which are used in the sequel. The second chapter contains a short review of works in the field of statistical diagnosis up to the end of 1998. The third and the fourth chapters deal with retrospective problems of statistical diagnosis. In the third chapter change-point problems for random sequences are considered. Here we analyse problems with single and multiple change-points in mean values of random sequences, in coefficients of the linear functional model, in derivatives, as well as gradual disorders. Methods for solving all these problems are proposed and investigated. Here we give asymptotic analysis of change-point estimates and compare asymptotic properties of our estimates with the maximum likelihood estimates. The special paragragh is devoted to a priori estimates in change-point problems: the Rao-Cramer type of inequality and the minimax boundaries for the estimation error probabilities in different

xiv Preface problems of statistical diagnosis. The fourth chapter deals with 'contamination' problems for random sequences. Methods of solving these problems are proposed and analysed. The fifth chapter deals with sequential problems of statistical diagnosis. Here we analyse nonparametric analogs Df the cumulative sums (CUSUM) method, the quasi-bayesian method of Girshick-Rubin and Shiryaev, the exponential smoothing method, and the 'moving sampie' methods. For each of these methods, the characteristics of the normed delay time in change-point detection, the normed 'false alarm' probability, and the rate of convergence of the normed delay time are investigated. The apriori informational boundaries for the normed delay time in detection and the rate of convergence of the normed delay time to its limit are proved. The asymptotic comparative analysis of the nonparametric change-point detection methods based upon the apriori informational boundaries is carried out. Then we formulate and analyse the problem of 'early detection' of non-stationarities. In the last section of this chapter the robust properties of nonparametric sequential change-point detection methods are investigated and robust modifications of these methods are proposed. The sixth chapter deals with problems and methods of statistical diagnosis for random fields. Retrospective problems for random fields, Le., multidimensional analogs of 'disorder' and 'contamination' problems, are considered, as weil as sequential problems for random fields. The multi-dimensional analog of 'disorder' problems consists in the search of a boundary which divides the domain of the field into two areas with different probabilistic characteristics of observations. The apriori low boundaries for the quality characteristics of detection in multi-dimensional 'disorder' and 'contamination' problems are established. An approach to solving sequential problems of statistical diagnosis for random fields is proposed. Chapter 7 has been written by our colleagues A. Kaplan and S. Shishkin. It deals with applications of statistical diagnosis methods to the analysis of EEG signals. Here our readers can find the professional description of the modern state of the art of this field and actual problems of statistical diagnosis of EEGs. Results of applications of our nonparametric methods to real EEG signals are presented. Chapter 8 deals with problems of statistical diagnosis in economic models and systems. Here we analyse the problem of the structural breaks in non-stationary econometric models, the problem of the statistical analysis of 'contaminated' econometric models, and the problem of the 'early detection' of crises in financial and economic systems. The Appendix is devoted to the description of nonparametric algorithms

Preface xv and programs of statistical diagnosis and some results of their experimental testing. This book iso in general, aimed at mathematicians working in mathematical statistics. However, the authors hope that it will be useful to postgraduate and undergraduate students, and the second part will be of interest to applied scientists. The authors are grateful to Professor M. Hazewinkel, Dr. P. Roos, and Kluwer Academic Publishers for the support of this project. B. Brodsky B. Darkhovsky Moscow, September 1999