Springer Series in Statistics Advisors: P. Bickel, P. Diggle, S. Fienberg, U. Gather, I. Olkin, S. Zeger The French edition of this work that is the basis of this expanded edition was translated by Vladimir Zaiats. For other titles published in this series, go to http://www.springer.com/series/692
Alexandre B. Tsybakov Introduction to Nonparametric Estimation 123
Alexandre B. Tsybakov Laboratoire de Statistique of CREST 3, av. Pierre Larousse 92240 Malakoff France and LPMA University of Paris 6 4, Place Jussieu 75252 Paris France alexandre.tsybakov@upmc.fr ISBN: 978-0-387-79051-0 e-isbn: 978-0-387-79052-7 DOI 10.1007/978-0-387-79052-7 Library of Congress Control Number: 2008939894 Mathematics Subject Classification: 62G05, 62G07, 62G20 c Springer Science+Business Media, LLC 2009 All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed on acid-free paper springer.com
Preface to the English Edition This is a revised and extended version of the French book. The main changes are in Chapter 1 where the former Section 1.3 is removed and the rest of the material is substantially revised. Sections 1.2.4, 1.3, 1.9, and 2.7.3 are new. Each chapter now has the bibliographic notes and contains the exercises section. I would like to thank Cristina Butucea, Alexander Goldenshluger, Stephan Huckenmann, Yuri Ingster, Iain Johnstone, Vladimir Koltchinskii, Alexander Korostelev, Oleg Lepski, Karim Lounici, Axel Munk, Boaz Nadler, Alexander Nazin, Philippe Rigollet, Angelika Rohde, and Jon Wellner for their valuable remarks that helped to improve the text. I am grateful to Centre de Recherche en Economie et Statistique (CREST) and to Isaac Newton Institute for Mathematical Sciences which provided an excellent environment for finishing the work on the book. My thanks also go to Vladimir Zaiats for his highly competent translation of the French original into English and to John Kimmel for being a very supportive and patient editor. Alexandre Tsybakov Paris, June 2008
Preface to the French Edition The tradition of considering the problem of statistical estimation as that of estimation of a finite number of parameters goes back to Fisher. However, parametric models provide only an approximation, often imprecise, of the underlying statistical structure. Statistical models that explain the data in a more consistent way are often more complex: Unknown elements in these models are, in general, some functions having certain properties of smoothness. The problem of nonparametric estimation consists in estimation, from the observations, of an unknown function belonging to a sufficiently large class of functions. The theory of nonparametric estimation has been considerably developed during the last two decades focusing on the following fundamental topics: (1) methods of construction of the estimators (2) statistical properties of the estimators (convergence, rates of convergence) (3) study of optimality of the estimators (4) adaptive estimation. Basic topics (1) and (2) will be discussed in Chapter 1, though we mainly focus on topics (3) and (4), which are placed at the core of this book. We will first construct estimators having optimal rates of convergence in a minimax sense for different classes of functions and different distances defining the risk. Next, we will study optimal estimators in the exact minimax sense presenting, in particular, a proof of Pinsker s theorem. Finally, we will analyze the problem of adaptive estimation in the Gaussian sequence model. A link between Stein s phenomenon and adaptivity will be discussed. This book is an introduction to the theory of nonparametric estimation. It does not aim at giving an encyclopedic covering of the existing theory or an initiation in applications. It rather treats some simple models and examples in order to present basic ideas and tools of nonparametric estimation. We prove, in a detailed and relatively elementary way, a number of classical results that are well-known to experts but whose original proofs are sometimes
viii Preface to the French Edition neither explicit nor easily accessible. We consider models with independent observations only; the case of dependent data adds nothing conceptually but introduces some technical difficulties. This book is based on the courses taught at the MIEM (1991), the Katholieke Universiteit Leuven (1991 1993), the Université Pierre et Marie Curie (1993 2002) and the Institut Henri Poincaré (2001), as well as on minicourses given at the Humboldt University of Berlin (1994), the Heidelberg University (1995) and the Seminar Paris Berlin (Garchy, 1996). The contents of the courses have been considerably modified since the earlier versions. The structure and the size of the book (except for Sections 1.3, 1.4, 1.5, and 2.7) correspond essentially to the graduate course that I taught for many years at the Université Pierre et Marie Curie. I would like to thank my students, colleagues, and all those who attended this course for their questions and remarks that helped to improve the presentation. I also thank Karine Bertin, Gérard Biau, Cristina Butucea, Laurent Cavalier, Arnak Dalalyan, Yuri Golubev, Alexander Gushchin, Gérard Kerkyacharian, Béatrice Laurent, Oleg Lepski, Pascal Massart, Alexander Nazin, and Dominique Picard for their remarks on different versions of the book. My special thanks go to Lucien Birgé and Xavier Guyon for numerous improvements that they have suggested. I am also grateful to Josette Saman for her help in typing of a preliminary version of the text. Alexandre Tsybakov Paris, April 2003
Notation x greatest integer strictly less than the real number x x smallest integer strictly larger than the real number x x + max(x, 0) log natural logarithm I(A) indicator of the set A Card A cardinality of the set A = equals by definition λ min (B) smallest eigenvalue of the symmetric matrix B a T,B T transpose of the vector a or of the matrix B p L p ([0, 1],dx)-norm or L p (R,dx)-norm for 1 p depending on the context l 2 (N)-norm or the Euclidean norm in R d, depending on the context N (a, σ 2 ) normal distribution on R with mean a and variance σ 2 N d (0,I) standard normal distribution in R d ϕ( ) density of the distribution N (0, 1) P Q the measure P is absolutely continuous with respect to the measure Q
x Notation dp/dq the Radon Nikodym derivative of the measure P with respect to the measure Q a n b n 0<lim inf n (a n /b n ) lim sup n (a n /b n )< h = arg min h H F (h) means that F (h ) = min h H F (h) MSE mean squared risk at a point (p. 4, p. 37) MISE mean integrated squared error (p. 12, p. 51) Σ(β,L) Hölder class of functions (p. 5) H(β,L) Nikol ski class of functions (p. 13) P(β,L) Hölder class of densities (p. 6) P H (β,l) Nikol ski class of densities (p. 13) S(β,L) Sobolev class of functions on R (p. 13) P S (β,l) Sobolev class of densities (p. 25) W (β,l) Sobolev class of functions on [0, 1] (p. 49) W per (β,l) periodic Sobolev class (p. 49) W (β,l) Sobolev class based on an ellipsoid (p. 50) Θ(β,Q) Sobolev ellipsoid (p. 50) H(P, Q) Hellinger distance between the measures P and Q (p. 83) V (P, Q) total variation distance between the measures P and Q (p. 83) K(P, Q) Kullback divergence between the measures P and Q (p. 84) χ 2 (P, Q) χ 2 divergence between the measures P and Q (p. 86) ψ n optimal rate of convergence (p. 78) p e,m minimax probability of error (p. 80) p e,m average probability of error (p. 111) C the Pinsker constant (p. 138) R(λ, θ) integrated squared risk of the linear estimator (p. 67) Assumption (A) p. 51 Assumption (B) p. 91 Assumption (C) p. 174 Assumptions (LP) p. 37
Contents 1 Nonparametric estimators... 1 1.1 Examples of nonparametric models and problems............ 1 1.2 Kernel density estimators................................. 2 1.2.1 Mean squared error of kernel estimators.............. 4 1.2.2 Construction of a kernel of order l... 10 1.2.3 Integrated squared risk of kernel estimators........... 12 1.2.4 Lack of asymptotic optimality for fixed density........ 16 1.3 Fourier analysis of kernel density estimators................ 19 1.4 Unbiased risk estimation. Cross-validation density estimators. 27 1.5 Nonparametric regression. The Nadaraya Watson estimator... 31 1.6 Local polynomial estimators.............................. 34 1.6.1 Pointwise and integrated risk of local polynomial estimators........................................ 37 1.6.2 Convergence in the sup-norm....................... 42 1.7 Projection estimators.................................... 46 1.7.1 Sobolev classes and ellipsoids....................... 49 1.7.2 Integrated squared risk of projection estimators....... 51 1.7.3 Generalizations................................... 57 1.8 Oracles... 59 1.9 Unbiased risk estimation for regression..................... 61 1.10 Three Gaussian models................................... 65 1.11 Notes.................................................. 69 1.12 Exercises............................................... 72 2 Lower bounds on the minimax risk... 77 2.1 Introduction............................................ 77 2.2 A general reduction scheme............................... 79 2.3 Lower bounds based on two hypotheses.................... 81 2.4 Distances between probability measures.................... 83 2.4.1 Inequalities for distances........................... 86 2.4.2 Bounds based on distances......................... 90
xii Contents 2.5 Lower bounds on the risk of regression estimators at a point.. 91 2.6 Lower bounds based on many hypotheses................... 95 2.6.1 Lower bounds in L 2...102 2.6.2 Lower bounds in the sup-norm...................... 108 2.7 Other tools for minimax lower bounds...................... 110 2.7.1 Fano s lemma..................................... 110 2.7.2 Assouad s lemma.................................. 116 2.7.3 The van Trees inequality........................... 120 2.7.4 The method of two fuzzy hypotheses................. 125 2.7.5 Lower bounds for estimators of a quadratic functional.. 128 2.8 Notes...131 2.9 Exercises...133 3 Asymptotic efficiency and adaptation...137 3.1 Pinsker stheorem...137 3.2 Linear minimax lemma................................... 140 3.3 ProofofPinsker stheorem...146 3.3.1 Upper bound on the risk........................... 146 3.3.2 Lower bound on the minimax risk................... 147 3.4 Stein s phenomenon...................................... 155 3.4.1 Stein s shrinkage and the James Stein estimator....... 157 3.4.2 Other shrinkage estimators......................... 162 3.4.3 Superefficiency.................................... 165 3.5 Unbiased estimation of the risk............................ 166 3.6 Oracle inequalities....................................... 174 3.7 Minimax adaptivity...................................... 179 3.8 Inadmissibility of the Pinsker estimator.................... 180 3.9 Notes...185 3.10 Exercises............................................... 187 Appendix...191 Bibliography...203 Index...211