A Constrained Risk Inequality With Applications to Nonparametric Functional Estimation

Transcription

1 Uiversity of Pesylvaia ScholarlyCommos Statistics Papers Wharto Faculty Research 1996 A Costraied Risk Iequality With Applicatios to Noparametric Fuctioal Estimatio Lawrece D. Brow Uiversity of Pesylvaia Mark G. Low Uiversity of Pesylvaia Follow this additioal works at: Part of the Statistics Probability Commos Recommeded Citatio Brow, L. D., & Low, M. G. (1996). A Costraied Risk Iequality With Applicatios to Noparametric Fuctioal Estimatio. The Aals of Statistics, 4 (6), This paper is posted at ScholarlyCommos. For more iformatio, please cotact repository@pobox.upe.edu.

2 A Costraied Risk Iequality With Applicatios to Noparametric Fuctioal Estimatio Abstract A geeral costraied miimum risk iequality is derived. Give two desities f θ f 0 we fid a lower boud for the risk at the poit θ give a upper boud for the risk at the poit 0. The iequality sheds ew light o superefficiet estimators i the ormal locatio problem also o a adaptive estimatio problem arisig i oparametric fuctioal estimatio. Keywords adaptive estimatio, superefficiet estimators, oparametric fuctioal estimatio, miimum risk iequalities, white oise model, desity estimatio, oparametric regressio Disciplies Statistics Probability This joural article is available at ScholarlyCommos:

3 The Aals of Statistics 1996, Vol. 4, No. 6, A CONSTRAINED RISK INEQUALITY WITH APPLICATIONS TO NONPARAMETRIC FUNCTIONAL ESTIMATION BY LAWRENCE D. BROWN 1 AND MARK G. LOW Uiversity of Pesylvaia A geeral costraied miimum risk iequality is derived. Give two desities f f0 we fid a lower boud for the risk at the poit give a upper boud for the risk at the poit 0. The iequality sheds ew light o superefficiet estimators i the ormal locatio problem also o a adaptive estimatio problem arisig i oparametric fuctioal estimatio. 1. Itroductio. The problems of estimatig a fuctio at a poit uder squared error loss the whole fuctio uder itegrated squared error loss have held a cetral positio i the oparametric fuctioal estimatio literature. I particular, each has received a fairly detailed aalysis i desity estimatio, oparametric regressio white oise models. Progress to date ca be cotrasted as follows. I both problems asymptotic rates of covergece are typically slower tha '. For itegrated squared error loss, asymptotically miimax procedures have bee foud whe the parameter space is a Sobolev space. For a give space these procedures may be chose to be liear. For the poitwise estimatio problem, typically there do ot exist liear procedures which are asymptotically miimax. However, uder mild regularity coditios appropriately chose liear procedures have maximum risk withi a small costat multiple of the miimax risk. See, for example, Ibragimov Hasmiskii Ž or Dooho Liu Ž I particular, miimax rates of covergece ca be achieved by liear procedures. Oe of the most importat results for the global estimatio problem was the costructio of adaptive estimators which are simultaeously asymptotically miimax over a large umber of Sobolev spaces; see Efromovich Pisker Ž 1984., Efromovich Ž Golubev Ž Such adaptive procedures have ot bee foud for the poitwise estimatio problem. Received February 199; revised February Work supported i part by NSF Grats DMS DMS Work supported by a NSF Postdoctoral Research Fellowship. AMS 1991 subject classificatios. Primary 6G99; secodary 6F1, 6F35, 6M99. Key words phrases. Adaptive estimatio, superefficiet estimators, oparametric fuctioal estimatio, miimum risk iequalities, white oise model, desity estimatio, oparametric regressio. 54

4 A CONSTRAINED RISK INEQUALITY 55 Recetly Lepskii Ž has show that for a white oise model it is ot possible to fid adaptive estimators for the poitwise problem which preserve miimaxity over a rage of Lipschitz classes. Furthermore Lepskii showed that if a estimator is asymptotically rate miimax over oe Lipschitz class, it must iflate the maximum risk over the other Lipschitz class by at least a logarithmic factor of the sample size. Lepskii eve showed that these bouds ca be attaied. See also Lepskii Ž 1991, 199., see Dooho Johstoe Ž 199, Efromovich Low Ž for some recet, related results. I this paper we develop a two poit iequality which is particularly well suited to providig lower bouds for adaptive estimatio problems. The iequality gives, i a geeral settig, a lower boud for the squared error risk at oe parameter poit subject to havig a small risk at aother parameter poit. The relatioship to adaptatio is spelled out i Sectio 4. Such a iequality is also related to the study of -miimax procedures to the otio of superefficiet estimatio. The otio of -miimax procedures Žalso called almost submiimax procedures. was itroduced by Robbis Ž Frak Kiefer Ž further studied by Hodges Lehma Ž 195. Bickel Ž 1983, A example of superefficiet estimatio was foud i Hodges Ž 195. may further results were obtaied by Le Cam Ž We apply our iequality to this topic i Sectio 3A.. A costraied miimum risk iequality. Let Z have distributio with desity f or f with respect to a measure. For ay estimator 1 based o Z, its risk is defied by.1 R, E Ž x. f x dx. H The followig theorem gives a lower boud for RŽ,. give that RŽ,. 1. I the theorem subsequetly let qž x. f Ž x. f Ž x.. I IŽ 1,. E q 1 Ž Ž X... 1 qž x. for some x is possible, with the obvious iterpretatio qž x. f Ž x. 1 f Ž x.. THEOREM 1. Let assume 0 I RŽ, The Ž '..3 R, I. Hece, also, ' I /.4 R, 1. ž '

5 56 L. D. BROWN AND M. G. LOW PROOF. By traslatio ivariace we eed oly prove the theorem for Ž. 1 0, 0. We eed to fid which miimizes R H x f x dx subject to R0 H x f0 x dx. By Lagrage multipliers the miimizig satisfies Ž Ž x.. f Ž x. Ž x. f 0 Ž x. 0 for some 0. If 0, the x it follows that, which cotradicts the assumptio that ' I sice I 1. Hece 0 f Ž x. qž x. 5. Ž x., f Ž x. f Ž x. 1 qž x. 0 where 1. If q x, the x. Furthermore, this satisfies R, so that 0 qž x. 6. H f0ž x. Ž dx.. 1 qž x. 7. The, by Cauchy-Schwarz, sice q x f x f x. Hece 0 8. qž x. ' I H f0ž x. Ž dx. 1 qž x. 1 ž H q Ž x. f0ž x. Ž dx. / q Ž x H. f Ž x. Ž dx. 1 qž x. 1 f Ž x. Ž dx ž H 1 qž x. /. 1 1 qž x. qž x. ' I 1 f Ž x. Ž dx. 1 qž x. H H f x dx R sice ŽŽ x.. Ž1 qž x.. by 5.. I the secod iequality of 8. we have agai used CauchySchwarz. This proves 3.. Equatio 4. follows because Ž a b. a Ž 1 ba. for a, b 0. It is of iterest to ote that the first boud i the theorem,.3, is sharp. As before, let 0,

6 A CONSTRAINED RISK INEQUALITY 57 PROPOSITION 1. Fix B 1 0 B. Let f0 be the uiform Ž 1 distributio o 0, 1 let f be the uiform distributio o 0, B.. The I B 4.9 mi R, : R 0, I. Hece the boud.3 is attaied. ' ' PROOF. The best estimate of subject to R 0, is ½ ' 0, if B 1 x 1, Ž x. B, if 0 x b shows that the best is costat o the regio 0 x B 0 o 1 the regio B x 1. The, R 0, yields 10.. This estimator Ž '. satisfies R, I verifies 9. i view of Theorem 1. REMARKS. It ca be see that apart from arbitrary measurable trasformatios of the sample space, the example i the precedig proof is the oly oe i which the miimum i 9. is the same as 3.. I other words, the boud i 3. is attaied if oly if the likelihood ratio, qž x., takes oly the two values 0 I. I may applicatios f f will be desities of idepedet ideti- 1 cally distributed rom variables X 1,..., X, each with desity f Ž1. 1. Lettig Ž1. Ž1. Ž1. Ž1. ŽŽ Ž1. q x f x f x I E q Ž X..., we see that the iformatio 1 1 measure I satisfies Ž1..10 I I. Furthermore, the boud i Theorem 1 is still sharp because lettig f Ž 1 uiform o 0, 1 f be uiform o 0, I. yields 1 be ' for I, as i Propositio Applicatios. 4 mi R, : R 0, I ' A. Superefficiecy i the ormal locatio model. As a first simple example, the iequality give i the last sectio ca be used to yield the followig result about superefficiet estimates i the stard ormal locatio model. Let X,..., X be i.i.d. NŽ, 1. with. Write RŽ,. 1 for the risk of a estimator based o Ž X,..., X.. Thus 1 Ž 3.1. RŽ,. EŽ..

7 58 L. D. BROWN AND M. G. LOW THEOREM. Let let Ž l. 1. If Ž 3.. lim sup RŽ 0,., the Ž 3.3. lim sup sup RŽ,. 0. l REMARK. Ž 3.3. with l deleted, Ž 1, 1. the right-h side replaced by is of course implied by Ž 3.3. is well kow. See, for example, pages i Lehma Ž PROOF OF THEOREM. Sice X is sufficiet, we eed oly cosider estimates which are fuctios of X. Now if Ž 3.. holds, the there exists N M such that for all N, M RŽ 0,.. Note that X NŽ, 1. we may apply Theorem 1 with f,, the desity of a ormal distributio with mea variace 1. Take Ž l The 1 f, Ž x. Ž l. Ž 3.4. I H dx exp exp. f 1 0, Now by.4, if N, Ž ž / l l Ž M. R, 1 l 1 l M Ž l. 1 The theorem is proved by takig limits sice M Ž l. 1 0 as. B. Superefficiecy i the white oise model. We ow tur to our mai applicatio of Theorem 1. Cosider the followig white oise model Ž 3.6. dxt f Ž t. dt dw t, t, f F, ' where W is Browia motio o 1, 1 t. This model has bee studied extesively as a prototypical model for may other fuctioal estimatio problems such as oparametric regressio desity estimatio. See, for example, Ibragimov Hasmiskii Ž 1981., Low Ž 199., Dooho Low Ž 199., Brow Low Ž Nussbaum Ž

8 A CONSTRAINED RISK INEQUALITY 59 Ž k We shall focus o the followig class of parameter spaces. Write f. Ž x. for the kth derivative of f let 4 Ž k. Ž 3.7. F Ž k. f L Ž 1, 1. : f Ž x. M x. Miimax rates of covergece, as, for estimatig fž 0. are well kow. For estimators Ž. based o the sigal 3.6, write Ef f 0 for the mea squared error for estimatig fž 0.. The Ž. k Ž k lim sup if sup Ef f 0. ff Ž k. THEOREM 3. Let, l let be estimators based o 3.6. If Ž k. 0 x 3.9 sup f x m M Ž. k Ž k lim sup Ef f0 0, 0 the kž k1. f Ž. l ff Ž k. Ž lim if sup E f Ž As oted i the itroductio, a similar statemet, but with a differet proof, appears i Lepskii Ž PROOF. Let g: R R be a k times differetiable fuctio such that: Ž i. gž 0. 0; Ž ii. g has compact support; Ž k iii for some M m, g. Ž x. M m x; iv H g Ž x. dx 1. Such a fuctio is easy to costruct. Set / / k k1 Ž 3.1. ž, l 1 k1 Ž ž. l The Ž l

9 530 L. D. BROWN AND M. G. LOW k Ž Let f : 1, 1 R be defied by g Ž t. Ž f Ž t. f Ž t.. 0 By 3.11, assumptio iii for g 3.15, f F k. Write Pg for the probability measure associated with the process dxt fž t. dw t, t. ' The a sufficiet statistic for the family of measures P, P 4 0 g is give by dpg Ž T l. dp 0 1 Ž Ž. Set H1 g t dt. The ž / 0 uder P, T N,, uder P g, T N,. Now sice, there exists a N such that for all N, gž t if t 1. The by assumptio Ž iv. for g Ž it follows that for N 1, l. If assumptio Ž of Theorem 3 holds, there exists a R N N1 such that for all N, R Ž Ef Ž f Ž k Ž k1. Sice T is sufficiet for P, P 4 we may apply Theorem 1 with f Ž 0. 0 g 1 0, Ž. f 0 f0 0 g 0, p the desity of T uder P0 p the 1 desity of T uder Pg, where we have used p istead of f for the desities i Theorem 1. By Ž 3.4., p Ž x. IŽ. H dx exp expž. p Ž x. 1 for N. Theorem 1, 4., yields for N, g Ž 0. Ž E f Ž 0. Ž 1., where f 1 R 1 R 1 Ž k Ž k1. 1 kž k1. Ž g Ž 0.. g Ž 0. Ž l.

10 A CONSTRAINED RISK INEQUALITY 531 The 0 as hece by it follows that kž k1. f Ž. lim if E f Ž 0. g Ž 0. 0 l the theorem is proved. REMARK. Oe may also also cosider Sobolev parameter spaces i place of the Lipschitz spaces defied i Ž Thus, suppose the parameter space is F Ž k. fl Ž 1, 1. : f Ž j. Ž 1. f Ž j. Ž 1. 0 j 0,..., k1, S ½ 1 Ž k. H Ž. 5 1 f x dx M. Ž k1. k I this case the optimal rate for estimatig f 0 is i place of the k Ž k1. rate of Theorem 3. See, for example, Dooho Low Ž For this case, Theorem 3 remais valid with the obvious modificatio to Ž 3.9. with this optimal rate substituted ito Ž Ž The proof is very similar to that of Theorem 3. The pricipal differece is the defiitios of, g i Ž i order to establish a suitably ufavorable two poit problem, as is doe above Ž These,, g are give via the hardest liear subproblem algorithm, are explicitly described i Dooho Low Ž Note that here g is really g, that is, its form as well as its scalig deped o. 4. Adaptatio. Theorem 3 of the last sectio sheds ew light o adaptatio i the white oise model. For a sequece of estimates to be adaptive over two classes F Ž k. F Ž k. give by Ž , where k 1, k are itegers with k1 k, we would require Ž. k Ž k lim sup 1 1 sup Ef f 0 ff Ž k 1. Ž. k Ž k lim sup sup Ef f 0. ff Ž k. I particular Ž 4.. must hold Ž with sup deleted. for some fixed f F Ž k. 0. Now sice F Ž k. F Ž k., f F Ž k., it follows that Ž. k Ž k 1. ŽŽ k k..žž k 1.Ž k lim sup Ef f Theorem 3 the yields that k 1 Ž k 1 1. f Ž. l ff Ž k. Ž 4.4. lim sup sup E f Ž 0. 0

11 53 L. D. BROWN AND M. G. LOW it follows that Ž 4.1. caot hold if Ž 4.. holds. Hece, adaptive estimatio over ay two classes F Ž k. F Ž k., k k, is impossible. Ž 1 1 A similar coclusio is also valid for the Sobolev situatio discussed i the precedig remark.. I fact, a similar result ca be proved uder fairly geeral coditios usig Theorem 1 hardest oe-dimesioal subfamily argumets foud i Dooho Liu Ž I particular, suppose that T is a liear fuctioal 1 are covex symmetric subsets of L with optimal rates of covergece give by Ž ˆ. q 0 lim if i if sup E T T Ž., ˆT i where 0 q q 1. 1 The it follows from essetially the same argumets used to prove the above that if Ž ˆ. q lim if sup E T T Ž., the q 1 lim if sup E Tˆ T Ž. 0. l 1 5. Noparametric regressio desity estimatio models. I this sectio we show, with oly mior modificatios i the proofs, that Theorem 3 also holds i the oparametric regressio desity estimatio models. This aturally yields aalogs for the lack of adaptivity statemet i Sectio 4. The oparametric regressio model is give as follows. For each, set 1 t Ž i., i 0,...,, let i Ž 5.1. yi f Ž ti. e i, i 1,..., f F Ž k., ei i.i.d. NŽ 0, 1., where F Ž k. is give by Ž For estimates based o Ž 5.1. the miimax rate of covergece is the same as i the white oise model is give by Ž Moreover Theorem 3 also holds for estimators based o Ž The proof is essetially the same as i Sectio 3. Defie g, c,, as before. Write Pg for the probability Ž measure o geeratig 5.1. The T l dp dp. g 0 is sufficiet for P, P 4. Set 0 g g Ž ti.. Ý i1

12 The Note that for large, A CONSTRAINED RISK INEQUALITY uder P, T N,, uder P g, T N,. 1 g H 1 Ž t. dt the distributios of T are close to those give i Sectio 3. Now defie g by g Ž t. g Ž t i., t, i1 t t, i, i 1,...,. The g H Ž t. dt dt dt 1 g t 1 g t H H H g Ž t. g Ž t. dt 1 t, i Ý H g Ž t. g Ž t. dt. i1 t, i1 Note that sice g k Ž x. M g has compact support, it follows that for some M 1, M, Ž 5.. g Ž x. M1 x, Ž 5.3. gž x. M1 x hece for t t t,, i1 i g t g t M1 M t t, i1 M t t, i1 t, i Ý H 1, i1 Ž, i1. i1 t, i1 M M t t M t t dt M1 M M 3 3 M1 M M 1. 3

13 534 L. D. BROWN AND M. G. LOW Now by , as hece, where 0 as. The rest of the proof follows that give i Sectio 3 except that for large, 3.19 becomes I expž. g Ž 0. Ž 5.4. Ef Ž fž exp, ž / where expž. 1 as. We ow tur to the desity estimatio problem. Let X 1,..., X be i.i.d. each with desity F F Ž k.. The oce agai Theorem 3 holds for estimators based o X 1,..., X. For simplicity we shall give a proof for the case where the poit of superefficiecy is the uiform desity. I other words, f Ž t. 0 1 o 1, 1. Oce agai defie g, c, as i Sectio 3, but with the added restrictio that HgŽ x. dx 0. To apply Theorem 1 we eed to fid upper bouds for f Ž x1. f Ž x. Ž 5.5. I H dx1 dx. f Ž x. f Ž x Now sice f t 1 o 1, 1, 0 Ž g Ž t. I 1 dt H 1 1 ž H / 1 g t l 1 by Ž expž l. sice if x 0 x 1 e x. The as i Sectio 3 for large, g Ž 0. E f Ž 0. Ž 1., f where is defied by Ž Now take limits the theorem follows just as i Sectio 3.

14 A CONSTRAINED RISK INEQUALITY 535 REFERENCES BICKEL, P. J. Ž Miimax estimatio of the mea of a ormal distributio subject to doig well at a poit. I Recet Advaces i Statistics ŽM. H. Rizvi, J. S. Rustagi D. Siegmud, eds Academic Press, New York. BICKEL, P. J. Ž Parametric robustess, small biases ca be worthwhile. A. Statist BROWN, L. D. LOW, M. G. Ž Asymptotic equivalece of oparametric regressio white oise. A. Statist DONOHO, D. L. JOHNSTONE, I. M. Ž Adaptig to ukow smoothess at a poit via wavelet shrikage. Upublished mauscript. DONOHO, D. L. JOHNSTONE, I. M. Ž Neo-classical miimax problems, threshholdig, adaptive fuctio estimatio. Beroulli 396. DONOHO, D. L. LIU, R. G. Ž Geometrizig rates of covergece III. A. Statist DONOHO, D. L. LOW, M. G. Ž Reormalizatio expoets optimal poitwise rates of covergece. A. Statist EFROMOVICH, S. Y. Ž Noparametric estimatio of a desity of ukow smoothess. Theory Probab. Appl EFROMOVICH, S. Y. LOW, M. G. Ž Adaptive estimates of liear fuctioals. Probab. Theory Related Fields EFROMOVICH, S. Y. PINSKER, M. S. Ž Self-learig algorithm of oparametric filtratio. Automat. i Telemeh Ž I Russia.. FRANK, P. KIEFER, J. Ž Almost submiimax biased miimax procedures. A. Math. Statist. 14. GOLUBEV, G. K. Ž Adaptive asymptotically miimax estimates of smooth sigals. Problemi Peredachi Iformatsii Ž I Russia.. HODGES, J. L. LEHMANN, E. L. Ž The use of previous experiece i reachig statistical decisios. A. Math. Statist HODGES, J. L. Ž Commuicated to Le Cam Neyma. IBRAGIMOV, I. A. HASMINSKII, R. Z. Ž Statistical Estimatio: Asymptotic Theory. Spriger, Berli. IBRAGIMOV, I. A. HASMINSKII, R. Z. Ž Noparametric estimatio of the values of a liear fuctioal i Gaussia white oise. Theory Probab. Appl LE CAM, L. Ž O some asymptotic properties of maximum likelihood estimates related Bayes estimates. Uiv. Calif. Publ. Statist LEHMANN, E. L. Ž Theory of Poit Estimatio. Wiley, New York. LEPSKII, O. V. Ž O a problem of adaptive estimatio i Gaussia white oise. Theory Probab. Appl LEPSKII, O. V. Ž Asymptotically miimax adaptive estimatio I: upper bouds. Optimally adaptive estimates. Theory Probab. Appl LEPSKII, O. V. Ž O problems of adaptive estimatio i white Gaussia oise. I Advaces i Soviet Mathematics 1. Amer. Math. Soc., Providece, RI. LOW, M. G. Ž Reormalizatio white oise approximatio for oparametric fuctioal estimatio problems. A. Statist NUSSBAUM, M. Ž Asymptotic equivalece of desity estimatio Gaussia white oise. A. Statist ROBBINS, H. Ž Asymptotically submiimax solutios of compoud statistical decisio problems. Proc. Secod Berkeley Symp. Math. Statist. Probab Uiv. Califoria Press, Berkeley. DEPARTMENT OF STATISTICS 3000 STEINBERG DIETRICH HALL UNIVERSITY OF PENNSYLVANIA PHILADELPHIA, PENNSYLVANIA