Mathematical Institute, University of Utrecht. The problem of estimating the mean of an observed Gaussian innite-dimensional vector

Size: px

Start display at page:

Download "Mathematical Institute, University of Utrecht. The problem of estimating the mean of an observed Gaussian innite-dimensional vector"

Ralph Wells
5 years ago
Views:

1 On Minimax Filtering over Ellipsoids Eduard N. Belitser and Boris Y. Levit Mathematical Institute, University of Utrecht Budapestlaan 6, 3584 CD Utrecht, The Netherlands The problem of estimating the mean of an observed Gaussian innite-dimensional vector with independent components is studied, when vector is known to lie in a l -ellipsoid and the variances of the components need not be equal. Under some general assumptions on the ellipsoid we provide the second-order behaviour of the minimax risk. Key words Gaussian noise, ellipsoid, minimax linear risk, asymptotically minimax estimator, second-order asymptotics. Introduction Pinsker (980) initiated the study of minimax estimation procedures for the ltration problem in Gaussian noise which can be described, in equivalent terms, as () Y k = k + k k k =; ; Here k 0; k =; ;, are given, i 's are independent standart Gaussian random variables, >0 is a small parameter and =( ; ;) is the unknown innite-dimensional parameter of interest,, where () =(Q) =f a k k Qg ; (a k ; k =; ;) is a nonnegative sequence converging to innity. The observation model () arise as the limiting experiment in many other estimation problems. This model has been actively pursued recently, see [], [], [5], [6] and further references therein. These papers demonstrate amply the importance of the asymptotic minimax estimators and their practical relevance. In [9] it was shown that the quadratic minimax risk over the elipsoids coincides asymptotically with the minimax risk within the class of linear estimators. A procedure for obtaining the minimax linear estimators and evaluating their risks was described.

2 In this article, developing further the approach of [9], we describe the second-order behaviour of the minimax estimators and the quadratic minimax risk for the model (){ (). These results are illustrated by a number of examples. The authors are grateful to G.K. Golubev for a number of comments resulting in the improvement of some results of the paper and their better presentation. Minimax linear estimation Let the model of observations be given by (). For the sake of simplicity we assume that the sequence (a k ; k =; ;) in () is positive and monotone. In this Section we investigate the minimax linear risk which will be shown later to be asymptotically equal, under some conditions, to the minimax risk. Denote x =(x ;x ;) and introduce the class of linear estimators (3) ^ = ^(x) =(^ ; ^ ;); ^k = x k Y k ; k =; ; Dene the risk of a linear estimator (4) R (x; ) =E k^(x), k and the minimax linear risk (5) r l = rl () = inf sup R (x; ); x where kk = P k. To formulate the result about the minimax linear risk, we introduce some notations. Let c be a solution of the equation (6) and (7) d = d () = k a k(, ca k ) + = cq k(, c a k ) + Here b + denotes nonnegative part of b. The following Theorem is due to Pinsker [9], but we give its elementary proof for the sake of completeness. Theorem. Let c and d are dened by (6) and (7). Then (8) inf x sup R (x; ) = sup inf R (x; ); x the saddle point (~x; ~ ) for the problem (5) is given by (9) (0) ~x k =(, c a k ) + ; ~ k = k (, c a k ) + =(c a k )

3 and the linear minimax risk satises the following equations () r l = d = sup k k=( k + k) Proof. It follows immediately that the risk of a linear estimator has the form () R (x; ) = kx k +(, x k ) k (3) Since, according to (6) Qc = P kc a k (, c a k ) +, inf x sup R (x; ) sup R (~x;) Q sup(, ~x k ) =a k + k Qc + = = k(, c a k ) + k((c a k (, c a k ) + +(, c a k ) +)) k(, c a k ) + = d Note now that equation (6) can be also rewritten as (4) a k ~ k = Q; so that ~. Taking into account (3), (4) and (), we obtain d inf x = sup sup R (x; ) sup inf k k k + k This completes the proof of Theorem. x R (x; ) k ~ k = d ~ k + k k~x k Remark. The equations (6) and (0) can be obtained by the Lagrange multiplier method for a problem of maximizing a functional P k k=( k + k) subject to the convex constraint (). Remark. Due to monotonocityof(a k ;; ;), d = P N k(,c a k ), where (5) N = N () = maxfk a k c, g One can easily derive the explicit formulas for c and N (cf. [3]) c = c () = P N ka k Q, + P N ; k a k 3

4 Note that (5) entails that ( N = max l lx ka k (a l, a k ) Q ) (6) 0 c a k ; k =; ;;N 3 Asymptotically minimax estimation In this section we investigate the asymtotic behaviour of the minimak risk with respect to all possible estimators. We dene the minimax risk (7) r = r () = inf ^ sup E k^, k ; where ^ is an arbitrary estimator based on Y =(Y ;Y ;). In the proof of Proposition we use the van Trees inequality [0, p. 7]. Now we describe the version of this inequality which we use below. Let dp (y), y =(y ;y ; ), denote distribution of the vector of observations Y =(Y ;Y ; ) in () and '(y k ; k )bethe marginal (gaussian) density ofy k. Assume that a prior distribution d(); =( ; ; ) is dened according to which k are independent random variables, with corresponding densities k (x). Let, for all k; k (x) be absolutely continuous, with nite Fisher log k (x) I( k )= k (x) Assume also that k (x) is positive inside a bounded interval of the real line and zero outside it. We write E for the expectation with respect to the joint distribution of Y and. Then, according to the van Trees inequality (cf. [4] and [5]), the Bayes risk E(^ k, k ) admits a lower bound (8) E(^ k, k ) I k + I( k ) ; where I k =, k, is the Fisher information about k contained in the observation Y k and ^ k = ^ k (Y ). Since our setup here is slightly dierent fron those of [4] and [5], below wesketch a short proof of (8). Let A = ^ k, k ; k log ('(Y k ; k ) k ( k )) Denote Y (k) =(Y ; ; Y k, ;Y k+ ; ), (k) =( ; ; k, ; k+ ; ) and let dp (k) (y (k) ) and d (k) ( (k) ) respectively be their distributions. 4

5 Use the Cauchy-Schwarz inequality EA (EAB) =EB One can assume, without loss of generality, that EA <. Our assumptions permit integration by parts and interchanging the order of integration in the following integral yields EAB = = = (^ k, k log ('(y k ; k ) k ( k )) dp (y) d() (^ k, k ('(y k ; k ) k ( k )) dy k d k dp (k) (y (k) ) d (k) ( (k) ) dp (y) d() = It remains to note that EB = I k + I( k ). Next theorem describes the lower bound for the minimax risk (7). The proof of this and the following results of this Section will be given in the Appendix. Theorem. Let (m k ;; ;) be a sequence such that, for some >0, (9) a km k + 8 log, a 4 km 4 k! = Q Then the following lower bound holds (0) r km k m k + k + O( ) ;! 0 To derive a good lower bound, one should, in principle, maximize the functional appearing in (0) under the restriction (9). However, the following Theorem shows that, under rather mild condition, this problem is asymptotically equivalent to the maximization problem () which has already been solved by Theorem. This implies, in particular, the asymptotic equivalence of the minimax risk and the minimax linear risk. Theorem 3. Let c and N be dened by (6) and (5). If condition holds, then log, P a k 4 k(, c a k ) + ( P a k k (, c a k ) + ) = o() ;! 0 ; r = N X k, c N X ka k! ( + o()) ;! 0 The next Proposition, although looking quite general, provides exact asymptotics of the minimax risk for a more restricted class of ellipsoids. In particular, it is convenient in applications where the sequence (a k ; k = ; ;) is increasing faster than k m for any m >0. Since in such cases the limiting behaviour of the minimax linear risk d 5

6 typically does not depend on Q (cf. Example 4{5 in Section 4 below), this Proposition leads also to the exact asymptotics of the minimax risk r. In the context of curve estimation this corresponds to estimating "very smooth" function, with rapidly decreasing Fourier coecients (cf. [5]). Proposition. Let d be dened by (7). Then () d ((Q= )) r d ((Q)) Note that these lower and upper bounds for the minimax risk are nonasymptotic. Corollary. Let c and N be dened by (6) and (5). If P k = < and c N X ka k = o 0 k=n + then the following asymptotic expansion holds r =, X k=n + k k A ;! 0 ; A ( + o()) ;! 0 Remark 3. There are two terms in asymptotic expression of the minimax risk in Theorem 3. They can be either of the same order or the second term can be of smaller order than the rst. In the latter case Theorem 3 provides at least two terms of asymptotic expansion of the minimax risk (cf. Example below). Remark 4. Recall that the sequence (a k ; k =; ;)was assumed positive. The results remain valid under the weaker assumption a k 0, k =; ;(cf. []). 4 Examples The results presented below illustrate the assertions in the previous Section. Example. Consider model (){() with a k = k, >0, k = k,, + >0.In this case it is easy to prove that c N! as! 0. Using this and (6), one can calculate N = ( + )( + )Q=( ) + ( + o()) ; c = =(( + )( + )Q) + ( + o()) Here we make use of the asymptotic relation () m = M + ( + o()) as M!; >, ( +) Now one can easily verify the condition of Theorem 3. By applying Theorem 3, we derive the asymptotics of the minimax risk. 6

7 Case >0. The asymptotics () and relations for N and c yield (cf. [9] for =) r = 4=(+), (Q( + )) =(+) (=( + )) =(+) ( + o()) In this case Theorem 3 gives only the rst-order term of the minimax risk. Note also that although ( k;; ;) can be increasing to innity, the minimax risk still converges to zero. Case = 0. By using () and the asymptotics (3) one obtains k, = log M + C e + o() as M!; r =, log, + (C e +(), log(q),, )( + o()) ; where C e = is the Euler constant ([7, ]). Case <0. Using the asymptotic relation we calculate m=m m, = M, ( + o()) as M!; > ;, r = + 4=(+), (Q( + )) =(+) (=( + )) =(+) ( + o()) Example a k = k, >0, k = k,(+). In this case the condition of Theorem 3 is again satised, and N =(Q, = log, ) = ( + o()) ; c = log, (Q), ( + o()) Then by Theorem 3, r =, 4 (log, ), Q, ( + o()) Example 3 a k = k, >0, k = k,(+), >. One calculates N = (, )Q, = ( + o()) ; c = ((, )Q), ( + o()) With these asymptotic relations, one can show that d ((Q)) =, 4 Q, (, ), ( + o()) By applying Proposition, we can obtain only the rate of the second-order term of the minimax risk r =, 4 Q, (, ), ; 7

8 where lim inf ;!0 lim sup!0 Example 4 a k = e k, >0, k = k,.from (5) one can see that (4) e, c e N Using (6), (4) and the asimptotics gives m e m = M (M +) e ( + o()) as M! (e, ) N =, log, +(), (, ) loglog, + O() By the last two relations and (4), we have c NX ka k = N, (N +) c e ( + o()) N, e e, e, ( + o()) = O (log, ), We apply Proposition to this Example. Case >. Since, according to [7, 0.], we calculate NX k = N m = M + ( +) + M ( + o()) as M!; >0 ; + N, and obtain that ( + o()) = (log, ) + (, )(log, ), loglog, ( + o()) ; r = (log, ) + (, )(log, ), loglog, ( + o()) Case =. In this case we have that c P N ka k = O() ; NX k = N =, log, + O() ; and therefore, r =, log, + O( ) 8

9 Case 0 <<. One can show that where m, =, (M + m),, (,) m+ m,! = M, + ()+o() as M!; 0 << ; (, ) () =, (,) m+ m, is the Riemann zeta function ([7, 7.4.]). Using this asymptotics, we obtain NX k = (log, ) + (, )+o() Consequently, r = (log, ),, + (, )( + o()) Case = 0. Since, by (3), NX k = log N + C e + o() ; we get r = loglog, + (C e + log, )( + o()) Case <0. In this case one can verify that k=n + k =, N Therefore, by Corollary we have ( + o()) =,(log, ) ( + o()) r = + (log, ),, ( + o()) Example 5 a k = e kr, >0, 0 <r<, k = k,. With the asymptotics where m e mr = C r e Mr M +(,r)+ ( + o()) as M!; C r = 8 >< > 0 <r< (r), ; e =(e, ); r = ; r> ; 9

10 one can obtain N =, log, =r ( + o()) By denition of N, weevaluate X N c ka k = C r N,+(,r)+ c e Nr ( + o()) C r N,+(,r)+ ( + o()) = O log, (,+(,r)+)=r ( + o()) Now the asymptotics of the minimax risk may be obtained in the same way as in Example 4. Case >0. r = (log, ) =r,=r, ( + o()) Case =0. Case <0. r = r, loglog, + (C e + r, log, )( + o()) r = + (log, ) =r,=r, ( + o()) Remark 5. Note that in most cases in Examples 4 and 5 both the rst- and the second-order terms of the minimax risk do not depend on the "size" Q of ellipsoid (Q). Remark 6. Let a k = a k (), k =; ;, be as in Example 4 or Example 5. Dene the correspondent hyperrectangle in l -space H = H (Q) =f j k j q Qa, k (); k =; ;g The assertions of Examples 4 and 5 concerning the rst-order behaviour (also the secondorder behaviour for the cases = 0 and <0) of the minimax risk remain valid with = replaced by H. This follows immediately from the following easily veried relation for any Q>0; >0; 0 << there exists Q > 0 such that (Q) H (Q), (Q ) Example 6 a k = k, k = ekr, ; ; r > 0. Let us establish rst an upper bound for the minimax risk r () (see (7)). Such a bound is provided by the minimax linear risk which, according to Theorem, equals d (see (6){(7)). Using the asymptotic expansions (M!) m e mr = M,r Mr M e r m e m = M e M e e,, + r,,! M,r ( + o()) ; 0 <r<; (r) e (e, ) M, ( + o()) m e mr = M e Mr + e (M,)r ( + o()) ; r>; 0! ;

11 one can solve (6){(7), thus obtaining c =, log,,=r ( + o()); d = Qc ( + o()) = Q, log,,=r ( + o()) The last formula exhibits a distinctive feature of this example, as compared to all previous ones. Indeed, analyzing the proof of Theorem (cf. inequality (3)), one realizes that the term Qc, contributing to d, arises solely as the squared bias term of the linear minimax estimator. Thus, only the bias of the estimator contributes to its maximal risk, up to the rst order. To show that d coincides asymptotically with the minimax risk r (), we choose a prior distribution on and use the obvious inequality r () R (), where R () denotes the Bayes risk. Let be a distribution on such that and N = ; with probabilities = i =0; i 6= N;,almost surely; where =(Q=a N ) = and N =[c, ]. Clearly () =, = Qa, N = Qc ( + o()) = d ( + o()) and N = e Nr =, e O(). Due to suciency considerations, the Bayes risk R () in estimating is equal to the Bayes risk in estimating N, based on the observation Y N only. Since it follows (see [8], proof of Lemma 3.) that lim =0;!0 Var Y N r () R () = ( + o()) = d ( + o()) Thus r () = Q, log,,=r ( + o()) 5 Appendix Proofs The proof of Theorem is based on the following elementary result. Proposition. Let ;; m be independent Gaussian random variables with E k =0, Var k = d k. Then P ( m X k >P ) exp (, P, P m d k 4 P m d 4 k )

12 Indeed, using Markov inequality and moment-generating function of m, one obtains for any >0 P ( m X It remains to set k >P ) e,p Ee P m k = exp (,P + ( ) mx mx exp,(p, d k )+ = P, P m d k P m d 4 k d 4 k mx log(, d k),= ) Proof of Theorem. We select a prior measure d() such that k ;; ;, are distributed independently and normally with zero means and variances m k, k =; ;. Let E denote the expectation with respect to the joint distribution of Y ;Y ; and ; ;. Since is closed and convex, r = inf sup ^ E k^, k. We bound the minimax risk from below as follows r = inf sup E k^, k inf E k^, k d()=() ^ ^ (5) inf ^ inf ^ Ek^, k, sup ^ because, by Cauchy-Shwarz inequality and (9), sup ^ C E k^, k d()=() E(^ k, k ), Qa, ( + p 3) ( C ) = =() ; C E k^, k d() ( C ) sup ^ Q( C )a, + = Q( C )a, E k^k + C k d() (( C )) = + p 3 ( C ) = X Qa, ( + p 3) ( C ) = Note that Proposition, together with (9), entails ( C ) = (6) 4 kd() Now we recall the following known result. If and are independent Gaussian random variables with E =0,Var =, Var =, and = +, then inf E(, f()) = E (, E(j)) = =( + ) f () m k =

13 Using this, we estimate the rst term of the right-hand side of (5) inf ^ E(^ k, k ) From the last inequality, (5) and (6) we obtain nally Theorem is proved. r km k m k + k km k m k + k + O( ) Proof of Theorem 3. The following upper bound for the minimax risk follows immediately from Theorem (7) r r l = d = N X k, c N X ka k Introduce = () = 8 (log, ) P a k 4 k (, c a k ) = + P a k k(, c a k ) + Note that 0 and = o() as! 0 for any >0 because of condition of Theorem. Now we take the sequence m k = ~ k( + ),, k = ; ;, with ~ k dened by (0). Equation (9) is satised. Indeed, by virtue of (4) we have a km k + = 8 log, a 4 kmk! 4 = a k ~ k = Q Applying now Theorem for the sequence (m k;; ;), we calculate r ( + ) = NX k, c k ~ k ~ k( + ), + k NX ka k, c + O( ) NX ka k (, c a k ) + c a k + O( ) From (6) it follows that c can not be of smaller order than. Choosing now some >4 and recalling that > 0, = o() as! 0, and 0 c a k for k =; ;;N (see (6)), we conclude that the last lower bound, together with upper bound (7), proves Theorem 3. Proof of Proposition. Let m k ;; ;be some sequence of positive numbers such that (8) a km k Q; 3

14 i.e. m =(m k ;; ;). Introduce k (x) =(=m k ) 0 (x=m k ); k =; ; ; where 0 (x) =Ifjxj g cos (x=) These are the probability densities with supports [,m k ;m k ] respectively. It is easy to calculate Fisher information of distribution dened by density k (x) (9) I( k )=E k [(log k ( k )) 0 ] = I( 0 )=m k = =m k ; where E k denotes the expectation with respect to density k. We select a prior measure d() such that k ; k =; ;, are distributed independently with densities k (x); ; ;, respectively. Since (8) provides that supp we proceed estimating the minimax risk (7) from below as follows (30) r = inf ^ sup E (^ k, k ) inf ^ = inf ^ E (^ k, k ) d() E(^ k, k ) For this case (see (9)) the inequality (8) yields E(^ k, k ) =m k +, k, From this inequality and (30) we get that for any m the minimax risk r satises (3) r From this one obtains the following lower bound k m k = m k = + k r sup km k= = sup km k m(q) m k = + k m(q= ) m k + k The last relation and Theorem complete the proof. = d ((Q= )) Proof of Corollary. The left-hand side of the inequality (3) does not depend on m. Therefore, we can take any m. Now we make use of the vector ( ~ k ; k =; ;) dened by (0). Relation (4) provides that ~. Substituting ~ k in (3), k =; ;, one calculates r k ~ k= X N = ~ k = + k Using now this and (6), we obtain that r N X k, N X k, N X kc a k kc a k (,, )c a k +, Combining last relation with condition of Corollary and the upper bound (7) completes the proof. 4

15 References [] D.L. Donoho and I.M. Johnstone, Minimax risk over l p -balls, Technical Report 04, Department of Statistics, University of California, Berkeley, 989. [] D.L. Donoho, R.C. Liu and B. MacGibbon, Minimax risk over hyperrectangles, and implications, Ann. Statist., 8 (990), pp [3] S.Y. Efroimovich and M.S. Pinsker, Estimation of square-integrable density of a random veriable, Problems Inform. Transmission, 3 (98), pp [4] R.D. Gill and B.Y. Levit, Applications of the van Trees inequality A Bayesian Cramer-Rao bound, Bernoulli, (995). [5] G.K. Golubev and B.Y. Levit, On the Second Order Minimax Esimation of Distribution Functions, Preprint 88, Department of Mathematics, University of Utrecht, 994. [6] G.K. Golubev and M. Nussbaum, A risk bound in Sobolev class regression, Ann. Statist., 8 (990), pp [7] I.S. Gradshtein and I.M. Ryzhik, Table of Integrals, Series, and Products, Academic Press, New York, 980. [8] I.A. Ibragimov, R.. Khasminskii, On nonparametric estimation of the value of a linear functional in Gaussian white noise, Theor. Prob. Appl., 9 (984), 8-3. [9] M.S. Pinsker, Optimal ltering of square integrable signals in Gaussian white noise, Problems Inform. Transmission, (980), pp [0] H.L. van Trees, Detection, Estimation and Modulation Theory, Part, Wiley, New York,

Minimax Risk: Pinsker Bound

Minimax Risk: Pinsker Bound Michael Nussbaum Cornell University From: Encyclopedia of Statistical Sciences, Update Volume (S. Kotz, Ed.), 1999. Wiley, New York. Abstract We give an account of the Pinsker